Update & Batch Performance

I have my site on a 512MB slice at slicehost.net and the site is slow and the server grinds to a halt at least once a day, only a hard-reboot will get it going again.
Using Ubuntu Hardy set up.
php_value memory_limit 128M
max_execution_time 8000

What are the recommended configurations for apache and mysql?
is 512MB of ram too small?

the cron updates are not working, I have to manually click the update buttons from the pirets admin page.

I've upgraded my server to a 754MB slice at slicehost, set memory limit to 256M and max_execution to 300000
and set the pirets batch collection to 500

Still can't run the update scripts for single family or land classes, the process always stops at 33%, the error page says increase php memory, max execution time, or reduce the batch size

How can I get the listings updated? cron never gets to them all and the manual update always fails.

I am running into an issue as well.

I was seeing some memory limit errors so I downed my chunk size to 500. However, now it is trying to fetch more listings than are present. In the example below, there are only about 1500 records in the class 1 but the batch keeps going and going, e.g.:

2500 listings fetched from RETS server for class 1.

I'll look into it further but let me know if there are any ideas out there.

I was seeing some memory limit errors so I downed my chunk size to 500. However, now it is trying to fetch more listings than are present.

Oh, goody. This problem tends to pop up when the RETS server ignores the "offset" part of the request for listings.

Normally, fetching listings might go like this:

PIRETS: Server, please send me 500 listings, starting with listing 1.

Server: Okay, here are listings 1 through 500, and I have more listings.

PIRETS: Okay, now send me 500 listings, starting with listing 501.

Server: I can send you listings 501 through 788. I don't have any more listings.

PIRETS: Okay, I won't ask you for any more. I'll examine the listings you gave me and create, update or delete nodes.

When servers ignore the request offset, though, transactions look like this:

Server: Okay, here are listings 1 through 500, and I have more listings.

PIRETS: Okay, now send me 500 listings, starting with listing 501.

Server: Okay, here are listings 1 through 500, and I have more listings.

PIRETS: Okay, now send me 500 listings, starting with listing 1001.

Server: Okay, here are listings 1 through 500, and I have more listings.

…And so on and so on. The server keeps telling PIRETS there are more listings to fetch because it's not properly sending the range of listings that would include the end. So PIRETS just keeps requesting more listings - and keeps being sent them, from its perspective.

A possible workaround is to set the chunk size to "Unlimited" and hope that the maximum number of listings the server will send in one go is larger than the actual number of listings in that category. Another option is to call up your RETS provider and let them know just how much they blow.

I'm interested to learn how other locally-caching RETS systems work around this problem, if it's even possible…

scott, could you remind me again which RETS provider you're using? I definitely want to put this on the compatibility chart.

Note to self: Perhaps do some detection for redundant listings in the batch process code and bail out if it looks like this is happening.

Thanks Garrett-

It's Interealty.

"...let them know just how much they blow."

heh heh

Ah, heh heh… I wrote that in a furor, then figured I'd go back and edit in some more gentle language before I submitted the post… Looks like I forgot to. Oops.

Anyway, I'll add that to the compatibility chart.

I am working around the issue for now using a limit size of "Unlimited". The memory error persisted but I ended up upping the mysql max_allowed_packet to 32M which fixed the memory issue (so not a php memory issue).

Reference:
http://drupal.org/node/321210

I have installed this module on a server with 512MB RAM and the php memory limit is set to 128mb. Updating the records manually works every time, but it's never successful using cron. I have tried every limit size for the number of records to grab per batch with no success. When I disable this module cron runs like a charm, so I've confirmed that it is the update process that is causing it to lock.

Any other suggestions? Are there any recommended php or other settings for folks who are getting cron to run successfully?

Just to clarify, cron always returns a Page Not Found error after spinning the wheels for a couple of minutes.

Here are a couple of things that I noticed.

I was running into the default 2 minute server timeout. I added this line to the pirets.module to give some more time (in this example 20 minutes):

@ pirets.module
function pirets_cron() {
  module_load_include('inc', 'pirets', 'pirets.batch');
+  set_time_limit(1200);
  // See if it's time to update any classes
  $classes = pirets_classes_fetch();

It was working for a while then stopped. Trying to debug it further, I came to the conclusion that it was not pirets_cron that was the issue at all, it was the search module.

I used this post to help determine it was the search module and not pirets:
http://drupal.org/node/123269#comment-644012

Further digging brought up the "seach module" and "drupal_goto" issue:
http://drupal.org/node/356903

@jbergeron - when you go to yoursite.com/admin/reports/status and click "run cron manually" do you get the Page not found message? Maybe it's a drupal_goto somewhere.

Scott, thanks for your info.

It does seem to be about two minute that go by before I get the page not found error.

I don't actually have the search module enabled yet at this point, so I don't think that's the hangup for me. Also since the last time I posted I have installed the supercron module and verified that the pirets hook is the only one that can't be invoked. All of the other system and date cron hooks that are running go off without a hitch.

To answer your question I have tried running cron using the Dev module, by clicking on the "run cron manually" link in the status area, and by going to cron.php directly.

I have about 30,000 listings that I need to fetch so I know it needs a lot of time, but it works perfectly every time I update manually.

Thanks again for your thoughts, let me know if you have any others about this issue.

I just looked at the error log directly on the server and it appears that the following is the only other clue in there:

"Premature end of script headers: index.php"
I'm going to do some research on this, specifically relating to my current host Dreamhost. I am using one of their Private Servers.

In my particular case it looks like this was an issue with the server using Fast CGI timing out. I switched it over to PHP as CGI and it appears to be working. Cross your fingers! I did get an error that Cron had reached it's time limit was was aborted, but the log still shows nodes being updated every time I refresh it. This looks very promising.

Cron run exceeded the time limit and was aborted. Bummer. Tried increasing max_execution_time in php.ini with no luck. Back to square one.

I strongly, strongly, strongly recommend you install Drush and run cron through Drush instead of doing it through the web server using wget or other such silliness, whether you're using PIRETS or not (but especially if you're using PIRETS). This method allows you to bypass the web server, and therefore any timeout issues implicit in that.

If that's not an option for whatever reason, try updating the listings through the administration interface before ever running cron with PIRETS installed. (Go to the Listings section under PIRETS configuration, then click the "Update all now" link.) This will cause PIRETS to fetch and process all listings while running under the batch API, which also avoids timeout issues. The next time a cron run happens, PIRETS will only have to process changed listings, which will take much less time.

Can you automate running cron through Drush?

I'm sorry if that's too dumb a question to warrant an answer... but I can update the site manually. My client isn't going to want to do that, much less do it through the command line. Am I missing something?

Yes, you can. You just set the command to run Drush as the command to execute in the crontab. It may be a bit tricky managing things like $PATH if your crontab is running as root, but as an example, here's a (simplified) line from the crontab on one of my servers:

7 */7 * * * /usr/local/bin/php -f /usr/local/www/drush/drush.php cron -r /usr/local/www/abweb.us/www

The paths will vary on your server, but hopefully that's enough to get you started.

.

I may have a similar issue, I realized the updates were going terribly slow, (but no duplicated nodes are being created I believe), and then I saw that the tables for content_pirets_number was circling, adding new lines for every time I run an update. I am adding a printout of the table since I don't think my explanation is very clear... I am not sure if that is normal, but I have now over 6000 rows in that table and only downloaded about 2000 properties, and if I run batch update again (or drush cron for that matter) It will create another couple thousand lines before adding a new property. I tried removing the line 181 from pirets_batch.inc but it had no effect on my problem.

As you can see in the photo, it reached the node 103, then cron timed out, then run a batch update and it started back from the node 55 (now there are a dozen lines for node 55...) the previous 54 nodes are not from pirets.

(let me know if it;s a related or unrelated problem with this post)

Thanks!

obviously forgot the screen shot...

RE: Using Drush for running cron:

Is that last chunk of the command the path to the drupal root directory?
I'm using CentOs and drush is in the /usr/local/share directory, would the command below be correct?

7 */7 * * * /usr/local/bin/php -f /usr/local/share/drush/drush.php cron -r /home/kruse/public_html

is there a specific drush command for updating listings? (pirets_cron)
should I disable Elysia Cron module?

HELP! What can it be?

I'm not sure if my problem is along these lines but it seems to be. When I set the record chunk size to anything other than unlimited it keeps fetching listing to infinity. Usually bombs out around 2 million. I know for a fact there are not that many active listings.

When I do set the chunk size to unlimited it almost immediately seems to detect the correct amount of listings for a class...the progress bar goes to 33% and says:
Remaining 2 of 3.
Processed 0 of 4592 (0 created, 0 updated) in class LSE.

But doesn't get the listings or create the nodes....Just hangs there

I have a dedicated server through godaddy with 2GB of RAM...php mem limit set to 2GB...I have a huge amount of time for execution time.

I'm watching the firewall monitor of input speed and output speed. At first it spikes running fast. This is during the Initializing phase of the update. When it goes to where it hangs it is only downloading at like 7kbs. When I set the chunk size to 250 the speed screams and just keeps downloading listings and never getting to where it says: Processed 0 of 4592 (0 created, 0 updated) in class LSE.

Server Information:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=ED2163F9582C02D5AE8B0C33C94CEFC3; Path=/
Set-Cookie: JSESSIONID=ED2163F9582C02D5AE8B0C33C94CEFC3; Path=/
RETS-Request-ID: 1552074745
MIME-Version: 1.0
Cache-Control: private
RETS-Version: RETS/1.5
Date: Thu, 29 Sep 2011 15:04:43 GMT
Expires: 0
Set-Cookie: RETS-Session-ID=34c5d4e2c05c0bf2e7bf8b477f34f5ce; path=/
Connection: close
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Cache-Control: private
WWW-Authenticate: Digest realm="rets@retsiq.com",qop="auth",nonce="19d014f68541e37556054a65a37fc248", opaque="d7c865f330114"
Content-Type: text/xml
Content-Length: 503

jday: The crontab entry you posted looks correct. A simple way to test it is to just paste the part without the time information into the command line and see if it runs - that is, just `/usr/local/bin/php -f /usr/local/share/drush/drush.php cron -r /home/kruse/public_html`. There's still some caveats, though; depending on your server configuration, cron tasks may be run under a different user account (such as root) than the one you're using, and that can confuse things. Since you're using entirely absolute paths, I don't think that will be an issue in your case, though. And no, there's no specific Drush command for PIRETS updates; just cron should work.

As for Elysia Cron, yes, it may be confusing things. Pathologic contains its own logic to assure that listing updates are not run too often by cron.

markusa: I've encountered problems like that before. Like most PIRETS problems, it's due to a crappy RETS implementation on the server. See this post to see what's going on behind the scenes.

Update performance issues:
It seems like it takes too long to create and update listings. It does about 600/hr using the manual batch update method. That seems pretty slow. I have a dedicated box with 2GB of ram and an Intel duel core processor. What are some reasons that the processing of listings should be so slow?

Its slower mainly for the largest class of properties...I guess the number of listings means a lot more processing?

Hey guys,

I've been struggling with batch performance issues on a fairly large site for a while now. PIRETS would basically never complete deleting properties from the admin interface.
I dug into the batch code and found the pirets_class_update_batch_del function.

I made an adjustment at line 445 (I've just commented out the original code)

//....

//$node = node_load($context['results']['db_props'][$context['sandbox']['to_delete'][$context['sandbox']['index']]]['nid'], NULL, TRUE);
//node_delete($node->nid);
node_delete($context['results']['db_props'][$context['sandbox']['to_delete'][$context['sandbox']['index']]]['nid']);

//...

I guess what was slowing it down was forcing a node load just to get the id of the node.. when that id was already accessible within that array.
Once I made this change the function has ran without issue for a while now.. Nodes are deleting properly on the admin batch AND it finishes in a timely manner.

Seeing as how node_delete() itself calls node_load(), along with passing a parameter telling it to explicitly delete the node_load() cache, I doubt your optimization did any good, actually.

That being said, I wonder why I originally wrote it that way. Is $node being used later on in the code? Because if so, you just created another problem…

The rest of the function makes no reference to the $node object and I've had 0 issues with the site updating since I've changed this.

//...
    unset($_SESSION['messages']);
    $context['finished'] = ++$context['sandbox']['index'] / $context['sandbox']['to_delete_count'];
  }
  $context['results']['counts'][$context['results']['class']]['del'] = $context['sandbox']['index'];
  $context['message'] = t('Deleted @index of @count in class @class.', array('@index' => $context['sandbox']['index'], '@count' => $context['sandbox']['to_delete_count'], '@class' => $context['results']['class']));
//function ends..
}

As far as the optimization wouldn't loading the node twice (considering node_delete does load_node as well) just unnecessarily hit the database?

I have no end of problems with the deleting of nodes....It always ends with an error in the middle of trying to delete nodes. Can anyone give some reasons why this might be so? Occasionally it will actually go through the entire process properly although this is the minority. The deleting process is extremely slow. I've tried the code modifications above and I've tried the way it is originally written. Seems to make no difference. Its very frustrating. Now the classes are getting out of sync where there are thousands of nodes to be deleted. I have to update manually using the pirets config page.

Why? Why? Why? It just says error and gives me nothing else.

This post may or may not help your issue with deleting nodes:

Possible to just run the delete old listings function?

Redid the whole website. This time I made the node, all tables dealing with CCK, and all tables dealing with pirets Innodb tables. Also included the files table, location,nd_location, Performance increased by about 50%!!!!! I highly recommend this to people trying to set up a large real estate site using pirets. Finally I am getting a smooth running system going.

In fact making the critical data tables use the Innodb engine increased the speed of the entire site dramatically. Innodb uses row-locking for inserting database records. the standard myisam uses table locking which means that for every insert the entire table, or tables are locked. All data input must be done sequentially that way. Nodes with many many fields and a huge number of records benefit greatly from only one row of a table being locked from other processes working simultaneously instead of the entire table. So if you are trying to quickly insert say 40000 records while people are viewing and querying the site you can get bogged down quick if the entire table of your listings is locked instead of just one listing being locked during inserts and updates.