I can fetch listings, photos everything works fine. I've been playing limiting to 1,000 fetch / time. Then I upgraded to 1,500 and somehow it can't import a single more listing. 1,000 is nowhere near the total, with some of the categories featuring over 10,000...

Funny enough if I bring it back to 1,000 it can update them, both with cron or ''update'' from the pirets module. And update, delete and create up to reaching the 1,000 limit.

The error I get (actually 100's at a time in a single second) is the following:

mysqli_num_rows() expects parameter 1 to be mysqli_result, boolean given in /home/ *** /public_html/includes/database.mysqli.inc on line 178.

To troubleshoot I did:

run cron
run update
flush the listings and download them again
deactivate the fields that generate a table in the database (I assume form the last dev version that allows to fetch same cck's appearing as different in each class)

and nothing...

I found this post:

http://drupal.org/node/512962

I may try to play with the patches posted there, but I would like to know if there may be any link between the post and my pirets problem.

Thanks in advance!

Comments

thomas.lucas’s picture

thomas.lucas commented

I've been doing some research and that is what I found so far:

this error (as it says..) it's triggered by mysqli.inc, and the line mysql_num_rows is the filling for the db_result function.

the function is called 2 times (I think) one at pirets.batch.inc and another at pirets.admin.inc

the one at batch is general, and since I am having no problem fetching all but 3 classes I don't think is it.

The one at pirets.admin.inc is class-related. I think the problem may be there. Google-ing the problem in general is some typo (comma, semi-colon, and using 1 or 2 quote marks seem to be the typical problems. But it may be that the forums I found where of newbies like me...). I've tried changing to previous releases. Like beta 5, beta 6, dev 6, dev 3 from February.

I can update every time with no problems 5 classes, all of them having less than 1,300 listings. The 3 I can't get to update have 2,200 5,000 and 13,000 properties. I am not sure if size may have anything to do with the problem. I increased the php memory from 96 to 512, the times to 100 seconds and the uploads and file sizes also to over 256 and nothing changed.

I am not a programmer, so I would very much appreciate any input about what may be causing the error and what other tests are out there I can try. I tried to install the devel module to run a query on the classes that give errors but I don't know how to write a query for the RETS....

Thanks in advance

jday’s picture

jday commented

For what it's worth, here's the error in my logs:

Warning: mysql_num_rows(): supplied argument is not a valid MySQL result resource in db_result() (line 189 of .../public_html/includes/database.mysql.inc).

I'm successfully fetching 4 classes: SF:12,272 listings, CD:2,604 listings, MF: 1,076 listings, LA: 8,192 listings.
php memory_limit is set to 300M
on VPS server with 3GB ram

I was not able to fetch all listings before moving to the 3GB RAM server, started at 512M, 1G, 1.5G now finally found a working environment. I'm getting the mysql_num_rows() warning but since all classes are updating daily I'm not too concerned.

Garrett Albright’s picture

Garrett Albright commented

thomas.lucas, you had this problem before, but said that you had fixed it. Is the fix no longer working? Did you not really fix it? Or…?

thomas.lucas’s picture

thomas.lucas commented

First it wasn't fetching the pictures or listings. The listings got fixed by switching the correlation from MLS# to sysid and the pictures adding and correlating the picture count. Then It fetched fine 1,000 listings for each category. So I assumed everything was fine, but in crons run after 8 hours wasn't adding more listings. Probably the interealty refusing to take offset. So I changed to unlimited (or 2,000 or 50,000 for that matter -I added the 50,000 to the code in case a number large enough to take all listings at once had anything to do). but all I got was updates to 1,500 and then the error popping up again. I also changed form batches of 50 to 500 with no success and to batches of 2 and 1 with some success. "some" meaning that I got to update 1,500 of each when before was rejecting to update 1,500 for 3 categories.

My server was first shared hosting, then vds with 1 gig and after jday post I upgraded yesterday from 1g to 2g on my vds. I will try again today. The real point is that I don't know what the error comes from. First I thought may be related to the listings, then to the pictures, then to the memory... I am now setting up the vds 2g and I will try again and post the results.

Resuming... I guess I never fixed the db_result problem. Since I don't know what triggers it I am having hard time coming with ideas of tests I can run ^-^

At any rate thanks jday and Garrett for your help!

thomas.lucas’s picture

thomas.lucas commented

Ok.. Just finished updating the server:
VDS 2G Ram
--

The problem was the packet size: http://drupal.org/node/744598

Now it imports the full +20k listings beautifully! I set up a crontab with the wget command, but it timed out every now and then. Then I set up a drush cron (did it with webmin, so the times are missing here bc webmin adds them, but makes no difference to the following question).

/usr/bin/php -f /usr/local/bin/drush/drush.php cron --root=/var/www/html --uri=http://example.com cron

Runs cron and everything fine, but some nodes don't get the pictures. If I run the cron with wget they get pictures (but not added to the ones missing them) and if I go and run the ''update all now'' batch the pictures are actually fetched for the ones that didn't get pictures with the drush crontab.

I wanted to ask if the pictures may be refetched in the next drush crontab (and then I can just run crons close on another so images get updated) or (and that would be ideal) there is an easy way to set up a drush / cron line to run the batch file.

At any rate, Thanks a lot Garrett for your great work and keep it up!

P.S. I am with interealty and I am able to fetch everything, "unlimited" easily fixes the offset problem. So you may want a update the compatibility char to say that at least 1 interealty based mls with over 20k listings works just fine.

Garrett Albright’s picture

Garrett Albright commented

It depends on what you mean by "get pictures." Pictures aren't downloaded immediately along with listings - otherwise, listing downloads would take even longer. See this page on the PIRETS.info site for more info on how PIRETS deals with property images.

dbaasi’s picture

dbaasi commented

Thanks for the module, what a great tool. I have also been experiencing this same issue. Works fine on unlimited (providing you have enough resources) but if you set the record chunk size and the rets server has more than that number, it will not get the next chunk of records. So if you set it to 250, you only get 250 even though there are thousands. The backtrace shows:
Type: php
mysqli_num_rows() expects parameter 1 to be mysqli_result, boolean given in /home/content/a/s/i/asidbatbd/html/includes/database.mysqli.inc on line 178.
Backtrace:
mysqli_num_rows(a:1:{i:0;b:1;})[database.mysqli.inc:178];
.db_result(a:1:{i:0;b:1;})[lock.inc:106];
..lock_acquire(a:1:{i:0;s:12:"pirets_fetch";})[pirets.batch.inc:221];
...pirets_class_update_batch_update(array)[?:?];
....call_user_func_array(array)[batch.inc:190];
....._batch_process(a:0:{})[batch.inc:106];
......_batch_do(a:0:{})[batch.inc:34];
......._batch_page(a:0:{})[system.admin.inc:1808];
........system_batch_page(a:0:{})[?:?];
.........call_user_func_array(a:2:{i:0;s:17:"system_batch_page";i:1;a:0:{}})[menu.inc:348];
..........menu_execute_active_handler(a:0:{})[index.php:18];
...........index.php

I noticed you have a message The RETS server may override this value (MAXROWS). Could this end the batch process prematurely?

Garrett Albright’s picture

Garrett Albright commented

Hmm, the "lock_acquire" partis interesting. That implies the problem doesn't actually lie with the RETS server or the communication therewith, but with Drupal's locking system, which PIRETS uses to assure two or more conflicting RETS operations aren't happening at the same time. Huh… strange.

dbaasi’s picture

dbaasi commented

I was able to get the batch to run to completion in chunk sizes less than "unlimited" by changing line 162 in the pirets.batch.inc
from // $context['finished'] = strpos($response->data_raw, ' $context['finished'] = strpos($response->data_raw, 'No Records Found') == TRUE;
For some reason, I was not seeing the maxrows response in each chunk.