HELP! What can it be?

When I set the record chunk size to anything other than unlimited it keeps fetching listing to infinity. Usually bombs out around 2 million. I know for a fact there are not that many active listings.

When I do set the chunk size to unlimited it almost immediately seems to detect the correct amount of listings for a class...the progress bar goes to 33% and says:
Remaining 2 of 3.
Processed 0 of 4592 (0 created, 0 updated) in class LSE.

But doesn't get the listings or create the nodes....Just hangs there

I have a dedicated server through godaddy with 2GB of RAM...php mem limit set to 2GB...I have a huge amount of time for execution time.

I'm watching the firewall monitor of input speed and output speed. Chunk size set to unlimitied at first it spikes running fast. This is during the Initializing phase of the update. When it goes to where it hangs it is only downloading at like 7kbs. When I set the chunk size to 250 the speed screams and just keeps downloading listings and never getting to where it says: Processed 0 of 4592 (0 created, 0 updated) in class LSE.

Server Information:
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=ED2163F9582C02D5AE8B0C33C94CEFC3; Path=/
Set-Cookie: JSESSIONID=ED2163F9582C02D5AE8B0C33C94CEFC3; Path=/
RETS-Request-ID: 1552074745
MIME-Version: 1.0
Cache-Control: private
RETS-Version: RETS/1.5
Date: Thu, 29 Sep 2011 15:04:43 GMT
Expires: 0
Set-Cookie: RETS-Session-ID=34c5d4e2c05c0bf2e7bf8b477f34f5ce; path=/
Connection: close
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Cache-Control: private
WWW-Authenticate: Digest realm="rets@retsiq.com",qop="auth",nonce="19d014f68541e37556054a65a37fc248", opaque="d7c865f330114"
Content-Type: text/xml
Content-Length: 503

Comments

markusa’s picture

markusa commented

Everything else seems to be fine....all green checks.....gets the available fields....

markusa’s picture

markusa commented

Get a ton of these errors:
warning: mysqli_num_rows() expects parameter 1 to be mysqli_result, boolean given in /var/www/vhosts/dfwrealestatehunter.com/httpdocs/includes/database.mysqli.inc on line 184.

with the occasional:
SimpleXMLElement::__construct() [simplexmlelement.--construct]: ^ in /var/www/vhosts/dfwrealestatehunter.com/httpdocs/modules/pirets/pirets.connect.inc on line 347.

markusa’s picture

markusa commented

Wow....turns out it must be a problem with having selected a field that gives error. Don't know which one but I went and unchecked most fields...took it to bare bones and it started to work....now the long task of figuring out which fields will work ok!

markusa’s picture

markusa commented

This is a wonderful module! Thanks for all your effort. Only took me about 3 days to figure it out!

markusa’s picture

markusa commented

Okay apparently its not a type of field....but I am running up against some kinda of limit of the number of fields. Got about 61 or 62 fields now....add one more and boom the behavior described above.

Any ideas on that? Some kinda query limit? Mysql limit?

Garrett Albright’s picture

Garrett Albright commented

As mentioned in the earlier thread, the "downloading forever" problem is probably related to this.

As for your other error, sorry, but I can't imagine what could be going wrong.

markusa’s picture

markusa commented

Yep...obviously there is some limit of records that can be fetched or something....

Two classes of listings work like a dream...the biggest has about 2500 records....the other three classes have much larger number of listings....when trying to update it says the correct number of listings but hangs saying:
Remaining 2 of 3.
Processed 0 of 17574 (0 created, 0 updated) in class LND.

And doesn't process any.

I even backed off the fields I was trying to fetch to the bare minimum and it still doesn't go. I guess what I really need to for their RETS server to allow chunked amounts and not have the goofy behavior that you mentioned where it keeps sending the first chunk over and over. Does the fact that I was able to get the smaller listings classes to go after I backed off the number of fields relate to this somehow....maybe its only one problem.

Can you recommend a course of action to troubleshoot/fix this? Other than telling the RETS company that they blow? I don't think they care what I think.

Ever got your module to work for the Dallas area NTREIS MLS server?

markusa’s picture

markusa commented

I have this information provided by NTREIS:

NTREIS RETS has a default download limit of 400 records. You can use the LIMIT parameter in your RETS Client to allow for more records. See the „RETS 1.5 Specification Manual Section 7.4.3‟ for information on the LIMIT parameter

When set to unlimited what value is sent for the LIMIT parameter?
Shouldn't I be able to set the limit to say 40000 records instead of unlimited?

They also say this:
If your attempts at an initial full download are unsuccessful, NTREIS suggest the large download be subdivided into smaller packages using MODIFIED.

I guess I'll be diving into the PIRETS code to solve this problem! By the way I have tried both the 3.0 beta 6 and the dev version available from the drupal module page. The dev is what I am currently using

markusa’s picture

markusa commented

Ends up not being a code problem....it was the mysql max_allowed_packet setting....upped it to 32MB and I was able to get all the listings using the unlimited chunk size setting.

Your module is awesome! Now I've got to try and enable all the fields and see if that works right.

markusa’s picture

markusa commented

Ended up having to set the max_allowed_packet to 128M in the my.conf: the mysql config file
and I also had to add query_cache_limit=2M
query_cache_size=64M

Garrett Albright’s picture

Garrett Albright commented
Status: Active » Closed (fixed)

Ah, good. :D