Problem/Motivation
Cron hangs when Visitor Info module is enabled and a lot of IP addresses must be queried from ipinfodb.com to update expired IP information. There is no limit to this process which can result in cron timeouts.
Proposed resolution
Proposed solution is to limit the number of IP addresses to process on each cron run.
Remaining tasks
Review the patch from comment #3.
Original report by malc_b
Hi,
This was all working but I've just noticed that cron hasn't run for 4 weeks. I installed supercron and with that tried each in turn. The problem one was visitorinfo. I see you have a version 2.0 but it doesn't look a simple upgrade. It would be nice if you provide instructions to get back to the same as 1.6 with a google map. A least that would be a simple upgrade then. I've got the geoip module installed but I've not used gmap. I've put the upgrade on the back burner for now. What I'd like to fix is 1.6 so I can at least get my visitor map back in the interim. What could be causing it to hang in cron? Looking at the code I see it pauses between IP request but doesn't seem to have an overall limit on the IP requests it will process. I wonder if this my be the cause with a backlog of IP addresses to lookup. How could I clear the backlog? Or do you have a recipe I can follow for 2.0 to get a basic visitor map like in 1.6?
Thanks.
| Comment | File | Size | Author |
|---|---|---|---|
| #9 | interdiff.txt | 298 bytes | berliner |
| #9 | visitorinfo-1133660-9.patch | 2.68 KB | berliner |
| #3 | visitorinfo-1133660.patch | 2.69 KB | berliner |
Comments
Comment #1
twooten commentedNot sure why your cron is hanging. Are there any errors in your logs? How often is your cron supposed to be running?
Version 2 is a pretty simple upgrade. The data the Visitor Info provides is meant to be used with Views.
From the README.txt:
The idea here is that Visitor Info collects, geocodes and stores all the
visitor information in the database. You can then list this information in
any way you want, either in a simple list, a table or you can use any of
the several available Views map plugins to display this data on a map.
For instance you can use the GMap module to display your visitor on a Google
Map. Because this works with Views, you can configure it to display only the
information you want and hide stuff such as latitude and longitud or IP which
could be considered sensitive information. It is also possible to use another
mapping solution such as OpenLayers (http://drupal.org/project/openlayers).
Tim
Comment #2
malc_b commentedHi,
Thanks for the fast reply.
It's fixed itself. I'd installed supercron and was invoking visitorinfo cron via that to test it. I was getting 500 errors and trying to find anything in the main (shared) error log. Then it worked. I reckon the problem is that visitorinfo_cron() is open ended. It will look up as many blank IP as there are and each time it waits 0.5s. Cron and functions have a timeout limit so I think I was just hitting that. The final time when it completed it had take 155s so the timeout must be longer than that. The visitorinfo cron function probably should be limited to do all blank IPs OR some maximum number of lookups so it doesn't take too long.
I've also now set the refresh time to 0 so that it never refreshes. That seems like it also might be part of the cause as at some time doesn't this mean it will try to do all the IPs again? Or at least 2 days worth.
Part of the problem might also be that my cron had stopped. Maybe because cron was interrupted (host had moved not long ago) and that broke it until I cleared the semaphore flag. By then I had 4 weeks backlog on visitorinfo.
Malc
Comment #3
berliner commentedI ran into this problem myself on a rather huge community site where cron suddenly stopped working. Then I remembered that this error has originally been introduced by my own patch from #783242: IP updates. To fix the problem and still get IP updates I decided to limit the number of records processed on each cron run.
So I propose another patch that adds this functionality including a new admin setting to control the number of records to process.
Comment #4
riversidekid commentedIs this patch likely to appear in the next version?
Comment #5
twooten commentedThis looks like a good solution. I'll get it in the next release.
Comment #5.0
twooten commentedIssue clarification in summary
Comment #6
malc_b commentedThe patch seems to be malformed if I'm reading it right. At line 5 it has
so line 233 should be "function visitorinfo_cron() {" which is it is (well 232 in .dev version). But then the next block in the patch file says
@@ -241,19 +242,22 @@ function visitorinfo_cron() {And you can't have the same text at both line 233 and 241. At 240 the module has " if ($interval > 0) {"
I ended patching by hand as that wasn't the only problem (well I could get patch to run so quicker to do it by hand). This was on the latest dev version which I would have thought would have included this.
Comment #7
berliner commentedThe patch still applies. I just pulled the latest dev and applied the patch successfully like this in the module root:
Concerning the function name:
This only serves as context information. Technically it's not part of the code changes inside a patch.
Comment #8
malc_b commentedThanks for the quick reply.
Must be have been the -p1 that was the problem them. The error from patch was VERY cryptic and when I looked at the patch file this looked like an error.
I do think line 30 is wrong though
$result = db_query_range("SELECT csid, ip FROM {visitorinfo} v WHERE v.city = '%s'", $nothing, 0, $limit);since you removed $nothing so '%s' should be ''.
I think that line should be
That way you fill in the latest results first (but maybe the natural result is csid DESC). The latest results are the more important as the map is filled with csid DESC.
The other thing I can't find is anything which trims the table or re-arranges it. The logic seems to be:
Get visitor's IP
Check if IP already in database, if not insert it.
At cron fill in new IP locations.
For map grab the latest, new, 200 IPs
The interesting thing is that if the IP is from a repeat visitor then nothing changes in the database. Since the map takes the latest 200 (or whatever figure you set of course) then only new visitors appear on the map. I would have thought that latest visitors (new or repeat) is more logically. Or most frequent visitors (which would mean recording a count of course).
If the aim is to show the latest visitors then after find the visitor is in the database the city etc. details could be grabbed and the visitor inserted again. Then cron could delete entries after 200 with city etc. details (or whatever is the map setting).
Alternately the table could have a visit_count which is inc'ed each repeat visit and a visit_date which is also updated. You could then opt for latest 200 or most frequent 200. Table trimming might be left to manual, maybe.
Comment #9
berliner commentedYou're right. I corrected that.
The rest of your comment addresses topics that are out of scope of this issue, which deals with the hanging cron. I would suggest to open a new issue for that.
Comment #10
malc_b commentedGood point. I'll start another specific thread.