Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
While importing a larger number of nodes, I get lots of OVER_QUERY_LIMIT errors. Here is a patch to sleep a small amount of time before each query, and possibly retry a configurable number of times with increasing sleep time for each retry.
Comment | File | Size | Author |
---|---|---|---|
#9 | google-geocoder-delay-1672742-9.patch | 1.53 KB | joelstein |
#1 | geocoder-over-query-limit-1672742-1.patch | 8.65 KB | gnucifer |
Comments
Comment #1
gnucifer CreditAttribution: gnucifer commentedComment #2
kevinquillen CreditAttribution: kevinquillen commentedAre there better ways of avoiding OVER_QUERY_LIMIT? I think I went over mine on Saturday, and 1000 coordinates were wiped out.
Comment #3
gnucifer CreditAttribution: gnucifer commentedBetter how? You have a limited number of queries per day if you choose not to pay for more. The problem for me is that there also seems to be a query frequency threshold that is rapidly exceeded with the current implementation. Sleeping 100ms or so in between each query resolves this, but if you are out of total queries this patch will not help you. The only solution in that case is to wait 24 hours, or purchase extra queries.
Comment #4
kevinquillen CreditAttribution: kevinquillen commentedHitting the limit is fine- what I want to prevent is replacing valid data with NULL if I am over the query limit, which is what appeared to happen to me a few days ago.
Is there no way to not geocode if the target field (Address) has not changed, or just a way to skip?
Comment #5
steinmb CreditAttribution: steinmb commentedIf you know you are doing a lot of queries to the same address (reimporting stuff etc) you could cache the result with Varnish. I did that on a dev. system where I had a few migrate tasks that was pulling content from an older legacy system into Drupal.
Comment #6
kevinquillen CreditAttribution: kevinquillen commentedHow did you do that?
Comment #7
steinmb CreditAttribution: steinmb commentedRequests to maps.googleapis.com is mostly HTTP GET, so they are not hard to cache, but you need to change your local DNS/local host file to point maps.googleapis.com to your local proxy server.
Comment #8
Rob_Feature CreditAttribution: Rob_Feature commentedthis no longer applies to the latest dev but is a much needed patch. changing to needs to work for updating...
Comment #9
joelstein CreditAttribution: joelstein commentedHere's a patch similar to the one rolled for the Location module (#697468: Add configurable delay for Google geocoding to avoid 620 errors), which adds an option to configure milliseconds of delay between geocoding requests. This one was committed to Location over a year ago, so I think we can safely use this same solution.
You can read more about Google's usage limits (and the OVER QUERY_LIMIT response) here.
200ms seems to keep my site under the rate limits (no more than 5 requests per second). It defaults to 0 (no delay). After applying the patch, you can add a value of "200" to the new field at "admin/config/content/geocoder".
Comment #10
joelstein CreditAttribution: joelstein commentedMy patch could probably be improved upon by adding in the exponential delays suggested in comment #1. I'm not sure if that would make a difference to Google, but maybe so.
Comment #11
pedrop CreditAttribution: pedrop commented#9 patch worked for me, thanks.
There is a 2500 query per hour limit in the google documentation so a delay over 1.44 sec seems to be the safe solution if you have to recode 2500 or more geofields.
Comment #12
Cauliflower CreditAttribution: Cauliflower commentedUnfortunately, the limit is 2500 request per day :-(.
Comment #13
kevinquillen CreditAttribution: kevinquillen commentedYeah, it is per day, and Google doesn't play around.
Comment #14
fonant CreditAttribution: fonant commentedFor a feature request for an alternative solution, more useful where site visitors are performing arbitrary proximity searches, see #1949954: Client-side geocoding for proximity searches, to avoid Google's new 2500 per day limit
Comment #15
gregglesUpdating the title to be more descriptive of the potential problems. Google returns OVER_QUERY_LIMIT for two scenarios and this patch is meant to fix only one of them. The other (2,500/24 hour period) can only be fixed by paying money to google or using a different backend. Discussing that issue here is off-topic.
I believe the solution in #9 will work for situations where there is only one process doing geocoding at the same time. If you have a url on the site that does geocoding and 6 people hit it at the same instant then this won't protect you. That's probably OK for most sites, but I wanted to note that limitation.
I'm using geocoder on a page where I actually use it 11+ times in a row to serve the single page (geocoding a zip, geocoding up to 10 addresses). I get these errors pretty consistently. I deployed this patch and increased the delay to 201ms and it seems to prevent the problem quite well. I'll humbly suggest it is RTBC.
Comment #16
gregglesI can now say with greater certainty that after I set it to 210ms there have been no more problems with OVER_QUERY_LIMIT in the last 7 hours where before I would get 1 or 2 per hour.
Comment #17
Brandonian CreditAttribution: Brandonian commented#9 committed. Thanks @joelstein!
http://drupalcode.org/project/geocoder.git/commit/c1a79dc