Avoiding OVER_QUERY_LIMIT due to more than 5 requests in a second in google geocode

Are there better ways of avoiding OVER_QUERY_LIMIT? I think I went over mine on Saturday, and 1000 coordinates were wiped out.

Hitting the limit is fine- what I want to prevent is replacing valid data with NULL if I am over the query limit, which is what appeared to happen to me a few days ago.

Is there no way to not geocode if the target field (Address) has not changed, or just a way to skip?

If you know you are doing a lot of queries to the same address (reimporting stuff etc) you could cache the result with Varnish. I did that on a dev. system where I had a few migrate tasks that was pulling content from an older legacy system into Drupal.

How did you do that?

Requests to maps.googleapis.com is mostly HTTP GET, so they are not hard to cache, but you need to change your local DNS/local host file to point maps.googleapis.com to your local proxy server.

this no longer applies to the latest dev but is a much needed patch. changing to needs to work for updating...

Here's a patch similar to the one rolled for the Location module (#697468: Add configurable delay for Google geocoding to avoid 620 errors), which adds an option to configure milliseconds of delay between geocoding requests. This one was committed to Location over a year ago, so I think we can safely use this same solution.

You can read more about Google's usage limits (and the OVER QUERY_LIMIT response) here.

200ms seems to keep my site under the rate limits (no more than 5 requests per second). It defaults to 0 (no delay). After applying the patch, you can add a value of "200" to the new field at "admin/config/content/geocoder".

My patch could probably be improved upon by adding in the exponential delays suggested in comment #1. I'm not sure if that would make a difference to Google, but maybe so.

#9 patch worked for me, thanks.
There is a 2500 query per hour limit in the google documentation so a delay over 1.44 sec seems to be the safe solution if you have to recode 2500 or more geofields.

Unfortunately, the limit is 2500 request per day :-(.

Yeah, it is per day, and Google doesn't play around.

For a feature request for an alternative solution, more useful where site visitors are performing arbitrary proximity searches, see #1949954: Client-side geocoding for proximity searches, to avoid Google's new 2500 per day limit

Updating the title to be more descriptive of the potential problems. Google returns OVER_QUERY_LIMIT for two scenarios and this patch is meant to fix only one of them. The other (2,500/24 hour period) can only be fixed by paying money to google or using a different backend. Discussing that issue here is off-topic.

I believe the solution in #9 will work for situations where there is only one process doing geocoding at the same time. If you have a url on the site that does geocoding and 6 people hit it at the same instant then this won't protect you. That's probably OK for most sites, but I wanted to note that limitation.

I'm using geocoder on a page where I actually use it 11+ times in a row to serve the single page (geocoding a zip, geocoding up to 10 addresses). I get these errors pretty consistently. I deployed this patch and increased the delay to 201ms and it seems to prevent the problem quite well. I'll humbly suggest it is RTBC.

I can now say with greater certainty that after I set it to 210ms there have been no more problems with OVER_QUERY_LIMIT in the last 7 hours where before I would get 1 or 2 per hour.

#9 committed. Thanks @joelstein!

http://drupalcode.org/project/geocoder.git/commit/c1a79dc