Optimization of updating geoip information

Actually, I tested something similar several years ago. The problem is that for any real data set, you run into database and web server limits when you try to do all the queries at once. AFRINIC is anomalous because it only serves a small subset of the IP data, but even then you'll see that the testbot ran out of memory during the test cases. (More common are database failures because the socket is open too long or the query is too large.) Drupal has no control over server and database resources, and there is a huge variation on the default values of these resources depending on what hosting you're using, so requiring any limits to be more than typical will prevent this module from being used on a lot of sites.

One other strategy I tried was to batch the queries to insert more than one row at a time. I performed detailed tests with different sized chunks of 10, 50, 100, 200, 500, and 1000 rows at a time, and I found there was no significant improvement in speed over the one-at-a time query. This is because each time you set up a new query you have a lot of overhead which dominates the total time.

Because of both these factors, I concluded it was better to keep the code simple and stay as far away from resource limits as possible. So I'm inclined to mark this as "won't fix" unless you can get the testbot to pass and unless you can demonstrate speedup of the inserts without increasing any web server or database limits.

I've attached a patch which contains my test code - you'll have to edit it, but at least it will show you what I tried.

As I said in #2, the proposed change simply will not work.

I am open to suggestions on how to make the process more efficient, but my testing shows that the speed is limited not by the network fetch time, nor by the PHP processing time, but by the DBTNG processing time. A complete update from the comprehensive RIRs (i.e., any other than AFRINIC) takes only 1 minute, of which 2/3 of the time (about 40 seconds) is spent in the Drupal DB routines.

If anyone has a patch or an idea for speeding up the process, please open a new issue. I have tried many different things, but haven't found anything that significantly improves the update time.

Grammar error fixed