I found that when module updates geoip info, it first clears table and then inserts data. If we keep in mind fact that this update is quite slow, it means that for a quite long time module will be useless and non-working.
I've posted a patch which makes update process faster but there are two nuances:
1. It won't insert data in table all the time but will do it once when finished parsing (all this time table will be empty).
2. Update is still too slow (and again all this time table will be empty).
Is there any ideas to change it?
I can suggest to play with tables: insert into tmp table -> rename ip2country to ip2country_old -> rename tmp table to ip2country -> remove ip2country_old table. Also may be this solution would be better if used with db_transation (not sure).
Comments
Comment #1
tr commentedYes, it's true that the table is dropped first then re-created with the new data. This can pose two problems: First is that if the update fails, you'll only have a partial set of IP data in your database. Second, as you mention, the IP data is incomplete for the 2 minutes or so it takes to do the update.
The first problem can be fixed by using a temporary table, but that should also be accompanied by a check to see that the new dataset is approximately the same size as the old dataset - this will prevent swapping out the tables in the case that the new data is incomplete due to a network or server error. Transactions should *not* be used, because the dataset is far too large to lock up everything in one transaction. Besides, if you put the database updates into a transaction you'll hang every client that tries to access that data, until the transaction is done. To me, that's worse than a simple failure to find the IP in the database during update which is what occurs now.
The second problem can be mitigated by controlling how and when you perform updates. It's not necessary to update very often, so this gap occurs maybe once per week for a normal site (although, many sites don't like to change their data even once per month). (Note that other modules use a third-party IP database which doesn't even get changed more than once per month, and those other modules require manual updating of that third-party database.)
If you're willing to create a patch which loads the new data into a temporary table then swaps the temp with the real table if and only if the temp is complete, then I'd like to include that into this module. Perhaps a better approach would be to use db_update() instead of drop/insert, as updating the existing data would preserve *all* the data during the update process. The problem then becomes culling obsolete rows from the table - I don't know how big a problem obsolete rows pose.
Comment #2
tr commentedThe latest -dev implements a scheme for updating the data safely, without first deleting the existing data. Also implemented are data integrity checks using the MD5 checksum and database summary information downloaded from the RIR server. The end result is that database updates happen completely and without error, or they don't happen at all, and the module continues to operate with a full database while the update is happening. Error reporting has been improved so that the watchdog will report update failures and their cause.
The 7.x-1.x and 8.x-1.x branches have this fix. The 6.x-1.x branch will not be receiving this fix, as it is near end-of-life. Also, database transactions are not supported in Drupal 6, and that is necessary for this feature to work (the old table is deleted and the temporary table is renamed within a transaction, to ensure that both operations succeed or fail together).