Early Bird Registration for DrupalCon Portland 2024 is open! Register by 23:59 PST on 31 March 2024, to get $100 off your ticket.
Under heavy load, we are experiencing deadlocks and lock wait timeouts within radioactivity_cron:
warning: Deadlock found when trying to get lock; try restarting transaction#012query: UPDATE radioactivity SET energy=energy * pow(2, (last_emission_timestamp*1.0-1349251314)/21600), last_emission_timestamp=1349251314 WHERE decay_profile=1 AND last_emission_timestamp<1349251314 in _db_query() (line 169 of .../www/includes/database.mysqli.inc).
Comment | File | Size | Author |
---|---|---|---|
#3 | deadlock-errors-1802118-2.patch | 11.55 KB | bropp |
#1 | radioactivity-fix_lock_timeouts-1802118-1.patch | 5.74 KB | pdrake |
Comments
Comment #1
pdrake CreditAttribution: pdrake commentedThis patch adds the option to process the update query in a batch format where a limited number of rows are processed per update hook. This is important because the query requires a full table scan which will ultimately lock every record in the table and the update query can take an appreciable amount of time on a large table. Processing in batches, combined with the new index, requires a lock on only a few records at a time. On busy sites, the time taken while locking the table can result in deadlocks or lock wait timeouts. By default, this query will behave as it did before, unless radioactivity_decay_batch_size is set. Additionally, I have wrapped the entire function in a lock_acquire to ensure this function is not called multiple times concurrently as that is a sure-fire way to cause deadlocks.
Comment #2
bropp CreditAttribution: bropp commentedWas getting this error in the 7.x branch as well, so rolled a patch based off of the one in #1.
It's not perfect and uses db_query because D7 doesn't support LIMIT in update queries. I haven't tested it on redis/memcache installs either. Otherwise, appears to be working.
Comment #3
bropp CreditAttribution: bropp commentedPatch attached.