Fast file-based locking

Quick review:

• Can you summarize why database locking can't be used today?
• Don't show developer speak in user output/documentation (eg. form description of lock settings). Reword these for clarity.
• I don't know why but I have problems with the names lock_rlock() and lock_rwlock(). I think I prefer to have one function, with an additional paramater to specify the lock-type. Maybe lock_obtain() and lock_release(). I'm also more familiar with the terminology 'shared' and 'exclusive' lock but that might be my particular background. Not sure.
• _lock_file_sanitize_name() and friends seems like bloat to me. It has only one call-site, plus developers can be instructed about what good names are. It is not like we are dealing with user input.
• If we don't need attributes now, please don't add them. If we need additional attributes, the API does not shield the programmer from internal implementation details.
• Are blocking locks a good idea? What happens when I own a blocking lock, and my process gets terminated/abrupted? Will all other processes come to halt until the GC comes along?

If database locking can be used efficiently, I'm in favor of standardizing on database locking. I'd think databases are optimized for this; no need to extend database functionality ourselves or to build our own layer on top of it.

Note that we already use database locking for the sequences table:

includes/database.mysql.inc:  db_query('LOCK TABLES {sequences} WRITE');
includes/database.mysql.inc:  db_query('UNLOCK TABLES');

that LOCK TABLE sequences statement causes lots of support headaches because shared web hosts almost never give the permission required to obtain this lock (applies to MySQL4 only). On several client sites I have commented out this line entirely ... File based locks avoid this permission issue nicely.

I can't imagine why file-based locking would be faster than SQL-based locking. If it would be, MySQL could re-implement their locking mechanism using files (if they aren't using files already)? I understand that finer granularity could improve performance, but according to the MySQL documentation table-based locking is usually much faster than row-based locking (except in rare cases). Details at http://dev.mysql.com/doc/mysql/en/table-locking.html.

Note that I'm just playing the devil's advocate here (and in the comment above). I want to make sure that the code bloat introduced by file-based locking is justified.

As Drupal matures it is attracting bigger projects using Drupal on load-balanced setups (eg. multiple webservers in front of a database server). Should we maintain a separate locking-backend for them, or use one that is compatible with their environment to begin with? I'm OK with using this approach for now, but in the end, I'd rather have us maintain a single locking backend, than two.

Also, the current locking API is not compatible with database locking, I'd think. It does not allow switching between database and file-based locking without rewriting code. This matters because the next logical step would be support for transactions to guarantee true database integrity. File-based locking does not offer true database integrity. In a way, it is not future-safe.

Jeremy: it might also be worth checking out MySQL's GET_LOCK(), RELEASE_LOCK(), IS_FREE_LOCK() and IS_USED_LOCK(). See http://dev.mysql.com/doc/mysql/en/miscellaneous-functions.html. These locks are implemented with pthread_mutex_lock() and pthread_mutex_unlock() for high speed and can be used as generic application locks. Might solve the NFS issue at a reasonable cost.

Not going to commit this as in. Not until this has been investigated further, and until we had some feedback. For now, I'm marking this 'active' again. Feel free to set it to 'patch' when there is a follow-up.

I added some locking to drupal.org's bootstrap.inc and evaluated it over the past 2 days:

SELECT GET_LOCK("cache-$cid", 10) <- takes 0.22ms
SELECT RELEASE_LOCK("cache-$cid") <- takes 0.21ms

LOCK TABLES cache WRITE <- takes 0.11ms
UNLOCK TABLES <- takes 0.15 ms

Most of the time, the performance difference can be ignored. Sometimes, table locking is slightly slower but not much. In general, table locking does not seem to be a problem on drupal.org. They don't represent the fastest queries, but they certainly aren't the slowest queries either. Quite the contrary.

I created a locking mechanism based on dir, since according to the php docu, that is the only file action m guaranteed to be atomic.

I created it to solve issues that I had with the file locking that was used in the banner module at that time.

This one really works great! I tested it intensively and it did not screw up once. However, I did not do a scripted test or something, just normal browsing. But still the improvement to the php method was obvious, since the original locking mechanism did even screw up, when it was only me accessing the file!

It is some time that I did it, but as far as I remember I implemented exclusive and shared lock both based on dirs! And the flowchart was created with reliability in mind. There should not be a single case, where this thing could screw up. Better save then sorry.

This is not a patch but since I only added something at the end and the file version information is at the very top, it should not be a problem. My basis was drupal 4.5.0.

I will upload some additional files (my banner module) with some instructions to test this set of functions.

The banner module brings some enhancements, I'll try to remember, what all
that was:
* I created a better statistics caching mechanism paired with a new file
locking mechanism.
* You can have more than one banner block and still, the thing does not screw
up, and use the same file to cache statistics.
* Even when more than one banner place is used, banners do not show up twice
at different places.
* You can show more than one banner per category.
* Banner statistics are only appended to the cache file, so it does not have
to be read before writing.

look at drupaldev.flachware.com for a demo.

I did never finish it though, but would feel very sorry, if the effort would
be lost, since it performs very well. It was based on whatever version was
available at drupal 4.5.0 release.

As far as I remember, the presentation side was finished, but not the file
upload, ... stuff. I changed some things in the database as far as I remember
and the form, how the cache files are written changed.

Here are the files for review and testing, concerning the cache files, I do
not remember exactly but one might be obsolet.

includes/file.inc - containing the file locking mechanism
banner_file.php
banner_db.php
misc/7.banner.count
misc/7.banner.cache
modules/banner.module

If you are not interested in my other improvements, you should at least check
the new file locking scheme, it is really worth it, I put a lot of effort in
making it bulletproof, ultra fast, and reliable.

Please let me know, if you could use the code and whether you need help
understanding it. my time is very limited at the moment, but I will try to
help you where ever possible.

It seems this was a performance problem related to mysql 3.x
Anyway, the problem about "LOCK TABLES" had already plenty of discussion around here, feel free to search for those discussions.