Creating severe load on system by logging thousands of unlink errors in DB log

I've seen a similar problem currently. Looking at possible fixes.

I've made a simple edit to the unlock method to say:

function unlock() {
if (file_exists($this->lock)) {
unlink($this->lock);
}
}
}

and it seems to remove this pesky warning from bloating the logs.

subscribing...
My error lists /tmp/cache_content_lock
Technically, it is correct; there is no such file in /tmp.

If you are using a check to see if file exists (as suggested in #2) then won't it increase diskio further decreasing performance?

I am getting same error while using APC and it creates drastic load on the server. Not because of the change you suggested, just by just installing cacherouter since it is creating hundreds of error messages in the log. I am using APC as engine.

my error is slightly different (same place line 124) but different error.
unlink(/tmp/www_-cache_filter_lock) [<a href='function.unlink'>function.unlink</a>]: No such file or directory in /var/www/vhosts/domain.com/httpdocs/sites/all/modules/cacherouter/Cache.php on line 124.

Why it is trying to unlink files when I am not using file engine at all?

Same error has been reported with xcache too. #575920: Xcache - unlink errors

function unlock() {
if (file_exists($this->lock)) {
unlink($this->lock);
}
}

Even with the above change, I am getting some errors:

	unlink(/tmp/cache_lock) function.unlink: No such file or directory in /<snip>/modules/cacherouter/Cache.php on line 125.
 

Backtrace (courtesy of drupal_tweaks module)

Backtrace: unlink(...)[Cache.php:125] <=unlock(Array)[apc.php:109] <=set(...)[CacheRouter.php:63] <=set(...)[cacherouter.inc:93] <=cache_set(...)[jq.module:50] <=jq_plugins(...)[jq.add.inc:6] <=_jq_add(...)[jq.module:34] <=jq_add(...)[twilio.module:17] <=twilio_init(Array)[:] <=call_user_func_array(...)[module.inc:471] <=module_invoke_all(...)[common.inc:2590] <=_drupal_bootstrap_full(Array)[bootstrap.inc:1326] <=_drupal_bootstrap(a:1:{i:0;i:8;})[bootstrap.inc:1209] <=drupal_bootstrap(a:1:{i:0;i:8;})[index.php:16] <=index.php

It appears that the origin of the error is a module Ive written ('twilio.module') that uses a jquery plugin invoked via the jq module, which attempts to use D6 core's cache_set, which meanders through APC (which I am obviously using) and on to the unlink error.

Is this related to the other unlink errors in the issue queue?
Is there any way to mitigate it at this time?

Thanks.

I do not have your twilo module still I see the error. :)

yeah I tried to word my post ("origin" !== "culprit" ) so as not to blame my own module, ;)

The file might be unlinked by another concurrent request right between file_exists() and unlink() - unless we use locks on deletion of files. I would try an atomic @unlink() instead.

There is a thread on the D7 issue queue regarding the implementation of a lock system to avoid race conditions like these, using semaphores. (http://drupal.org/node/251792)
There is a patch for D6.14 (http://drupal.org/files/issues/lock-251792-224-D6.patch) that implements the locking framework and fixes a few race conditions (menu_rebuild(), and something to do with translations).
Perhaps this would be helpful in this case?

I am not sure this is a good long-term option, because it remains to be seen whether this will make it into D6 core at all.

EDIT: It is curious to me that, if this is indeed due to a race condition, that it has never been reported before - this is a type of module that should be used on a high-volume site, and one would think that this situation would occur frequently on such a site. The project that I've rolled cacherouter out on is new, and besides se spider traffic, I've had only maybe 1,000 hits in the past 5 days.

I don't think this is a race condition related to file unlink. Please note the issue is against APC cache. The issue is cache router is trying to unlink APC cache entry as if it was a file. And it can't because there is no file by that name.

e.g in #4 above "unlink(/tmp/www_-cache_filter_lock)", "www_-cache_filter_lock is a memory based entry in user data portion of APC not a disk based file under /tmp.

So cacherouter somehow needs to understand it is dealing with APC and should delete the cache entry instead of "unlinking" thinking it is a file. It should not even try to do any diskIO to check if the file exists since it can't exist. That is why #6 is not a solution. It will just supress the error message but 1) it won't delete the cache entry when it is supposed to and 2) unnecessarily increase disk io, negating the very reason APC is used.

I never really used APC, but from reading the code I see, that this is quite not the case. Apart from storing actual data in APC, cache router also implements a locking mechanism to avoid collisions between concurrent requests. This is implemented with the use of dummy empty files in tmp directory (with a good descriptive names employing a "lock" suffix), and obtaining flock() locks on these. Unless APC have some native locking mechanism (I really don't know this bit), this implementation is fine. The unlink only just removes the dummy file when the lock is lifted (which is in fact just a cleanup operation, not strictly required).

Looking at the code I can see your point. However looking at engines/memcache.php it has own implementation of lock() and unlock() which does not use filesystem at all. So perhaps APC need similar handling of lock /unlock?

Ok the orginal version before rc1 , APC engine had its own locking mechanism. This was specifically deleted by this commit http://drupal.org/cvs?commit=259630
"tweaked apc/eacc/xcache to use files for locks for better reliability and true atomic locks."

see the diff
http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/cacherouter...

So since it no longer has lock/unlock implementation in the file, it executes the default one from from parent class.

Subscribing.

I installed Cache Router last night and was alarmed to wake up to a screen full of error messages.

I've suppressed the error messages for now, but I agree that if no call to the file system is necessary it should be avoided.

@ajayg: I'm not sure I understand. If there are files for locks, why are we all getting these messages? Is there nothing that needs to be deleted, or are there files being created but not deleted?

It seems the files are for lock but some how they don't exists or getting deleted somehow before cacherouter needs to delete at appropriate time. Perhaps they are not getting created as part of lock and unlock is trying to delete.

I haven't see any lock file getting created at all, but I could be just looking at wrong moment after they are deleted.

subscribe

  function unlock() {
    if ($this->lock_fp) {
      flock($this->lock_fp, LOCK_UN);
      fclose($this->lock_fp);
      unlink($this->lock);
      $this->lock_fp = null;
    }
  }

This will fix your problem. We need to unlock the file first, close the handle and then unlink the file. I think this problem only occurs on windows system.

Using a debian box here, still have the problem.

try the fix. does it solve ur pb ?

<?php $conf['cache_inc'] = './sites/all/modules/authcache/authcache.inc';
$conf['cacherouter'] = array(
  'default' => array(
    'engine' => 'apc',
    //'server' => array(),
    //'shared' => TRUE,
    //'prefix' => '',
    //'path' => '',
    'static' => FALSE,
    'fast_cache' => FALSE,
  ),
);

?>

i see you commented the //path

why? cause cache needs to know where to store files. thats why you have error "no such file in directory" etc

FWIW, facebook uses the pthread mutex lock option.

I've also been seeing some issues with this new locking system in place. I think the default APC file locking system is supposed to be pretty efficient.

I am also seeing these errors. Mostly for cache_filter_lock.

The proposed fix in #18 did not work for me. I am on a CentOS system and it gives me:
PHP Parse error: syntax error, unexpected $end, expecting T_FUNCTION in /var/www/vhosts/domain.com/httpdocs/sites/default/modules/cacherouter/Cache.php on line 131

Also, I have set my path as suggested in #21 to no avail. I am still getting errors for files in /tmp and nothing is created in the defined path.
Setting "lock_dir" (called at line 48 of Cache.php) does change the error to show the defined directory
(ie - 'lock_dir' => 'sites/default/files/filecache',). Using this I had to manually create the filecache directory. However, I still get the errors.

I am seeing the files being created and deleted from time to time though.

Same problem here:

Warning: unlink(/tmp/cache_lock) [function.unlink]: No such file or directory in /var/www/drupal2/sites/all/modules/cacherouter/Cache.php on line 124

Cacherouter 6.x-1.0-rc1
Drupal 6.14
CentOS 5.3

change the path for cache it may work then.

@fred it looks like a parse error, maybe you don't update the code properly ?

We're seing this too. Is the maintainer available for a reply?

Unfortunately maintainer hasn't replied for two months.

i've see the warning too.. but after i add this line in my php.ini. its gone!! and seems it works well

apc.mmap_file_mask=/tmp/apc.XXXXXX

may this could help

#29 works, I used:
http://www.innovatingtomorrow.net/2008/01/17/improve-php-performance-apc

hey guys, if you are running this on a production server you should have all the logging for php warnings turned off :/

i will have a new release to fix this issue before christmas (hopefully sooner)

sorry for the delay on this fix.

I am currently using this to stop the flood of errors. Looking forward for the fix :)

function unlock() {
$x = error_reporting();
error_reporting(0);
unlink($this->lock);
error_reporting($x);
}

Applied proposed solution at #29. Seems to work.

subscribe.. have this too on Ubuntu 9.10. Will try #29 and post back if it works.

Correction from #33: the unlink problems did appear again so no solution after all.

#29 worked for me, but APC still doesn't seem to giving a performance boost at all.

@q0rban and others for which #29 is working: What do you see in admin/reports/status/php for apc.mmap_file_mask?

I have /tmp/apc.KFc2WP for both Local and Master Value. This changes everything I restart apache. Is that the correct value?

@toemaz, I have something similar, but when I ls /tmp I don't see that directory. I wouldn't think I'd need to create it. Shoudn't it create the dir itself?

I don't see in ls -al /tmp neither. It's a mystery to me how this is supposed to work.

Hmm, copying apc.php into my webroot shows that it does appear to be working.. I think. :/

I think you could change the 'lock_dir' from cacherouter config in settings.php
I am testing a windows XP.

$conf['cache_inc'] = './sites/all/modules/cacherouter/cacherouter.inc';
$conf['cacherouter'] = array(
'default' => array(
'engine' => 'apc',
'server' => array(),
'shared' => TRUE,
'prefix' => '',
'path' => 'sites/default/files/filecache',
'static' => FALSE,
'fast_cache' => TRUE,
'lock_dir' => 'C:/WINDOWS/Temp',
),
);

[edit]
Thanks to #18 from "nquocbao" (October 27, 2009 - 13:42)
after I found the 'lock_dir' in configs, used #18 to solve this on my windows machine.
before I ad #18, lock files where created in 'C:/WINDOWS/Temp' but they where not getting deleted by unlink (Permission denied), afterward things seems to be working well :)
[/edit]

Subscribing

In my case, APC, Authocache, Cache Router and Block Cache Alter. From yesterday this error appear 10 times:

unlink(/tmp/domain-cache_menu_lock) [function.unlink]: No such file or directory in /var/www/vhosts/domain/httpdocs/sites/all/modules/cacherouter/Cache.php on line 124.

Unlinks errors should be suppressed by the patch for file engine lets focus on #578522: File engine: Performance, filenames, and other cleanup

About locking: filelocks are very slow suppose locking should be implemented by http://php.net/manual/en/book.sem.php is possible

Marked as duplicate #575920: Xcache - unlink errors

Digging deeper into this problem today...

base lock|unlock was introduced for engines that does not have own "native" locking so file locking was implemented

Seem this is a wrong way, because
1) file-locking is too slow - much slower then apc and memcache
2) brings troubles with Drush - different users for apache and cli #642544: Weird thing. Keeps happening.

Locking is needed to track lookup arrays which used by apc, shared memcache, xcache to make possible cache cleaning by "wildcard"

I was wondering about #1 above too. File locking may be negating any performance gains from memory caching e.g. APC.

I doubt it will negate performance, but it is definitely not as fast. #2 is not true. APC/XCache/eAcc bring problems with drush because command line php (php-cli) does not load opcode compilers. So what we need to do to fix drush support is to disable cacherouter or fall back on database caching if drush is being used.

We should fix this for 1.0 release, i'll add it to the roadmap.

On highload site I got a lot of troubles with filelocking - suppose it depends on FS-type but sometimes it just hangs

All the more reason to use native locking where possible. In rc1 i tested a new method, it obviously brings more harm than good, so let's revert back. I'll revert it back tonight and commit.

Is there a possibility of a dev release with this code reverted, I think most people would appreciate it. Thank you very much for your efforts.

I was checking the commit logs - doesn't look like a fix has been committed for this yet.

I think that much of what people have said in this thread so far is inaccurate - I don't think that this has anything to do with either file caching or with APC's mmap_file_mask option. Just to be sure, I tried what was suggested in #29 and it made no difference.

#44 (+ the correction from #46) are the truth about this issue, at least from what I can tell. So is there a native locking mechanism for APC? Because if there is, that would definitely be the way to go.

Eagerly anticipating a fix. Till then, I am going to put @ signs in front of all the calls to unlink.

Actually, I just found #14, and it solved the issue for me. So if you on APC, then take the code from http://drupalcode.org/viewvc/drupal/contributions/modules/cacherouter/en..., and add back in the lock() and unlock() functions and you will be back in business.

I think we should implement a global locking method which should be possibly configurable or by default check for available locking methods like: eaccelerator_lock() semaphores or fallback to file locking

+1

@EvanDonovan

I don't suppose you have the code from your fixed file handy do you?

I've edited mine but I keep getting white screens, that said, it is early on a Monday and my brain is still in neutral :)

Many thanks,
Danielle

@Dinis: Sorry, I don't. I switched from CacheRouter to Memcache module due to performance issues with APC user caching. It was causing too much fragmentation.

Yes, I am keeping APC for opcode cache , and removed cacherouter in favor of memcache API

I just started noticing this in my db logs. The exact error is:

unlink(/tmp/cache_rules_lock) [<a href='function.unlink'>function.unlink</a>]: No such file or directory in /home/domain/public_html/sites/all/modules/cacherouter/Cache.php on line 124.

I applied #29 but its still showing up. Could someone give some insight on how to resolve this? Thanks!

Is it possible this issue creates deadlocks on systems with high load?
(may be due to missing real semaphore based locking mechanism and lock / unlock race condition in)
We're facing deadlocks on a daily basis (debian lenny, apache2, apc)

Yeah... it's possible... I'm still trying to find a good locking solution...

Right now only eaccelerator have native locking, also probable to use semaphores as I pointed in #43

Andpost - it seems to me like apc_add contains atomic operations with semaphores. So only one instance at a time will get return TRUE for the valid key set - until apc_delete is called for it again.
This was in the code as you've previously pointed out. Why don't we add this again?

The php semaphore things are SysV semaphores with integer identifiers (ftok available)... i'm very concerned we'll find a reliable way for our flexible keys/identifiers...

Seems like building reliable locks is a huge problem.

Memcache has solution #493448: Memcache and cache_clear_all wildcard

@miro_dietiker about dealocks #766748: Caution when running w/APC older than 3.1
Also #245341: Unthrottled while loop in memcache engine causing high cpu utilization.

Subscribing

Same error :

unlink(/tmp/cache_menu_lock) [function.unlink]: No such file or directory in /home/drupal/sites/all/modules/cacherouter/Cache.php on line 125.

with APC cache enabled

subscribe

subscribing

same problem.

using xcache instead of apc

there is a mistake in my post. i think in my case it was not cacherouter bug but misconfigured xcache at my linuxbox

i had a line in my php.ini:
xcache.mmap_path = "/dev/zero"

when i setted a new value:
xcache.mmap_path = "/tmp/xcache"

the php warning disappeared

update:
lol. i posted it too fast. the error is still there.

holy moses, this thread has been open since September 2009 and yet still there isn't a fix

This already fixed in dev but still open while there's no release

So, what is the fix?

adresaklumea - use the -dev version!

I'm using the dev version and I still get
unlink(/tmp/cache_lock) [<a href='function.unlink'>function.unlink</a>]: No such file or directory ...

We should really roll-back this. I'm going to roll rc2 so is there any patches?

This is a rollback for APC

Commited as part on http://drupal.org/cvs?commit=410214

Code looks good.

For me, this fixed the "permission denied" problem on APC under Windows.

@janusman this was already commited so you can test this change using 6-dev version

I've been using dev for the past couple of weeks and am having a serious problem. I'm getting GIGABYTES of cache slam averted messages every day (tons of them EVERY SECOND). I was getting cache slam averted messages before upgrading to the latest dev, but the quantity was a tiny fraction of what I'm seeing now.

This is the message that keeps recurring:

[Sun Sep 5 05:49:39 2010] [apc-warning] Potential cache slam averted for key '/tmp/www-cache_content_lock' in /var/www/vhosts/mywebsite.com/httpdocs/sites/all/modules/cacherouter/engines/apc.php on line 212.

I'm running APC 3.1.3p1, and PHP 5.2.5, with apc.write_lock=1

The problem is discussed in a couple of places:
http://pecl.php.net/bugs/bug.php?id=16843
http://pecl.php.net/bugs/bug.php?id=16814&edit=1

It appears that the ability to turn off slam_defense was added in APC 3.1.4 (and available via patch for 3.1.3p1), but I'm unclear as to whether one SHOULD disable it. None of the discussions I've found seem to indicate that you should disable it in production, yet, strangely, the PHP docs seem to indicate that slam_defense has been deprecated in favor of write_lock: http://www.php.net/manual/en/apc.configuration.php#ini.apc.slam-defense -- so with write_lock on am I fine?

@azinck - I was about to post the same thing! Hopefully we can get some more info on this!!

I hope someone can let us know soon. My DB logs are inundated with these errors! :(

RikiB:
I updated to 3.1.4 which I initially thought had helped a lot. But then I saw that, for some reason, any Views I had stored in code were not being loaded. I'm assuming it's some sort of problem relating to require_once and APC. Anyhow, I reverted to APC 3.0.19 for opcode cache and am using cacherouter's file caching engine instead of the APC user cache for now. It's been working great.

Thanks. Everyone raves about how stable and fast APC is with Drupal but there is nothing but problems. I was using 3.0.19 for a while and it was causing major crashes on my site and also made the site very slow. This post probably explains why my site cant run APC under version 3.1 http://drupal.org/node/766748

So now I am using the latest beta 3.1.4 and while it doesnt seem to crash my server and is very fast, there are these (literally) thousands of db errors. I guess I could try 3.13 or something but I hate to reinstall APC after I finally got it working.

I tried using eAccelerator but cacherouter doesnt seem to work with the newer versions anymore as many people have discovered. The strange thing is the site was really fast even without cachrouter, but I dont understand how that all works. Is cacherouter even needed? I tried the "db" caching system which was very fast but Im not sure if there are potential problems with using that on a medium sized site, also the file caching engine was pretty slow compared to APC. I guess I could try xCache or something, but with all my research APC is supposed to be "the best".

Im hoping someone can post a solution to using APC 3.1.4 with Drual. Thanks again!!

Well I just disabled the module and APC seems to be working without cache router installed and doing a great job. Im not even exactly sure if I needed cache router in the first place, haha. It seems simply installing APC on your server will speed it up dramatically, I also installed memcache and together they are doing AMAZING!!

RikiB:

APC will work as an opcode cache on basically any PHP application by simply installing it on the server. Cache Router allows you to use the separate user cache functionality of APC to replace the database-based caching in Drupal.

If you have apc.php on your server you should see the user cache get populated with entries from Drupal when Cache Router is installed and properly configured. Now that you have Cache Router disabled you should no longer see those entries.

Thanks for the clarification. Is there a solution for the thousands of db errors? Perhaps downgrade APC to an earlier version, or is there something in the cache router code to support the latest APC 3.1.4?

subscribing

Subscribing.

#51 solved the issue for me. seems logical also.

I'm testing it on a test server.

I'm still seeing this problem with the newest DEV of CacheRouter + Xcache on Centos
warning: unlink(/tmp/xcachename-cache_menu_lock) [function.unlink]: No such file or directory in /public/sites/all/modules/cacherouter/Cache.php on line 129.

where is the code outlined in 51? I am not sure how to get to it after the git migration.

I modified the code from above
http://drupal.org/node/588820#comment-2195862

function unlock() {
    if ($this->lock_fp) {
      flock($this->lock_fp, LOCK_UN);
      fclose($this->lock_fp);
      if ($this->lock){  //added this check.
      unlink($this->lock);
      }
      $this->lock_fp = null;
    }
  }

Even after above I keep getting unlink errors to /tmp/xxxx-cache-y

I am also using xcache.

in Cache.php around line 55

  if (isset($options['lock_dir'])) {
      $this->lock_dir = $options['lock_dir'];
    }
    elseif (isset($default_options['lock_dir'])) {
      $this->lock_dir = $default_options['lock_dir'];
    }
    else {
      $tmp_dir = ini_get('upload_tmp_dir');
      $this->lock_dir =  (!empty($tmp_dir)) ? $tmp_dir : '/tmp';
    }

Where is upload_tmp_dir defined?

Maybe one of these might help?

1. Define lock_dir

2. Maybe need to define upload_tmp_dir in php.ini?
(It wasn't defined for me)
http://drupal.org/node/161948#comment-255653

3. Maybe define in settings.php or global.inc file (aegir)

 $thishost = $_SERVER['SERVER_NAME'] ;
 conf['upload_tmp_dir'] = 'sites/'.$thishost.'/files/tmp';

Edit :(
still getting error:
warning: unlink(sites/mysite/files/filecache/mysite-cache_lock) [function.unlink]:

There is an issue for missing views #899820: Missing code-based views when using cacherouter with APC

If anyone is getting errors like "Permission denied" for unlinks when using cacherouter in a WAMP environment, here's what helped me:

First, use the little documented "lock_dir" cacherouter config parameter to put the lock directory under your website root. Here's an example of some code for the settings.php that will automatically find the Drupal site's file directory path and put both the files and lock_files under it. Of course you can use static path entries if you want.

// Need to use variable_get in bootstrap process.
$site_files = variable_get('file_directory_path', conf_path() .'/files');
$site_files_path = realpath('.') . '/' . $site_files;
$conf['cache_inc'] = './sites/all/modules/cacherouter/cacherouter.inc';
$conf['cacherouter'] = array(
  'default' => array(
    'engine' => 'xcache',
    'servers' => array(),
    'shared' => TRUE,
    'prefix' => '',
    'path' => "{$site_files}/filecache",
    'static' => FALSE,
    'fast_cache' => FALSE,
    'lock_dir' => "{$site_files_path}/filecache/locks",
  ),
); 

Make sure the directories defined above exist and are r/w to the web server.

Next, since Windoze and possibly some Linux implementations won't delete files if they are open, you need to modify the Cache.php unlock() function (approx line 127 in file) so that it will close the lock file which the lock() function opens. Here's what is should look like:

  function unlock() {
    if ( $this->lock_fp ) {
      fclose($this->lock_fp);
    }
    unlink($this->lock);
  }

This solved the unlink errors for me. YMMV

I'm quite impressed by the wrongness of the code discussed in this thread and the fact that no one talked about the actual correct solution...

I'll try to explain:

Say you have 3 processes: A, B, and C.

Process A acquires the lock (function says: create a new file or truncate existing file, lock that file read/write -- waiting if necessary)

Process B wants the lock too (function says: create a new file or truncate existing file, lock that file read/write -- waiting if necessary) -- B waits on A

Process A is done (function says: delete the lock file, I don't need it anymore)

Note that under MS-Windows the operating system prevents the unlink() because the file is still opened. So you get the unlink error, but Process C will block as expected.

Process C acquires the lock (functions says: create a new file or truncate existing file, lock that file read/write -- waiting if necessary) -- C does not wait on anything since the file does not exist, it runs the code that is otherwise expected to be run by ONE process at a time.

Process B and C are done (function says: delete the lock file, one of the function generates the unlink() error since the file will be gone)

IMPORTANT NOTE: With just A and B, B also generates the unlink() error since the file was deleted by A.

BUT, note that Process C fails to wait for process B to be done because the file was deleted by A so when C comes along, it creates a new file which is NOT locked.

Solution 1.

DO NOT DELETE THE DAMN FILE. Is that so complicated?!

  /**
   * unlock()
   *   lock the cache from other writes. [<<-- this is wrong too, but anyway... I'm not here to fix the docs]
   *
   * @param none
   * @return bool
   *   Returns TRUE on success, FALSE on failure
   */
  function unlock() {
    flock($this->lock_fp, LOCK_UN);
    return TRUE;
  }

One potential concern with this solution, albeit easy, is that you are going to end up with one "temporary" file per cache variable. So if you have some 50 variables and are running 100 websites (multi-site install) you'll end up with 5000 files in /tmp. Not the end of the world, but just so you know...

Another concern, as mentioned by different people, using files can be slow, although frankly, in comparison to the suggestion of using the Database for a semaphore, that's still going to be dead fast. Plus, by using a different name for each variable, it makes it unlikely that more than 2 or 3 processes attempt to access the same variable (really? I'm most certainly wrong on that one if you have high traffic because the "variable" cache is written to each time the variable_set() function is called.)

As a proof of concept, I have a little piece of code that you can write to a file named lock.php and run "simultaneously" in 3 terminals with the following command:

 php lock.php

  $wrong = TRUE;
  $f = fopen("/tmp/php-lock.tmp", "w+");
  echo "Waiting for the lock... ";
  flock($f, LOCK_EX);
  echo "Got the lock!\n";
  sleep(10);
  if($wrong) {
    unlink("/tmp/php-lock.tmp");
  }
  else {
    flock($f, LOCK_UN);
    fclose($f);
  }
  echo "Done.\n";

If you run all 3 pretty much at the same time with $wrong set to TRUE, you'll get errors saying that the file was deleted.

However, if you:

1. start the first (A)
2. wait a couple of seconds
3. start the second (B)
4. wait for the 1st to exit (say "Done.")
5. start the third (C)

As you will notice, C will NOT wait for B to be done because the file does not exist anymore.

(Additional note: for the crazy people out there who use MS-Windows as a web server, A won't have permission to delete the file because B has it open. So the problem under MS-Windows is "just" that you get unlink errors.)

There is one other problem with file locking, much less obvious. In many cases, the other in which the lock is requested is not the order in which it is acquired. This means if processes A, B, and C ask for the lock in that order, A will get it first, and then either B or C get it. That can cause problems in that the last value, offered by C, should be the one saved in the cache. Yet, if C is given priority over B, you'll get the value of B instead. In most cases Solution 2 palliates to that problem all by itself.

Solution 2.

The other solution, which is faster, but require an identifier and that the PHP semaphore work on your operating system (I have no clue whether it works under MS-Windows!), is to use the Semaphore API. That runs in memory only and is handled by the kernel. See the sem_get() to get started:

http://www.php.net/manual/en/function.sem-get.php

Then click on sem_acquire() and sem_release(). Note that just like with the files, you DO NOT want to make use of the sem_remove() because that does not work as long as new processes may still get started and unless your website goes to sleep once in a while...

There is a test code sample that shows the same thing as the with the flock() function but using the semaphore. The main problem here is the first parameter of the sem_get() function (123):

  $s = sem_get(123, 1, 0600);
  echo "Waiting for the lock... ";
  sem_acquire($s);
  echo "Got the lock!\n";
  sleep(10);
  sem_release($s);
  echo "Done.\n";

That parameter has to represent a unique number, but that does not really tells you how to convert a variable name in a 32 bit identifier. The semaphore documentation mentions the use of the ftok() function. Unfortunately, if we do not have a file, ftok() is useless. Since all PHP processes need a unique, but equal, identifier for each key, the only solution that comes to mind is a hash of the existing path. Although a hash is not guaranteed to be unique, it does not matter much since that just means the same lock will be used for 2 or 3 different variables which is acceptable here.

Patch Details

Notice that the patch should work for MS-Windows as well as I checked to see whether sem_get() exists. From what I understand, it won't exist under MS-Windows. The fallback is the file system flock() mechanism.

If lock()/unlock() are called multiple times, the lock reuses the same resource (lock_rsc) is reused. There is no need to reopen the file. It will automatically be closed when the object gets destroyed.

The fopen() uses "w+" so it does not attempt to destroy the file if it exists. As far as I know, it will be the same as the "w", just a bit safer. Do not try to use the "x" flag because that will prevent the file from being opened in the first place and the function will return FALSE.

The lock() and unlock() functions now both return TRUE or FALSE as documented although I would imagine that the unlock() function could just completely ignore that return value.

I fixed up the documentation in the patch...

I hope someone finds this valuable and maybe the author will put the patch in for D6.

Suppose we could solve this by providing only a platform dependent solution!

This needs a some line in README to point about performant solution

Once sem_get() available use it (some php installs could have this extension disabled)
http://www.php.net/manual/en/intro.sem.php Note: This extension is not available on Windows platforms.
But for windows we have wincache, and suggested functions is http://www.php.net/manual/en/function.wincache-lock.php

Once both unavailable fallback to flock() but a bit cleaned-up code

PS: I'm actually not using D6 now so have no ability to test but glad to commit patch after peer review

Note that I put a test on the sem_get() function in the patch and have flock() as the fallback. I would assume that will work as is under MS-Windows, but we'd need to have someone test on that operating system to confirm that it works for them. flock() is not so bad, but if you can use the sem_xxx() functions, then you have a speed advantage.

I would assume that will work as is under MS-Windows

There's no such extension for MS Win - see #97

Andy,

If you look at the test, if sem_get() does not exist, then I do not use the semaphore and fallback to the fopen+flock+fclose which is slower, but will most certainly work under MS-Windows.

In any event, the sem_get() works great for me under Linux. 8-)

Thank you.
Alexis Wilke

Alexis, yes. As I said in #97 this makes a lot of sense but I glad to have more reviews before commit the one.
The follow-up for this could be a patch that allows other cache backends to re-use "sem_*" functionality, as I remember "memcached" locking is a lot more expensive...

EDIT: memcache locking still a botleneck

I noticed that I was still using one lock_fp variable (fp = file pointer) which should have been changed to lock_rsc (rcs = resources).