Memcache add may not be thread safe

Following some strange occurrences on a site where we are using memcache for semaphore we have been investigating the memcache lock_acquire and believe that it may be possible for two processes to acquire a lock on the same lock key.

This appears to be a known issue with memcache add - the following page on php.net (http://www.php.net/manual/en/memcache.add.php#96278) indicates that race conditions can occur under heavy load.

Bearing in mind the suggested fix in that thread I have created a script to try and reproduce the issue in a controlled manner:

<?php

require_once './includes/bootstrap.inc';
drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

global $locks;

$name = 'My-Lock';
$timeout = 600;

// Ensure that the timeout is at least 1 sec. This is a limitation
// imposed by memcached.
$timeout = (int) max($timeout, 1);

if (dmemcache_add($name, _lock_id(), $timeout, 'semaphore')) {
  $locks[$name] = _lock_id();

  // check key is correct - to ensure thread safety
  if ($locks[$name] == dmemcache_get($name, 'semaphore')) {
    drupal_set_header($_SERVER['SERVER_PROTOCOL'] . ' 201 Memcache safe add');
    print 'Memcache set safely';
  }
  else {
    drupal_set_header($_SERVER['SERVER_PROTOCOL'] . ' 599 Memcache unsafe add');
    print 'Memcache set unsafely';
  }
}
else {
  drupal_set_header($_SERVER['SERVER_PROTOCOL'] . ' 200 Memcache no add');
  print 'Didnt add to memcache';
}
?>

The idea behind setting different response headers was to be able to use apache bench to send a high volume of concurrent requests and use its built in response header reporting to show any errors:

We then used:

ab -n 100 -c 2 -v 3 http://www.example.com/lock_test.php

And got:

Document Path: /lock_test.php
Document Length: 19 bytes

Concurrency Level: 2
Time taken for tests: 18.267015 seconds
Complete requests: 100
Failed requests: 1
(Connect: 0, Length: 1, Exceptions: 0)
Write errors: 0
Non-2xx responses: 1
Total transferred: 34118 bytes
HTML transferred: 1902 bytes
Requests per second: 5.47 [#/sec] (mean)
Time per request: 365.340 [ms] (mean)
Time per request: 182.670 [ms] (mean, across all concurrent requests)
Transfer rate: 1.81 [Kbytes/sec] received

The one "Non-2xx" response was confirmed in the output as being the 599 header for an unsafe memcache add (i.e. a memcache key set with the correct name, but not the lock_id of the request that is currently executing (and so in theory two processes would have gained a lock)