I know there were some issues on storing files of cache to retrieve them fast.

My solution was - creat subdirectiry structure like squid-cache uses.

But there was a problem - we do not know when cache old or not.
My solution was - checkpoint variable to differ old cache content from new content to know when just recache it.

Of course this structure can grow without limitations -but it's not a problem make cron script to just remove files from cache

cache need files/cache directory
Here my custom cache.inc is:

/*
*
*   Filecache by Ilya V. "brainstorm" Azarov, brainstorm.name
*
*
*/
class cache_cc{
var $data, $created, $expire, $headers, $checkpoint;
 function cache_cc($c){
    $this->data = $c['data'];
    $this->created = $c['created'];
    $this->expire = $c['expire'];
    $this->headers = $c['headers'];
    $this->checkpoint = $c['checkpoint'];
 }
}

function filecache_get_checkpoint($table){
	if($table == 'filecache_checkpoint') return 0;
	$key = $table;
	$lock = filecache_lockpath('filecache_checkpoint', $key);
	$dir = filecache_md5path('filecache_checkpoint', $key);
	$src = filecache_name('filecache_checkpoint', $key);
	$f = fopen($lock, 'w+');
	if(flock($f, LOCK_SH) && file_exists($src) ){
		$cache = @unserialize(file_get_contents($src) );
        $checkpoint = $cache['data'] ['checkpoint'];
		flock($f, LOCK_UN);
        fclose($f);
        return $checkpoint;
	}
	fclose($f);
	return 0;
}
// here we do custom combined rewrite of cache_get && cache_set
function filecache_increment_checkpoint($table, $expire = CACHE_PERMANENT/* 0 */, $headers = NULL){
	if($table == 'filecache_checkpoint') return 0;
	$key = $table;
	$lock = filecache_lockpath('filecache_checkpoint', $key);
	$dir = filecache_md5path('filecache_checkpoint', $key);
	$src = filecache_name('filecache_checkpoint', $key);
	// exclusive lock for an increment time
	$f = fopen($lock, 'w+');
	if(flock($f, LOCK_EX) ){
        if(file_exists($src) ){
        	$cache = @unserialize(file_get_contents($src) );
        	$checkpoint = $cache['data'] ['checkpoint'];
        	$checkpoint = ($checkpoint + 1)  % 2000000;
        	$cache = array(
				'data' => array('checkpoint' => $checkpoint),
				'created' => time(),
				'expire' => $expire,
				'headers' => $headers,
				'checkpoint' => 0,
			);
			$cache = serialize($cache);
			$cnt = @file_put_contents($src, $cache);
        }else{
			$cache = array(
				'data' => array('checkpoint' => 0),
				'created' => time(),
				'expire' => $expire,
				'headers' => $headers,
				'checkpoint' => 0,
			);
			$cache = serialize($cache);
        	$cnt = @file_put_contents($src, $cache);
        }
        flock($f, LOCK_UN);
        fclose($f);
        return $cnt;
    }
    fclose($f);
    return false;
}
function filecache_md5($key){
    static $cache_arr;
    if(isset($cache_arr[$key] ) ) return $cache_arr[$key];
    $md = md5($key);
    $cache_arr[$key] = $md;
    return $md;
}
function filecache_tablepath($table, $key){
    return variable_get('filecache_path', 'files/cache') . '/'. $table;
}

function filecache_md5path($table, $key){
    static $cache_arr;
    if(isset($cache_arr[$table . $key] ) ) return $cache_arr[$table . $key];
    $md = filecache_md5($key);
    $path = array($md{0}, $md{1}, $md{2}/*, $md{3}, $md{5}*/ );
    $path = filecache_tablepath($table, $key) . '/' . implode('/', $path);
    $cache_arr[$key] = $path;
    return $path;
}
function filecache_fname($table, $key){
    return base64_encode($key);
}
function filecache_name($table, $key){
    return filecache_md5path($table, $key) . '/' . filecache_fname($table, $key);
}
function filecache_pmkdir($dir){
    if(is_dir($dir) ) return true;
    return mkdir($dir, 0755, true); // PHP5 ONLY
}
// checks file exists and create path to them if not :)
function filecache_lockpath($table, $key){
    if(!is_dir(filecache_md5path($table, $key) ) ){
        filecache_pmkdir(filecache_md5path($table, $key) );
    }
    return filecache_md5path($table, $key) . '/' . 'lock';
}
function filecache_safefile_get_contents($table, $key){
    $lck = filecache_lockpath($table, $key);
    $src = filecache_name($table, $key);
    $f = fopen($lck, 'w+');
    if(flock($f, LOCK_SH) ){
        $cnt = file_get_contents($src);
        flock($f, LOCK_UN);
        fclose($f);
        return $cnt;
    }
    fclose($f);
    return false;
}
function filecache_safefile_put_contents($table, $key, $cache){
    $lck = filecache_lockpath($table, $key);
    $f = fopen($lck, 'w+');
    $src = filecache_name($table, $key);
    if(flock($f, LOCK_EX) ){
        $cnt = @file_put_contents($src, $cache);
        flock($f, LOCK_UN);
        fclose($f);
        return $cnt;
    }
    fclose($f);
    return false;
}

function cache_get($cid, $table = 'cache') {
  global $user;
  $key = $cid;
  if (!file_exists(filecache_name($table, $key) ) ) {
    return 0;
  }
  $cache = @unserialize(filecache_safefile_get_contents($table, $key) );
  if (isset($cache['data'] ) ) {
    // If the data is permanent or we're not enforcing a minimum cache lifetime
    // always return the cached data.
    if ($cache['expire'] == CACHE_PERMANENT || !variable_get('cache_lifetime', 0)) {
      if(filecache_get_checkpoint($table) != $cache['checkpoint']) return 0;
      //echo filecache_get_checkpoint($table) . '-' . $cache['checkpoint'] . "\n";
      return new cache_cc($cache);
    }
  }
  return 0;
}


function cache_set($cid, $table = 'cache', $data, $expire = CACHE_PERMANENT, $headers = NULL) {
  $cache = array(
    'data' => $data,
    'created' => time(),
    'expire' => $expire,
    'headers' => $headers,
    'checkpoint' => filecache_get_checkpoint($table)
  );
  $cache = serialize($cache);
  $key = $cid;
  filecache_safefile_put_contents($table, $key, $cache);
}
/*
* Removes all files from given cache directory
*
* OBSOLETE
*/
function filecache_clear_directory_clear($dir){
    if($dh = @opendir($dir) ){
        $dirs = array();
        $files = array();
        while($f = readdir($dh) ) if($f != '..' && $f != '.') {
            $f1 = $dir . '/' . $f;
            if(is_dir($f1) ) array_push($dirs, $f1);
            elseif(strcmp($f, 'lock') != 0) array_push($files, $f1);
        }
        @closedir($dh);
        while($f1 = array_pop($files) ) @unlink($f1);
        while($f1 = array_pop($dirs) ) filecache_clear_directory_clear($f1);
        return true;
    }else{
        return false;
    }
}

function cache_clear_all($cid = NULL, $table = NULL, $wildcard = FALSE) {
  global $user;
  if (!isset($cid) && !isset($table)) {
    cache_clear_all(NULL, 'cache_page');
    return;
  }
  if(empty($cid) || $wildcard){
      //filecache_clear_directory_clear(filecache_tablepath($table, $cid) );
      filecache_increment_checkpoint($table);
  }else{
      @unlink(filecache_name($table, $cid) );
  }
}

Comments

kbahey’s picture

This is useful, thanks for sharing it.

However, there are other modules that already do that, such as:

http://drupal.org/project/fastpath_fscache

http://drupal.org/project/boost

Also, I benchmarked your caching vs. normal caching with Drupal 5.6, and here are the results:

Drupal 5.6, normal caching

Concurrency Level:      20
Time taken for tests:   3.140041 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      28982000 bytes
HTML transferred:       27920000 bytes
Requests per second:    636.93 [#/sec] (mean)
Time per request:       31.400 [ms] (mean)
Time per request:       1.570 [ms] (mean, across all concurrent requests)
Transfer rate:          9013.26 [Kbytes/sec] received
Drupal 5.6, filecache

Concurrency Level:      20
Time taken for tests:   3.515108 seconds
Complete requests:      2000
Failed requests:        0
Write errors:           0
Total transferred:      28498000 bytes
HTML transferred:       27436000 bytes
Requests per second:    568.97 [#/sec] (mean)
Time per request:       35.151 [ms] (mean)
Time per request:       1.758 [ms] (mean, across all concurrent requests)
Transfer rate:          7917.25 [Kbytes/sec] received

So, for some reason, database caching is better, perhaps because MySQL's query cache which is kept in memory?
--
Drupal performance tuning, development, customization and consulting: 2bits.com, Inc.
Personal blog: Baheyeldin.com.

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

ilya1st’s picture

Note: Because flock() requires a file pointer, you may have to use a special lock file to protect access to a file that you intend to truncate by opening it in write mode (with a "w" or "w+" argument to fopen()).

you must open file for writing to lock them. Open for reading - undocumented feature.

Fastpath cache uses more plain cache structure. so....

And last: flock on filesystem ia much more easier call to decrease DB load.

On shared hostings sometimes database cache is NOT better cause of permanent SQL server load by other users.

http://brainstorm.name

ilya1st’s picture

Depends on hosting configuration, default type of drupal tables and other factors. Who gives guarantee that MySQL on hosting is fine tuned.

My benchmarks on my notebook: http://brainstorm.name/blog/filecache-benchmarks

http://brainstorm.name