Attaching MongoDB cache backend (ported from MongoDB D7 cache). This needs MongoDB module installed.

MongoDB is very cool for caching. In fact I think it's best cache engine available for Drupal, because it's only slightly slower than Memcache, but has great features. Besides being general fast data store, it can be very useful for cache because it can do queries on cache objects so we can perform wildcard cache flushes without ugly workarounds (finally!).

TODO: atm it assumes MongoDB is installed at "sites/all/modules/mongodb". Perhaps we should make it a settings.php variable.

CommentFileSizeAuthor
#21 mongodb.php_.txt3.06 KBcrea
#14 mongodb.php_.txt3.93 KBsyamphu
#8 mongodb.php_.txt3.71 KBcrea
#10 mongodb.php_.txt3.78 KBcrea
#1 mongodb.php_.txt3.87 KBcrea

Comments

crea’s picture

crea commented
StatusFileSize
new mongodb.php_.txt3.87 KB
andypost’s picture

andypost
he/him
Primary language Russian
commented

Yay!

I think it's better to make setting as memcache-compression to point a mongo folder.

Is this dependency so hard required? It seem that could not work for "fast_cache" when database is not started yet... because it depends on variable_get(). Is it hard to remove this dependency ?

Now we need testers and some updates for docs!
Also it's good to provide links or own testing results.

crea’s picture

crea commented

Is this dependency so hard required ? Is it hard to remove this dependency ?

It is possible but that would mean code duplication between the projects. Also since MongoDB project will be heavily used in D7 would be stupid to not use their backported code and ideas. Also afaik if we don't use their collection objects that means we are going to create our own duplicate connections, wasting memory, etc.
In summary: it's bad idea.

It seem that could not work for "fast_cache" when database is not started yet... because it depends on variable_get()

It is possible to remove variable requirement somehow and also it's possible to ignore "fast_cache" too ;)
Also maybe it's possible to get rid of garbageCollection(). As far as I understand this garbage collection is necessary to kill the unsafe window between expiration of flushed temporary item and next flush() so any invalid item that is expired won't be returned. If that window is acceptable we could remove the garbageCollection() function.
Best way is to store that variable straight in MongoDB. But that could require additional modifications of the Cacherouter module.

andypost’s picture

andypost
he/him
Primary language Russian
commented

We should care about using CR as caching backend and it's bad idea to depend on some module here.
I think better to make it configurable to use own implementation or mongo.module

fast_cache is one on the key features so...

garbageCollection() lives in cache.inc and cacherouter.inc this is a part of original cache implementation. I think it make no sense to execute it on every _get()

Anyway lets wait for some opinions\reviews

crea’s picture

crea commented

We should care about using CR as caching backend and it's bad idea to depend on some module here.

D7 MongoDB cache depends on D7 mongodb.module. There's no big problem about it, except for the fast_cache problem. And it's fixable without abandoning MongoDB module completely.
We could also ask that D6 MongoDB collection-management code is moved to a separate inc file to exclude any module-level code. Maintaining duplicated version of their code just to not load module would be unwise IMO.

garbageCollection() lives in cache.inc and cacherouter.inc this is a part of original cache implementation.

Cacherouter's cacherouter.inc doesn't perform garbage collection in cache_get() on it's own. Original cache.inc does.

I think it make no sense to execute it on every _get()

If you had minimum cache lifetime = 10 minutes, and run cron every 5 minutes, you could have a situation where cron is running on 9th minute of cache lifetime so between 10 and 14 minutes (next cron run) you would have unsafe window where already invalid entries are returned as valid.
Garbage collection on get() ensures this situation doesn't happen.

crea’s picture

crea commented

On variables problem:
Garbage collection is necessary both in full boostrap and in fast_cache mode. Thus to properly work in fast_cache mode CacheRouterEngine class needs separate methods to store and fetch last flush time in cache backend itself, so that it's available for garbage collection before returning an entry. Then these methods can be used instead of variable_get() and variable_set().
The fact that it doesn't work this way now is a bug: that means currently in fast_cache mode CacheRouter can return invalid cache entries.

andypost’s picture

andypost
he/him
Primary language Russian
commented

fast cache is a custom implementation so there's no requirements at all. Regardless of normal page cache which affected by the flush_time variable.

crea’s picture

crea commented
Assigned: crea » Unassigned
StatusFileSize
new mongodb.php_.txt3.71 KB

I've opened a separate feature request: #780418: garbage_collection() method. This way garbage collection requiring variables (and thus DB connection) will be only performed in cache_get() instead of mongodbCacheRouterEngine::get(). Thus it will become compatible with page_cache_fastpath(). Also other engines may implement this method too, if needed.

Attaching new version which implements this method.

crea’s picture

crea commented
Assigned: Unassigned » crea
Status: Active » Needs review
crea’s picture

crea commented
StatusFileSize
new mongodb.php_.txt3.78 KB

Better version.

andypost’s picture

andypost
he/him
Primary language Russian
commented
Assigned: Unassigned » crea

Agreed, last step - add a new param for config - to point mongodb.module not from module's code

crea’s picture

crea commented
Status: Needs review » Needs work

I've found couple of bugs in the code.

XerraX’s picture

XerraX commented

subscribing

syamphu’s picture

syamphu commented
StatusFileSize
new mongodb.php_.txt3.93 KB

looks like i found some bug on delete function when im going to clear theme_registry cache.
And this code fix my problem.

  // function delete on mongodb.php 
  function delete($key) {
    // Delete from static cache
    parent::flush();
    
    if ($key == '*') {
      // Whole bin delete
      $this->collection->remove();
    }
    elseif (substr($key, strlen($key) - 1, 1) == '*') {
      // Wildcard delete
      $key = substr($key, 0, strlen($key) - 1);
      $search = new MongoRegex('/^'.preg_quote($key).'.*/i');
      $this->collection->remove(array('_id' => $search));
    }
    else {
      // Single entry delete
      $this->collection->remove(array('_id' => (string)$key));
    }
  }
crea’s picture

crea commented

@syamphu
Yes as I pointed above there are bugs, I fixed most of them locally, but haven't updated the engine to use provided mongodb path. As soon as I finish it, I'll upload the fixed engine file.

febbraro’s picture

febbraro commented

subscribe

XerraX’s picture

XerraX commented

Any progress?

crea’s picture

crea commented

Are you in hurry huh ? It won't be committed anyway, cause andypost seems to be busy.

XerraX’s picture

XerraX commented

Thats a catch question.. ok i will be quiet. sorry

andypost’s picture

andypost
he/him
Primary language Russian
commented

@crea I'll commit this as it ready and tested.

Right now I involved in some optimization of site which using cacherouter, so there'll be progress in next weeks

crea’s picture

crea commented
Status: Needs work » Needs review
StatusFileSize
new mongodb.php_.txt3.06 KB

I've implemented include mechanics in the engine but I suggest that we move it to the parent class so it becomes available for all engines which need to include files.
Example config:

$conf['cache_inc'] = './sites/all/modules/cacherouter/cacherouter.inc';
$conf['cacherouter'] = array(
  'default' => array(
    'engine' => 'mongodb',
    'include' => 'sites/all/modules/mongodb/mongodb.module',
    'shared' => TRUE,
    'prefix' => 'mysite',
    //'servers' => array(),
    'static' => FALSE,
    'fast_cache' => FALSE,
  ),
);
andypost’s picture

andypost
he/him
Primary language Russian
commented

Great idea! Suppose we should always add include key as array to remove
is_array($options['include'])

chx’s picture

chx commented

What the hell? Why are you wasting your time on this instead of helping mongodb_cache inside the mongodb project?

crea’s picture

crea commented

@chx
I don't know what you are smoking. Cache Router in D6 is superior because there's no DrupalCacheInterface in D6 so it's best place to put efforts.

chx’s picture

chx commented

Ah, D6. That's , I guess, fine.

igorik’s picture

igorik commented

subscribe, I am interesting about this new caching method - cahe router with mongo db, I have som problems with memcache so this could be great replace for it.

andypost’s picture

andypost
he/him
Primary language Russian
commented

I think MongoDB is most promising caching solution for drupal today and I glad to see @crea as co-maintainer
So going to test it deeper and commit

dakku’s picture

dakku commented

++

digi24’s picture

digi24 commented

The prefix handling seems a bit strange, i did not expect it to use the db_prefix as well.

rizqi’s picture

rizqi commented

function delete is slow when there are lots of cache data, this problem is because using MongoRegex
according to the mongodb documentation there is a solution for that http://www.mongodb.org/display/DOCS/Advanced+Queries

While /^a/, /^a.*/, and /^a.*$/ are equivalent and will all use an index in the same way, the later two require scanning the whole string so they will be slower. The first format can stop scanning after the prefix is matched.

how to fix, change this code:

$this->collection->remove(array('cid' => new MongoRegex('/'. preg_quote($cid) .'.*/')));

to :

$this->collection->remove(array('cid' => new MongoRegex('/^'. preg_quote($cid) .'/')));
crea’s picture

crea commented

rizqi, have you reported this to the MongoDB project ? I took this code from D7 version so D7 mongodb maintainers may want to fix this too.