Currently we record the length all all wildcard clears against a bin, then fetch any wildcard shorter than the current cid in cache_get().
On D6, this potentially means that the {cache} bin will see a lot of wildcard fetches.
For example (made up, didn't look for specifics):
cache_clear_all('mykey:', 'cache', TRUE);
cache_get('variables');
Because variables is longer than 'locale:', each time we need to request the variable cache, we'll need to fetch the wildcards 'mykey:' just in case.
If we were to change the wildcard lengths array to be keyed by $string[0], then we'd change the array structure to look like:
$wildcard_lengths = array(
5 => 10,
3 => 20,
);
to
$wildcard_lengths = array(
'm' => array(
5 => 10,
3 => 20,
),
);
Now, when fetching variables, it's only necessary to check for wildcard clears against 'mykey' for cids that start with 'm', instead of all cids - so cache_get('variable'); wouldn't trigger a wildcard fetch at all (until a contrib module does cache_clear_all('vog', 'cache', TRUE));
(There's also a more extreme version of this, but it would require some learning. Instead of keying by the first letter, we could key by substr(0, min($lengths));
Say the minimum length is 5 characters for a wildcard clear, this would mean the clear on 'vog' wouldn't affect variables either, but we'd need a lot more complexity than there already is to learn and update what the shortest cid is for each bin, so I think if we did that at all, it should be a separate issue to the simpler change here).
Comments
Comment #1
catchHere's a patch. Since we now invalidate cache items where $cache->flushes is greater than the number returned from wildcards, this patch invalidates all items that have seen a wildcard flush when it's installed, but is otherwise transparent more or less.
Comment #2
catchAnd here's one using the minimum length of all cid clears against the bin. This adds some complexity to memcache_wildcards(), but on a complex Drupal 6 site, it reduced the number of calls to getMulti on the front page from 16 to 2.
Comment #3
catchFix a notice.
Comment #4
catchFix a notice, introduce three syntax errors, this patch gets those..
Comment #5
catchRan simpletests and there was no regression. Also ran load tests, and while there as no improvement, there was no regression either. So I'm going to commit this to -dev and we can refine later if something comes up. This should bring the hit rate close to pre-wildcard support, and may have a small performance benefit - at least on systems that don't run memcache locally.
Comment #6
catchCommitted.
Comment #7
catchI hadn't committed this, just did that, also moving this to 7.x.
Comment #8
catchBumping to critical, I'd like to try to get this in before 7.x-1.0
Comment #9
catchForward ported this to 7.x-1.x. Tests ran with no regressions.
Comment #10
das-peter commentedLooks like the commit introduced an issues:
Notice: Undefined variable: table in MemCacheDrupal->wildcards() (line 248 of sites/all/modules/memcache/memcache.inc).I guess the variable
$tableis deprecated in favour of$this->binThe attached patch replaces all
$tablevariables with$this->bin.Comment #11
das-peter commented*grml* forgot to change status
Comment #12
catchThanks, committed.
Comment #13
das-peter commentedThanks for committing so fast.
Unfortunately I didn't catch all issues.
The variable
$wildcard_invalidateseems to be replaced by$this->invalidate.Attached fixes that and also one trailing space.
Comment #14
catchThanks again. Committed and pushed.
Comment #15
das-peter commentedNext minor issue.
Pisco told me about this strict warning:
Strict warning: Only variables should be passed by reference in MemCacheDrupal->wildcards() (line 248 of …/modules/memcache/memcache.inc).It's caused because
reset()expects a reference to a variable. Thus a function return like fromarray_keys()provokes the warning.Comment #16
catchThis was already fixed by #1244296: Strict warning: Only variables should be passed by reference in memcache_wildcards().
Comment #17
das-peter commentedFound another construct that wasn't properly ported to D7. Unfortunately the code was written defensive enough to prevent warnings ;)
Comment #18
catchThanks (again). Looks like the test cases could do with tightening up a bit more. Committed and pushed this one!
Comment #20
berdirOk, so either I'm not getting this or this patch makes no sense at all. The wildcard tests are currently failing for me and I've tried to fix them, but didn't get far except adding like a million debug statements ;)
The following code attempts to get the shortest prefix:
- This only gets the shortest prefix because the code below is broken and completely clears the array whenever a shorter wildcard is found, see below.
- Second and even more important, this never actually verifies that the prefix matches the cid. If the cid is 'abc' and the prefix which happens to be the first is 'c', then we're using the prefix 'a' for the wildcard lookup. Whut?
The following lines of code empties the wildcard_flushes array for the given bin if the key length is shorter:
- Because we never actually verify that the prefixes have something in common (e.g. 'a' and 'abc' as opposed to 'a' and 'cba'), this always resets the wildcards array when any shorter wildcard is found, e.g. when you have the wildcards 'abc' and 'bca' and then get a new wildcard 'c' you end up with just 'c'.
- Also, even if the wildcards are common, e.g. 'abc' and 'ab', we loose the information about the number of flushes. which means that with the following sequence of calls, the last cache_get() does *not* fail: cache_set('test_string:1', 1), cache_clear_all('test_string'), cache_set('test_string:1', 2), cache_clear_all('test_'), cache_get(''test_string:1'). This is because flushes of our cache is 1 but we also only have a single wildcard with a single flush registered. This is the reason the tests currently fail, the other issues aren't detected.
It is not trivial to fix and correctly implement this. I'm wondering if we should actually roll back this patch and start with a much more simple solution that uses a fixed prefix size of 1.
Comment #21
catchOK this bit is 'by design' but I think we are missing adding a full cache flush for the bin. The idea was that his happens very rarely, so it'd be OK to flush the full bin at that point. If we add the full flush then everything's invalidated and it starts again.
Comment #22
berdirOk, here is a patch that adds the full bin clear (Note that I also had to set $this->cache_flush directly, the whole code is screaming for some refactoring that moves things like that in separate methods).
I also added some tests to test this.
Comment #23
catchThanks for the patch, tests look great!
I've committed this to 7.x-1.x. We need to backport the full bin flush and tests to 6.x too.
Comment #25
jeremy commentedBack-port to 6.x committed.