The attached patch is a very early pass at a plugin caching architecture for Views. It allows each display to have a specified caching mechanism, modeled after the access plugins. (See screenshot below.) The plugin architecture is in place and working, with a no-op "no caching" implementation, and a dirt-simple "cached for [n] minutes" plugin. The latter is unfinished, but should show the direction things are going in.
After conversations with Earl, I'm anticipating three points for caching: the build_info that stores a rendered query and its arguments, the raw results of a query/pager_query combination, and the fully rendered output of a given display. The first mechanism is in place in views.inc, but work to get the query-result and display-rendering cache hooks in place are continuing.
Comment | File | Size | Author |
---|---|---|---|
#19 | views_plugin_cache.inc_.patch | 582 bytes | pavel.karoukin |
#14 | updated-cache.patch | 16.62 KB | merlinofchaos |
#13 | updated-cache.patch | 10.69 KB | merlinofchaos |
#11 | views_2_caching.patch | 29.54 KB | eaton |
#9 | views_caching.patch | 23.72 KB | eaton |
Comments
Comment #1
eaton CreditAttribution: eaton commentedI've ported this to the DRUPAL-6--3 branch as per merlinofchaos' request. There are a number of hitches, though, that make things tricky. Queries and raw results in particular are very difficult to cache using the new plugin caching mechanism. I've manged to implement rudimentary output caching, but I'm not sure that it will work with any filters or arguments that use contextual information (the currently logged in user, the current date, etc).
Further updates as events warrant.
Comment #2
eaton CreditAttribution: eaton commentedIn the light of day, there are still some thorny problems that need figuring out.
Those are the two biggies I'm looking at right now. The existing views_plugin_cache_time.inc implementation is very dumb -- it just uses the view name, the display ID, and a hash of $_GET to generate its key. That means it can generate a key very early in the views building process, but it only works for page displays and also ignores defaulted arguments and context-sensitive filters like 'author is current user'.
Work continues.
Comment #3
eaton CreditAttribution: eaton commentedWoo! After chatting with Earl for a while and poking around, I stepped back from trying to cache the queries and focused on output caching. It's definitely a bit easier to work with -- the attached version of the patch includes ONLY output caching implementation, but it reduces a page with the example 'frontpage' view from 107 queries to 34. It currently spits out a 'Cached output used' message via dsm(), and thus will choke if you don't have devel module installed. Further testing to follow....
Comment #5
eaton CreditAttribution: eaton commentedHere's a cleaned up version of the patch that uses vpr() instead of dsm() for better compatability. I still need to address cache invalidation; right now old data is never cleared out. We should probably clear anything related to a given view or display whenever it's validated. But it's definitely ready for testing at this point.
Comment #6
eaton CreditAttribution: eaton commentedNow both result AND rendered output caching are working, including paged views.
The *_time cache implementation allows users to set a lifespan independently for results and rendered output -- depending on the output style, there can be problems caching the output so it's nice to be able to turn that off while still getting the result caching.
Next up will be cache invalidation so the tables get cleared out automatically when saving views, etc. Right now, though, it's a perfectly usable patch and can probably be used for some benchmarking.
Comment #7
eaton CreditAttribution: eaton commented...And here's a version that ignores caching when the view is being previewed, flushes the cache for a given view whenever it's saved, and properly clears out old cache data when cache plugins are changed.
I'm doing to say that it's starting to hit 'needs review' rather than 'needs work' -- it might not be perfect but it's now fully functional, and doesn't have any major areas of scary wrongness that I'm aware of.
Comment #8
eaton CreditAttribution: eaton commentedAnd here's the actual patch.
Comment #9
eaton CreditAttribution: eaton commentedComment #10
eaton CreditAttribution: eaton commentedWhoops. Looks like the notes got lost for that one -- the latest version of the patch properly handles clearing out the cache when the 'Clear cache' button is pressed on the Performance page, or in Devel module. In addition, it properly stores JS, CSS, and HEAD additions made by output styles that use them. Carousels and AJAX pagers now cache properly. Huzzah!
Comment #11
eaton CreditAttribution: eaton commentedHere's a re-rolled version of the patch for Views 2.x, as well. The patches applied cleanly to that branch with on tweak in $views::execute(), where the logic had been dramatically simplified. It just required wrapping a larger chunk of code in the caching, since it hadn't been pulled into the query class yet.
Tested with exposed filters, ajax paging, and CSS/JS. w00t.
Comment #12
merlinofchaos CreditAttribution: merlinofchaos commentedCommitted! Let's see how this place in the real world.
Comment #13
merlinofchaos CreditAttribution: merlinofchaos commentedUpdated patch, putting here for testing before I commit:
Comment #14
merlinofchaos CreditAttribution: merlinofchaos commentedMore refactoring
Comment #15
ChrisBryant CreditAttribution: ChrisBryant commentedWOW, nice work here Eaton & Earl! This really rocks and I'm looking forward to test this out on some projects.
Comment #16
nicksanta CreditAttribution: nicksanta commentedYou guys have done an awesome job! I had a little play on some low traffic sites and it seems to be working nicely! Next week I'll be upgrading to views 2.6, a site which has been suffering really severe performance issues due to the sheer volume of content being displayed views on every page. This will hopefully put a stop to that!!!
Great job again!
Comment #17
Scott Reynolds CreditAttribution: Scott Reynolds commentedI have just committed a caching plugin for Apache Solr Views (http://drupal.org/cvs?commit=224746). One thing I noticed that probably needs to be addressed, is that the standard caching plugins show up as options on the settings page. This is a problem because those caching plugins wont work with the Solr facet blocks.
Be nice to define the plugin with an additional param in hook_views_plugins().
I looked to find where that check would go and it looks like
views_fetch_plugin_names()
is the spot just need to figure out how to pass in the query type to it.Comment #18
Scott Reynolds CreditAttribution: Scott Reynolds commentedhmm I think I have decided to retract my suggestion :-D. The standard caching plugins do work just fine with Apache Solr Views project, the facet blocks just don't work with the caching but with my cache plugin they do work.
Seems like its just a documentation problem for me, and not and issue for Views.
Comment #19
pavel.karoukin CreditAttribution: pavel.karoukin commentedI am using panels and embed views into it. Noticed following: views cache do not respect atleast "Offset" parameter. Small patch attached to fix this, but I am pretty sure we need more generic code for this issue
Comment #20
Scott Reynolds CreditAttribution: Scott Reynolds commentedWrong status
Comment #21
merlinofchaos CreditAttribution: merlinofchaos commentedHmm. The amount of data that can be tweaked externally is going to make it very difficult to do caching when you do it like this. Besides, Panels has its own caching mechanism, you're much better off using that. Not accepting the patch.
Comment #22
pavel.karoukin CreditAttribution: pavel.karoukin commentedCase: I am allowing users create their own layout via Panels. And not all of them going to switch on Panels caching. That's why I need caching in more deep level like Views in this case.
Comment #23
merlinofchaos CreditAttribution: merlinofchaos commentedThe provided cache mechanisms are not expected to be appropriate for every scenario. The scenario you outline, sadly, is one of them. If the Panels caching mechanism doesn't wokr, then you may need to create a customized caching plugin that fits your scenario better.
Comment #24
pavel.karoukin CreditAttribution: pavel.karoukin commentedOk. got you point. Can you point me somewhere where i can see good tutorial on creating plugins for views? I can use views_cache as example, but I'd like to place this plugin into my own module. So how I can hookup it?
Comment #25
merlinofchaos CreditAttribution: merlinofchaos commentedThe system is new enough that there are not yet tutorials. However, nodecomment.module (2.x line) and apache solr both have caching plugins already, so there may be examples you can get from there.
Comment #26
Scott Reynolds CreditAttribution: Scott Reynolds commentedApache Solr Views has the caching plugin: http://drupal.org/project/apachesolr_views
Apache Solr module does not. (and ironically there is actually two Apache Solr modules..)
Comment #27
dawehneradding tag.
Comment #28
te-brian CreditAttribution: te-brian commentedHey, a little late, but I wanted to thank you guys for the work your doing on this. The more we start to use memcached and other performance enhancing techniques the greater we appreciate thoughtful and configurable caching.
Comment #29
sbydrupal CreditAttribution: sbydrupal commentedDo we apply both Eaton's and Merlin's patches or, Merlin's patch is sufficient ? Thanks
Comment #30
dawehneri think you need #11 and #14
Comment #31
sbydrupal CreditAttribution: sbydrupal commentedthanks dereine
Comment #32
dawehneror you could just use the current version of views(dev :) ), i think they are both included there.
Comment #33
eaton CreditAttribution: eaton commentedThe caching feature is included in Views 2.6 -- no need to grab the bleeding edge. Just the cutting edge. ;-)
Comment #34
dawehnerlet's say the working version , put thats true cutting edge :p.
See #503774: Cache Time based does not reset
Comment #35
sbydrupal CreditAttribution: sbydrupal commentedCase:
There is a simple "View" that lists node titles.
When the cache is enabled,
tried 2 options:
- query 1 min, html output 1 min
- query 1 min (no html output caching)
When a node is deleted, the view still is showing the Node title in the cache after 1 minute...
(I am not clicking any tabs on the browser to activate the cache after the node deletion,...)
(No query is made on the view during this period of testing cache)
what would be the best way to use views caching ? Node is deleted and it is still showing indefinitely ???
Thanks
Comment #36
merlinofchaos CreditAttribution: merlinofchaos commentedsbydrupal: http://drupal.org/node/503774
Let's end this thread. Can we have any further issues with this patch as new issues?
Comment #37
sbydrupal CreditAttribution: sbydrupal commentedthanks Merlin.