[meta] Profile/rationalise cache tags

Bumping to critical as per #2204159-16: (Responsive) Image styles do not add correct cache tags, nor do they invalidate cache tags upon flushing and #2204159-17: (Responsive) Image styles do not add correct cache tags, nor do they invalidate cache tags upon flushing.

Important quote from catch in #2204159-17: (Responsive) Image styles do not add correct cache tags, nor do they invalidate cache tags upon flushing:

Agreed on a critical task for rationalising the tags once we have them. I think it's better to use them consistently then optimise them away afterwards. In the back of my mind I'm thinking about implementations that can do direct deletion and then won't have a runtime hit fetching the tags at all, but since both memcache and db backend will have the performance hit we can't rely on that.

A sample test case was articulated by catch over at #2183017-19: Views doesn't bubble up rendered entities' cache tags.

While working on #2304987: Don't invalidate cache tags of referenced entities, use entity list cache tags correctly, add test coverage for entity list cache tags, I thought of one possible approach to reduce the number of DB queries to retrieve cache tags. So I hacked something together (see attached patch), but despite a decrease of 10–15 DB queries even on a simple page load, the observable performance impact for /node/1 (an article node) is 5 to 10 ms, a 2 to 5% page load time improvement.

This will likely make a more noticeable difference once we've optimized kernel booting (~bootstrap) and once almost everything is rendered. Posting this very rough prototype-quality patch for when we continue this issue.

Yes, but that number of 10–15 DB queries being saved is only going to be the case on sites with relatively little content. On sites with tens or hundreds of thousands of entities, far fewer DB queries would be saved. I.e. I doubt this strategy is going to scale beyond simple sites.
Of course, sites with such numbers could then choose to implement a different cache tag preloading strategy, this could become the default strategy, which is sufficient for simpler sites.

Another angle to profile is how often we invalidate certain cache tags.

I noticed today while working on something else that a single drush cr resulted in 27 or so cache tag invalidations for the 'extension' cache tag. For some reason, ViewsPluginManager is using that, and also extension:views.

Did not track down where exactly that happened.

What's looking very weird is this:

**
 * Implements hook_ENTITY_TYPE_update() for 'field_config'.
 */
function views_field_config_update(FieldConfigInterface $field) {
  Cache::deleteTags(array('extension' => 'views'));
  \Drupal::cache('render')->deleteAll();
}

That's not a valid way to specify a cache tag, I think that's clearing 'views', not 'extension:views'. The same is also in views_invalidate_cache() and in views_invalidate_cache(). Given that this is not clearing the cache tag it thinks it is (extension:views), I'm wondering if we need this at all.

Plan to look a bit more at this during the weekend.

Great catch on those old, unupdated cache tags. They should've been updated as part of #2340123: Setting cache tags can be tricky: use strings instead of nested arrays to improve DX. The fact that everything still works with them being invalidated incorrectly indeed suggests that it's not necessary (unless there's no test coverage and it's for a rarely exercises code path and hence nobody has encountered problems caused by it yet).

I don't understand why drush cr should result in any cache tag invalidations at all; why doesn't it simply call ::deleteAll() on every cache bin? Then again, I don't know how drush cr works exactly, so perhaps I'm making a silly statement here.

Drush's cache-rebuild calls drupal_rebuildwhich does a deleteAll() on all bins and then calls drupal_flush_all_caches() which does the same thing, again.

I believe there's an issue open for that somewhere. rebuild.php does the same. Not a huge problem I think, as there shouldn't be much data cached between those so not much to delete, but some implementations have to set flags and so on on deleteAll().

Discussed with @catch, @alexpott, @effulgentsia, and @webchick. We agreed that this is release-blocking.

Looking at a different angle from this, cache tags that are invalidated and how often.

Looking at my cachetags table of a site that has been running for a few days now, these are the most frequent invalidations based on select * from cachetags order by invalidations desc limit 100;:
- configFindByPrefix: 4k
I don't like this cache tag ( I added it myself). It was written when we still used the file system for active store. I doubt that the cache is considerably more performant than just doing a like 'someprefix%' query on an indexed column and we have to invalidate it on every single config write. If not remove, then we should look into maintaining a single cache entry for all lists, but it could possibly grow a lot.

I just checked on my site, I have 30 cache entries about 'find:somePrefix', field.field has a length of 15k characters, the others are below 4, total is 46k.

- entity_field_info 2k, entity_bundles 1.2k, entity_types, 1.2k
I don't know yet why they are called so often. I'm seeing them in my logs every few hours, I don't know why yet. I've just written a cache tags logger that logs them including a backtrace, I'll update this as soon as I know more.

- local_task: 1.2k
Looks like this is invalidated in MenuRouterRebuildSubscriber and ThemeHandler::resetSystem(), but that is only called when a theme is installed/uninstalled. So I guess we have a lot of router rebuilds, which is not that suprising, a lot of drush cr, view saves and so on happening at the moment.

- config:system.menu.* 1k each
Based on my logs, something is frequently invalidating *all* menu cache tags together, maybe we should use a common cache tag for them for this? Will keep an eye on this...

- config:core.extension 600, extension:views: 600
I still don't really understand the second one, looks like something that was mostly added for views? I guess most invalidations are coming from system_list_reset(), which is the one mentioned above, that is called a lot during drush cr as well.

- file_list 992, node_list 680
Those are obvious. I also have node_view high up there due to a custom call, a workaround until #2381217: Views should set cache tags on its render arrays, and bubble the output's cache tags to the cache items written to the Views output cache is fixed.

Thanks Berdir, very valuable info!

- configFindByPrefix: 4k

Not surprised. Would be great if we could indeed remove this one.

- entity_field_info 2k, entity_bundles 1.2k, entity_types, 1.2k

Interesting…

I've just written a cache tags logger that logs them including a backtrace, I'll update this as soon as I know more.

Awesome!

- local_task: 1.2k

Yeah, it sounds logical that lots of site building activity going on causes this. Not worried about this.

- config:system.menu.* 1k each

I'm betting this is MenuTreeStorage::rebuild(), which does this:

    // Invalidate any cache tagged with any menu name.
    $cache_tags = Cache::buildTags('config:system.menu', $affected_menus, '.');

In other words, the real question is: what is triggering those menu rebuilds? I wouldn't be surprised if it was all that Views site building activity.

- config:core.extension 600, extension:views: 600

I also don't understand the Views one. IIRC it's mostly undocumented. So would be great if we could at least at docs explaining its purpose.

- file_list 992, node_list 680

Yep, these are indeed expected, if lots of files are being uploaded and nodes created/modified.

Ok, opened #2442041: Remove CachedStorage::listAll() caching

I've just written a cache tags logger that logs them including a backtrace, I'll update this as soon as I know more.

#16: Could you share?

Sure, it's part of our past logging/debugging project: http://cgit.drupalcode.org/past/tree/src/PastCacheTagLogger.php?id=3a07e..., #2441991: Log cache tag invalidations.

Wow, past is awesome. How could I have missed such a useful module :).

Thanks, will add some logging, too.

I figured out where the entity_types/entity_bundles/entity_field info cache clears are coming from, at least some of them: system_requirements(). Which is called regularly from our monitoring.

#0 core/lib/Drupal/Core/Plugin/DefaultPluginManager.php (164, invalidateTags)
#1 core/lib/Drupal/Core/Entity/EntityManager.php (218, clearCachedDefinitions)
#2 core/lib/Drupal/Core/Entity/EntityDefinitionUpdateManager.php (157, clearCachedDefinitions)
#3 core/lib/Drupal/Core/Entity/EntityDefinitionUpdateManager.php (63, getChangeList)
#4 core/modules/system/system.install (490, needsUpdates)
#5 (0, system_requirements)
#6 core/lib/Drupal/Core/Extension/ModuleHandler.php (395, call_user_func_array)
#7 core/modules/system/src/SystemManager.php (118, invokeAll)
#8 core/modules/system/src/SystemManager.php (102, listRequirements)
#9 core/modules/system/src/Controller/SystemController.php (119, checkRequirements)

That's a problem I think, because it is invalidating *a lot* (like all the entity caches). I think we should try to find a way to avoid that, e.g. just do a refresh of that data, without invalidating those cache tags.

I'd say we should refactor the Entity Manager to expose a method to clear just definition caches.

clearCachedDefinitions() is generally one of the worst for the plugins, too from my installer benchmarking.

I find it problematic that we do a general cache clear all and nothing more granular in ModuleInstaller and support that as a service.

is there really no other way?

Opened #2443409: Add a way to entity manager to get fresh entity and field definitions without invalidating all caches.

Another one: #2443485: Remove extension:views cache tag and other views related cache improvements

So, our cache tag headers are starting to get huge for some of the big overview pages.

Looking at them a bit, we have 200+ cache tags on the frontpage now, which is displaying various views with nodes, a ton of files, various blocks and menus.

I'm seeing (only relevant stuff, ignoring special tags that only exist once or twice, so it won't add up ;)):

• node: 63 + _view
  Not much that we can do about that, except display less of them, do more lazy loading of things below the fold.
• file: 62 IDs + _view
  I think we can get rid of file tags. In core, they're basically immutable anyway, and even if not, given that we can not add the cache tags to the actual files, having a cache tag per file is not so interesting. Instead, I would suggest to rely on the file_usage table. We know where each file is used, and we only care about them being embedded there. So each invalidated file can invalidate all the nodes/entities that are displaying it. I'll open an issue to discuss this.
• block_plugin: 17 different plugins (many of them very long)
  block_plugin. This I find completely weird. There is no reason for those. The only block_plugin cache tag invalidation I can find in core is in a test, for the branding block ;) *If* a block plugin needs to invalidate cache tags, it can just add a cache tag itself, we have a nice API for this. And I think the need to do this is quite rare, if anything, you add cache tags for something you display, and not yourself. Will open an issue for this.
• config:block.block: 18 (16 blocks + list + _view)
  Those are a bit harder to get rid of. But I'm wondering if they're worth it. Many blocks are displayed in header/footer/sidebars usually and many don't have any visibility settings, so saving one of them will invalidate all pages anyway. They likely are interesting for the render cache, but maybe we can optimize something for the cache tags header?
• config:system.menu: 16
  We have lots of small menus embedded in the site. They are unlikely to change frequently. And as mentioned above, they're also all invalidated together often. Again, wondering about an easy way to alter the default implementation and return a single, common tag for all menus? Still need to figure out why they are invalidated so often.
• taxonomy_term: 8
  Not so many yet, will probably still grow.
• block_content: 5 (4 ID's + _view)
  Not a lot, but possibly still interesting to optimize? We already disabled render caching for block_content entities, because we say they're always inside a block config. Could we optimize cache tags too, somehow? Actually, thinking about it, I guess not, not all will be in config entities.

  Looking at this, I'm actually not seeing as many block plugins and configs as I would expect to. Because I don't have a cache tag for the block plugin for those, which I think is because we have a block plugin that is a group of other block plugins, and I'm not implementing getCacheTags() properly. So we'd quite a bit more of those block_plugin cache tags, actually.

• image style: 4
• user: 6

Some parts will definitely still grow quite a bit. We could have a lot more files for examples, because we have a gallery that alone displays 20+ files (which I think shows how problematic they are).

Thanks for your great work on this, Berdir!

Looking at another real world example, from drupal.com:

X-Drupal-Cache-Tags: block:anchortagholder block:crazyeggcode block:drupalcom_content block:drupaleverywhere block:footerblock block:googleanalyticstrackingcode block:homepagebannerparallax block:homepagedrupal8beta block:joindrupalorg block:joindrupalorgmobile block:jumplinkmenu block:socials block:views_block__advertisments_block_1 block:views_block__get_started_conten_block_1 block:views_block__showcases_block_1 block:views_block__testimonials_block_1 block:wheredoistart block:whyusedrupal block_content:1 block_content:106 block_content:11 block_content:126 block_content:26 block_content:46 block_content:51 block_content:6 block_content:61 block_content:76 block_content:81 block_content:91 block_content_view block_plugin:block_content__4b1b68d3-1672-4b9f-8406-6dfcd993610e block_plugin:block_content__4f39af08-6aab-4835-ae1d-2907cd4310bd block_plugin:block_content__5045bc21-1955-4505-b200-a499a03ed95a block_plugin:block_content__653acc64-595c-44c5-9903-3bde886c829d block_plugin:block_content__65b0fe3a-3106-4be4-9300-a9a7a8361bce block_plugin:block_content__6ca13320-6b7d-432e-9b00-ab8eb4ffeae1 block_plugin:block_content__78103474-019d-4346-8c0d-136f8e1dd6a1 block_plugin:block_content__7ba78417-d703-4c5a-9916-2213040431e8 block_plugin:block_content__8bde6d39-947f-452c-8605-5ba254aac2ad block_plugin:block_content__992f3646-09a3-45b3-8d0b-115508f932e3 block_plugin:block_content__a62ab01e-50bf-410e-9b00-b71368cb9e45 block_plugin:block_content__e7a0ee10-d603-4530-8b00-cf0b212b2acb block_plugin:system_main_block block_plugin:system_menu_block__socials block_plugin:views_block__advertisments-block_1 block_plugin:views_block__get_started_conten-block_1 block_plugin:views_block__showcases-block_1 block_plugin:views_block__testimonials-block_1 block_view filter_format:full_html menu:socials node:36 node_view rendered theme:drupalcom theme_global_settings user:11

Coming from that I totally agree, lets remove block_plugin:X. With that we would be looking at:

X-Drupal-Cache-Tags: block:anchortagholder block:crazyeggcode block:drupalcom_content block:drupaleverywhere block:footerblock block:googleanalyticstrackingcode block:homepagebannerparallax block:homepagedrupal8beta block:joindrupalorg block:joindrupalorgmobile block:jumplinkmenu block:socials block:views_block__advertisments_block_1 block:views_block__get_started_conten_block_1 block:views_block__showcases_block_1 block:views_block__testimonials_block_1 block:wheredoistart block:whyusedrupal block_content:1 block_content:106 block_content:11 block_content:126 block_content:26 block_content:46 block_content:51 block_content:6 block_content:61 block_content:76 block_content:81 block_content:91 block_content_view block_view filter_format:full_html menu:socials node:36 node_view rendered theme:drupalcom theme_global_settings user:11

This looks pretty good and not that redundant.

--

Theoretically its possible to de-duplicate that output, but unsure if the main consumers like Varnish / NGINX could parse it as well still:

X-Drupal-Cache-Tags: block:anchortagholder,crazyeggcode,drupalcom_content,drupaleverywhere,footerblock,googleanalyticstrackingcode,homepagebannerparallax,homepagedrupal8beta,joindrupalorg,joindrupalorgmobile, jumplinkmenu,socials,views_block__advertisments_block_1,views_block__get_started_conten_block_1,views_block__showcases_block_1,views_block__testimonials_block_1,wheredoistart,whyusedrupal block_content:1,106,11,126,26,46,51,6,61,76,81,91 block_content_view block_view filter_format:full_html menu:socials node:36 node_view rendered theme:drupalcom theme_global_settings user:11

So need to check what the current approaches are in matching those headers to expire.

Created #2449069: Remove default block_plugin cache tags, because they're useless

Here's an issue for the file idea: #2449087: Don't add cache tags for files, instead, use {file_usage} to invalidate whoever embeds them

Talked with cache, fastly e.g. has a 1024 byte limit on the surrogate header, so proposal would be to keep track of seen cache tags in a table and use some intelligent algorithm to shorten the keys.

The above would look like:

[
'node' => 'n',
'node_view' => 'nv',
'block' => 'b',
'block_content' => 'bc',
'filter_format' => 'ff',
'menu' => 'm',
'rendered' => 'r',
'theme' => 't',
'theme_global_settings' => 'tgs',
'user' => 'u',
]

A hypothetical 'blockify' cache tag would then be:

'blockify' => 'b1',

X-Drupal-Cache-Tags: b:anchortagholder b:crazyeggcode b:drupalcom_content b:drupaleverywhere b:footerblock b:googleanalyticstrackingcode b:homepagebannerparallax b:homepagedrupal8beta b:joindrupalorg b:joindrupalorgmobile b:jumplinkmenu b:socials b:views_block__advertisments_block_1 b:views_block__get_started_conten_block_1 b:views_block__showcases_block_1 b:views_block__testimonials_block_1 b:wheredoistart b:whyusedrupal bc:1 bc:106 bc:11 bc:126 bc:26 bc:46 bc:51 bc:6 bc:61 bc:76 bc:81 bc:91 bcv bv ff:full_html m:socials n:36 nv r t:drupalcom tgs u:11

Because we know all cache tags, we can also use the invalidation table to theoretically create a short code for the values as well.

-

Can use some shortening for tags as well with configurable algo, my favorite text algo would make an output like:

X-Drupal-Cache-Tags: b:AnEr b:CrDe b:DrOm_CoNt b:DrRe b:FoCk b:GoDe b:HoAx b:HoTa b:JoRg b:JoLe b:JuNu b:SoLs b:ViWs_Blck_AdTs_Blck_1 b:ViWs_BlCk_GeT_StEd_CoEn_BlCk_1 b:ViWs_BlCk_ShEs_BlCk_1 b:ViWs_BlCk__TeLs_BlCk_1 b:WhRt b:WhAl bc:1 bc:106 bc:11 bc:126 bc:26 bc:46 bc:51 bc:6 bc:61 bc:76 bc:81 bc:91 bcv bv ff:FuLl_HtMl m:SoLs n:36 nv r t:DrOm tgs u:11

Besides those really long views block names, it looks pretty good ...

#28:

• block_plugin — yes, let's kill it. This is from the first thing that added cache tags to core. Our understanding of cache tags was much worse back then. Kill it with fire.
• config:block.block:* — perhaps we can optimize this by only having a single cache tag for all block config entities in a given theme? But if we do that, we almost might as well just use the rendered cache tag like we do for theme settings?
• config:system.menu.* — would indeed be great to know why they're invalidated so often, that should not be the case and IMO is the bigger problem

Here's an idea that I need to get out of my head.

We have that issue about cache context hierarchy. I think we need the same for cache tags.

Take nodes as an example, we have the following cache tags:
* node:N
* node_view
* node_list

Now, you create a view that lists nodes 1,2,3,4,5 as rendered nodes. Each node render array has node:$id and node_view. And then the wrapper around that that contains those 5 nodes has node_list.

Now, the bubbling means that the next-higher element (could be a block cache, or the whole page), gets the cache tags node_list, node:(1-5) and node_view. However, we know that we will *always* invalidate node_list as well (but not node_view) when we invalidate a specific node:N. That means that bubbling up node:(1-5) when we already have node_list is pointless.

Question is, how to do know that? Can we find a syntax that allows us to describe those relationships? Something like node:* instead of node_list?

If we can find a way to do that, then I think we can remove a huge amount of cache tags from block and page caches, which means we have less cache tags to check on a page/block cache hit and the http headers should get *a lot* smaller as well... Thoughts?

That's interesting for sure. :)

Initial thought: we want to evolve our entity listing cache tags (node_list etc.) in future versions of D8. We want them to become much more optimized/granular/targeted. Your proposal only works because our current listing cache tags are so incredibly coarse. Making the optimization you propose prevents future improvements in that area, I think, or at least severely limits us there.

(You could of course apply that optimization in a kernel.response subscriber yourself for now: if node_list is present, automatically omit node:* cache tags.)

+100 to creating an issue to create a cache tag hierarchy.

I think the easiest is indeed to use either:

- node (encompasses all node:x things)

OR

- node:*

instead of node_list / [something]_list.

I am not sure which one is simpler to parse / merge, but lets definitely open an issue.

I think I would go with a naming convention to describe the hierarchy.

With the old way even the merging with just 'node' would make sense:

A: [ 'node:1', 'node:2' ] LIKE [ 'node' => [1,2] ]
B: [ 'node' ] LIKE [ 'node' => TRUE ]
R: [ 'node' ] LIKE [ 'node' => TRUE ]

As TRUE is no array and overwrites the multiple values and hence takes priority.

#36: I don't think it limits us, if we apply a more granular tag, then don't apply 'node' OR 'node_list' or 'node:*' or whatever is the optimized hierarchical tag ...

Then optimization does not take place.

For the excessive menu cache tags invalidations, we have #941970: Only set router rebuild needed when something related to routing actually changes.

#2442041: Remove CachedStorage::listAll() caching also landed, solving the biggest problem reported in #16.

Officially changing this into a meta issue,which it has been for a while ;)

That said, I need to do new tests once I've updated the site where I'm doing that so that I can benefit from the recent improvements.

But at the moment, I'm not seeing any specific problems that would be critical. My suggestion would be to move this down to major, we can also create new major/critical issues as we find new things to improve.

I agree with #41. Especially considering that:

• changing when cache tags are invalidated is not an API change; the API remains unchanged, optimizing when/how often cache tags are invalidated does not break anything
• 99% of cache tags are retrieved using EntityInterface::getCacheTags(), so if we change cache tags at that level (e.g. node:35 -> n:35), that'd hardly constitute an API change, and the chances of this needing to happen are very slim
• optimizing the X-Drupal-Cache-Tags header can happen at a higher level: it can be done in a KernelEvents::RESPONSE subscriber, hence not needing any API changes

@berdir: since you have had exposure with at least one major D8 site already, did you encounter outstanding issues with cache-tags in the past few months? The idea in #33 seems to be the same approach than you took at https://gist.github.com/Berdir/fae8136a615a59abd3ea and I'm wondering if minifying cache tags is still an option we'd like to explore in this issue?

Also does anybody have any feedback on the overall performance hit when requiring large headers (16k+) in the stack? People could virtually get beyond fixed limits (e.g. complex view) simply by doing regular site building, and there's only so much you can tweak in your stack anyway (for instance, you can't really ask Cloudflare or Fastly to change their headers size every morning)

#43: I think it still makes sense to investigate shortening, but on the other hand any Response Subscriber can do so in contrib/ without any API change, so I am not sure how important it is for 8.0.0 - compared to other things.

Response Subscriber can do so in contrib/ without any API change

As I've learned quite painfully, be careful when you do this. This will also affect the cache tags that the page cache sees, so invalidations don't work anymore unless you also explicitly minimize invalidated cache tags and invalidate the minimized version too.

I currently disabled that part of my optimization, but I still have the logic that drops e.g. the single node cache tags if the list is present and I've also implemented #2449087: Don't add cache tags for files, instead, use {file_usage} to invalidate whoever embeds them in file_entity.

Re fastly, note that they recently increased the limit. We're not 100% sure to what exactly. They said to 16KB, but I found a post somewhere from them where they said that the old value was 8KB/1024 characters. So it is probably 2048 now.

That said, I'm totally OK with doing stuff like this in contrib. I think @WimLeers mentioned that drupal.org/project/purge already contains something like this and also deals with the page cache problem according to him.

#43 See the port of Fastly to D8 that I did for a generic cache tag minification implementation: #2491561: Port to Drupal 8.

Therefore +1 to what Fabianx said.

I like how Wim has handled this in the Fastly port but I can't help thinking we should still be handling this in core and not put the burden on contrib to implement. I'm not saying it has to be necessarily implemented for 8.0.0, but what about 8.1.x if there's any API change required? As to the severity, we need to keep in mind that pages exceeding the max headers size will throw an ugly 502 Bad Gateway error which might not be easy to track down, depending where the limitation in the stack is.

I can't help thinking we should still be handling this in core and not put the burden on contrib to implement

No. Different sites choose different optimizations. Doing this in core requires invalidating not only the "actual" cache tags but also the minified cache tags. Which means making Drupal core slower, because the page cache is using the built-in cache APIs. Invalidating cache tags on Fastly does not affect Drupal core in any way, so there it's fine to send potentially useless cache tag invalidations (the minified ones) — that's Fastly's problem then.

but what about 8.1.x if there's any API change required?

The above already demonstrates no API changes are necessary.

As to the severity, we need to keep in mind that pages exceeding the max headers size will throw an ugly 502 Bad Gateway error which might not be easy to track down, depending where the limitation in the stack is.

This is IMO the strongest argument. It's the only one that could make us consider this critical IMO. Poorly configured shared hosting (i.e. with low header size limits) + 502 = frustrated Drupal 8 user.

Thanks for the clarification, Wim.

Doing this in core requires invalidating not only the "actual" cache tags but also the minified cache tags.

I was actually thinking only the 'minified' version would exist and no longer the "actual" cache tags as we know them. 'node' would permanently become 'n', 'block' would permanently become 'b', etc. I trust you to know what's best, though.

I was actually thinking only the 'minified' version would exist and no longer the "actual" cache tags as we know them. 'node' would permanently become 'n', 'block' would permanently become 'b', etc. I trust you to know what's best, though.

If that were the case, I would run away screaming from Drupal as a developer. That'd seriously impede the DX IMO. You wouldn't be able to look at your cache backends (DB, redis, whatever) and look at the cache tags stored there without looking up every single one. Plus, you would not actually even have a way to look them up, unless we maintained a reverse lookup table.

Drupal 8 out of the box is going to break some hosting providers and CDNs if the cache tags aren't managed to not grow larger than some limit. The nginx default large_client_header_buffers is 8K which at Acquia we have already seen at least one fairly simple D8 site hit that limit (it's fairly common to keep the default at that size). So it would be nice if core had a setting along the lines of "if there will be more than N bytes of cache tags, just don't send the cache tags in the header" because in some cases it would be better not to cache a page with lots of entities on it than to throw a 500. Even if we raise the limit to 16K or 32K, somebody eventually is going to break it.

And minimizing tags without a limit isn't sufficient because at some number of entities we will still grow beyond the header limit or the memory limit. Additionally we have a concern with the use of cache tags when the number of active bans in Varnish could become extremely high to support every different cache tag. After a few thousand, apparently it can cause a performance problem.

Additionally Cloudflare told us at Acquia when we asked about raising the header limit above 8K that "We discourage large response headers since they significantly downgrade performance."

Sure, we can solve this in contrib, but it's one more module that every D8 site should install.

"if there will be more than N bytes of cache tags, just don't send the cache tags in the header" because in some cases it would be better not to cache a page with lots of entities on it than to throw a 500

That's definitely a possibility. The problem is that it'll be mighty difficult to agree on what the default limit should be. But be ware that the part outside of quotes is wrong: cache tags don't cause a page to be cached, but allow a page to be invalidated from the cache. So in case such a situation occurs, what you really want is to set a header that causes reverse proxies not to cache that response. (Think Surrogate-Control: max-age=0.)

Additionally we have a concern with the use of cache tags when the number of active bans in Varnish could become extremely high to support every different cache tag. After a few thousand, apparently it can cause a performance problem.

Not according to the Varnish Software CTO: https://www.varnish-software.com/comment/74099#comment-74099

Additionally Cloudflare told us at Acquia when we asked about raising the header limit above 8K that "We discourage large response headers since they significantly downgrade performance."

Performance of what? It sounds to me like they're referring to the client, which indeed would have to download 8K of headers. But you could just configure your reverse proxy to not pass that header along to the client.

Sure, we can solve this in contrib, but it's one more module that every D8 site should install.

Untrue; only the largest/most complex sites would have to do this.

To the people who want to see this move forward: you can start working on generic optimizations in a ResponseSubscriber that runs after FinishResponseSubscriber, in order of impact/importance:

1. If node_list is present, but we also have node:X cache tags: only keep node_list
2. Same for other entity types.
3. node:999 -> 999, config.block:999 -> b:999
4. Consider having a single config_all cache tag and stripping all config.* cache tags (this would also require a CacheTagsInvalidatorInterface service), or do it per config.[category] e.g. config.block_list
5. et cetera

That'd be a very easy patch to experiment with.

(I'd love to do this, but this is a very simple issue to fix, that with 98% probability doesn't require API changes. So I need to be working on criticals instead. We can easily fix this even during the RC phase.)

The problem is that it'll be mighty difficult to agree on what the default limit should be.

I talked to Wim about this - from Acquia Cloud's perspective it would be fine if the default limit was "no limit" as long as it was easy to change. We could then set the limit in the Acquia hosting include on behalf of our customers to match what our platform supports.

I think #54 is a great idea.

The suggestion in #2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them seems to help with that.

+1 to #56.

Here's some more feedback from the field: we're basically coming across this issue for any D8 brochure site that has enough items on the page to cause cache-tags to populate very quickly. Last example being a slideshow getting beyond a 4K limit.

X-Drupal-Cache-Tags: block_content: (snipped) file:1006 file:1081 file:1241 file:1246 file:1251 file:1261 file:1266 file:1271 file:1276 file:1281 file:1286 file:1291 file:1296 file:1301 file:1306 file:1311 file:1356 file:1451 file:1456 file:1461 file:1466 file:1471 file:1476 file:1481 file:1486 file:1491 file:1571 file:1576 file:1581 file:1586 file:1591 file:1596 file:1601 file:1606 file:1611 file:1616 file:1621 file:1626 file:1631 file:1636 file:1641 file:1646 file:1651 file:1656 file:1661 file:1666 file:1671 file:1676 file:1681 file:1686 file:1691 file:1696 file:1701 file:1721 file:1731 file:1736 file:1746 file:1751 file:1756 file:1761 file:1766 file:1771 file:1781 file:1786 file:1796 file:1816 file:1826 file:1831 file:1836 file:1851 file:1856 file:1861 file:1866 file:1876 file:1886 file:291 file:316 file:371 file:446 file:451 file:456 file:471 file:476 file:481 file:486 file:491 file:496 file:501 file:506 file:511 file:516 file:521 file:526 file:531 file:536 file:541 file:546 file:551 file:556 file:561 file:566 file:571 file:576 file:581 file:586 file:591 file:596 file:601 file:611 file:696 file:701 file:706 file:746 file:751 file:756 file:761 file:766 file:771 file:776 file:781 file:786 file:791 file:801 file:806 file:811 file:816 file:821 file:826 file:831 file:961 file:966 file:971 node:11 node:131 node:196 node:201 node:206 node:216 node:221 node:226 node:231 node:236 node:241 node:246 node:251 node:256 node:261 node:266 node:271 node:276 node:281 node:286 node:291 node:296 node:301 node:306 node:311 node:316 node:321 node:326 node:336 node:341 node:346 node:351 node:356 node:361 node:386 node:391 node:396 node:401 node:406 node:411 node:416 node:421 node:426 node:431 node:436 node:441 node:446 node:451 node:456 node:461 node:466 node:471 node:481 node:561 node:566 node:571 node:576 node:616 node:631 node:651 node:661 node:666 node:671 node:676 node:681 node:686 node:691 node:696 node:701 node:706 node:711 node:716 node:721 node:726 node:731 node:736 node:741 node:746 node:751 node:756 node:76 node:761 node:766 node:771 node:776 node:781 node:786 node:791 node:796 node:801 node:806 node:81 node:811 node:816 node:821 node:826 node:831 node:836 node:841 node:846 node:86 node:861 node:871 node:876 node:886 node:891 node:896 node:901 node:906 node:91 node:911 node:916 node:921 node:926 node:941 node:946 node:951 node:956 node:966 node:971 node:976 node:981 node:986 node:996 (snipped)

#56 (#2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them) is done, now let's do #54: specify a header size limit.

This isn't solving the root problem but what about looking at shortening the tags where possible? For example, n:206 and f:1596 rather than node:206 & file:1596. It is a stupid hack that doesn't solve the real problem but it would buy some more breathing room.

A generic shortening is not that easy, it could result in clashes and there are tricky problems. The *optional* internal page cache *middleware* is using the same cache tags:

* optional: We can't rely on it being there and only do the shortening as part of that class
* middleware: Meaning, nothing except another middleware can be executed after it, and that would then again slow down the page cache because one more class needs to be loaded.

And the page cache shouldn't see the shortened cache tags otherwise you need to invalidate them too which then results in additional invalidations in e.g. the fast chained backends (but we need to improve them anyway). I stopped shortening cache tags but what I kept is dropping the node:N cache tags when the node_list cache tag is present and similar optimizations for blocks and menus (which are a good target to optimize.. we should consider a single cache tag or optimize invalidations.. right now, all menus are invalidated on a menu rebuild anyway).

The problem with the max length are similar when combining with page cache. You don't want to just cut off but prevent caching.. but not the internal page cache, as that one has no limit on the cache tag length. And a lot of cache tags also means a lot of stuff being loaded, so those are exactly the pages you're most interested in caching ;)

#58 is the *perfect* example why I still think we should do #2449087: Don't add cache tags for files, instead, use {file_usage} to invalidate whoever embeds them. It's already implemented in our file_entity port and so far, works very well for us. The result is that those file cache tags are just *gone*, instead, changing a file (which is not something that core allows in any way anyway) just invalidates the entities that use the file.

Well, the fastly implementation has a generic use a 3-character hash shortening, when size_limit reached.

Not sure how much it would help on the 4K limit.

Yes, maybe if file_entity implements it we should indeed do that in core - I misunderstood earlier.

Obviously one can also get fancy and create groups of things similar to how wildcard flushes in memcache 6.x/7.x worked then subscribe to cache tag invalidations.

I'm heavily hand-waving here based on third-hand information, but my understanding is that various components of a front-end proxy stack (Varnish, Ngnix, etc.) default to 4K for headers, and I've seen at least 2-3 references to Berdir's gist in https://gist.github.com/Berdir/fae8136a615a59abd3ea on internal Acquia tickets, Twitter, etc. as a way around this problem. It seems like this is something we need to do something about in core. Is #54 enough?

Yes, my gist is flawed, it breaks page cache. I've partially reverted that part and just kept the optimizations (dropping cache tags that are covered by list cache tags).

To avoid the issues between the 'real' cache tags and the shortened ones, it might be possible to have a cache tags service which just does the shortening then hands it off - then you're using the short tags for all invalidations, not just for the headers.

The problem is, that just covers invalidations. We'd also need to convert the actual cache tags that are passed to cache writes, and each backend is responsible for dealing with that.

I'll roll a patch; I'm going to do a hybrid of #54 and what I did for the Fastly module in #2491561: Port to Drupal 8.

That should solve all immediate problems, but we should still be looking at the suggestions in #53.

As promised in my previous comment: #2542868: Allow a header value size limit to be specified.

Wim and myself talked a bit about that problem, given that even 4k is not necessarily enough, which was the recently introduced number of Acquia hosting.

We should group those headers by prefix, and maybe if any prefix exists more than 5 times, replace it with the "foo_list" tag.

Wim and myself talked a bit about that problem, given that even 4k is not necessarily enough, which was the recently introduced number of Acquia hosting.

4K is not recommended. 16K is recommended. Fastly does this. Acquia Cloud does this.

We should group those headers by prefix, and maybe if any prefix exists more than 5 times, replace it with the "foo_list" tag.

Simpler: find all cache tags that have a _list suffix (e.g. node_list, block_list) and then take their prefix (node, block), append a colon (node:, block:) and remove any matching cache tags.

Simpler: find all cache tags that have a _list suffix (e.g. node_list, block_list) and then take their prefix (node, block), append a colon (node:, block:) and remove any matching cache tags.

Yeah the trick is, how do we skip too much processing on every request.

Exactly.

We can continue to evolve, improve, refine our use of cache tags, to get more granular cache invalidation, or more efficient, or whatever other trade-off we want to take into consideration.

Hello guys,

Didn't read all the topic yet, but have to say about my cachetags table.

Now it has 12 016 071 entries and takes 1,3 Gb of disk storage.

The top entries is:

4xx-response 9108
address_list 4243
breakpoints 152
callcamp:1076 52
callcamp:1083 52
callcamp:1141 77
callcamp:1149 60
callcamp:1159 67
callcamp_list 3525
config:core.extension 78
config:system.menu.account 189
config:system.menu.admin 190
config:system.menu.hsuser-menu 163
config:system.menu.tools 190
config:views.view.hsorders 87
config:view_list 57
contextual_links_plugins 162
element_info_build 162
entity_bundles 188
entity_field_info 242
entity_types 179
hsorder_list 5200
local_action 162
local_task 226
person_list 4328
phone_list 2140
route_match 196
taxonomy_term_list 60
theme_registry 77
user_list 140

I have around 12 M entities of different types. Overall database entries is 66 M (migrations, revisions, etc), overall storage size is 9,4 Gb.

Well... not sure, but may be we need to do something with that? probably disable caching for given entity type... not sure. Do you see any problem here?

We have #2250033: Garbage collection for cache tag invalidations for that. But until that is committed, it is perfectly safe to do a truncate on that table followed by cache rebuild with drush or in the UI.

Thank you Berdir!

Do we have anything like an X-Drupal-Cache-Debug header we can send so we can do cache tag debugging with an FOSHTTPCache-like VCL? I'm working on some testing/debugging/profiling with cache tags and varnish, and it's annoying having to tweak Varnish config constantly to test in a production-like state vs debugging (though not the end of the world). I could add a new issue requesting it if needed.

Also, I may be reading things completely incorrectly, but as a result of #2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them, the following is in services.yml:

  # Cacheability debugging:
  #
  # Responses with cacheability metadata (CacheableResponseInterface instances)
  # get X-Drupal-Cache-Tags and X-Drupal-Cache-Contexts headers.
  #
  # For more information about debugging cacheable responses, see
  # https://www.drupal.org/developing/api/8/response/cacheable-response-interface
  #
  # Not recommended in production environments
  # @default false
  http.response.debug_cacheability_headers: false

AFAICT, the only way to get cache tags working with Varnish at all (other upstreams too, right?) is to send the headers... so why would this property have anything to do with 'debug' or not be recommended in production environments. Isn't it required to use with Varnish?

I would think the setting http.response.debug_cacheability_headers should be for sending across an X-Cache-Debug or X-Drupal-Cache-Debug header, while http.response.cacheability_headers would determine whether the Cache-Tags and Cache-Contexts headers are sent; and the warning about production environments would not apply to that.

Found while digging through #2527126-75: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them:

For purpose B, just having that header is insufficient, you also need code that talks to the reverse proxy (Varnish, a CDN …) to inform it about the invalidations as they happen. (And given that I've ported both the Fastly and CloudFlare modules to Drupal 8, and added support for this, I can add an additional data point: they each require a specifically named header anyway, so we already need to remove/rename the X-Drupal-Cache-Tags header.)

So... is the idea that every reverse proxy/caching module (e.g. Akamai, Fastly, Varnish, etc.) will manually output whatever header it recommends (e.g. X-Drupal-Cache-Tags, or X-Cache-Tags, or X-Fastly-Cache-Tags, etc.)? And the X-Drupal-Cache-Tags header will not be the recommended header to be used for cache tag support in upstream proxies?

The cache tags docs page currently points to the FOSHTTPCache documentation for enabling cache tag support in Varnish, and I'm working on adding support in Drupal VM, so I'm just trying to get things straight and make sure I'm not adding any redundant information, and also so I'm understanding the purpose of the http.response.debug_cacheability_headers correctly.

The docs page also states that:

Just like Drupal 8 can send an X-Drupal-Cache-Tags header for debugging, it can also send a Surrogate-Keys header with space-separated values as expected by some CDNs or a Cache-Tag header with comma-separated values as expectected by other CDNs. And it could also be a reverse proxy you run yourself, rather than a commercial CDN service.

...but without any pointers for doing so—is there something I can enable to output these various types of headers?

After digging through a few more issues in related modules and efforts, I finally found this, which seems to be the answer I'm looking for:

@Berdir - the idea is that cache tag headers will be emitted by Cloudflare module, Varnish module, etc. Core is no longer in the business of sending production headers.

From #2542868-25: Allow a header value size limit to be specified, which was closed as 'works as designed'.

So I guess the idea is that Varnish users should wait for #2365949: Varnish module for Drupal 8 to land, and then get purge_purger_http configured for Varnish purging?

@catch - Thanks for the clarification; it looks like @nielsvm is planning on using core's headers for the Purge module, at least, so I'm targeting that implementation (e.g. in this blog post), but it'll be interesting to see whether most upstream modules/proxies decide to build their own cache tags header or use the 'debug' one built into core.

IMO, why write your own (like the Fastly module does) and maintain that extra baggage when it's already baked into core?

[Edit: also, cache tags are awesome. Going to start using them on a prod site or two and see if I can track how well it's working with Varnish's ban lurker, and also monitor need for garbage collection in the related issue.]

it looks like @nielsvm is planning on using core's headers for the Purge module, at least,

No he's not. He's planning to remove them. He's been aware for months that that is the wrong way to go.

IMO, why write your own (like the Fastly module does) and maintain that extra baggage when it's already baked into core?

Because, again, like the CR says (https://www.drupal.org/node/2592471), and like I wrote in my comment to you at #2527126-176: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them:

Yes, send your own. The CR already explains that very explicitly. See https://www.drupal.org/node/2592471.

I'll state the reason *again* then: because different reverse proxies have different expectations wrt the name of the header and/or the format of the value of the header.

And the much bigger reason: you need a Drupal module to talk to the reverse proxy anyway, to inform them which cache tags are being invalidated.

So you want a header with a particular name, a particular format, that your reverse proxy REMOVES before forwarding the response to clients, otherwise you send kilobytes of irrelevant information to the client, and you need to talk to your reverse proxy anyway, which requires a module anyway. So, yes, it is a bit of duplication, but it's worth it, because you can impose whatever strict limits/expectations your reverse proxy has.

So you want a header with a particular name, a particular format, that your reverse proxy REMOVES before forwarding the response to clients, otherwise you send kilobytes of irrelevant information to the client, and you need to talk to your reverse proxy anyway, which requires a module anyway. So, yes, it is a bit of duplication, but it's worth it, because you can impose whatever strict limits/expectations your reverse proxy has.

Being a pragmatist, though... only Fastly, CloudFlare, et all (the for-pay services) require a strict naming convention (Surrogate-Keys for Fastly, Cache-Tag for CloudFlare)... Varnish (and any other reverse proxy that allows defining custom logic for cache tag support) allows me to use whatever... and Drupal core already provides X-Drupal-Cache-Tags, and it doesn't look like that debug feature is going away, so if I were the Purge maintainer (I'm not, just hypothetical), and I saw that I didn't need to build a bit of code that I'd be responsible for, and can work with an existing feature in Drupal (albeit one that was renamed to a 'debug' feature very recently)... why not?

Just being a devil's advocate. I'm happy to update my blog post if it's truly a best practice of every upstream module to build it's own header implementation and logic. But if it's already in core... and it works great... and it works right now (which it does), the pragmatist in me likes it.

I would read this as saying: although it doesn't look like it's going away, the behaviour of http.response.debug_cacheability_headers is for debug purposes, and may change without warning at any point. Perhaps becoming much more verbose in the future, or perhaps becoming silent except under certain edge-conditions.

Technically there's nothing to stop you from using them however you like in your own modules, just as technically there's nothing to prevent you from calling internal/private functions…but if you'd prefer not to have the rug swept out from under you without warning, it might be best to simply generate your own headers.

Where "recently" means almost half a year ago. It took. Lot of discussion and thinking in #2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them to arrive at this conclusion. I wish we wouldn't have had to do this, but we did it for good reasons.

#83++

I'd love to hear @Fabianx, @alex.pott and @catch's thoughts on this though.

Sorry for the bikeshed, but yes, I read through all of #2527126: Only send X-Drupal-Cache-Tags and -Contexts headers when developer explicitly enables them, and about five other entire cache-tag/metadata threads before coming to the questions I've had in this thread. I'm okay with the answer, I just wanted some more justification... in 2527126, it seemed most of the thread was talking about simply allowing people to enable/disable cache tag headers... then between #99 and #101 the variable name switched from not having the word "debug" associated to having it, based on a few comments that were here and there between #75, #78, and mentioned a couple other places.

I'm guessing there was some further offline discussion/decisions made outside of that issues comments, but I just wanted to clarify that the messaging seemed a little sparse behind why core went from being the source of truth/ultimate implementation for the cache tags header to just offering a debug option to inspect headers (but as @manarth mentioned, don't rely on it!).

The current solution works and is extremely painless to use (which is why I decided to finally test it out and post a guide on my blog about it)—luckily the 'how the header is set' part of it is a very small implementation detail, and I'll be sure to update the post once Purge/Generic HTTP Purger updates to start building its own headers.

To that end, I opened #2692523: Generate own Purge-Cache-Tags header (I can't speak for @nielsvm, but I didn't see any other indication that he planned on switching the implementation).

[Edit: also, just for clarity, I was asking these questions just to make the answers as clear as possible (to someone who hasn't had a chance to remain embedded in D8 development for the past 8 months), so thank you for answering and providing better justification!]

I know you are writing about this, to help make things clearer for D8 users. And that is *awesome*. I know you want to imorove the documentation in https://www.drupal.org/developing/api/8/cache/tags#reverse-proxies to become more concrete, which I applaud you for :) But, my big concern, and hence the reason for me reacting so sharply and firmly is that you're making a recommendation that goes *against* what we collectively arrived at the consensus. Perhaps we can change that recommendation over time. (Hence the reason I'd like feedback from the people I mentioned in #84.) But *for now*, please follow the consensus.

@geerlingguy++

(Another problem with using this header btw is that you cannot effectively enable a debug header anymore, because the reverse proxy (which is eg always present on Acquia Cloud in the form of Varnish) will strip this header. So, by repurposing the debug header, it is no longer possible to be a debug header. A similar problem arises when you have multiple reverse proxies layered on top of each other: a Varnish instance to offload origin, and then a CDN to reduce latency. If the CDN uses this same header, then Varnish will already have stripped this header.)

I think it is important to highlight the following when referring to cache invalidation strategies and methods in Varnish Cache:

Bans:
In Varnish you can use bans when flushing objects in the cache and it works great to a certain point, think in the hundreds of bans per second, as the ban list size is manageable and will not incur any performance degradation. This applies specially when using so-called ban-luker friendly bans. The only drawback is that this cache invalidation method does not free memory right away.

Purges:
Purges are normally used to flush one specific object from the cache and evicts it from memory immediately. You can do as many evictions as your server can take purge requests, which should be in the thousands per second.

XKEY / HashTo / Surrogate Keys:
Evicts from the cache all objects with a common key, also called cache tags. It is Varnish Softare implementation of Surrogate Keys and it is provided through the usage of the XKEY VMOD, which was open sourced as part of release of Varnish Cache 4.1. This module and other are available on the VMOD bundle available in github; https://github.com/varnish/varnish-modules

When "flushing the cache" in Varnish you have plenty of options and I think that it is important to highlight the differences between these, specially bans and purges. You can read more about this here

Feel free to reach to me if you have questions or envision ways of making these goods available for the wider Drupal community (i.e. envolve our experts in reviewing VCL, a VCL library for Drupal Core out-of-the-box).

For a simple front page, I have hundreds of config entities and a bunch of different content entities, mainly nodes, in the list of cache tags. Would it make sense to implement a cache tag counting on bubbling, and replacing them with ENTITY_TYPE_list tags? This could massively reduce the amount of cache tags associated per cache entry.

That might work, it depends on your use case. See \Drupal\Core\Entity\Entity::invalidateTagsOnSave().

34 Kilobytes. Please fix.
X-Drupal-Cache-Tags: block_view config:block.block.architonic_breadcrumbs config:block.block.architonic_content config:block.block.architonic_help config:block.block.architonic_local_actions config:block.block.architonic_login config:block.block.architonic_messages config:block.block.architonic_page_title config:block.block.architonic_primary_local_tasks config:block.block.architonic_secondary_local_tasks config:block.block.demodateblock config:block_list config:easy_breadcrumb.settings config:shortcut.set.default config:system.menu.admin config:user.role.administrator config:user.role.authenticated config:views.view.account_object_product http_response local_task media:271600 media:271601 media:271602 media:271603 media:271608 media:271610 media:271613 media:271614 media:271615 media:271620 media:271622 media:271625 media:271626 media:271627 media:271629 media:271630 media:271631 media:271632 media:271635 media:271637 media:271644 media:271647 media:271648 media:271649 media:271650 media:271651 media:271652 media:271653 media:271654 media:271655 media:271656 media:271657 media:271658 media:271659 media:271660 media:271662 media:271663 media:271664 media:271666 media:271667 media:271668 media:271669 media:271671 media:271672 media:271673 media:271674 media:271675 media:271679 media:271680 media:271681 media:271682 media:271683 media:271684 media:271685 media:271686 media:271687 media:271691 media:271692 media:271693 media:271694 media:271695 media:271696 media:271697 media:271698 media:271699 media:271700 media:271701 media:271704 media:271705 media:271706 media:271707 media:271709 media:271710 media:271711 media:271712 media:271713 media:271714 media:271715 media:271716 media:271717 media:271718 media:271720 media:271721 media:271722 media:271723 media:271724 media:271726 media:271727 media:271728 media:271729 media:271730 media:271731 media:271732 media:271734 media:271737 media:271738 media:271740 media:271741 media:271742 media:271751 media:271752 media:271753 media:271754 media:271756 media:271758 media:271762 media:271763 media:271767 media:271770 media:271781 media:271782 media:271783 media:271784 media:271788 media:271789 media:271791 media:271792 media:271793 media:271795 media:271796 media:271797 media:271798 media:271801 media:271802 media:271806 media:271807 media:271809 media:271810 media:271821 media:271822 media:271823 media:271826 media:271827 media:271829 media:271830 media:271831 media:271832 media:271835 media:271837 media:271839 media:271840 media:271841 media:271843 media:271855 media:271889 media:271891 media:271892 media:271894 media:271895 media:271901 media:271902 media:271908 media:271914 media:271915 media:271916 media:271918 media:271919 media:271920 media:271921 media:271922 media:271923 media:271924 media:271925 media:271926 media:271927 media:271928 media:271929 media:271930 media:271931 media:271933 media:271934 media:271944 media:271945 media:271953 media:271954 media:271955 media:271956 media:271957 media:271958 media:271959 media:271960 media:271962 media:271963 media:271965 media:272105 media:272107 media:272111 media:272115 media:272117 media:272119 media:272121 media:272122 media:272144 media:272145 media:272146 media:272147 media:272171 media:272177 media:272186 media:272195 media:272196 media:272204 media:272207 media:272208 media:272210 media:272211 media:272274 media:272277 media:272280 media:272281 media:272282 media:272286 media:272288 media:272290 media:272292 media:272293 media:272294 media:272295 media:272296 media:272297 media:272298 media:272299 media:272301 media:272317 media:272318 media:272321 media:272327 media:272328 media:272330 media:272331 media:272332 media:272334 media:272335 media:272338 media:272342 media:272343 media:272344 media:272345 media:272346 media:272348 media:272350 media:272351 media:272352 media:272353 media:272356 media:272358 media:272359 media:272361 media:272362 media:272364 media:272365 media:272366 media:272367 media:272368 media:272369 media:272370 media:272372 media:272373 media:272374 media:272375 media:272377 media:272378 media:272379 media:272380 media:272381 media:272382 media:272383 media:272384 media:272385 media:272386 media:272387 media:272388 media:272389 media:272393 media:272394 media:272395 media:272400 media:272402 media:272403 media:272404 media:272405 media:272406 media:272407 media:272408 media:272409 media:272410 media:272455 media:272457 media:272469 media:272470 media:272471 media:272472 media:272473 media:272474 media:272475 media:272477 media:272480 media:272530 media:272539 media:272541 media:272542 media:272544 media:272546 media:272548 media:272550 media:272553 media:272555 media:272556 media:272557 media:272571 media:272574 media:272576 media:272579 media:272587 media:272588 media:272592 media:272593 media:272595 media:272596 media:272597 media:272598 media:272599 media:272600 media:272602 media:272608 media:272618 media:272619 media:272622 media:272623 media:272628 media:272631 media:272637 media:272639 media:272640 media:272643 media:272646 media:272651 media:272652 media:272654 media:272657 media:272658 media:272662 media:272665 media:272667 media:272675 media:272676 media:272680 media:272684 media:272685 media:272687 media:272688 media:272690 media:272696 media:272698 media:272701 media:272702 media:272730 media:272733 media:272734 media:272739 media:272744 media:272745 media:272746 media:272748 media:272749 media:272751 media:272752 media:272753 media:272754 media:272755 media:272756 media:272758 media:272759 media:272858 media:272862 media:272866 media:272868 media:272869 media:272870 media:272871 media:272872 media:272873 media:272874 media:272876 media:272878 media:272883 media:272887 media:272888 media:272889 media:272894 media:272909 media:272910 media:272911 media:272913 media:272914 media:272915 media:272916 media:272917 media:272918 media:272919 media:272920 media:272921 media:272922 media:272926 media:272930 media:272931 media:272934 media:272949 media:272954 media:272978 media:272979 media:273051 media:273052 media:273064 media:273069 media:273111 media:273112 media:273113 media:273117 media:273119 media:273120 media:273121 media:273122 media:273129 media:273130 media:273131 media:273132 media:273133 media:273134 media:273135 media:273136 media:273137 media:273138 media:273139 media:273140 media:273141 media:273147 media:273148 media:273149 media:273164 media:273178 media:273192 media:273199 media:273219 media:273251 media:273252 media:273253 media:273254 media:273256 media:273257 media:273258 media:273262 media:273263 media:273268 media:273269 media:273270 media:273272 media:273273 media:273275 media:273277 media:273278 media:273280 media:273281 media:275621 media:275624 media:275625 media:275626 media:275627 media:275631 media:275633 media:275634 media:277642 media:277794 media:277796 media:277799 media:277802 media:277980 media:278051 media:278202 media:278372 media:278374 media:278377 media:278387 media:278395 media:278455 media:278457 media:278459 media:278460 media:278618 media:278619 media:278622 media:278640 media:278642 media:278648 media:278651 media:278719 media:278725 media:278728 media:278730 media:278771 media:278772 media:278773 media:278774 media:278811 media:278812 media:278813 media:278814 media:278815 media:278817 media:278818 media:278824 media:278827 media:278831 media:278832 media:278834 media:278893 media:278894 media:278895 media:278898 media:278930 media:278931 media:278932 media:278933 media:278934 media:278935 media:278936 media:278937 media:278939 media:278940 media:278954 media:278955 media:278956 media:278964 media:278965 media:278966 media:278973 media:279027 media:279028 media:279029 media:279030 media:279032 media:279033 media:279035 media:279036 media:279037 media:279040 media:279047 media:279048 media:279049 media:279050 media:279054 media:279060 media:279061 media:279063 media:279064 media:279065 media:279066 media:279067 media:279068 media:279069 media:279071 media:279072 media:279077 media:279081 media:279083 media:279085 media:279086 media:279094 media:279095 media:279122 media:279125 media:279128 media:279129 media:279235 media:279236 media:279237 media:279238 media:279239 media:279241 media:279242 media:279243 media:279249 media:279257 media:279313 media:284206 media:284208 media:284257 media:284258 media:284263 media:284281 media:284282 media:284283 media:284284 media:284292 media:284293 media:284309 media:284332 media:284334 media:284335 media:284338 media:284339 media:284358 media:284359 media:284360 media:284361 media:284362 media:284363 media:284364 media:284365 media:284369 media:284374 media:284385 media:284395 media:284399 media:284401 media:284405 media:284448 media:284449 media:284450 media:284452 media:284457 media:284463 media:284472 media:284473 media:284474 media:284475 media:284476 media:284480 media:284487 media:284488 media:284491 media:284492 media:284497 media:284504 media:284505 media:284507 media:284516 media:284517 media:284520 media:284523 media:284524 media:284527 media:292928 media:292935 media:292976 media:292978 media:293047 media:293049 media:293051 media:293061 media:293064 media:293067 media:293070 media:293074 media:293079 media:293081 media:293087 media:293121 media:293130 media:293133 media:293135 media:293138 media:293143 media:293145 media:293146 media:293147 media:293148 media:293149 media:293150 media:293151 media:293152 media:293153 media:293154 media:293155 media:293157 media:293158 media:293159 media:293160 media:293161 media:293162 media:293163 media:293164 media:293165 media:293166 media:293167 media:293168 media:293169 media:293170 media:314159 media:314168 media:314174 media:314206 media:314212 media:314217 media:314230 media:314236 media:314256 media:314282 media:314292 media:314300 media:314301 media:314302 media:314305 media:323398 media:323412 media:323478 media:323517 media:323518 media:323520 media:323521 media:323522 media:323524 media:323526 media:323527 media:323528 media:323529 media:323530 media:323531 media:323535 media:323537 media:323538 media:323539 media:323544 media:323551 media:323552 media:323554 media:323555 media:323556 media:323559 media:323561 media:323563 media:323565 media:323567 media:323572 media:323573 media:323575 media:323576 media:323578 media:323579 media:323581 media:323582 media:323583 media:323584 media:323585 media:323586 media:323587 media:323591 media:323592 media:323594 media:323595 media:323596 media:323597 media:323598 media:323599 media:323600 media:323601 media:323602 media:323718 media:323723 media:323725 media:323727 media:323729 media:323731 media:323735 media:323737 media:323738 media:323744 media:323772 media:323774 media:323775 media:323778 media:323782 media:323785 media:323802 media:323809 media:323819 media:323823 media:323833 media:323835 media:323840 media:323848 media:323856 media:323857 media:323859 media:323860 media:323861 media:323875 media:323877 media:323883 media:323884 media:323896 media:323909 media:323918 media:323922 media:323924 media:323928 media:323931 media:323935 media:323936 media:323947 media:323955 media:323956 media:323960 media:323961 media:323963 media:323964 media:323970 media:323971 media:323972 media:324036 media:324040 media:324043 media:324044 media:324052 media:324055 media:324060 media:324063 media:324072 media:324082 media:324084 media:324086 media:324088 media:324093 media:324110 media:324112 media:324113 media:324116 media:324120 media:324124 media:324126 media:324128 media:324133 media:324139 media:324143 media:324145 media:324146 media:324147 media:324150 media:324151 media:324152 media:324193 media:324199 media:324206 media:324209 media:324211 media:324215 media:324227 media:324228 media:324229 media:324234 media:324236 media:324238 media:324239 media:324240 media:324245 media:324249 media:324254 media:324255 media:324257 media:324258 media:324259 media:324261 media:324263 media:324264 media:324266 media:324269 media:324272 media:324285 media:324286 media:324287 media:324288 media:324289 media:324290 media:324291 media:324292 media:324293 media:324294 media:324308 media:324309 media:324310 media:324313 media:324318 media:324321 media:324326 media:324335 media:324341 media:324343 media:324344 media:324345 media:324350 media:373184 media:373192 media:373195 media:373200 media:373204 media:373207 media:403907 media:403908 media:403909 media:403910 media:403911 media:403912 media:403913 media:403919 media:403920 media:403921 media:403922 media:403924 media:403925 media:403927 media:403928 media:403929 media:403930 media:403932 media:403933 media:403934 media:403935 media:403936 media:403937 media:403940 media:403941 media:403943 media:403945 media:404392 media:404393 media:404394 media:404395 media:404396 media:404397 media:404398 media:404399 media:441742 media:441759 media:441777 media:441779 media:441780 media:441781 media:441788 media:441797 media:441806 media:441810 media:441814 media:441819 media:441821 media:441825 media:441837 media:441840 media:441854 media:441858 media:441875 media:441933 media:441936 media:441944 media:441954 media:442079 media:442204 media:442216 media:442219 media:442220 media:442221 media:442222 media:442223 media:442224 media:442226 media:442229 media:442231 media:442243 media:450500 media:450502 media:450506 media:450507 media:450509 media:450510 media:450512 media:450513 media:450519 media:450524 media:450526 media:450534 media:450536 media:450564 media:450568 media:450569 media:450570 media:450588 media:450590 media:450592 media:450596 media:450597 media:450608 media:450610 media:450612 media:450613 media:450617 media:450619 media:450620 media:450627 media:450628 media:450629 media:450630 media:450631 media:450637 media:450638 media:450639 media:450640 media:450641 media:450642 media:450643 media:450644 media:450645 media:450672 media:450675 media:450676 media:450677 media:450680 media:450718 media:450736 media:450740 media:450745 media:450751 media:450756 media:450757 media:450768 media:450773 media:450779 media:450783 media:450784 media:450786 media:450788 media:450789 media:450792 media:450794 media:450798 media:450801 media:450805 media:450808 media:450814 media:450818 media:450843 media:450844 media:450846 media:450847 media:450848 media:450849 media:453915 media:453916 media:453917 media:453918 media:453919 media:453920 media:453921 media:453922 media:453923 media:453924 media:453925 media:453926 media:453927 media:453928 media:453929 media:453930 media:453931 media:453932 media:644196 media:644200 media:644208 media:644230 media:644234 media:644235 media:644238 media:644241 media:644372 media:644377 media:644378 media:644379 media:644380 media:644383 media:644384 media:644403 media:644418 media:644438 media:644440 media:644441 media:644444 media:644445 media:644457 media:644458 media:644459 media:644460 media:644463 media:644466 media:644468 media:644622 media:644623 media:644624 media:644625 media:644627 media:644628 media:644630 media:644631 media:644632 media:644634 media:644635 media:644637 media:644639 media:644641 media:644642 media:644644 media:644646 media:644647 media:644655 media:644656 media:644657 media:644659 media:644660 media:644661 media:644662 media:644663 media:644664 media:644668 media:644670 media:644671 media:644675 media:644682 media:644701 media:644704 media:644705 media:644708 media:644724 media:644725 media:644726 media:644727 media:644730 media:644731 media:645221 media:645224 media:645226 media:645227 media:645228 media:645230 media:645232 media:645234 media:645235 media:645236 media:645237 media:645239 media:645240 media:645241 media:645242 media:645243 media:645244 media:645245 media:645246 media:645247 media:645249 media:645250 media:645251 media:645252 media:645253 media:645254 media:645350 media:645354 media:645357 media:645364 media:645370 media:645376 media:645383 media:645390 media:645393 media:645399 media:645407 media:645413 media:645416 media:645417 media:645418 media:645420 media:645421 media:645425 media:645426 media:645428 media:645429 media:645431 media:645433 media:645434 media:645436 media:645438 media:645440 media:645441 media:645442 media:645443 product:1147569 product:1147570 product:1147571 product:1147572 product:1147578 product:1147580 product:1147583 product:1147584 product:1147585 product:1147590 product:1147592 product:1147595 product:1147596 product:1147597 product:1147599 product:1147600 product:1147601 product:1147602 product:1147605 product:1147607 product:1147614 product:1147617 product:1147618 product:1147619 product:1147620 product:1147621 product:1147622 product:1147623 product:1147624 product:1147625 product:1147626 product:1147627 product:1147628 product:1147629 product:1147630 product:1147632 product:1147633 product:1147634 product:1147636 product:1147637 product:1147638 product:1147639 product:1147641 product:1147642 product:1147643 product:1147644 product:1147645 product:1147649 product:1147650 product:1147651 product:1147652 product:1147653 product:1147654 product:1147655 product:1147656 product:1147657 product:1147661 product:1147662 product:1147663 product:1147664 product:1147665 product:1147666 product:1147667 product:1147668 product:1147669 product:1147670 product:1147671 product:1147674 product:1147675 product:1147676 product:1147677 product:1147679 product:1147680 product:1147681 product:1147682 product:1147683 product:1147684 product:1147685 product:1147686 product:1147687 product:1147688 product:1147690 product:1147691 product:1147692 product:1147693 product:1147694 product:1147696 product:1147697 product:1147698 product:1147699 product:1147700 product:1147701 product:1147702 product:1147704 product:1147707 product:1147708 product:1147710 product:1147711 product:1147712 product:1147721 product:1147722 product:1147723 product:1147724 product:1147726 product:1147728 product:1147732 product:1147733 product:1147738 product:1147741 product:1147752 product:1147753 product:1147754 product:1147755 product:1147759 product:1147760 product:1147762 product:1147763 product:1147764 product:1147766 product:1147767 product:1147768 product:1147769 product:1147772 product:1147773 product:1147777 product:1147778 product:1147780 product:1147781 product:1147792 product:1147793 product:1147794 product:1147797 product:1147798 product:1147800 product:1147801 product:1147802 product:1147803 product:1147806 product:1147808 product:1147810 product:1147811 product:1147812 product:1147814 product:1147826 product:1147860 product:1147862 product:1147863 product:1147865 product:1147866 product:1147872 product:1147873 product:1147879 product:1147885 product:1147886 product:1147887 product:1147889 product:1147890 product:1147891 product:1147892 product:1147893 product:1147894 product:1147895 product:1147896 product:1147897 product:1147898 product:1147899 product:1147900 product:1147902 product:1147903 product:1147905 product:1147906 product:1147916 product:1147917 product:1147927 product:1147928 product:1147929 product:1147931 product:1147932 product:1147933 product:1147934 product:1147935 product:1147937 product:1147938 product:1147940 product:1148084 product:1148087 product:1148091 product:1148095 product:1148097 product:1148099 product:1148101 product:1148102 product:1148127 product:1148128 product:1148129 product:1148130 product:1148157 product:1148163 product:1148172 product:1148190 product:1148191 product:1148201 product:1148204 product:1148205 product:1148207 product:1148208 product:1148280 product:1148283 product:1148286 product:1148287 product:1148288 product:1148292 product:1148294 product:1148296 product:1148298 product:1148299 product:1148300 product:1148301 product:1148302 product:1148303 product:1148304 product:1148305 product:1148307 product:1148324 product:1148325 product:1148328 product:1148334 product:1148335 product:1148337 product:1148338 product:1148339 product:1148341 product:1148342 product:1148345 product:1148349 product:1148350 product:1148351 product:1148352 product:1148353 product:1148355 product:1148357 product:1148358 product:1148359 product:1148360 product:1148363 product:1148365 product:1148366 product:1148369 product:1148370 product:1148372 product:1148373 product:1148374 product:1148375 product:1148376 product:1148377 product:1148378 product:1148380 product:1148381 product:1148382 product:1148383 product:1148385 product:1148386 product:1148387 product:1148388 product:1148389 product:1148390 product:1148391 product:1148392 product:1148393 product:1148394 product:1148395 product:1148396 product:1148397 product:1148402 product:1148403 product:1148404 product:1148409 product:1148411 product:1148412 product:1148413 product:1148414 product:1148415 product:1148416 product:1148417 product:1148418 product:1148419 product:1148471 product:1148473 product:1148485 product:1148486 product:1148487 product:1148488 product:1148489 product:1148492 product:1148493 product:1148495 product:1148498 product:1148548 product:1148557 product:1148559 product:1148560 product:1148562 product:1148564 product:1148566 product:1148568 product:1148571 product:1148573 product:1148574 product:1148575 product:1148589 product:1148592 product:1148594 product:1148597 product:1148605 product:1148606 product:1148610 product:1148611 product:1148613 product:1148614 product:1148615 product:1148616 product:1148617 product:1148618 product:1148620 product:1148626 product:1148636 product:1148637 product:1148640 product:1148641 product:1148646 product:1148649 product:1148655 product:1148657 product:1148658 product:1148661 product:1148664 product:1148669 product:1148670 product:1148672 product:1148675 product:1148676 product:1148680 product:1148683 product:1148685 product:1148693 product:1148694 product:1148698 product:1148702 product:1148703 product:1148705 product:1148706 product:1148708 product:1148714 product:1148716 product:1148719 product:1148720 product:1148748 product:1148751 product:1148752 product:1148757 product:1148762 product:1148763 product:1148764 product:1148766 product:1148767 product:1148769 product:1148770 product:1148771 product:1148772 product:1148773 product:1148774 product:1148776 product:1148777 product:1148878 product:1148882 product:1148886 product:1148888 product:1148889 product:1148890 product:1148891 product:1148892 product:1148893 product:1148894 product:1148896 product:1148898 product:1148903 product:1148907 product:1148908 product:1148909 product:1148914 product:1148929 product:1148930 product:1148931 product:1148933 product:1148934 product:1148935 product:1148936 product:1148937 product:1148938 product:1148939 product:1148940 product:1148941 product:1148942 product:1148946 product:1148950 product:1148951 product:1148954 product:1148969 product:1148974 product:1148998 product:1148999 product:1149072 product:1149073 product:1149085 product:1149090 product:1149133 product:1149134 product:1149135 product:1149139 product:1149141 product:1149142 product:1149143 product:1149144 product:1149151 product:1149152 product:1149153 product:1149154 product:1149155 product:1149156 product:1149157 product:1149158 product:1149159 product:1149160 product:1149161 product:1149162 product:1149163 product:1149169 product:1149170 product:1149171 product:1149186 product:1149200 product:1149214 product:1149221 product:1149241 product:1149277 product:1149278 product:1149279 product:1149280 product:1149282 product:1149283 product:1149284 product:1149288 product:1149289 product:1149294 product:1149295 product:1149296 product:1149298 product:1149299 product:1149301 product:1149303 product:1149304 product:1149306 product:1149307 product:1151730 product:1151733 product:1151734 product:1151735 product:1151736 product:1151740 product:1151742 product:1151743 product:1153777 product:1153929 product:1153931 product:1153934 product:1153937 product:1154116 product:1154187 product:1154338 product:1154508 product:1154510 product:1154513 product:1154523 product:1154531 product:1154591 product:1154593 product:1154595 product:1154596 product:1154756 product:1154757 product:1154760 product:1154778 product:1154780 product:1154786 product:1154789 product:1154858 product:1154864 product:1154867 product:1154869 product:1154910 product:1154911 product:1154912 product:1154913 product:1154950 product:1154951 product:1154952 product:1154953 product:1154954 product:1154956 product:1154957 product:1154963 product:1154966 product:1154970 product:1154971 product:1154973 product:1155032 product:1155033 product:1155034 product:1155037 product:1155069 product:1155070 product:1155071 product:1155072 product:1155073 product:1155074 product:1155075 product:1155076 product:1155078 product:1155079 product:1155093 product:1155094 product:1155095 product:1155103 product:1155104 product:1155105 product:1155113 product:1155167 product:1155168 product:1155169 product:1155170 product:1155172 product:1155173 product:1155175 product:1155176 product:1155177 product:1155180 product:1155187 product:1155188 product:1155189 product:1155190 product:1155194 product:1155200 product:1155201 product:1155203 product:1155204 product:1155205 product:1155206 product:1155207 product:1155208 product:1155209 product:1155211 product:1155212 product:1155217 product:1155221 product:1155223 product:1155225 product:1155226 product:1155234 product:1155235 product:1155262 product:1155265 product:1155268 product:1155269 product:1155375 product:1155376 product:1155377 product:1155378 product:1155379 product:1155381 product:1155382 product:1155383 product:1155389 product:1155397 product:1155453 product:1160424 product:1160426 product:1160475 product:1160476 product:1160481 product:1160499 product:1160500 product:1160501 product:1160502 product:1160510 product:1160511 product:1160527 product:1160550 product:1160552 product:1160553 product:1160556 product:1160557 product:1160576 product:1160577 product:1160578 product:1160579 product:1160580 product:1160581 product:1160582 product:1160583 product:1160587 product:1160592 product:1160603 product:1160613 product:1160617 product:1160619 product:1160623 product:1160667 product:1160668 product:1160669 product:1160671 product:1160676 product:1160682 product:1160691 product:1160692 product:1160693 product:1160694 product:1160695 product:1160699 product:1160706 product:1160707 product:1160710 product:1160711 product:1160716 product:1160723 product:1160724 product:1160726 product:1160735 product:1160736 product:1160739 product:1160742 product:1160743 product:1160746 product:1169250 product:1169257 product:1169298 product:1169300 product:1169369 product:1169371 product:1169373 product:1169383 product:1169386 product:1169389 product:1169392 product:1169396 product:1169401 product:1169403 product:1169409 product:1169443 product:1169452 product:1169455 product:1169457 product:1169460 product:1169465 product:1169467 product:1169468 product:1169469 product:1169470 product:1169471 product:1169472 product:1169473 product:1169474 product:1169475 product:1169476 product:1169477 product:1169479 product:1169480 product:1169481 product:1169482 product:1169483 product:1169484 product:1169485 product:1169486 product:1169487 product:1169488 product:1169489 product:1169490 product:1169491 product:1169492 product:1191023 product:1191032 product:1191038 product:1191070 product:1191076 product:1191081 product:1191094 product:1191100 product:1191120 product:1191146 product:1191156 product:1191164 product:1191165 product:1191166 product:1191169 product:1200493 product:1200507 product:1200573 product:1200612 product:1200613 product:1200615 product:1200616 product:1200617 product:1200619 product:1200621 product:1200622 product:1200623 product:1200624 product:1200625 product:1200626 product:1200630 product:1200632 product:1200633 product:1200634 product:1200639 product:1200646 product:1200647 product:1200649 product:1200650 product:1200651 product:1200654 product:1200656 product:1200658 product:1200660 product:1200662 product:1200667 product:1200668 product:1200670 product:1200671 product:1200673 product:1200674 product:1200676 product:1200677 product:1200678 product:1200679 product:1200680 product:1200681 product:1200682 product:1200686 product:1200687 product:1200689 product:1200690 product:1200691 product:1200692 product:1200693 product:1200694 product:1200695 product:1200696 product:1200697 product:1200813 product:1200818 product:1200820 product:1200822 product:1200824 product:1200826 product:1200830 product:1200832 product:1200833 product:1200839 product:1200867 product:1200869 product:1200870 product:1200873 product:1200877 product:1200880 product:1200897 product:1200904 product:1200914 product:1200918 product:1200928 product:1200930 product:1200935 product:1200943 product:1200951 product:1200952 product:1200954 product:1200955 product:1200956 product:1200970 product:1200972 product:1200978 product:1200979 product:1200991 product:1201004 product:1201013 product:1201017 product:1201019 product:1201023 product:1201026 product:1201030 product:1201031 product:1201042 product:1201050 product:1201051 product:1201055 product:1201056 product:1201058 product:1201059 product:1201065 product:1201066 product:1201067 product:1201131 product:1201135 product:1201138 product:1201139 product:1201147 product:1201150 product:1201155 product:1201158 product:1201167 product:1201177 product:1201179 product:1201181 product:1201183 product:1201188 product:1201205 product:1201207 product:1201208 product:1201211 product:1201215 product:1201219 product:1201221 product:1201223 product:1201228 product:1201234 product:1201238 product:1201240 product:1201241 product:1201242 product:1201245 product:1201246 product:1201247 product:1201288 product:1201294 product:1201301 product:1201304 product:1201306 product:1201310 product:1201322 product:1201323 product:1201324 product:1201329 product:1201331 product:1201333 product:1201334 product:1201335 product:1201340 product:1201344 product:1201349 product:1201350 product:1201352 product:1201353 product:1201354 product:1201356 product:1201358 product:1201360 product:1201365 product:1201370 product:1201375 product:1201388 product:1201389 product:1201390 product:1201391 product:1201392 product:1201393 product:1201394 product:1201395 product:1201396 product:1201397 product:1201411 product:1201412 product:1201413 product:1201416 product:1201421 product:1201424 product:1201429 product:1201438 product:1201444 product:1201446 product:1201447 product:1201448 product:1201453 product:1250700 product:1250708 product:1250711 product:1250716 product:1250720 product:1250723 product:1281772 product:1281773 product:1281774 product:1281775 product:1281776 product:1281779 product:1281780 product:1281786 product:1281787 product:1281788 product:1281789 product:1281791 product:1281792 product:1281794 product:1281795 product:1281796 product:1281797 product:1281799 product:1281800 product:1281801 product:1281802 product:1281803 product:1281804 product:1281807 product:1281808 product:1281810 product:1281812 product:1282271 product:1282272 product:1282273 product:1282274 product:1282275 product:1282276 product:1282277 product:1282278 product:1320076 product:1320093 product:1320111 product:1320113 product:1320114 product:1320115 product:1320122 product:1320131 product:1320140 product:1320144 product:1320148 product:1320153 product:1320155 product:1320159 product:1320171 product:1320174 product:1320188 product:1320192 product:1320209 product:1320267 product:1320270 product:1320278 product:1320288 product:1320413 product:1320538 product:1320550 product:1320553 product:1320554 product:1320555 product:1320556 product:1320557 product:1320558 product:1320560 product:1320563 product:1320565 product:1320577 product:1328956 product:1328958 product:1328962 product:1328963 product:1328965 product:1328966 product:1328968 product:1328969 product:1328975 product:1328980 product:1328982 product:1328990 product:1328992 product:1329020 product:1329024 product:1329025 product:1329026 product:1329045 product:1329047 product:1329049 product:1329053 product:1329054 product:1329065 product:1329067 product:1329069 product:1329070 product:1329075 product:1329077 product:1329078 product:1329085 product:1329086 product:1329087 product:1329088 product:1329089 product:1329095 product:1329096 product:1329097 product:1329098 product:1329099 product:1329100 product:1329101 product:1329102 product:1329103 product:1329131 product:1329134 product:1329135 product:1329136 product:1329139 product:1329177 product:1329195 product:1329199 product:1329204 product:1329210 product:1329215 product:1329216 product:1329227 product:1329232 product:1329238 product:1329242 product:1329243 product:1329245 product:1329247 product:1329248 product:1329251 product:1329253 product:1329257 product:1329260 product:1329264 product:1329267 product:1329273 product:1329277 product:1329302 product:1329303 product:1329305 product:1329306 product:1329307 product:1329308 product:1332429 product:1332430 product:1332431 product:1332432 product:1332433 product:1332434 product:1332435 product:1332436 product:1332437 product:1332438 product:1332439 product:1332440 product:1332441 product:1332442 product:1332443 product:1332444 product:1332445 product:1332446 product:1524497 product:1524501 product:1524510 product:1524534 product:1524538 product:1524539 product:1524542 product:1524545 product:1524677 product:1524682 product:1524683 product:1524684 product:1524685 product:1524688 product:1524689 product:1524710 product:1524725 product:1524750 product:1524752 product:1524753 product:1524756 product:1524757 product:1524769 product:1524770 product:1524771 product:1524772 product:1524775 product:1524778 product:1524780 product:1524938 product:1524939 product:1524940 product:1524941 product:1524943 product:1524944 product:1524946 product:1524947 product:1524948 product:1524950 product:1524951 product:1524953 product:1524955 product:1524957 product:1524958 product:1524960 product:1524962 product:1524963 product:1524971 product:1524972 product:1524973 product:1524975 product:1524976 product:1524977 product:1524978 product:1524979 product:1524980 product:1524984 product:1524986 product:1524987 product:1524991 product:1524998 product:1525017 product:1525020 product:1525021 product:1525024 product:1525040 product:1525041 product:1525042 product:1525043 product:1525046 product:1525047 product:1525549 product:1525552 product:1525554 product:1525555 product:1525556 product:1525558 product:1525560 product:1525562 product:1525563 product:1525564 product:1525565 product:1525567 product:1525568 product:1525569 product:1525570 product:1525571 product:1525572 product:1525573 product:1525574 product:1525575 product:1525576 product:1525578 product:1525579 product:1525580 product:1525581 product:1525582 product:1525583 product:1525679 product:1525683 product:1525686 product:1525693 product:1525699 product:1525705 product:1525712 product:1525719 product:1525722 product:1525728 product:1525736 product:1525742 product:1525745 product:1525746 product:1525747 product:1525749 product:1525750 product:1525754 product:1525755 product:1525757 product:1525758 product:1525760 product:1525762 product:1525763 product:1525765 product:1525767 product:1525769 product:1525770 product:1525771 product:1525772 product_list rendered user:61

From the IS:
> Since the basic implementation of cache tags has a runtime overhead based on the number of unique cache tags requested, in some cases it might be better to skip adding those cache tags all together, and go back to a full cache clear for them.

This Issue originated before ChecksumInvalidater, and i don't see, and doubt, that in the current situation this is still the case.

> Profile/rationalise cache tags

Also i could not find profiling in this issue.

Bringing this up with the intent that when we add more complexity and maintenance cost, we've done due diligence on the win we get. Or am i missing something?

Having this issue might not be necessary, but that quote is absolutely still relevant and correct.

Cache taqs have a cost and it always makes sense to think about whether or not they should be used.

Some very recent issues on this topic are #3335768: Manually clear cache keys from plugin managers with finite variations instead of using cache tags about removing some plugin discovery cache tags and #3334489: ChainedFastBackend invalidates all items when cache tags are invalidated on completely changing how the ChainedFastBackend deals with cache tags. Both could be considered child issues of this. Or we could close it and just deal with these issues as they appear.

One example that I think would make sense to evaluate are the ENTITY_TYPE_view cache tags, there's an argument to be made that they change so infrequently that we could just as well invalidate the rendered cache tag instead.

Thanks for elaborating.

> Cache taqs have a cost

This is the very claim that i challenge. What are they? What if the cost of avoiding cache tags is far beyond the cost of that cache tags? We can't balance pros and cons thoroughly without that answer.
We've had gazillions of requests to avoid this or that abstraction, because it has some cost, and the good decisions were made with solid profiling results.

> > Cache taqs have a cost
> This is the very claim that i challenge.

As @catch said each unique cache tag results in an extra query (if multiple per cache item, then grouped) every single time a cache item is fetched. It's not a claim, it's a fact.

And as @catch also said, nobody wants to remove cache tags entirely, it's about specific use cases and tags.

> What are they? What if the cost of avoiding cache tags is far beyond the cost of that cache tags? We can't balance pros and cons thoroughly without that answer.

That's exactly what this issue is about and the issues that I and @catch linked are specific examples for cases we are looking intoi, but you can not answer that generally.

Thanks for elaborating. We're getting closer, so please understand and/or forgive my insistance...
> each unique cache tag results in an extra query (if multiple per cache item, then grouped) every single time a cache item is fetched.

This is the one that is still missing documented evidence.

If you are referring to \Drupal\Core\Cache\DatabaseCacheTagsChecksum::getTagInvalidationCounts, there we have one query for all tags, which was the big win of the ChecksumInvalidator.

Or are you referring to a different query? Please educate me.
(I'll soon be pushing some resources into this topic too, and what i'll do and maybe join this quest depends on these insights.)

PS:
> nobody wants to remove cache tags entirely
Who opened that bottle? Not me.

> there we have one query for all tags

For all tags *per cache get call*. We call that method in \Drupal\Core\Cache\DatabaseBackend::prepareItem() for example. It's not even optimized if you do a getMultiple() but that doesn't really happen for most cases where cache tags are used anyway. We don't know ahead of time which caches with which cache tags we're going to load on a certain request.

Quoting myself from #3334489: ChainedFastBackend invalidates all items when cache tags are invalidated:

> Testing the patch on default umami installation frontend as admin with disabled render caching, I'm seeing 6 extra queries to the cachetags table (142 vs 148 in total).

So there's your answer for that scenario (umami frontpage, admin, no render cache), the cost of cache tags is 142 extra database queries in HEAD for a single request. With enabled render caching, it's likely fewer, but depends on what kind of cache hits and misses you have.

FWIW, the number might be off because disabled caches cause a lot of cache writes that invalidate the static cache in there, but still, it can be a lot.

Thanks a lot @catch and @Berdir for elaborating this. So even if we have no hardcore profiling, and it is not one DB query per tag, it's "every additional tag has some likelyhood to trigger an additional DB query" or in other words the factor from query to DB queries may be small but is greater than zero.

Thinking about ideas like preloading, subqueries, mapper services, but will first think and elaborate.

Working on #3322514: Explicitly support Relay (drop-in replacement for PhpRedis), noted in the example once again how many block config entity cache tags we have on a default installation. Similar to how we have group shortcuts into the set and only have cache tags for this, I think it might be worth looking into having a single block:$theme cache tag.

Block configs usually don't change that much, if they do it's often multiple (drag and drop reorder) and at the same time, for a use case with multiple themes, the list cache tag will invalidate all other themes (page and dynamic page cache) as well not just the one that's affected the block that you just saved.

As promised in #120, i have thought and worked on this.
I now see the reasons why imho removing cache tags from the API (rather than from the DB) is a breaking change and wrong, and we should rather do the latter.
Elaborates the reasoning in the IS.

I agree with a), removing cache tags is indeed a problem and #3335768: Manually clear cache keys from plugin managers with finite variations instead of using cache tags has one example where a clear call needs to change. It will require a release manager decision whether that change is OK or if we need to find some sort of BC.

On b) plugin discovery/managers have and always had an invalidation API that should be used instead of the underlying cache tags. I commented on #3001284: Allow plugin derivers to specify cache tags tags for their definitions that I disagree with that approach.

I also disagree on c) and the issue you created, that is not the direction we want to go, we want fewer cache tags, not more.

Another edge case is help topics (each topic is plugin) and its cache needs precise clean on code deploy to update search index for new topics