Caching HTML reliably currently has several inter-related issues. This affects both caching stuff inline with the cache APIand *SI requests. The latter have their own special issues on top.

Adding stuff to HTML head

This counts for CSS/JS/page title/link tags, whatever.

Inline + *SI caching:
#1839338: [meta] Remove drupal_set_*() drupal_add_*() in favour of #attached/response for out-of-band stuff
#1871596: Create a PartialResponse class to encapsulate html-level data [Moving to sandbox]
#1762204: Introduce Assetic compatibility layer for core's internal handling of assets

There's also an issue if any of this is done in an *SI callback since we currently rely on PHP to collect that information globally after rendering the rest of the page has happened (i.e. <head> is built based on information in <body>). This will mean that if there's a controller for those callbacks in core (i.e. #1830854: [meta] The ESI pipeline battle plan) it'll need to be able to present that information in a format that's accessible to JavaScript to dynamically add those tags etc.

For assets, at DrupalCon Munich and BADCamp we discussed rendering them as a json object with the block HTML, and then have a script loader in the parent that finds these, compares against global assests already in the page (which is in Drupal.settings now anyway), and loads them asynchronously. This bit doesn't need to be done in core, as long as aggregation itself is pluggable and the information is provided cleanly.

Some of this is discussed in #1014086: Consider using imagecache style generation of CSS and JS aggregates however that's also a straight performance issue (i/o on shared filesystems, aynchronous asset generation) that's not strictly related to ESI.

Arbitrarily breaking caching of HTML by adding current user-specific markup

Modules that hook into rendering then do arbitrary things based on the global user mean that the HTML can only be cached per user. However the pipeline for rendering that HTML doesn't know it's per-user usually. #914382: Contextual links incompatible with render cache is a current example that's also broken in Drupal 7 quite badly.

When inline editing went in, the final patch made it so that user/role-specific information happens in JavaScript post-processing, leaving the main HTML content cacheable the same way it was before.

#1874664: Introduce toolbar level "Edit" mode that shows all contextual links has further exacerbated the original bug.

- #1991684: Node history markers (comment & node "new" indicator, "x new comments" links) forces render caching to be per user

Creating markup that's specific to the current request

All links generated by l() attempt to determine whether the link is "active" based on the current request which means that any markup containing rendered links can't be cached on anything but a per-page basis #1979468: ".active" from linkGenerator(), l() and theme_links() forces an upper limit of per-page caching for all content containing links.

Not actually trying to cache rendered output in core

EntityNG, the new router system, menu links as entities and various other things have slowed down entity rendering and menu block generation. Two issues are open to improve caching of these. There isn't a dependency for these on the above issues (which are mostly enforcing best practices that core should already be following).

#1605290: Enable entity render caching with cache tag support
#1805054: Cache localized, access filtered, URL resolved, (and rendered?) menu trees

Comments

Issue summary:View changes

Updated issue summary.

Issue summary:View changes

Added per-user crap.

Issue summary:View changes

Added more stuff

Title:Meta: support *SI callbacks properly[meta] support render caching properly
Priority:Normal» Major

Updated issue summary and title.

Issue tags:+caching

Issue tags:-caching+D8 cacheability

There was issue to cache menu-trees, suppose it needs added to summary

It's the last issue in the summary, or do you mean a different one?

Yes, this one... sorry missed. Also coming here from #1848648: [meta] Convert menu system to new routing system
So currently related #1971384: [META] Convert page callbacks to controllers

Issue summary:View changes

Updated issue summary

Some (rather raw) notes from the Portland discussion on this topic: https://docs.google.com/a/acquia.com/document/d/1sgxD0l2OM91EqTekXryhdaX...

Component:base system» cache system

Moved to cache system per #2050763-16: Refine "base system" component (notes on refactoring of "base system" category here: https://docs.google.com/a/acquia.com/spreadsheet/ccc?key=0AusehVccVSq2dF...).

Issue summary:View changes

Added an issue

Issue summary:View changes

Updated issue summary.