Ensure it is possible to use AJAX with GET requests

Thanks for the work on this. Subscribing, and intend to review next week.

Note that while #561858: [Tests added] Fix drupal_add_js() and drupal_add_css() to work for AJAX requests too by adding lazy-load to AJAX framework introduced the ajax_page_state variables, it was #384992: drupal_html_id does not work correctly in AJAX context with multiple forms on page that originally introduced the ajax_html_ids variables, and that alone prevents using a GET request for AJAX. Also note that ajax.js never supported GET requests out of the box. But all that aside, I agree there's value to reducing the payload both for performance, and to enable contrib to change some AJAX requests from POST to GET if it wants to.

Sub - without HTTP GET AJAX no reverse proxy caching for ajax pagers.

Updated patch fixes HTML ID test.

Some ideas I've had in terms of keeping track of what is inside the aggregated css/js files
http://groups.drupal.org/node/53973#comment-279159

I like this feature, however I am not sure if it qualifies as a critical as it may be able to go in a post 7.0 release of core.

I think you're right. I'm just not 100% positive.

To re-iterate from #7:

But, I do think it is critical that a contrib solution be possible. I suspect it already is, between hook_ajax_render_alter(), the ability for JS functions to be overridden, drupal_html_id() using a drupal_static() that can be hacked into from another function, and PHP allowing $_POST to be written to even if it's a GET request.

...

Once we're confident that such a contrib module is possible without hacking core, I think ctools would be the right home for the implementation, and for this issue to move to.

In other words, my inclination is to move this issue to the ctools queue, and I'll probably do so in a few days barring further comments. I'm leaving it in the core critical queue for now, just to make sure people have a chance to flag any simple core API change that may be required if my assumption above is wrong.

Switch drupal_html_id() from using integer increments to a randomized short hash. Makes collision between requests extremely unlikely and drops the need for ajax_html_ids in the request.
...
Generate a page state css hash and js hash for any given array of css/js files. Store it in the cache table. Transfer this in ajax_page_state instead of the full list of CSS/JS files [...] Upon receiving an AJAX request, use cached list of css/js files looked up by page state hashes. Eliminates the need for client to send the list of css/js files back to Drupal.

I understand and basically agree with the outlined problem. I especially agree with the idea that the AJAX framework currently prevents to implement true REST callbacks, and also kills proxy/reverse-proxy caching, since every request is a HTTP POST.

However, both solution proposals make the system more fragile:

1. Only the client knows about the actual HTML IDs on the page.

   We still have lots of theme functions and templates that output arbitrary IDs, sadly. Also, third-party/non-Drupal scripts may inject further HTML IDs that are unknown to Drupal. Lastly, even though it's less likely, a random hash can lead to duplicate IDs.

2. Relying on an additional page state cache means that we'd need another, non-volatile cache like {cache_form}.

   Right now, the entire system has a single defined state only. The actual client/browser holds and defines that state. Drupal only reacts accordingly to that state, which means that we purposively do not try to synchronize the state information between client and server. Exactly that intentionally missing synchronization is what makes this part of the current AJAX framework rock solid.

   When discussing this problem space, then we also need to take into account that the AJAX framework doesn't provide means to remove loaded files currently. While I couldn't believe it myself, I've heard various rumors that there are some contrib projects for D6, which implement a third-party library that actually supports full unloading of JS and CSS files (it's the JS part that baffled me). Given that (not verified) info, I suspect that a contrib module could relatively easily add an AJAX framework command to unload JS/CSS files from the page. Due to the current design, the AJAX framework would simply send those removed files for "reloading" if they happen to be needed in a later request again.

Food for thought.

If all fails, then we should merely make sure that a contributed module is able to replace resp. plug in to the affected functionality without having to hack core.

All theming by HTML ID was already completely and permanently broken by the rework of drupal_html_id() a very long time ago. So that's not a happy thing, but it did happen like 8 months ago, so any theme that uses ID is already up the creek without a paddle.

In Drupal (IMO), CSS cannot target IDs at all, as they are unique (and can be volatile). At least IDs that are generated by drupal_html_id(). So every time AJAX generates some new code it comes in with a new ID. If CSS is targeting IDs, #fail.

IDs are the fastest way to select elements in JavaScript, so IDs are still helpful for operations like AJAX. It's true Drupal's auto-generated IDs should not be used for theming - for that sake I already discussed with eff some time ago that we should actually make drupal_html_id() generate much shorter and completely random machine IDs, because the current human-readable output sends the wrong signal and makes people believe it is safe to rely on them.

We already changed lots of CSS and also JS code in D7 to no longer target IDs, but CSS classes instead - and to aid facilitating that, CSS classes have been heavily improved, too.

However, the impact of Drupal 7's HTML IDs on themers is not the point here. Regardless of auto-generated, human-readable, cryptic, or not, the AJAX framework must not generate potentially duplicate IDs. Currently, that is ensured by making the client send all known IDs to the server.

The real problem we're facing here is that AJAX always performs POST requests, which auto-disables all kind of built-in caches. In order to appropriately use GET requests, we need to find a way to decrease the HTML ID and page state information that is sent to the server in order to prevent duplicate IDs and lazy-load asset files.

Some quick notes on Sun's comments:

Only the client knows about the actual HTML IDs on the page. We still have lots of theme functions and templates that output arbitrary IDs, sadly. Also, third-party/non-Drupal scripts may inject further HTML IDs that are unknown to Drupal.

This is true but I'm not sure there is anything core can really do about this. If a contrib module chooses not to use drupal_html_id() for the initial page response it's likely it won't be used for the AJAX response, in which case whether the client sends back all the HTML IDs or not makes no difference. Basically, if you aren't using the core AJAX framework you're on your own (with or without this patch).

Relying on an additional page state cache means that we'd need another, non-volatile cache like {cache_form}.

Yes. A cache for looking up sets of JS/CSS files via hash would be equivalent/similar in many ways to {cache_form}. If we were to establish such a cache I would not be very concerned about fragility - if the site successfully sent the page then it will be able successfully respond via AJAX for that page.

It would be nice to see some discussion on this issue around the actual patch as it does let you test these assertions and behaviors against a real site : )

That's a good move for sure. *everything* needs a hook_alter :D

@webchick: it's certainly within your right to not like this, and to leave it for Dries.

But I'd like to point out that it is already the case that you shouldn't target IDs (except ones you really know to be unique on a page, like ones that are in page.tpl.php) from CSS, and precisely *because* Drupal is a framework. Even the D6 precursor to drupal_html_id(), form_clean_id(), made it so that if you have two form elements on a page, that both would have otherwise had the ID 'foo', that instead the second one is changed to 'foo-2'. It was correct that in D7, we changed the function name from form_clean_id() to drupal_html_id(), and started using it from contexts other than forms.

That we didn't finish the job and still have this in node.tpl.php:

is really a shame, but unaffected by this patch. This *should* be changed to call drupal_html_id(). But until it is, is unaffected by drupal_html_id() potentially returning a different ID than what was passed. In the meantime, what this means is that a page that renders the same node in two different places on the page will fail HTML validation.

So, we have this dilemma. CSS that targets #node-42 currently works, but a page that outputs the same node twice on a page fails validation. If we fix this to use drupal_html_id(), then we fix HTML validation, but the CSS only targets the first one and then fails to target the other ones, since they're changed to "node-42--2", "node-42--3", etc.

All this is true with or without this patch, and with or without AJAX. What AJAX introduces is the idea that "uniqueness on a page" needs to be tracked across page requests.

What the patch introduces is a way for a contrib module to implement uniqueness in some other way than the way done by core, because the way done by core, while working, comes with a performance and caching limitation that a contrib module may want to solve. A well-written contrib module could still implement a strategy by which the first time an ID is used, it is unmodified, so that CSS that targets an ID only breaks precisely in the exact same situation in which it already would have broken anyway even without that contrib module.

So yes, we're a framework: one that allows (and encourages) pieces to be combined in flexible ways. But IDs must be unique per page (blame the W3C for that one). So, the price of this is to not rely on a hard-coded ID of a piece that can appear more than once on a page.

Would more docs in hook_html_id_alter(), saying something like "WARNING: don't break CSS and JS for no good reason: consider the implications of altering Drupal's logic for outputting unique IDs!" help?

I don't understand how not committing this patch means that AJAX GET requests are impossible. I understand that this is the only part of the rendering chain that's not alterable, but the idea that this needs to happen in order to support GET requests is currently a theory, not a proven rule, since no such code exercising these alters exists. And if it did, we'd be considering it for core inclusion. Or am I mistaken?

Has to go into 8.x first.

The last submitted patch, ajax-html-id-956186-21.patch, failed testing.

Wow, working on ctools and views that ajax ids and css made me crazy too.

I think it comes from a biais of the "typical Drupal developer", everything in done in PHP. I'd like to talk about CSS files, there is already lots of good ideas about how to handle ids.

From what I understand, css/js files are sent to avoid adding them again if they are already present in the page, and only for this purpose. In ajax.inc there is

Now why is this done on the PHP side? there is already an ajaxPageState variable with what's already there, we should add a couple of ajax command that would load CSS and JS files where we can do this check and forget about posting all CSS and JS files in the request. The diff would have to be made before the commands potentially changing ajaxPageState are processed obviously, that's pretty easy to do.

Actually with a few tweaks it would even be possible to get css files from the head and contents of the aggregated css files directly with a regex targeting a specially formated comment containing the filename, i'm not for this particular solution but just to point out there is a lot more that can be done on the js side of things.

Not sure what you're talking about, in the other issue, the architectural discussion ended at #24 after merlinofchaos said xLazyLoader will not work because it doesn't look for duplicate (to which I said it's our job to worry about that) and it wouldn't work for aggregated CSS files (to which I gave a solution, a bad one, but still), but more importantly I'm not even talking about some kind of loader. The rest of the thread is focused on how to implement merlinofchaos approach best.

The thing is, it's this patch that introduced ajaxPageState, and I'm talking about using precisely that to check for dups. It's a different issue because the context here is different 1 year after. That was an interesting read but that was no architectural discussion.

I'm not saying ajaxPageState should be remove, I'm saying is should only be posted back from PHP not to PHP. Instead of fat ajax requests and responses we'll only have fat ajax responses. Which is not ideal but better and will make it possible to use GET request which is what it's all about here.
To avoid fat responses we'd have to look into aggregated files to know what's going on which is dirty but possible (and I'm not saying we should do that just yet).

The last submitted patch, core-remove-files-from-ajax-requests-956186-35.patch, failed testing.

Thanks for pointing it out, I didn't give enough attention to the linked post it seems. That makes things simpler, the html_id part of this thread which is fixable can be ported to another issue and this one closed as it works as designed. There are a few solutions to limit the impact but nothing will solve this issue properly unless the whole stack is revisited.

To get small ajax http request, you just can't use Drupal AJAX framework. It should be possible to refactor the ajax commands system to be used easily by third party but that's an other issue as well. It depends on what op wanted to do. Or am I once again wrong?

Hello @Steven Jones, have you had any progress on this? I am very much for GET ajax calls. That is the architecture of the web, GET to retrieve content (and for caching) and POST to create new content. If Drupal wants to be a player on the modern web it needs to respect the architecture of the web... So a nice skinny GET request would be wonderful!

This problem is forcing us to do a lot of custom callbacks just so we can use views + ajax + varnish.

It must be rather common, so I wonder if anyone knows of a D7 module or recipe which would allow all views to use cacheable GETs for Ajax in a simple way?

Filed separate issue about serialization of ajax_html_ids #1575060: ajax_html_ids are broken for forms with file element (encoding=multipart/form-data)
Currently this functionality broken for forms with file element