Jump to:
| Project: | Drupal core |
| Version: | 8.x-dev |
| Component: | base system |
| Category: | task |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
| Issue tags: | frontend performance, JavaScript, Needs profiling, Performance |
Issue Summary
Questions to resolove, see #55:
- How to aggregate all the js/css files used in the current page the answer belongs #1048316: CSS and Javascript aggregation improvement - meta issue (use this or that module as basis to put in core, etc.)
- How to load said scripts and CSS into the page so that they can be used on the front end, i've started a little something here #1423500: Use RequireJS to load all JS,
- And the most important one for last: How are we going to use theses script?
---
Drupal 7 improved the CSS and JavaScript aggregation methods so that we don't get dozens of different large aggregates for different pages.
However, this also means more js files loaded on each page, which has it's own issues, since browsers can only load and execute one javascript file at a time.
However, there is lovely, lovely http://labjs.com/ which allows for parallel loading and execution of js, just seen a contrib project for this http://drupal.org/project/labjs. - LABjs is being used on twitter and examiner.com amongst others.
It's MIT licensed so we'd have to speak to the author about dual licensing. I haven't reviewed the contrib project yet to see how well it works, however it's the sort of thing which could likely benefit from being built into core drupal_add_js()/#attached
Comments
#1
Subscribe. That could be great. Currently the labjs module must replace misc/drupal.js with its own version, thus I'd love to see it in core.
#2
Subscribing.
#3
subscribe
#4
sub+
there also have
ControlJS
http://stevesouders.com/controljs/
HeadJS
http://headjs.com/
#5
Very nice...
#6
Subscribing.
Also see http://blog.getify.com/2010/12/on-script-loaders/
#7
Related; #784626: Add explicit "default" scope for drupal_add_js() which defaults to "footer" so that pages render fast
#8
If LABjs is in core, we won't need #784626: Add explicit "default" scope for drupal_add_js() which defaults to "footer" so that pages render fast any more. It will be equivalent whether a script is in header or in footer. It will be equivalent to put drupal.js, jquery.js... in footer. The only problem are scripts with document.write(). Even in that case, the temporary solution that I use in Google Ad Manager shows that it works perfectly with lazy loading mode.
#9
Subscribe... #1071368: Breaks Add This and Administration menu; integration with labjs
I'll be looking at various js loaders and seeing what ones I can put into AdvAgg. This is a 6.x module as there are a lot of sites out there still at 6.x
#10
subscribe
#11
No patch here yet.
#12
subscribe
#13
#1048316: CSS and Javascript aggregation improvement - meta issue and #352951: Make JS & CSS Preprocessing Pluggable might be the first steps. Seems like this should be swappable before making it into Drupal core.
#14
http://yepnopejs.com/ looks promissing too (via John Resign - http://twitter.com/#!/jeresig/status/45150930932482048)
#15
Re-titling.
#16
Not sure which Script Loader to use? http://t.co/JpwIqN0
Dion posted the comparision on Twitter. Instead of settling with one for Drupal core, we have to make this pluggable.
#17
I think we should include 1 in core as a module. It will force all JavaScript added to be compatible with JS loaders as a result. And like Rob Loach said, this should be pluggable.
Not sure how many care, but I've made advagg fairly pluggable now. You can select a different function to render your script tags; what that means is js files added by drupal_add_js could be included in the JSON response and loaded on the client side. Same for css... actually js/css use almost the same codepath. Checkout readme.txt for a high level overview.
#18
I agree that this should be pluggable. However, the required features should not be rich so that it can be pluggable. Being maintainer of labjs.module for a while, I think these features are required:
- Parallel loading (of course)
- Reserved execution order
- As much compatible as possible
Other features are not necessary, or we'll stick with one loader, like we're sticking with jQuery. Well, even in that case, it could be good. But I just don't see why we need other features, supposing that this module/subsystem is optional, they are just unnecessary and introduce more bugs.
#19
+1
#20
+1 I like head.js
#21
Subscribing. Marking #1149770: Consider using RequireJS for loading JS files needed after an AJAX request a duplicate. Adding http://requirejs.org/ as a candidate for us to consider.
#22
#23
Some great ideas here - the getify blog post and spreadsheet are both interesting resources. Identifying what the API for a pluggable system might look like is going to be interesting.
One thing that has not been mentioned is that sufficiently small script loaders could (perhaps optionally) be placed inline on the page, eliminating an additional (high latency to data ratio) http request.
#24
One thing that just occurred to me, is that next generation JS aggregation would greatly benefit from the possibility to load (and cache) JS without executing it. This allows us to include commonly needed JS in a page where it is not needed (aggregated with other JS that is needed), without needing to execute it, preventing a likely additional http request on a subsequent page view. Drupal has several JS files that could benefit from this approach - they are tiny and make no measurable impact on the aggregate size (but a very measurable http latency impact), yet we are unable to aggregate them because doing so also executes them. Looking at the comparison grid (line 29) it looks like ControlJS, YepNope.js, curl.js, PINF JS Loader, BravoJS support this. Of these curl.js looks particularly interesting, although I haven't gone through the implications of using this in a Drupal context (using AMD seems like a benefit, but I think may imply a lot of work).
Another implication of using a loader with (very likely) parallel behavior is that to get the most benefit we _may_ want to split the aggregate file up into a small number of roughly equally sized sub-aggregates (perhaps ~3...would need research) to maximize throughput. This was discussed some in another issue, but I can't find it now...
#25
@Owen Barton
Advagg's bundler uses a default value of 4 aggregates; in firefox 4 the max is 6 on my box. Because of the ability to change the function that writes out the script tags, advagg should allow any script loader to be used in a 3rd party module.
Once 6.x-1.0 is out; I'll work on a 7.x version of advagg. Once 7.x is stable I would like take the lessons learned and bring the important parts into core for D8. Here is an example of a lesson learned #1148204: Setting JSON values to objects/callbacks via drupal_add_js extension (or similar)
#26
Subscribing.
#27
Mark.
#28
subscribe
#29
Nice. I think a client side loader could be an ace little modernisation/optimisation of the Drupal front end. Certainly needs some careful thought though.
As for the question of which loader we use… might be worth noting #865536: drupal_add_js() is missing the 'browsers' option. The current patches assume bringing feature parity with drupal_add_css to be the best solution: that is, IE conditional comments. Conditional comments are massively limited, and it would be a whole heap better if there were some kind of clever client side conditional loading built into drupal_add_js.
With that in mind, this is a massive +1 for yep-nope, or if we go the pluggable route, that we include conditional loading in our api. Yep-nope does all the asynchronous preloading with decoupled (and thus ordered) execution, resource fallbacks and whatnot like the other libraries. I believe it was originally based off labjs, but was re-written with a really simple interface for making the loading of certain resources conditional.
It would open the door to replacing the icky/limited browsers option that's currently headed for drupal_add_js with a simple client side test option. Simple tests could just be included inline (like checking for HTML5 geolocation support with '!!navigator.geolocation;') or contrib code could even rely on external tests (such as 'Modernizr.geolocation'). Yep-nope even includes IE conditions built in as 'prefix' options... so we could still use drupal_add_js for 'just IE' if we wanted to (similarly to the 'browers' option issue).
I don't think it's much bigger than any of the others (I think it might be slimmer than labjs, at 1.6kb gzipped?), and I've heard no complaints in terms of speed, but that conditional loading functionality could be really useful. And by useful, I'm talking about improving Drupal as a framework here — I'm currently not aware of any core functionality that 'needs' it. Having said that, if the overlay were being written for D8/HTML5 it could've been a bit clever with using native hashchange events and only loading a polyfill where needed (though I fear the polyfill may be somewhat engrained in the BBQ plugin we used, so can't say how easy that would've been!). My point is core use-cases in D8 might come up; we'd have to see.
But for contrib, this would give modules that want to use, say, geolocation or web sockets a way to do so elegantly (providing fallbacks only as required, whilst not having to implement their own client side testing and script loading solution); at seemingly no additional cost to core over any of the other script loaders. There are some really cool things contrib could do with these new technologies; it'd be cool if the 'framework for innovation' was already there in core.
Another good reason for yep-nope is that it's wtfpl licensed, so we're good to use it as is, adapt it to fit more neatly into Drupal or whatever; we don't even need to worry about attribution other than as a courtesy.
I should also note, there is an issue in the html5 queue #1252178: Add Modernizr to core which expands the scope of this issue to bundling a feature test library such as Modernizr along with a script loader that might leverage those tests (yep-nope can come bundled with Modernizr). I'm yet to be convinced about bundling feature tests — but a script loader in core would certainly mean that feature test libraries such as Modernizr could live happily in contrib, should it not be deemed core-worthy.
Any thoughts? I may be entirely wrong :o)
#30
Here from the modernizr thread -- from what I've read, yepnope is less robust than some of the other asynchronous/polyfill js loaders, but if it is good enough for the majority of use cases I think that its bundling/compatibility with modernizr should be taken into account. Whether or not modernizr is included in core, it is shaping up as a (very) widely used tool in front end dev and is likely to be included and included often on the module or theme level.
Planning for that integration would be a plus IMHO.
#31
Subscribing.
#32
I'm going to break this down into a few thoughts and some recommendations.
Thoughts:
Recommendations:
#33
Totally in agreement, mfer. In fact, I recently signed on to co-maintain Modernizr, and have a couple fun projects in the works that will give it the love it needs. The community hasn't had the opportunity to generate real use patterns yet, which needs to happen before we decide on a use case for all of Drupal.
#34
I have posted some additional thoughts over in the Modernizr issue queue - #1288248: [Meta] Develop a Modernizr API for other Drupal modules
#35
+1
#36
Let's at least postpone this on #352951: Make JS & CSS Preprocessing Pluggable.
There's so many options it'd be good to see things solidify and what goes on with async support too so I agree with mfer we may not want this in core after all (although we should look at the hacks that LABjs module had to do to drupal.js and similar to make it simpler).
#37
sub.
#38
...coming from #865536: drupal_add_js() is missing the 'browsers' option
#39
Sub
#40
As with LabJS, there is already a Drupal module (6 and 7) for HeadJS:
http://drupal.org/project/headjs
#41
Here's a very detailed comparison of over 20 script loader libraries, including those named in this thread for reference:
https://docs.google.com/spreadsheet/ccc?key=0Aqln2akPWiMIdERkY3J2OXdOUVJ...
#42
nod_ brought this up on IRC, and I think we definitely need to push this forward. Having our JavaScript libraries load asynchronously will make our page loading/rendering performance extremely fast. RequireJS is another option. Whichever library we end up using will be infinitely better than what we bake ourselves.
#43
Yes. We need this for proper static analysis, and thus to allow for more efficient, fine-grained performance enhancements: it will enable smart bundling/aggregating of JS files.
At Facebook, they are currently overhauling all JavaScript to use CommonJS in favor of their own internal JS dependency/packaging system (which, in a nutshell, boiled down to a
@requires foo barcomment at the top of the file to make it dependent on thefooandbarpackages).#44
Sounds good!
We should run some benchmarks and feature/stability comparisons to decide which one we should gun for and just do it!
I've had tons of problems implementing HeadJS and keeping it playing nice with contribs. Having this in core would definitely make things awesome! :)
#45
In regards to CommonJS, is this the async Loader Code? https://github.com/amdjs/amdjs-api/wiki/AMD Will it also load CSS async? If not we should look as being able to load JS and CSS async; gives us a lot more options for speed improvements when things like Embedded CSS are taken into account.
#46
Let me start by saying I'm asking questions to drive out a solution good for a number of use cases. For example, we have the case of a site that someone interacts with and spends a bit of time on the site. Then we have the case of a publishing platform where someone visits a single page or a few pages of the same type. In the former cases you may have a number of cases different scripts used on different pages. In the latter case you have one set of scripts sent to a user. Optimizing for each of these cases is different.
An issue with performance (especially on mobile networks and devices – which will overtake desktop usage by the time D8 is popular) has to do with the number of files included, fetching them, and dealing with that. Sure, there will be improvements from what we have now but some of what we have now will still be used.
So, if we async individually load all the JS files on a page we will open up a bunch of network connections. The best browsers open up 6 concurrent connections to a single domain at a time (across all tabs and windows). This is 3 times the HTTP 1.1 spec. On each of these connections the current tcp spec has 3 packets transmitted before waiting for a response (though Google is working to change this). If we move from loading 2 or 3 aggregated JS files to a bunch (I've seen 15 or 20 on some sites) of individual files being loaded async the gains we get from async will be more than lost in additional network effects. This will really hit the site that has someone just reading a couple news articles or blog posts that have the same JS included.
So, when we talk about and architect a solution lets think about multiple use cases and the different parts involved in it.
Some other (random?) pieces of information to include in this conversation:
This being said, I'm all for some good change here. I like CommonJS and RequireJS could be nicely used. If we go with something like RequireJS we should look at incorporating the optimization technique it can use for aggregation into our aggregate setup. Since we are not going to have a dependency on node/JS/Java on the server side we would need a PHP port of the aggregation (for now without the minification).
Thoughts? (@Wim, I'd appreciate anything you have for direction here as you are more up on this stuff than me.)
#47
I'm no expert in JS optimization, and definitely not in the JS "asynchronous modules" realm. I do know we can and should do better. Depending on what my job will entail, I may be able to work on this.
This answer probably belongs more in #1048316: CSS and Javascript aggregation improvement - meta issue, but that issue has been silent for over 6 months.
Less bytes
To give an extreme example: on wimleers.com (which is on D7), I don't want *any* JS, except for Google Analytics. I don't need any JS, hence I don't want it. Yet, by using the Google Analytics module, all of drupal.js and all default jQuery stuff is loaded. Almost 100 KB, or ±65 KB gzipped. More than all HTML/CSS/images combined.
On the front page: "37.2% of CSS (estimated 24.5kB of 65.8kB) is not used".
My point is: we really need to move forward on this. Aggregation reduces the number of requests, which is good. But loading less data, and thus also needing less requests, is better. Of course, this is easier said than done in the case of Drupal, which must also work out-of-the-box, without deployment scripts and all that jazz.
Plugabble
Different use cases will alway require different solutions. Plus, this will keep evolving as browsers, webservers, standards and WPO techniques evolve. The single most important thing for Drupal 8 is to have all CSS/JS stuff pluggable, so that we can keep evolving in core, and not force outdated techniques upon Drupal users.
#48
I made a separate issue for the actual loading part #1423500: Use RequireJS to load all JS.
There is still lots to talk about here beside loading, let's get to it :)
#49
@mfer
AdvAgg's Bundler takes care of some of the concerns about aggregate size. AdvAgg also has hooks that make removing of JS files a fairly painless process.
AdvAgg makes most parts of it replaceable via variable function calls.
#50
I think cleaning up AdvAgg to make it core-worthy would be an interesting start. We'd definitely learn a lot from it, collectively, instead of just mikeytown2 :)
#51
Agreed, AdvAgg is amazing and standard on all our Drupal 6 sites and we're anxiously waiting on a D7 version! :)
#52
As for minification, I'm working on that. Before DrupalCon I'll have something there and if my core conversation gets picked I'll be talking about it there. At the very least I'll be talking about that in my session Front End Performance Improvements. JS minification for Drupal has some GPL legal stuff surrounding it making it not as simple as just doing it.
#53
Although this discussion is mainly client-side, I just wanted to point out that Assetic is probably the direction we want to go for handling all this.
#54
A lot of great discussion here, and I look forward to seeing and perhaps participating in some of the cool work that will come out of this discussion. In this comment, I just want to respond to Wim's #47:
With hook_js_alter(), you can remove the JS files you don't want. Is there something about that that's not working for you, or are you just pointing out that D8 core should support this better without requiring an alter hook for fixing bad assumptions?
In D7, how the CSS is grouped, aggregated, and markup for it generated is pseudo-pluggable, since all of that is controlled via the rendering of a 'styles' render element, whose definition can be altered in hook_element_info_alter(). In D8, the JS is now also using the same pattern, with a 'scripts' render element. So, in the above statement, are you asking for something different than what we already have, or simply to retain pluggability as we add new features, like an async script loader?
#55
I'd like to recenter a bit, there are three sides to this issue:
First two issues will have patches but we should get "uses-cases" out of this one, like Wim explained about his website. There need to be a "decision framework" which will allow choices to be made for #1 and #2 or this could go off track.
You can see that for #2 it's actually pretty easy to have "script loader support in core" the hard part is knowing what to do with it. And I making the case that js shouldn't be managed in PHP that much, for example drupal_add_library() should go away and be managed as JS dependencies of AMD modules.
All in all, like effulgentsia said, D8 should support this better without having to use hooks :)
#56
@mfer
Here's my frontend performance presentation I gave in PDX: http://pnwdrupalsummit.org/sessions/front-end-performance (checkout the slides & notes in the slides). AdvAgg uses JSMin+ which is released under the GPL; allows for scripts to move from the header to the footer or to any other region of a theme; Aggregates are multi-process, each one is built in the background (code for this created HTTPRL).
@Rob Loach
assetic looks pretty cool! Thanks for the heads up, didn't know about it.
#57
Regarding 3):
Whatever script loader is being proposed, it needs to be able to cope with Drupal's #ajax framework, which means to dynamically lazy-load files and settings when needed, based on the previously existing state that performs a request, and even in the situation where all JS is aggregated into optimized groups, and even more so in the situation where individual files shall be cached.
Aside from that, I don't see a proposal à la "ditch PHP hooks/registries for JS" to fly for various reasons - it's essential for the Drupal backend to know which files it delivers, and also, that they can be altered, adjusted, or amended by modules. This information ties deeply into version control (hence, caching), but also contextual ESI injections.
#58
"it needs to be able to cope with Drupal's #ajax framework", that's not much of a requirement, everything is done by
ajax.js, so any loader able to load this file would work really :) the issue with requirejs has a working patch for it, it's NW only because I haven't fixed the tests I broke with my patch yet, otherwise it's working fine, try it out and break it so it can evolve towards a better solution :)Concerning #3 I blamed the wrong function, I meant to get rid of
drupal_get_library(), my apologies. I totally agree with you thatdrupal_add_library()is necessary, I mean it's from there that we can have paths to specific js/css files, so any loader would need this. I got a bit distracted while writing and got confused between the two of them.I don't mean to get rid of hooks for js files, I'd just like to find a way where it's not necessary to use them to do simple tasks like not loading default drupal js files/settings. It's still fuzzy even for me, the only thing I'm pretty sure of is that js needs to be treated differently than PHP so the usual "Drupal way" is not necessarily the right one. I just want to make sure we have a go at another different approach more fitted to js.
#59
AdvAgg has a way to export the JS and CSS files that are loaded on the page: advagg_get_js_css_get_array(). Patch for QuickTabs is where the code came from #1172010-6: Quicktabs, AdvAgg - Loaded via AJAX do not pull advagg bundles which is my attempt at the #ajax issue.
#60
Also see #1279226: jQuery and Drupal JavaScript libraries and settings are output even when no JS is added to the page for why we need a JS dependency system.
#54: see #1279226: jQuery and Drupal JavaScript libraries and settings are output even when no JS is added to the page: this should work correctly out-of-the-box, without any altering. JS that's not used at all should not be loaded, period.
#62
Agree with Wim.
Despite jQuery being included in Drupal core and being essential to many administration aspects, it is not always used in the front-end (edge case IMO).
drupal_add_js() is also "dumb" in that it doesn't take dependencies into account or prevent the same script being added multiple times.
The fact that JS is alterable in D7/8 is a huge step forwards but we would really take this implementation further and if not add a script loader in core (which one is another debate), at least implement some form of dependency system to make implementing custom script loaders less of an uphill battle.
#63
#1140356: Add async, onload property to script tags
#784626: Add explicit "default" scope for drupal_add_js() which defaults to "footer" so that pages render fast
We only need one direction, we should choose. I vote for this one.
#64
#1423500: Use RequireJS to load all JS gets the benefit of the package-based dependency tree.
#65
@nod_ and @Rob Loach, with the new RequireJS thread, can this be marked as a duplicate yet?
#66
not really, I made another issue on purpose. Read up #55 to see what kind of things we have to agree on that have nothing to do in the RequireJS issue.
#67
Trimming some redundant tags.
#68
With #1737148: Explicitly declare all JS dependencies, don't use drupal_add_js having solved the "we don't know about JS dependencies" issue, as well as #1279226: jQuery and Drupal JavaScript libraries and settings are output even when no JS is added to the page, we are now able to continue this issue.
#69
#70
Just a heads up. AdvAgg uses render arrays for JS (like CSS). It has these 3 patches included in some form #865536: drupal_add_js() is missing the 'browsers' option, #1140356: Add async, onload property to script tags, & #1664602: drupal_add_js() defer and async options break when aggregation is enabled (Replace with universal attributes array).
Long story short we can sorta testout the solution in D7. As an example, this D8 only module works if AdvAgg is installed (and the info file is hacked) http://drupal.org/project/async_script_shim