Add explicit "default" scope for drupal_add_js() which defaults to "footer" so that pages render fast

None of Drupal's behaviors do that.

That's not strictly true. Take collapse.js as an example.

This will require too many changes to the D7 API to be done properly.

Essentially, the default should indeed be 'footer'. Setting the defer attribute was quite pointless 2 years ago, I'm not sure how that has evolved. I'm quite sure it's still not properly supported by all browsers. In any case, it doesn't allow for sufficient granularity in the first place.

To elaborate a bit more:

Put JS files at the bottom by default. However, when at least one JS file is added to the header, both jquery.js and drupal.js should be added to the header as well. JS files that alter the appearance of the site heavily (such as carousels) should always be added to the header, and guidelines for this should be written, and should be very clear. This is for rule 6.

source: http://wimleers.com/blog/battle-plan-drupal-7

So, the basic needs:
- JS in footer by default
- API to mark JS dependencies, so that if a file is marked as being required in the header, the JS it depends on is loaded in the header (and before it) as well. An obvious example is the requiring of jquery.js.
- Strict guidelines on when it is necessary to override from the default of 'footer' to 'header'. Two cases that can't be "footered". One: JS that provides absolutely essential styling that cannot be added later during the page load without annoying the user. E.g. a carousel.Two: JS that provides absolutely essential behaviors, that can't be loaded later during the page load because the user may want to and be able to use it immediately, while the page is still loading. E.g. an autocomplete on a search page (Google's search suggestions are a prime example).

And if we want to go crazy about optimizing Drupal's page loading performance:
- It'd be nice if we'd no longer have to rely on jQuery.extend to set the Drupal settings, or have the ability to move the Drupal settings also to the footer when no JS is required initially. This allows for the ability to make pages load blazingly fast (i.e. the time until the user sees useful content, not until the page is 100% done loading).
- A JS API to load additional JS files when it is needed. This is related to #561858: [Tests added] Fix drupal_add_js() and drupal_add_css() to work for AJAX requests too by adding lazy-load to AJAX framework, but not the same. That issue is about loading additional JS files when performing an AJAX callback to the server, which may return HTML that requires additional JS to be loaded. This is about when JS decides that it's time to load additional JS. E.g. the JS for a complex text editor, map editor, form element/widget could be loaded just-in-time.
- A JS API *and* a Drupal API that gives the ability to preloading JS files that are likely to be needed in the future, so that they're in the browser cache. E.g. preload that text editor JS. This is not limited to JS, it can just as well be applied to CSS, images, fonts …

I'm probably forgetting some things because it's so late and I've been programming C++/Qt all day.
I talked to John Resig about *the* best technique to prevent flicker some time ago. I'll look that up as well.

It should be clear that the full range of enhancements is out of scope for D7. It is fully doable though to make the simple API change for D7 that makes 'footer' the default. When modules find this is a problem, they only have to change a single line of code. That is a perfectly acceptable change at this point of time IMO.
However, I have the feeling I'm forgetting about something important, but then again I can't think of what that could be.

<script defer> is not supported by Opera, but Opera 10.5 does support the new HTML5 <script async> attribute.

<script async> is supported by Firefox 3.6. Note that Firefox have/has had some issues with defer ([1], [2], [3]). This includes a change to DOMContentLoaded being fired before deferred scripts have been loading - this may affect how behaviors work.

Neither <script defer> nor <script async> is supported by Webkit.

• Using async would be optimal, but that is only supported by few browsers, and D7 does not use an HTML5 DTD.
• We could use defer. The page load in IE and Firefox would be optimized, while status quo is preserved in Opera, Webkit and others. We would need to find out whether the bugs related to defer+DOMContentLoaded in Firefox affects behaviors.
• Adding scripts to the footer would work across all browsers. The question is whether it actually improves performance. When inserted into the header, Firefox stops the rendering while the script is downloaded, but it does seem to download the script in parallel with other resources (images etc.) references further down in the HTML page. Moving the script to the footer would likely decrease the time until the page rendering starts and finishes but possibly increase the time until behaviors are attached.

To summarize: If we go with defer, we reduce the time until rendering starts for users of IE5.5+ and Firefox 3.5+ and preserve status quo for others, while the footer approach will reduce the time until rendering starts for everybody but possibly increase the time until behaviors are attached. Do you agree with this analysis?

Putting scripts at the bottom can be a good idea. But, this needs to happen on a case by case basis. It cannot be done for everything.

For example, if a script takes the html and alters it, like vertical tabs does, you want that to be in the header. If that script is moved to the footer the html will render as fieldsets for just a moment before the JS loads and changes it to vertical tabs. If the JS is in the header you won't see the page layout change.

My measurements, on the drupal.org frontpage, indicates that the javascript and CSS files (in the header) finish downloading way before the main page (on Webkit: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_4; en-US) AppleWebKit/534.5 (KHTML, like Gecko) Chrome/6.0.489.0 Safari/534.5). In this case, moving the javascript files to the footer is perfectly pointless.

There is a tonne of evidence and statistics showing how dramatically page performance improves when scripts are loaded at the end of an HTML document. Steve Souders' High Performance Web Sites blog is a good and up to date source for such data.

Steve is working on the cutting edge of web performance optimization and is a respected leader in the front-end performance community as well as in multiple W3C working groups, where he contributes to standards working groups to help the standards become more performant and implementors (browser vendors) make more performant browsers. His recent post on the Evolution of Script Loading summarizes the best techniques to use today to get the best out of front end web browser performance with modern browsers (and without breaking older browsers). His biggest piece of advice is "Move Scripts to the Bottom". He goes into more detail on this technique in "Render first. JS second.".

I have not measured loading and rendering times for the javascript-heavy admin UI of Drupal 7 (though we should) but I expect that it is mostly impacted by javascript loading (40-90%). (Of course the ratio also depends heavily on the network layer as well as the server, but I assume in my estimates that both are performing normally.) In other words, I think we could cut page load times for overlays and other admin UIs down to as little as 10-60) of their current load times. These numbers are not based on any tests with Drupal, but front end performance improvements this radical are often and easily achieved just by moving scripts to the end of the document.

Further, more and more research is showing that page-load time directly impacts the usage of, and the perception of the quality of a website or service. Or in the case of Drupal's admin UIs, the Drupal software itself. I.e. How much people like using Drupal. Steve Souders' post on WPO – Web Performance Optimization summarizes the research well and links to the actual reports by Google, Yahoo Bing, Netflix, Mozilla and Shopzilla. Google search reveals more.

On a number of projects I have worked on, it has been a challenge simply move <?php print $scripts; ?> to the bottom of page.tpl.php because some scripts are written with an assumption that they will execute before the browser starts parsing anything after the opening <body> tag. There are some cases where scripts actually need to be in the <head>, but in my experience these cases are rare, and the scripts are simply not written correctly or do no fully support progressive enhancement. Often they were easily re-factored to function performantly.

Writing or re-writing scripts in this way also naturally encourages two other good web-performance practices;

1. Progressive enhancement; The page should look and work just fine without the scripts. That applies to the case where javascript is disabled, and to the case where the scripts are still being loaded.
2. Separation of code that must execute on page-load from code that can be lazily-loaded to run in response to events after page load. This requires a little more thought, but is not at all difficult. Steve Souders writes about this technique here, though it is more in reference to Control JS, a new project he has just started that is an API for loading scripts in a performance-optimized manner.

If Drupal makes the default scope for drupal_add_js() 'footer' there will be several major improvements for Drupal core and contrib;

1. Pages will render faster! :)
2. Users will like using Drupal more. (And become frustrated and impatient less often.)
3. Scripts will become cleaner, more performant and better support progressive enhancement.
4. Drupal implementors will not need to manually move <?php print $scripts; ?> to the bottom of html.tpl.php just to make pages render fast.
5. Drupal implementors will not need to repeatedly fix scripts that do not execute correctly in the footer, just to make pages render fast.

If we leave the moving of scripts to the bottom, to be done "on a case by case basis" or as a "per module/feature", and thus do not make 'footer' the default scope;

• Drupal will be perceived to be not as good as other systems that do perform well.
• Making sites perform well will continue to be really difficult and require a lot of duplicate effort.
• Scripts will continue to be written poorly, and authors will have the right to ignore the fact that their script only works in the header scope.

Responses to previous comments

• defer="" and async="" are useless until they are implemented equivalently in webkit, gecko/firefox and IE (and opera ideally). Steve Souders points this out, though I can not find that article right now.

• Wim Leer's comment #4 is an excellent guide for how to implement this. The starting point (make 'footer' the default scope for drupal_add_js()) is easy. The script dependency API is a harder, but can probably be a separate issue/feature/patch.

  The dependency API should not assume any script will require Drupal.settings, misc/drupal.js or misc/jquery.js. It should require that modules explicitly declare a dependency on any of these.

  Additionally, it would be nice to be able to declare which property of Drupal.settings is the dependency, since contrib modules tend to bloat Drupal.settings, and breaking out parts of it into the top (if required through a dependency) or the bottom will result in still further performance enhancements.

• Strict guidelines on when it is necessary to override...

  If we are to avoid all module authors simply adding array('scope' => 'header') to their calls to drupal_add_js(), we will need to be strict about this and provide very clear documentation and guidelines that is easy to find. It should also help script authors understand how to write their scripts for progressive enhancement.

• Two cases that can't be "footered". One: JS that provides absolutely essential styling that cannot be added later during the page load without annoying the user. E.g. a carousel.

  Why can't the style be provided with CSS for a carousel?

  Two: JS that provides absolutely essential behaviors, that can't be loaded later during the page load because the user may want to and be able to use it immediately, while the page is still loading. E.g. an autocomplete on a search page (Google's search suggestions are a prime example).

  I disagree that this would be a candidate for 'header' scope (assuming that 'footer' were the default). The autocomplete feature is not required for the page to render and on most Drupal sites the main purpose of a page is not the search form.

  To support progressive enhancement, the search form should function without any javascript at all. I expect most uses of search forms occur after page load anyway, except in exceptional cases of Users Who Always Search and a few sites like Google.com and other search engines.

• if a script takes the html and alters it, like vertical tabs does, you want that to be in the header. If that script is moved to the footer the html will render as fieldsets for just a moment before the JS loads and changes it to vertical tabs. If the JS is in the header you won't see the page layout change.

  Vertical Tabs is a difficult one to fix without major page flicker. I have some untested ideas to maintain the same layout for vertical tabs in both JS and no-JS modes;

  • Show and hide drawers on :hover. This will require refactoring Vertical Tabs default no-JS markup so that drawers are child elements of tabs. It can be refactored back to a list of tabs followed by a list of drawers by jQuery when JS is enabled. I think this is roughly how Drupal 7's no-JS Vertical Tabs work already (fieldsets are transformed to a list of tabs and a list of drawers), but am not certain.
  • Make tab-links point to a page with a different default vertical tab, e.g. /my/path?vertical_tab=2. This will cause data to be lost when the vertical tab is part of a form. We could fix that if the tab were a form button. But that would require we make the tab name the button, and that the summary description is not clickable, since the summary can not be styled if it is inside a button label.

  These types of page-flicker issues should be addressed after the initial change to make 'footer' the default scope, so that we can see how severe they actually are. I suspect modern browsers may cause page-flicker to be quite infrequent, by pre-fetching scripts before they need to be executed and by caching them more intelligently.

• I'm not convinced it ever makes a sensible difference for the user.

  Please read the articles I linked to.

• My measurements, on the drupal.org frontpage, indicates that the javascript and CSS files (in the header) finish downloading way before the main page

  That is probably because the the browser stopped loading the page while it loaded and parsed the javascript, which is all the more reason to move them to the footer. I refer you again to Steve Souder's blog where he explains page-blocking quite well. Note also the point "when the page become usable" relative to all of those events.

  Also, we need to compare measurements, not just take them. We also need to note when the page become usable, as well as when the document.ready event fires, and when resources have finished loading.

• Moving JS to the footer is about optimization not developing solutions. Drupal defaulting to the footer could fall into the category of premature optimization.

  I disagree. Loading scripts in the footer and footer only is becoming the convention. Any high-budget website that has spent time or money on WPO does this or would like to.

I firmly believe this is a critical feature for Drupal. Clearly it is too late for Drupal 7, but if we get this in early in the Drupal 8 cycle there will be time to re-factor scripts as necessary in both core and contrib, as well as address the worst page-flicker issues.

Thanks Matt! A BoF is a great idea. I hope that we can move on this earlier than that though, since D8 development will probably be underway by then. For the record, I do not think that "move a lot of scripts to the footer" will achieve much. Because most of our scripts are dependent on jquery.js and/or drupal.js, the bulk of the executable script and network-transfer will still be in the head. Lazy loading can drastically reduce that initial overhead, but that is an awfully large undertaking and a separate issue to this one.

Subscribe

I spoke to Dries (in person) about this and he suggested we should also consider this for Drupal 7. It needs a lot of manual testing because we still do not have a unit testing framework in place for javascript, so I am doubtful that it would be wise to commit this to D7. However I am interested to hear others' thoughts.

We need to keep jquery.js on the top as some nodes might use inline JavaScript (ugly, but possible). Anything in JS_SYSTEM on top? We should also consider moving Drupal.settings to the bottom of the page too since most of that stuff is run in behaviors anyway, which is run when $(document).ready is run.

For Drupal 7? Seems a bit late as some of contrib might expect drupal_add_js() to add things to the header. But I'm more then happy moving it :-) .

LABjs allows JS scripts to be transferred in parallel with little effort on the part of the developer, so that they can all get there faster. (Older browsers will load them one at a time. Newer browsers load them in parallel anyway, but still maintain execution order and block page rendering in order to support expectations of many scripts and websites which are poorly optimized on this point.)

This issue is about moving the scripts to the footer so they do not block page rendering.

Both techniques can yield front end performance techniques, but they are different techniques each with their own problems and advantages. They are however quite related, especially #1033392: Script loader support in core (LABjs etc.).

IMO LabJS is more likely to have more problems than this technique, because it is more complex and does not respect execution order by default. It would be good to continue development and testing of both techniques.

I have re-attached the patch to see if I can not-break the testbot. ;)

We need to keep jquery.js, drupal.js, and - potentially - also all libraries using weight JS_LIBRARY at the top (unless a specific weight has been assigned).

That would be at the cost of most speed improvements, and thus defeats the purpose of this enhancement.

The last submitted patch, js_footer.patch, failed testing.

#1140356: Add async, onload property to script tags
#1033392: Script loader support in core (LABjs etc.)

anyone still having strong feelings about this one?

With a script loader, it's better to load scripts in header to take advantages of parallel loading. I think we should postpone this issue when we have not decided whether there will be a script loader in core.

If we don't have a script loader, we can still load *all* scripts in footer. When I tried to put $scripts (D6) in footer, the only problem that I had is inline scripts that calls Drupal or Drupal.settings. This problem was remedied with something like this patch.

Benchmarks will be favored for scripts at the bottom. However, the question is: when we have a script loader, will this patch be reverted? It's likely.

I'll make a benchmark later today.

Disclaimer: I'm the author of LABjs module.

Here are the results. Three testcases for D7 (as I have something that is ready for D7):

• Standard: D7 standard profile, anonymous with most of permissions, frontpage of 10 nodes and 4 of them have an image. 8 scripts and an inline script block.
• Bottom: same as standard, but I modify modules/system/html.tpl.php to move $script to the bottom.
• LABjs: same as standard, with LABjs 7.x-1.3.

Tested with webpagetest.org, with Chrome and IE8 (only with IE8 there is the filmstrip feature). I tested a few times with each testcases (webpagetest also for each request, it tests several times and take the median result).

There are 3 images Chrome timing, IE8 filmstrip, IE8 waterfall, each for 3 testcases (standard, bottom, LABjs).

Chome:

IE8 filmstrip:

IE8 waterfall:

Trimming some redundant tags.

Here's a re-roll from the patch in #22.

It still needs the tests to be updated and maintenance-page.tpl.php needs a way to print $page_bottom to include scripts with the footer scope.

Issue summary still mentions D7. Needs an update.

That's a really nice idea. I'm not comfortable putting footer as the default so that'd be a cool way to let people choose for themselves.

I agree, for the most part.

I disagree that it should be configurable to make the 'default' scope "footer". If a developer wants to do this through alters or other code, that's fine, but not through any sort of standard configuration UI. Drupal should Just Work—Fast—Out of the box with no configuration, hacking nor bugs. This issue is about making the scripts run after the page renders so that Drupal can render fast. This implies that poorly written scripts may need to be fixed.

Agreed. I don't want to offer this as a configuration option in the core UI, but the additional 'default' scope would allow a contrib project to offer this functionality much more cleanly.

Tagging.

I still think this makes sense. Any chance to move forward here? Looks like we're bound to feature freeze with this.

While this is performance-related, and performance-related stuff generally seems to be post-feature-freeze material, the concrete change proposal involves to add an entirely new notion of how developers should specify the JS they want to load, so I'd personally not bet on that.

That said, any idea how this maps to and will play with the foreseen removal and replacement of drupal_add_js() with the everything-is-a-library design?

Will this even be needed with that?

@vrazz
If you want to do this in D7, checkout AdvAgg. It can do it for you.