[meta] Various asset (JavaScript) libraries have to be updated to a (minified) stable release prior to 8.0.0

I can speak to CKEditor: we're working with the CKEditor developers to upgrade to the CKEditor 4.3 stable over at #2039163: Update CKEditor library to 4.4.

Also: is this really a beta blocker?

Definitely not a beta blocker, but should be a release blocker (and try to upgrade some things around the first beta so it's less of a jump later if possible).

jQuery 2.1.0 is out.

http://blog.jquery.com/2014/01/24/jquery-1-11-and-2-1-released/

That or getting #1663622: Change directory structure for JavaScript files going (it's about using bower for doing this).

is there any issue|policy about to ship core with full or min version of library?

#1341792: [meta] Ship minified versions of external JavaScript libraries

Updated script + latest results.

Library                   Current        Latest
------------------------- -------------- --------------
backbone                  1.1.0          1.1.2
classList                 master         -
ckeditor                  4.4.0          4.4.3
domready                  1.0.4          1.0.5
html5shiv                 3.6.2          3.7.2
jquery                    2.1.0          2.1.1
jquery.cookie             1.4.0          1.4.1
jquery.farbtastic         1.2            2.0.0-alpha.1
jquery.form               3.5            3.50
jquery.joyride            2.0.3          2.1.0
jquery.once               1.2.3          1.2.6
jquery.ui                 1.10.2         1.11.0
jquery.ui.touch-punch     0.2.2          -
matchmedia                -              0.2.0
modernizr                 2.6.2          2.8.3
normalize                 3.0.1          3.0.1
picturefill               -              2.1.0
underscore                -              2.1.0

Adapting title to not mix it up again.

:)

In a similar vein: #2234277: Composer update (includes security fixes)

Not only a theme system issue.

As the maintainer of Drupal 7 in Debian, I find tarekdj's comment (#17) most useful: We have to track where do all of the sources in our packages come from, and we do not consider minified Javascript files to be a valid substitute for sources.

I'll just go a little step further, and add a wishlist request:

It is in our view very important to have the project source (in this case, the Drupal code, or at least the Git repository) distributed with full sources to all of its components. Think, for example, on cases where an upstream project (that is, an included library) dies and its website disappears. Drupal could be left with only its minified version.

We would mostly appreciate if, besides shipping the minified libraries, the source from which they were minified would also be part of the Drupal source tree or, at least, of its Git tree. Or, at least, ship with a list of URLs where to download the actual sources used. Do you think that would be possible?

@gwolf sounds useful, otoh most of sources could be found at google and others cdn. also for debug there's https://github.com/mozilla/source-map/ in Firefox 23+ and webkit

andypost: Right. Sources can be found, and what I'm describing is somewhat a theoretical situation — One that has happened, however, in several other projects. And, you might know, Debian is quite a specialist in nitpicking ;-) That's why I'm asking "just" the Drupal community to help us do this.
This is important, among other things, so we can keep track of issues in the included versions of said libraries. If a bug is found in libjs-foo 1.2.3, it's very important for us to know that Drupal includes 1.2.0 (and is vulnerable), or that Drupal includes 1.2.5 (and is OK). It will also allow us to decrease code duplication, as we will be able to use systemwide libraries instead of the ones included (as it might be the case with frameworks, such as Symfony).
Again, this is something I'll end up doing if Drupal does not, but I will deeply appreciate (and of course, am willing to step in and help) if it is accepted!

@gwolf you can get this information for assets from https://api.drupal.org/api/drupal/core!core.libraries.yml/8 and PHP libraries from composer.json - does that help?

catch: Excellent! It will really help me. Of course, I still have to do some manual work, but *lots* less than what I was expecting to.

Thanks!

On a related note: #2375997: Avoid tying Drupal 8's composer.json to specific package commits.

Also, many bower/npm packages can be managed through Composer: https://packagist.org/packages/fxp/composer-asset-plugin

Included is a patch which requires jQuery 2.1.1 through npm.

Manual Upgrade is easier than you thought. Just nobody would review & commit patches in right time and then endless reload.

npm would include a lot of more files.

@droplet: That's the point. It doesn't use npm or bower itself, just reads in their packages and installs them.

@Mile23
Does that mean we could more easily automate our update process for those libraries?

Reading https://github.com/francoispluchino/composer-asset-plugin/blob/master/Re... this seems to be the case.
I really like that, though I don't know why we can't rely on bower as well.

Bower would have extra requirements, such as installing node. Bower is actually an npm package. :-)

I experimented with this a little more and it's a little bit hit-or-miss in terms of what specific files you can count on being where. For instance, the bit above where I demo requiring jQuery gives you the whole jQuery repo, and maybe not just the bits you care about.

Worth experimentation.

Bower doesn't work, lots of library exclude minified files (I think that's even the recommendation) so we can't count on bower giving optimized production files and we're not in the business of minifying vendor scripts.

#2276785: Evaluate Bower for managing third party scripts
#1663622: Change directory structure for JavaScript files

Updated the change notice with some steps to get this resolved.

While we might run into issues from particular updates, checking the version number, opening issues and the patch to update the library itself could all be done by contributors new to core, so tagging novice.

The composer/bower discussion here is interesting and could be useful, but feels like a new, major issue to discuss trying to use something like that.

Oops :)

There, did a bunch.

Should we do #1341792: [meta] Ship minified versions of external JavaScript libraries at same time ?

@droplet yes when upgrading we should use the minified version of the file, then we can close that issue too.

Postponed #1341792: [meta] Ship minified versions of external JavaScript libraries on this since we can do them all in one go.

Tagging blocker - although we may just be able to mark that issue duplicate or downgrade to major once this is fixed here.

Put backbone and underscore in same issue.

#51: wow, neat!

Cool. Updated Summary and sent my very last patch this year :)

Added the update of #2387027: Upgrade PHPUnit to the latest stable release to the list. PHPUnit is not yet part of the core.libraries.yml file. The current version in HEAD is 4.1 and the latest stable version is 4.4. On PHPUnit is to be downloaded from https://github.com/sebastianbergmann/phpunit.

What we mean by assets is Javascript really. Everything that you'd find in core/assets/vendor, not core/vendor. PHPUnit is out of scope (also look at the issue component: "javascript").

@nod_: I have chatted with webchick about this on IRC. And she said that I should add PHPUnit to this meta-issue. If you have a better solution, please let me know.

@daffie we've been individually opening critical issues for vendor library updates. I've just opened #2400407: [meta] Ensure vendor (PHP) libraries are on latest stable release as a meta issue to track those and added the PHPUnit issue as a child there. i think it's worth keeping JavaScript vs. PHP updates separate, but not having a meta to track the PHP updates seems like an omission.

Removed #2387027: Upgrade PHPUnit to the latest stable release from this list. Thanks catch for creating a new issue for the PHP-libraries.

I have seen many libraries have been switched to minified versions within last 14 days. Please see #2400287: Remove all occurences of sourceMappingURL and sourceURL when JS files are aggregated for an issue we need to fix before release or we will see many 404 errors.

Please join #2400675: Missing .map files causing 404 file not found errors to decide if we remove sourceMappingURL from minified JS files in asset folder or if we add the missing .MAP files to core.

Note to the folks filing sub-issues:

I talked this issue over with the other branch maintainers the other day. Normally, we would indeed make any issues that are hard blockers to solving critical issues critical as well. However, in the case of meta issues like this where every child issue is essentially a copy/paste of the other ones, it's actually polluting the list of criticals quite a bit, which has a number of down sides: it makes it hard to tell how we're progressing against release; the volume "drowns out" other issues that are actually critical blockers; it also makes it impossible to determine whether an issue involves just routine library updates or if upgrading a library actually is critical (for example, if it solves a security issue).

So please go ahead and just file these children as "normal" unless there's something about the library upgrade that would meet the standard issue priority definitions. Rest assured, we're all constantly looking at the RTBC queue for these library update issues because a) we know they help resolve this critical issue and b) they are normally pretty easy to sign-off on and commit. (If for some reason one of them sits at RTBC for longer than a couple of days feel free to ping one of us about it in #drupal-contribute.)

CKEditor 4.4.7 is out. Issue to update it: #2415111: Update CKEditor library to 4.4.7.

jquery.farbtastic 1.2 2.0.0-alpha.1

Stable version is : 1.3u ( https://github.com/mattfarina/farbtastic/tree/1.3u )

The underscore.js version number was in the backbone.js row.

jquery.ui 1.11.3 released Feb 12th.

Outdated* as of 3/20/2015
underscore 1.8.2 (current: 1.7.0)
picturefill 2.3.0-beta (current 2.2.0)
jquery.once 2.0.0 (current 2.0.0-beta3)
jquery.ui 1.11.4 (current: 1.11.2)

*as seen on the table in the initial post

Updated underscore, picturefill, jquery-once, jquery-ui versions.

Updated the 'last checked' date to today for every library.

Updated the instructions as per https://www.drupal.org/node/2203431#comment-9526163 so that newly created issues aren't marked as Critical

Updated underscore version 1.8.2 => 1.8.3 and Date checked for all libraries

Just a little summary of open JS updates:

#2462259: Update underscore to 1.8.3
Needs review, testing

#2427649: Update to jQuery UI 1.11.4
Needs patch, review, testing

#2393713: Update JS lib: jquery.form to 3.5.1
Has been tested but needs further review after using different minify method

#2462261: Update picturefill to 2.3.0
RTBC

https://www.drupal.org/node/2427649
Has patch. Needs review, manual testing.

Picturefill now updated to 2.3.0.

Library                   Current        Latest        
------------------------- -------------- --------------
backbone                  1.1.2          1.1.2         
classList                 2014-12-13     2014-12-13    
ckeditor                  4.4.7          4.4.7         
domready                  1.0.7          1.0.7         
html5shiv                 3.7.2          3.7.2         
jquery                    2.1.3          2.1.3         
jquery.cookie             1.4.1          1.4.1         
jquery.farbtastic         1.2            2.0.0-alpha.1 
jquery.form               3.50           3.51          
jquery.joyride            2.1.0          2.1.0         
jquery.once               2.0.0          2.0.0         
jquery.ui                 1.11.4         1.11.4        
jquery.ui.touch-punch     0.2.3          -             
matchmedia                0.2.0          0.2.0         
modernizr                 2.8.3          2.8.3         
normalize                 3.0.2          3.0.3         
picturefill               2.3.0          2.3.1         
underscore                1.8.3          1.8.3        

All external libraries currently in core have been updated to minified versions: #1341792: [meta] Ship minified versions of external JavaScript libraries

We now need to make sure that any additional external library updates we do retain the minified versions. Updating title accordingly.

...and make sure they all have source maps :)

+1 to #91: we must not forget to ensure sourcemaps are present and kept in sync!

Can we put something in the issue summary that links to how to provide a source map? I'm not familiar with how to do that.

source maps are cool ...

http://blog.teamtreehouse.com/introduction-source-maps

We need "how to" instructions for source maps so folks making patches do the right thing.

Issue referenced in #45 is now fixed, removing the blocker tag.

This is now captured as part of #2485119: [meta] The Drupal 8.0.0-rc1 Release Checklist. Downgrading.

Updates are available for domready, jquery, picturefill, jquery.once and jquery.farbtastic

@MustangGB: Thank you! :)

Gave jquery.once a new issue instead of re-using the existing one.

Making manual checks for each library.

Thanks @tohesi, you can use this script to check the updates :)

Thanks @droplet! Created new issues for Backbone and normalize and updated the table with current core and latest releases. Wasn't sure about farbtastic 1.3u with the u(nofficial) suffix sounding all so suspicious.

Updating / adding tasks for updated assets. Also fixed a minor hiccup on the version check script.

Library                   Current        Latest        
------------------------- -------------- --------------
backbone                  1.1.2          1.2.1         
classList                 2014-12-13     2014-12-13    
ckeditor                  4.4.7          4.5.0-beta    
domready                  1.0.8          1.0.8         
html5shiv                 3.7.2          3.7.2         
jquery                    2.1.4          2.1.4         
jquery.cookie             1.4.1          1.4.1         
jquery.farbtastic         1.2            2.0.0-alpha.1 
jquery.form               3.51           3.51          
jquery.joyride            2.1.0          2.1.0         
jquery.once               2.0.1          2.0.2         
jquery.ui                 1.11.4         1.11.4        
jquery.ui.touch-punch     0.2.3          -             
matchmedia                0.2.0          0.2.0         
modernizr                 2.8.3          2.8.3         
normalize                 3.0.3          3.0.3         
picturefill               2.3.1          3.0.0-alpha1  
underscore                1.8.3          1.8.3 

Updated/added issues for backbone and jquery.once. Updated summary.

Forgot to update the table with the new jquery.once issue reference.

Updated CKEditor, Backbone.js and html5shiv current versions.

Updated the table.

Note: jquery.ui.touch-punch states the version number 0.2.3 inside the js file only (https://github.com/furf/jquery-ui-touch-punch/blob/master/jquery.ui.touc...)

Isn't the ckeditor-dev development repository supposed to be replaced with a production release https://github.com/ckeditor/ckeditor-releases in addition to latest version number?

Digging into #2400407: [meta] Ensure vendor (PHP) libraries are on latest stable release uncovered all manner of things that still need doing, so I can only assume this one will too. Escalating to critical, since it blocks RC1, and tagging as an actionable critical.

nah we were diligent and updating regularly. The messy updates are in the past.

Library                   Current        Latest
------------------------- -------------- --------------
backbone                  1.2.1          1.2.3
classList                 2014-12-13     2014-12-13
ckeditor                  4.4.7          4.5.3
domready                  1.0.8          1.0.8
html5shiv                 3.7.2          3.7.3
jquery                    2.1.4          3.0.0-alpha1
jquery.cookie             1.4.1          1.4.1
jquery.farbtastic         1.2            2.0.0-alpha.1
jquery.form               3.51           3.51
jquery.joyride            2.1.0          2.1.0
jquery.once               2.0.2          2.1.1
jquery.ui                 1.11.4         1.11.4
jquery.ui.touch-punch     0.2.3          -
matchmedia                0.2.0          0.2.0
modernizr                 2.8.3          2.8.3
normalize                 3.0.3          3.0.3
picturefill               2.3.1          3.0.0-beta1
underscore                1.8.3          1.8.3

Updated the proposed resolution to include information about priority of child issues, copied from #2400407: [meta] Ensure vendor (PHP) libraries are on latest stable release.

jQuery 3.0.0-alpha1
picturefill 3.0.0-beta1

Does anyone know what the timeline of jQuery 3.0.0 looks like? That alpha has been out since mid-July: http://blog.jquery.com/2015/07/13/jquery-3-0-and-jquery-compat-3-0-alpha...

Conversely does anyone know the support cycle for the 2.x branch once 3.0.0 is out?

My gut is that we should try to get onto the 3.0.x branch for RC if we can to avoid trying to do a major version upgrade in a minor, but releasing with on an alpha version would not be great either. If we opened an issue for that, we could see if there's any problems.

Same questions for picturefill.

ckeditor and its blocker were already bumped to critical.

farbtastic is abandoned, and #2268955: Deprecate farbtastic library is open. It'd be nice to be able to mark a js library as @internal.

https://github.com/scottjehl/picturefill/issues/492 for picturefill, which can done later as well, no need to rush.

I believe we are done here. Setting to RTBC.

Oops. Silly me, there is still #2521820: Update CKEditor library to 4.5.3. However, that is independently critical.

#2521820: Update CKEditor library to 4.5.3 is in, this one is toast. :) GREAT work, all!!!

#2505649: Update jquery.once to 2.1.1, #2559299: Update JS lib: backbone to 1.2.3 and #2521820: Update CKEditor library to 4.5.3 have all landed, IS updated.

https://github.com/scottjehl/picturefill is now on 3.0.1 :(

https://github.com/modernizr/modernizr is on 3.1.0

There is an issue for #2568387: Update JS lib: Modernizr to 3.1

CKEditor has a patch release: #2581291: Update CKEditor library to 4.5.4.

#130,
#2577895: Update JS library picturefill to 3.0.1

Library                   Current        Latest        
------------------------- -------------- --------------
backbone                  1.2.3          1.2.3         
classList                 2014-12-13     2014-12-13    
ckeditor                  4.5.4          4.5.5         
domready                  1.0.8          1.0.8         
html5shiv                 3.7.3          3.7.3         
jquery                    2.1.4          3.0.0-alpha1  
jquery.cookie             1.4.1          1.4.1         
jquery.farbtastic         1.2            2.0.0-alpha.1 
jquery.form               3.51           3.51          
jquery.joyride            2.1.0          2.1.0         
jquery.once               2.1.1          2.1.1         
jquery.ui                 1.11.4         1.11.4        
jquery.ui.touch-punch     0.2.3          -             
matchmedia                0.2.0          0.2.0         
modernizr                 3.1.0          3.2.0         
normalize                 3.0.3          3.0.3         
picturefill               3.0.1          3.0.1         
underscore                1.8.3          1.8.3         

#2614682: Update JS lib: Modernizr to 3.3.1
#2321583: Update CKEditor library to 4.5.5