Because #2158771: CSS hardening was addressed and you're looking for feedback, I figured I'd spend 30 minutes on my Saturday morning to test it and provide that feedback. Unfortunately, I could not get Navbar to work because of the Modernizr/Underscore/Backbone library detection that was seemingly completely broken.

That happens for various reasons:

  1. Libraries API can only detect versions on the non-minified file. So if you want to use the minified Backbone/Underscore/Modernizr, you must either download both the unminified (backbone.js) and the minified (backbone-min.js), or only download the minified and rename it to backbone.js. Otherwise Libraries API will fail.
  2. The instructions don't explain *at all* how to use the minified libraries. They assume you want poor front-end performance.
  3. At the same time, the instructions implicitly assumed you *did* download the minified libraries and then renamed them as if they weren't minified. If you did download the non-minified libraries as per the instructions, then the version detection wouldn't work!!!
  4. Incorrectly, though understandably, navbar_libraries_info() pretended it supported a "minified" variant, but it could never have worked, because it did not implement the required "variant callback". If no variant callback is enabled, then Libraries API … just assumes that the variant is installed (!!! WTF !!!). This makes sense when downloading packages (zip/tar.gz/…) that contain both the minified and unminified files, but this does not reflect reality. So, in fact, you should always implement the variant callback, but Navbar did not yet do this.
  5. To deal with this variant craziness, Navbar had lots of logic to deal with checking if the minified version actually existed or not (instead of letting Libraries API do this), leading to more code complexity and $variants = variable_get('navbar_libraries_variants', array()); which was introduced recently. If we use Libraries API correctly (even though that's ridiculously complex), then we can just ask Libraries API whether the minified variant exists, and if so, we just use that. In other words: gone is that navbar_libaries_variants variable, instead you just remove (or rename) the minified JS file if you want to use the non-minified JS. Much simpler.

So, I wanted to fix this once and for all (and then Edit module could use the same techniques). Unfortunately, the tests are so artificial that they don't really test real-world use cases, the docs are so theoretical that they're useless and all of the example modules listed at https://drupal.org/node/1570568 fall in either of two categories: A) they stopped using Libraries API (gee, I wonder why), B) they use Libraries API in the most simple way possible (that is often also incorrect): they don't support a minified "variant".
That's why I ended up doing deep debugging of the Libraries API module and reading its "docs" and "tests" in-depth.

I'm probably forgetting about a significant percentage of frustration and misery, but I'm hopeful that the collective pain endured so far will have led us to this patch that will hopefully allow us to erased this part of our memories, to be happily forgotten, forever.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Wim Leers’s picture

Status: Active » Needs review
Issue tags: +Spark, +sprint
FileSize
10.74 KB
jbeckers’s picture

Have not yet tried the patch, but two things come to mind (probably follow-ups though):

  1. Since a special version of modernizr is needed, it probably doesn't make sense for it to be the site-wide version. Can we just name it modernizr-navbar so other modules can add other special builds? Or would that still clash when multiple versions of modernizr are loaded at the same time? I guess it would, as the docs suggest: https://drupal.org/node/1342220#libraries-version
  2. Does Libraries API have a way to disable the minified version without removing it from sites/all/libraries/xyz/ ?
jbeckers’s picture

Patch seems to work perfectly.

Wim Leers’s picture

#2:

  1. I've thought of that also. The solution is simpler though: the few sites that have multiple modules that need Modernizr should just create a build that contains the superset of all required parts of Modernizr. Note that it is okay to use the full Modernizr build, but then you're loading many unneeded KiBs of data. I think the README should mention that though :)
  2. Surely you can do that through one of the many hooks that Libraries API offers. Plus, as I said, if you want to use the unminified version, you can also just temporarily delete the minified version.
Wim Leers’s picture

The part I didn't mention yet about this patch: it ensures that things *also* work when you download the minified version and rename it as if it wasn't minified. Even though that's not entirely correct, the patch tries to make the site admin's life as easy as possible.

jbeckers’s picture

One thing I noticed: with this patch the minified file should be named modernizr-min.js, while the README says modernizr.min.js

Wim Leers’s picture

#6: good catch. The README should be updated. The file already had to be named "modernizr-min.js" before this patch.

jessebeach’s picture

+++ b/navbar.module
@@ -924,38 +916,33 @@ function navbar_libraries_info() {
+    'variants' => array(
+      'minified' => array(
+        'files' => array(
+          'js' => array(
+            'modernizr.min.js',
           ),
         ),
+        'variant callback' => '_navbar_libraries_minified_variant_exists',
+        'variant arguments' => array('modernizr-min.js')
       ),
     ),

Two different file names. I'll change the variant check to modernizr.min.js

jessebeach’s picture

+++ b/navbar.module
@@ -924,38 +916,33 @@ function navbar_libraries_info() {
+    'versions' => array(
+      // Means ">=2.6.2": matches 2.6.2, 2.7.1, etc.
+      '2.6.2' => array(),
+    ),

Ok, so by declaring 'versions', we get a minimum version number for Libraries API to check against.

jessebeach’s picture

Alright, this is finally the implementation that I'm happy with. Wim, I hope you're cool with it as well.

All files are now declared in a named variant under a version. We can do this because of the variant existence checking that Wim added in the previous patch.

A library is determined to be installed if at least one of its version's variants is installed. Previously, a library was assumed to be installed if it had a version that matched the physical file's version even if eventually nothing got loaded.

The variant loading order is determined by the variant order property on the library (custom to navbar). This property can be overridden by another module using hook_libraries_info_alter. So no where now does the code impose the minified version; the variants and their preferred selection order are determined by configuration of the library info. Another module could conceivably add a variant to the library info of Navbar if needed for the installed library version.

Also, the status report now lists what variant will be loaded.

Navbar: Backbone library      1.0.0 (minified)
Navbar: Modernizr library      2.7.0 (source)
Navbar: Underscore library      1.5.0 (minified)
Wim Leers’s picture

Hehe, nice cleverness in there :) It's also impressive how many hoops we have to jump through to get reasonable behavior that most people expect.


I have only one nitpick:

+++ b/navbar.module
@@ -1090,6 +1170,27 @@ function navbar_convert_libraries_to_library($library, $options = array()) {
+  return '';

I'd return FALSE or NULL here.


… and one significant concern. AFAICT, the current patch pretends as if it is possible to install only the minified version of a library (e.g. backbone-min.js). But that's not really the case, because the version check will still always check the unminified file. In other words: you must still have the unminified file available, otherwise Libraries API will say the library is missing, because no version could be detected, because the unminified file does not exist.

tstoeckler’s picture

FileSize
8.84 KB
17.12 KB

As @Wim Leers already mentioned the version detection is currently quite fragile. This fixes that which also allows to remove the post-detect callback. (In other words the logic in the post-detect callback is moved into the version callback.)

I also added a @todo to _navbar_convert_libraries_to_library(). That function is really unfortunate and Libraries API should do that for you.

The major conceptual problem with this is that you are trying to enforce that if the 'minified' variant is available that it be used. There is no reason to do that, however. If people want to load the 'source' variant for some reason, they should totally be able to do that. One thing that Libraries API *should* do however, is to let libraries provide some default. To put this into code:

// Both of these are perfectly valid and should work in any case.
// (Although, of course, only one of them per page.)
libraries_load('backbone', 'source');
libraries_load('backbone', 'minified');

// However, Libraries API should be smart and load the minified variant
// (in case it's available) when I do the following:
libraries_load('backbone');

This might mean moving the 'variant order' that you introduced into Libraries API proper and then checking that in libraries_load(). Not sure yet, but I will open an issue to discuss that.

tstoeckler’s picture

tstoeckler’s picture

Oh, I forgot to say: I did not actually test that #12 works, so statistically it's probably broken.

Also, the one thing I don't understand about the patch is the version specification in hook_libraries_info(). Why is that done?

jessebeach’s picture

re #11, I opened an issue with a patch to make the error messages for version detection more explicit: #2167249: Make version/variant callbacks provide error messages themselves for better DX/UX.

tstoeckler, I'm psyched we're collaborating on this. I really want the Spark modules to have THE definitive implementation of Libraries.

tstoeckler’s picture

Yes, I saw that. Haven't commented there yet, but it's a great idea. Thanks for that! That would also help the version callback I introduced here.

jessebeach’s picture

tstoekler, I don't quite understand this comment from your patch:

// If different versions have been found, there is no way to
// determine the correct one. We cannot use the information on the preferred
// variants because we cannot guarantee that a less preferred variant will not
// be loaded.

Does this mean that a variant label such as minified is declared under two numbered version arrays? e.g.

'versions' => array(
  // Means ">=1.0.0": matches 1.0.0, 1.1.0, etc.
  '1.0.0' => array(
    'variants' => array(
      'minified' => array()     
    ),
  ),
  '2.0.0' => array(
    'variants' => array(
      'minified' => array()
    ),
  ),
),

Would this result in no file being loaded?

jessebeach’s picture

As @Wim Leers already mentioned the version detection is currently quite fragile. This fixes that which also allows to remove the post-detect callback. (In other words the logic in the post-detect callback is moved into the version callback.)

Given the current code in libraries_detect, I don't think we can move the variant callback logic to the version callback.

For one, the callback in invoked around line 527 of libraries.module, but the variants are not moved to a top-level property in the $libraries variable until around line 558, when the versions property is processed, thus the version checking fails here:

function _navbar_libraries_get_version(array $library, array $options) {
  $versions = array();
  foreach ($library['variant order'] as $variant_name) {
    debug($library['variants'], '', TRUE);  <-- THIS IS UNDEFINED.
    $variant = $library['variants'][$variant_name];

    // @see libraries_detect()
    if (!isset($variant['variant callback'])) {
      $variant['installed'] = TRUE;
    }

The version callback is meant to return a version for the library, a string.

So, I thought about this some more and made the attached changes. Now, version callback compares the variant versions. If the versions are the same or NULL (meaning the version couldn't be determined), but not all NULL (meaning no version could be determined), then we return the unique single version string.

We unfortunately lose some error information here because we get one error message (maybe return an array instead?). So I've tried to return at least some useful info with additional error messages. I tested all the scenarios.

Wim Leers’s picture

Status: Needs review » Needs work

So, what is left to be done here? I think just some thorough testing to find any remaining problems?

Well, I've unfortunately still found some problems.

  • When you have only modernizr.js (no minified variant), then the Status Report won't complain, but Navbar will try to load the minified one, even though it does not exist.
  • OTOH, if you have only modernizr.min.js, then the Status Report will fail (and importantly, Navbar will also not load modernizr.min.js, even though it exists!). This is why I said that Libraries API assumes you always have the non-minified variant. I don't think it's possible to work around that assumption. The same applies to having only backbone-min.js or underscore-min.js.
Wim Leers’s picture

Here's the sister issue for Edit module: whatever we do here, I must apply to Edit module as well — issue created: [#2169507.

jessebeach’s picture

Status: Needs work » Needs review
FileSize
345.84 KB
329.87 KB

re: #19, to your first point, I'm not seeing this behavior with the latest Navbar code. If I change the name of the minified Modernizr file from modernizr-min.js to zmodernizr-min.js (note the prepended 'z' character), then the source file is loaded. Note that the name of the minified file should be modernizr-min.js, not modernizr.min.js so that the minified file name matches the Backbone and Underscore minified file name pattern.

And the opposite case where the modernizr.js file is renamed and made unfindable.

I think it's working about as well as we can get it to for the moment. Can you retest and see if you replicate the behavior I'm seeing? If you can, I vote we go with the code in Navbar as the pattern for the Spark module Libraries implementation.

jessebeach’s picture

Wim Leers’s picture

Status: Needs review » Reviewed & tested by the community

#21: ah, you changed the naming scheme! After testing this again, I can confirm that this works flawlessly!

AWESOME WORK!

tstoeckler’s picture

Sorry for not following up earlier! #18 is certainly an improvement over the status quo so I won't un-RTBC, but I still think we can do better.

I at first didn't understand what you meant in #18 because I skipped over the code snippet. I just re-read that and now I get where you're coming from. Thanks for providing that snippet, that was helpful.

The problem is that the actual library (variant) file information is currently "hidden away" in the 'versions' property. Per definition this property can only be accessed once the version is known. However, since #12 it contains the version detection information for the variants. This is paradox, in short: You need to know the version to figure out the version. Instead the variant version detection information should be moved into the top-level 'variants' property at which point #12 starts working.

As I already mentioned above, but maybe did not explain well enough: I don't understand the reason why the 'versions' property is used at all. It is used when e.g. libraries move around files in newer versions and thus the library definition needs to differ depending on the version. In the code comment, it says something about the 'min' version.

What I think is happening, now that I think about it, is that the 'navbar' code depends on a certain minimum library version in order to work. Is that correct? In that case, what should really happen is that the navbar declares a *versioned* dependency on the navbar library. But since modules can't currently declare dependencies on libraries at all, I think it would be fine to leave in the 'versions' property for now.

The paragraph from above is still correct, though: The version-detection information should move up. I will provide an updated patch soon, but it might be tomorrow or Wednesday. (Obviously) Feel free to commit this, for now, though.

jessebeach’s picture

Status: Reviewed & tested by the community » Fixed
Issue tags: -sprint

Commited to 7.x-1.x e2e2a44d

This change is incorporated in 7.x-1.3

Wim Leers’s picture

Wim Leers’s picture

Status: Fixed » Active
FileSize
766 bytes

When using *only* the minified Underscore or Backbone, version detection fails, because the patterns are wrong.

Wim Leers’s picture

Status: Active » Needs review

The last submitted patch, 10: navbar_libraries_api_revamp-2167021-10.patch, failed testing.

The last submitted patch, 18: navbar_libraries_api_revamp-2167021-18.patch, failed testing.

The last submitted patch, 12: 2167021-12.patch, failed testing.

The last submitted patch, 1: navbar_libraries_api_revamp-2167021-1.patch, failed testing.

jessebeach’s picture

Status: Needs review » Fixed

Thanks Wim! Pushed to 7.x-1.x (503dae83). I'm about to cut 7.x-1.4.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.