Allow modules and themes to package external libraries

I think this is a good idea. I don't have an immediate need for it but I can see where it would make sense for a lot of modules and make life easier for the user.

I'm pretty sure there's an older issue about this somewhere, but I can't find it with a brief search. Maybe it's just comments embedded in other issues about getting all this working for distros.

It wouldn't be *that* hard to do this, since so much of the plumbing is already in place. Mostly, it's a policy thing. And it seems like if this is allowed for distros, it should be allowed for modules. But, it'd be good to get more of the infra team to chime in before we start actually planning the implementation.

FWIW I would be fine with this. If anything, I think it would only help to enforce our existing policy of not checking in third-party code to Drupal.org's Git repo.

Distributions are great, but only if you're starting your site from scratch. Lots of people want to add WYSIWYG editors, slideshows, and the like to their existing Drupal sites, and this would make these modules 200x easier to install for mere mortals. +100.

And ideally, the resolution to this issue would simply be to make this packaging capability project-type agnostic. I could see a theme shipping with some kind of suckerfish menu thinger too. No reason not to let it, as long as the library's GPL-compatible. We're just saving users a step or 10, and giving module/theme authors incentive not to stick their dependent third-party code in our repos.

Sure, this doesn't have to be hard-coded to only work with modules. The packaging script could just *always* look for a drupal-org.make file. However, even still, distros are Special(tm), all the crap of the core, no-core, etc versions, etc. So, we still need the distro-specific packaging to be separate from everything else.

Also, we definitely don't want non-distro projects that have drupal-org.make files that refer to other projects. So, I think we're going to need another set of validation rules for these so that you can't use the projects key at all and can only define libraries. Although this requires more thought and perhaps discussion.

That's what I mean -- this isn't *trivial*, but thankfully a ton of existing plumbing can easily be shared/reused.

I think that it would be nice to have this.

Would it be easier to allow (no-core) distributions to be added to an existing site, like modules? Then just make the module into a single module distribution™…

@zeta: You can already add no-core distributions to an existing site. That's part of the point. But no, that doesn't help this case.

Probably the biggest technical hurdle here is that if you have a given module foo that depends on a library bar, you want to end up with something like this:

sites/all/modules/foo/foo.module
sites/all/libraries/bar/bar.inc

Currently when you open the tarball for foo, you get:

foo/
foo/foo.module
...

The tarball can't really have paths like:

../libraries/bar

So, we'd need the tarballs to instead have paths more like this:

modules/
modules/foo
modules/foo/foo.module
libraries/
libraries/bar
libraries/bar/bar.inc

And instead of all the documentation (and everyone's habit for how to work with Drupal contrib tarballs) saying this:

cd [docroot]/sites/all/modules
tar -zxvf foo-7.x-1.0.tar.gz

People would have to stop at [docroot]/sites/all and untar from there.

While it's easy enough to create tarballs like this on the packaging side, tons of things (drush pm, drush make, update manager in D7 core, etc) are written to assume that module tarballs start with foo as the root, not modules vs. libraries. So, not only would we have to retrain all the humans and fix all the docs, we'd also have to fix those systems to handle this change.

That really sucks. I'd rather avoid all that, but I'm honestly not sure how to do so. :/ The only other alternative I can think of is to just put bar somewhere as a subdirectory of foo, but that's problematic for other reasons (e.g. if other modules want to share the same library).

If anyone has an elegant solution to all this, please share it! ;)

Thanks,
-Derek

@derek

Well if we're gonna make the change might as well make it to DRUPAL_ROOT ( not necessarily use of the constant but the same place ) so later when something thats not in sites/all/* can use the same packing scripts and muscle memory and not go thought this all over again ( if that would even be feasible )

I can see use cases later for adding and install profile and such below an existing multisite install that would then be picked up by future "install" runs on the same multi-site codebase ( this would be a HUGE boon for sites like cartoonnetwork{.de,co.uk,com} etc that all run from the same codebase when spun up, sure there are other ways to accomplish the same thing obviously but this is what came to mind

As well as other minor use cases for things that I had thought of when starting this reply but have slipped my mind, wow I feel old ... oh the second thing was if done with DRUPAL_ROOT then the same system can be used for D8 and D9 that will make use of PSR{-0} class loading and such that may not follow the same or similar dir structure as D6 and D7 in the placement of their Classes and Libs, so later there would be less of a disconnect in the docs of things like "For D7 do this .... for D8 do this ..." etc etc

Just a braindump but definitely something to think about ... hows that saying go ? "If your gonna get wet , might as well go swimming!" heh

Cheers!

We can't have the paths in the tarballs hard-coded to DRUPAL_ROOT for all sorts of reasons:
- We don't know if things belong in sites/all/* or sites/[some-specific-site]/*, etc
- We don't know if things belong in sites/* or profiles/*
...

If anything, trying to hard-code paths like sites/all/modules/* into the tarballs just makes us even less future-proof in case of directory restructuring later.

Installing other profiles in the profiles directory of an existing multi-site install is already possible via the 'no-core' distribution packages as I mentioned at the start of comment #8.

The PSR-0 class loading stuff isn't going to change how contrib is distributed. Classes provided by a module will remain in subdirectories of the module's root directory. Contrib isn't going to be sprinkling its classes into core's directory hierarchy.

So, thanks for offering your ideas, but sadly, I don't think there's anything actionable in there.

I *do* agree with the "if we're going to change it, let's try to change it all the way" sentiment, but it's hard to foretell the future... ;) But yeah, to the extent we can, let's try to Get This Right(tm) if we're going to do it at all.

Thanks,
-Derek

@dww Sure, and i'm totally fine with that, like i said it was a braindump to think about a lil, I can see what your saying about the PSR-0 stuff not changing the structure by follow the conversations here and on the PSR mailing list about it the last 6 months or so, and the sentiment agrees with what you say but I was more thinking about how it could/should be i guess, but thats for another place and way OT here.

That being said though I do have to question one thing though then, if moving fully to the root ( and thus allowing the tarballs to place files anywhere TECHNICALLY but i'm sure still governed in some way by the package manager setup ) whats the real difference in that and moving it one more dir up to ../ aka ROOT/sites/all/ ? i mean the technical feats you mention are an issue in both cases thus the "hey if we're going for it !!" sentiment , just curious if I'm missing your point somehow on that part ( which is very likely , been a very long day )

Cheers!

Re-titling based on #4 and #5 here. See for example #1471638: Add respond.js and matchMedia() polyfill to the whitelist.

If it helps, I have archived a module including a library;-

created with:

tar -cPzf timeago.tar.gz timeago ../libraries/timeago

extracted with:

tar -xPf timeago.tar.gz

I know we’re trying to minimise changes to the extraction procedure; omitting the P switch results in a warning, and a misplaced library. ../libraries/ need not exist.

We are working on cleaning up Drupal.org-related issue queues. Many issues have been open for years with no resolution because there were no volunteers interested in taking on the task or staff time available to complete the work.

We are going through old issues and marking them as Closed (won't fix). If you feel your issue has been closed in error, please do comment on the issue and let us know. If we know something is critical, we may reconsider the closure of a feature request or bug report.

Thank you for your understanding while we work to clean up the Drupal.org queues.