Here is the problem:

Installation profiles have the potential to help end-users get sites done faster and accelerate the adoption of Drupal. However, this promise has yet to be realized, and this core Drupal feature is currently being vastly underutilized. (source)

Great progress has been made on this front due to ""Fully packaged Drupal distributions now deployed on drupal.org".

Unfortunately, since 3rd party code was decided to be excluded from the packaging script, the Drupal distributions have been greatly crippled because many modules require 3rd-party libraries to be included in the distribution to run properly. There has been quite a bit of discussion regarding this issue on the following issue: #594704: Allow packaged install profiles on d.o to pull in code from other sources + sites

A solution I would like to formally propose is to allow 3rd party libraries to be included in the CVS. The benefit from this is that it makes packaged Drupal distributions usable, driving the adoption of Drupal among new users and it makes installing modules for current users much easier, especially when modules like Plugin Manager are being used. Including libraries in CVS means site builders don't need to hunt down, download, unpackaged and FTP the libraries when they need them. It achieves the original goal, "help end-users get sites done faster and accelerate the adoption of Drupal"

The current policy on this issue can be found here:

This is the policy for 3rd party code libraries in Drupal's contributions CVS repository.

  1. In general 3rd party libraries are forbidden, so do not commit it. Document for your users how to find and install it.
  2. They are absolutely and always forbidden unless if they are licensed under GPL or GPL v2+ or they have a broader license and can be re-licensed under the GPL. GPL "compatible" code is not allowed.
  3. Exceptions can be made if the 3rd party library:
    1. had to be modified to work with Drupal and this modifications wasn't accepted by the original author
    2. is generally difficult to find in the needed version. This is true or used to be true for some jQuery plugins.
    3. is no longer maintained by the original author

Maintainers who violate this policy or refuse to correct violations brought to their attention may have their project unpublished or CVS access revoked. For more discussion see: http://drupal.org/node/418844.

This policy does not apply to original code written by a project maintainer. For example, if you write an integration library to connect a Drupal module to another API, you may include it in CVS (licensed under the GPL), since this will be the original version of the library. (source)

I believe the new packaging script for distributions and modules like Plugin Manager are enough to warrant a re-evaluate this policy. I recommend revoking the policy and allowing 3rd party libraries to be included if they are licensed under GPL.

Frequently Asked Questions

Won't this reduce performance of the repository by increasing its size with 3rd-party libraries being duplicated by every module that needs them?

For many projects, the library they use is not used by other projects. For example, the Image FUpload module is the only project using the swfupload library. For cases like this, the library should be included in the module directly.

For libraries like MooTools, which are used by multiple project, there should be a module created simply for the purpose of redistributing the MooTools library. This module would become the dependency for all projects which require the MooTools library.

By including the measures mention above in the new policy, the size of the repository should not grow unsustainably and performance should remain intact.

Doesn't this have the potential to be a security problem?

If libraries are only enabled when they are needed, it shouldn't be a problem. That being said, this is something that the module maintainers need to manage as they already ought to be.

For modules like Wysiwig, the module maintainer (if he decides to include libraries) should make the libraries optional. If someone wants to use TinyMCE, they can enable the TinyMCE library and leave CKeditor disabled. If another person likes CKeditor, they can enable it and leave the other libraries disabled.

Won't this make it harder to moderate the CVS repository

It shouldn't change the difficulty of moderation because nothing has really changed. Everything still needs to be verified that it is able to be licensed as GPL. 3rd party libraries are already allowed, just under special circumstances.

Won't this fork the 3rd party code?

No, 3rd party libraries are designed to be used this way. The libraries would be used in their stable versions, and not as dev code. Allowing them in Drupal's CVS would not be "forking" them, but rather "redistributing" them as they were intended to be used.

Will GPL-compatible Libraries be allowed as well?

This policy should also be re-evaluated, but not until the GPL libraries are allowed. Consider this question "postponed". There are enough factors with just including GPL libraries to consider that I would like the Drupal infastructure team to decide that issue before the conversation gets side tracked with legal and licensing issues.

For reference, here is the policy on that matter, but lets stick to GPL libraries first, then we can talk about GPL-compatible:

Drupal.org's package management system automatically adds the GPL license to all packages. If we allow other licenses in CVS, it is going to get messy, and sooner or later we're going to run into licensing issues. Already, we get quite a few questions about Drupal's license. If we are going to add more licenses to the mix, it is going to be harder to audit, or provide answers to such questions. So, not allowing other licenses in CVS is a deliberate choice.

We've also decided against mirroring other projects in our CVS repositories--unless there are good reasons to do so. So when people need a non-GPL library, it is best to instruct them to download that library from that project's website. (source)

Comments

philbar’s picture

philbar commented

NOTE: This discussion started in a previous issue entitled "Allow packaged install profiles on d.o to pull in code from other sources + sites", starting with comment #110. I felt like I was hijacking the issue, so I decided to make this topic an issue on its own.

Crell’s picture

Crell commented

Subscribing.

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

I still think that not allowing to put third-party libraries in CVS is still a good idea.
If it would be allowed, then probably there would be many modules that would include the same library, with the effect of having multiple copies in different directories.

I would be more favorable to third-party libraries in CVS if they would be handled by a central module used from the other modules.

tstoeckler’s picture

tstoeckler
he/him
Primary language German
Location Essen, Germany
commented

An option that was also discussed (but IIRC without resolution) in #594704: Allow packaged install profiles on d.o to pull in code from other sources + sites:
What about using Drush Make to pull in the 3rd party libraries when putting together the tarballs. Then people using CVS wouldn't get them directly, but if the modules include a .make file anyway, it shouldn't be too much of a burden. Then we wouldn't bloating our CVS but still getting the same thing.

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

This could be a solution; it would avoid to update the files in CVS when a new version of the library is available.

dww’s picture

dww
we/he/they
commented
Issue tags: +Legal

@tstoeckler #4: Yes, that's exactly the point of #594704: Allow packaged install profiles on d.o to pull in code from other sources + sites -- instead of having people directly commit 3rd party code to our CVS repository, which is potentially problematic for many reasons, they just commit .make files that point to external "upstream" locations for that 3rd party code, and it would be packaged automatically. End-users don't care about the distinction, from their perspective, they still download a single package, install it, and it works. Our CVS repository doesn't have to withstand the bloat, we don't get into legal issues with licensing, and we don't have to be holding onto stale, potentially insecure versions of other people's code (at least not in the CVS repo, only in the packaged tarballs themselves). The only potential legal problem is that we'd be redistributing this other code. I really think #594704 is a vastly better solution to this problem than this proposal (but thank you very much for splitting it into a separate issue, since the hijacking was a bit unfortunate). I'd rather see agreement and progress on #594704 and then mark this as "won't fix". But, so far, there's been lots of "yay, +1 subscribe" but very little concrete action in the other issue...

Crell’s picture

Crell commented

The conclusion to #594704 was that our only legally comfortable recourse is import-on-package-build with a whitelist of allowed 3rd party URLs. Who's going to maintain that whitelist and who's going to write that code are still up in the air, as far as I know. :-)

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

Just to understand: is there are way to avoid the Drush .make file download a file from a web site that not in a white list? If it is possible, in which module should the code be added? drupalorg.module?

dww’s picture

dww
we/he/they
commented

@kiamlaluno #8: That's a technical question for #594704: Allow packaged install profiles on d.o to pull in code from other sources + sites. This issue is about *not* solving this problem via drush make and packaging, and instead just committing all the 3rd party code directly to CVS.

hunmonk’s picture

hunmonk commented

i think the policing of the whilelist would probably have to happen in the drush_make extension for drupal.org (in the drush_make contrib folder). not sure of the best approach though. the simplest would probably be to hard code the whitelist right into the plugin, but that wouldn't make it very flexible. or we could figure some external secure location to store the whitelist, and have the plugin suck it in during the verification process.

philbar’s picture

philbar commented

Everyone who is oppose to this proposal, and favors packaging the libraries from remote sources, please discuss it at the correct locations:

#594704: Allow packaged install profiles on d.o to pull in code from other sources + sites
#684788: Verify Library URLs against a White-list for drupal-org.make

This issue is for discussion on the policy of 3rd-Party Libraries in CVS. I prefer this option to white-listing because it

  • reduces the number of steps to install a module that contains a library (Principle 1 of D7UX: Make the most frequent tasks easy) and...
  • is easier to maintain than keeping an issue queue with library URLs to be verified.

It would be beneficial to more than just drupal-org.make.

avpaderno’s picture

avpaderno
he/him
Primary language Italian
Location Brescia, 🇮🇹 🇪🇺
commented

dww: Thanks for the explanation; I was getting confused between this, and the other report.

Then, I am definitively against allowing the commit of GPL libraries if not in the cases already reported in the third-party libraries policy.
The risk of seeing committed in CVS jQuery or JQuery UI is real. I can assure you that in some of the CVS applications I reviewed users included jQuery 1.3, or jQuery UI. When I point out the policy about third-party libraries, they always try to get an excuse that to me sounds like Yes; I read what you write, but who cares?.
Even in the case you point out that there is already a module that makes possible to other modules to use a specific library, you get Yes, but I don't want to create a dependency from that module as answer.

pwolanin’s picture

pwolanin commented

Given that this is a long-standing policy, and we have a roadmap to addressing some of the impact on end-users via packaging (in the future) with install profiles, can I am going to close this issue tomorrow as a "won't fix" barring some development that has not been hashed over above or in previous discussions of this issue.

dww’s picture

dww
we/he/they
commented
Status: Active » Closed (won't fix)

@philbar #11: While I appreciate your interest and persistence, I disagree with both of these motivations:

reduces the number of steps to install a module that contains a library (Principle 1 of D7UX: Make the most frequent tasks easy) and...

The steps would be identical -- download a single archive file and unpack it. If the packaging script pulls the library out of our CVS repository or some remote location before it puts it in the tarball, the end user sees the same result either way. If you're worried about the small subset of power users who deploy directly from CVS, there's no reason they can't just use drush_make themselves to pull both the code from d.o CVS and the remote 3rd party library.

is easier to maintain than keeping an issue queue with library URLs to be verified.

I don't see how. If we build a public whitelist system, it's obvious what's allowed and what's not, and there's an obvious place to ask to be included in the whitelist. If we just have a "go ahead and commit stuff you think is GPL-compatible" policy, we basically have to watch CVS commits, try to track down license info ourselves, etc. It's much easier to maintain a whitelist in the open than relax the policy on commits and then have to deal with violations manually. From the d.o infrastructure maintainer perspective, it's much easier to make people ask for permission than forgiveness. ;)

@pwolanin #13: Agreed. I see no point in waiting.

philbar’s picture

philbar commented
Status: Closed (won't fix) » Active

The steps would be identical -- download a single archive file and unpack it. If the packaging script pulls the library out of our CVS repository or some remote location before it puts it in the tarball, the end user sees the same result either way. If you're worried about the small subset of power users who deploy directly from CVS, there's no reason they can't just use drush_make themselves to pull both the code from d.o CVS and the remote 3rd party library.

Are you saying the packaging script would apply to modules? I was under the impression it would only apply to install profiles.

If it doesn't then I don't understand how you can disagree with this point. If someone doesn't use drush_make or install profiles (which currently is quite a few people), they have to perform more work searching and downloading libraries. By implementing my proposal, this step would be eliminated.

I don't see how. If we build a public whitelist system, it's obvious what's allowed and what's not, and there's an obvious place to ask to be included in the whitelist. If we just have a "go ahead and commit stuff you think is GPL-compatible" policy, we basically have to watch CVS commits, try to track down license info ourselves, etc. It's much easier to maintain a whitelist in the open than relax the policy on commits and then have to deal with violations manually. From the d.o infrastructure maintainer perspective, it's much easier to make people ask for permission than forgiveness. ;)

This issue is not about GPL-compatibility. I'm not proposing a "go ahead and commit stuff you think is GPL-compatible" policy. This proposal is only about GPL libraries...not GPL-compatible.
Drupal is already policing the CVS, they already have to make sure that things are ready for the GPL license to be applied. They already allow GPL libraries, just under certain circumstances. By allowing all GPL libraries in the CVS, Drupal will not need to change anyone's role or add people to new tasks. It's already in place.

Now you are saying what is already in place is more work than building a whole new whitelist system. Who's going to manage the list? Where are people going to make there proposals for additions to the list? (Both questions to be answered in the other issue). These are obvious problems that have not been addressed for the whitelist system. Unless the whitelist system is going to make managing the CVS easier, the whitelist system will not be easier than allowing GPL libraries in the CVS.

dww’s picture

dww
we/he/they
commented
Status: Active » Closed (won't fix)

"Are you saying the packaging script would apply to modules?"

Uhhh. ;) The packaging script applies to everything. The question is if modules could also use .make files. And yes, that'd be the idea, if we solved #684788: Verify Library URLs against a White-list for drupal-org.make. Sorry that the title of #594704: Allow packaged install profiles on d.o to pull in code from other sources + sites still references "install profiles", that's a bit misleading...

"Now you are saying what is already in place is more work than building a whole new whitelist system."

What's already in place is manageable given that most people aren't trying to commit 3rd party libraries. If we relaxed that policy, it'd become a lot more work.

A "whole new system" is not that complicated. It's either a new issue queue, or a new component of the infra or webmaster queue, and some actual mechanism to maintain a list of allowed URLs. I'm not personally going to write it myself for free, but it's not like it's rocket science beyond the scope of what this community is capable of building. Especially given all the people and groups interested in solving this problem (e.g. devseed, et al)...

Component: CVS » Other