Allow packaged install profiles on d.o to pull in code from other sources + sites

If you want my input from a legal standpoint, please tag it accordingly. I'm lucky I noticed this issue. :-)

From a legal/copyright standpoint, we can only distribute software through d.o that we have a license to distribute. In practice that means that it's under a "free to redistribute" license of some kind. If we automate pulling in 3rd party code, then we cannot vet that said 3rd party code is under such a license. That's opening ourselves up to a world of hurt.

Additionally, we currently have a standing policy of "if you check it into CVS, you're authorizing it to be under GPLv2+". The reason for that is so that we can say "if you download something from d.o, you can safely know that it's under GPLv2+". If we start allowing automated 3rd party aggregation, then again we are, guaranteed, going to have some BSD, MIT, LGPL, AGPL, PHP/PEAR licensed (which is not even GPL compatible), Apache 2 (only GPLv3+-compatible), etc. get aggregated at some point, probably by someone who doesn't know better. That means we're then either automagically redistributing someone else's code under a license they didn't intend (which is sometimes legal, sometimes not, but probably impolite either way) or we're distributing tarballs with a statement of "well most of this is GPLv2+, but some of it isn't, we don't actually know which is which, check inside for LICENSE.txt files that are scattered all around somewhere." I don't want to go there, and neither do you. :-)

There's absolutely advantages to automating pulling in 3rd party dependencies once the code has left the drupal.org server. I've started adding drush commands to some of my modules to pull in whatever 3rd party code they need, for instance. If we ever solve the write-to-disk problem, we could look into ways to automate that even more, potentially. But IMO we should maintain the "if it comes off of d.o it's GPLv2+, kthxbye" policy. The only way to make sure that stays true is to only allow the distribution of code that has been put on d.o explicitly by someone who knows (or at least has stated they know by signing up for a CVS account) that doing so means that it's under GPLv2+.

So a huge -1 to allowing automated packaging of 3rd party code. It's a can of worms I do not want to open.

The main reason that we're asking this question is that it's certain that install profiles are going to want to include externally hosted code. Given this certainty we should have a good plan in place for supporting this usage.

By way of example, the Managing News install profile is a aggregation application that depends on the simplepie parsing library. If a user is going to be able to use a site based on this profile they'll need the simplepie library. If we don't allow 3rd party code to be included in the tarball of this profile we'll be offering a broken download. If that is going to be the case I'd suggest that many profile authors will host complete tarballs elsewhere that are complete and functional. I'd also suggest that we should expect people to link from profile project page on drupal.org to complete tarballs that aren't hosted on drupal.org and contain this non-allowed code.

@Crell If the legal team says that we cannot ask the same of profile authors as we do module maintainers (only make GPL compatible code available through drupal.org or have your CVS access revoked) that is fine. But, given that profile and module maintainers and the same group of people that are used to the rules we have in place, and fundamentally we're asking them to adhere to these same rules, with the exception we're now saying "don't reference" and not "don't commit" the explanation above seems like it could be made more robust.

@dww shipping profiles with a secondary make file and the code for drush_make I think targets the wrong audience. The 'more complete' nature of install profile packages (vs. just downloading drupal itself and some modules) will hopefully make drupal more attractive to people that are not very familiar with the drupal project's specifics (like drush) and who simply want a php+mysql application that does something. It would be great if we could keep things simple for these folks and not have to introduce them to the command line, drush installation and running drush make just so the can use and install profile.

I would like this to happen. There are no reasons (technical or legal) to not package external GPL dependencies into our packages. It makes a lot of sense to package the TinyMCE module with the javascript part and the FeedAPI with the SimplePie parser, or RDF with the ARC2 library.

This makes *so much sense*, we cannot shoot the idea just because it's not that easy to implement securely, or just because people "might" point to stuff that we don't have the license to distribute.

How is it not holding module and profile developers to the same standard? The problem comes if the 3rd party code gets bundled up into a tarball that lives on d.o. That's true for both modules and profiles. If after downloading the tarball from Drupal.org it's very easy to download any 3rd party dependencies, that's fine, because the 3rd party code is not passing through our server. That's true for both modules and profiles.

Making 3rd party code easier to download directly onto a user's system, for modules or profiles, is a worthwhile goal either way.

The purpose of a Drupal packaging solution is to make Drupal easier to use for non-technical users. Obviously there is a barrier to entry when you introduce the "now download this module/theme and put it here". Removing this barrier will make drupal a more viable solution for more people. Arguably the barrier of "now go to this other site, download this other code, and place it in the location specified by in the install.txt" is much greater than downloading and installing a module/theme.

I can definitely understand the possible legal issues of simply including random code, without any requirements. As jmiccolis points out, we already have a rudimentary system of preventing this. If we can't trust cvs users enough to take the licensing requirements seriously, I would suggest that we create a white list of allowed external packages, which profile's can use.

IMO, to allow third-party code to be put in Drupal.org CVS creates two problems:

• The first problem is that if I commit third-party code into my module, then I should make a new commit every time the third-party library is updated. The users of my project would then wait for me to notice there is a new version of the library, when they could download the new library themselves.
• The second problem is that space in CVS would be occupied from files that I could find in some other site. See the space occupied from jQuery UI and how much modules occupy the same space, and you get an idea of what I mean.

I actually was assuming that one of the purposes of install profiles was to allow for external code to be included.

The legal aspect is unfortunate, but is actually not much different from our cvs repository: We in principle need to check each and every new checked in file for GPL compatibility. The same would apply to externally hosted files or software bundles.

I am of course ok with postponing this feature and getting something simpler to work first.

It seems possible that people are speaking past each other, and all might be in heated agreement. The options, as I understand them, are:

A - Allow install profiles (in whatever form) to invoke calls to outside code/library repositories, which happens directly on a client/peer basis, not through d.o.

B - Allow install profiles to call and aggregate outside code/library repositories, with tarballing happening on d.o.

C - Skip the entire idea.

"A" seems to be preferable for pragmatic reasons (e.g., pull code/libraries from the source) and legal reasons (keep non-GPL stuff off of d.o). And from what I can tell, everyone agrees on this.

Am I missing something?

@dww: Hum, why does that require core to support pulling 3rd party libraries on install from remotely? Install profiles could include any contrib module or API which supports this as you've already explained, since that would be hosted on drupal.org and under the GPL. Even for Drupal 5, so we don't need to wait for Drupal 8 :). I would not consider this an install profile pain, especially if only 10% of profiles would pull in remote stuff as you've estimated (although I do not agree with your 10% estimation :).

If we agree that install profiles will be shipped without 3rd party code but will include a component if required to remotely download and install 3rd party libraries, then as long as that remote source is available, we solved the issue, did we?

@dww: Sorry I didn't address the idea of having the install wizard retrieve remote code. I'm afraid of all the security hoops that need to be jumped through and I'd hate to make this new packaging infrastructure depend on something that as complex and controversial as the plugin manager has been. This is why I'm trying to completely understand all the reasons we can't aggregate 3rd party code. I'm also not tied to the idea that we have to aggregate these libraries, but rather that the experience for users who downlaod my-profile.tar.gz is easy.

A couple questions that I have:

Is it simply that we don't mind if people link to remote tarballs from project pages, but we do care if that tarball sits on drupal.org infra?

Do we think that enforcing a 'gpl only' or gpl-compatible' only install profile policy is too difficult or puts us at too much risk? Is there a way we can pass this risk to the profile maintainer?

Cheers,

Jeff

couldn't we have a white list of allowed sources for packages ?

I haven't played around much with Drush Make, but it seems really promising and +1 for aggregating around a standard that does the larger packing. Install profiles have *so much promise*, its a real shame to see it squandered. I authored two install profiles (an academic journal profile + a conference website profile) and in the last 1.5 years have received 50+ personal emails from frustrated users who were having trouble getting all the right modules and just wanted it to work out of the box.

I know that dmitrig01 is working to rewrite Drush Make to allow for exclusion of remote sources and hopefully this should unblock this issue and allow the generic packing to happen. However, including remote sources is a *huge win* for Drupal and will allow us to build really powerful applications that take advantage of useful, third party tools.

The legal and security issues should be taken seriously, but these are solvable problems and with the right people can certainly be solved. DWW + Gerhard can speak specifically to the security challenges on the D.O infrastructure, but proper sandboxing packaging scripts are definitely possible and are used with far less trusted and auditable sources than Drupal community members with CVS access.

The legal issues around third party code in CVS are obviously full of opinions, but given the relatively small number of install profiles that would need third party support (and given the trust we already place in the project maintainers) perhaps some sort of required checkbox on the project release page to certify that, yes, all code behaves could solve this problem. As with many legal questions, perhaps we could ask a lawyer specifically to solve this problem for us? (i.e. we want to do this thing, how do we do it legally?)

As the server distributing the code, we would be liable if we distributed something inappropriately. (In this case "we" is probably the DA who would get sued, plus various d.o maintainers. I really do not want that. :-) ) As long as 3rd party code doesn't go through the d.o infrastructure my legal bone is satisfied.

Yes, it's a PITA, but it's not unprecedented. Many Linux distributions ship without support for, say, mp3 playback or DVD playback because they're not allowed to distribute the necessary code (for copyright, patent, or flat out extortion reasons). Instead, the first time you start a certain program it will prompt you to click a button to download the necessary 3rd party library from another source. The process is still heavily automated, but is user-instigated and the code doesn't go through the distribution's servers. Sometimes it's a command line script instead of a GUI button, depending on the library and application. So far that has kept the major distributions safe.

If we could properly police the 3rd party code and ensure that only GPLv2+ code got autopackaged (possibly selected GPL-compatible 3rd party packages, but I'd have to go over those details with our attorney), that would be OK from a legal standpoint. However, that's a lot of work to ask of the CVS / drupal.org maintainers.

Gabor makes a valid point, I think, that an improved "download necessary 3rd party stuff" routine could be developed in contrib, and then just that engine gets packaged into profile tarballs. I'd be comfortable with that. It's already not hard to setup with Drush, but I agree that's suboptimal to something that works through the GUI.

*We need full fledged on-board packaging or we don't need to go through the pains of building packaging on d.o. at all.*

Getting install profile packaging right could boost Drupal's growth like nothing else. If we get a working packaging infrastructure into the average developer's hands out there, we can fully expect a flourishing install profile landscape comparable to contributions/modules. In my mind, this IS the future of Drupal, the one thing that could set it miles apart from its contenders. I firmly believe that a central contributions/modules is an essential part of Drupal's success story and I believe we have a chance to do the same with contributions/profiles if we get it right.

I'm seeing two problems between us and install profiles bliss at the moment:

• Packaging of external libraries
• Packaging of external custom modules/themes

Off-site packaging would start to address both of these problems. Packaging libraries actually seems to be the smaller of the two problems as writing some sort of white list is thinkable.

However the BIG ONE in my mind is custom modules and themes: There won't be good install profiles without serious custom code. If we don't have off site packaging, the only place for this custom code will be d.o. and we will wind up somewhere between two undesirable scenarios:

• A The "same" scenario: Many practically non functioning install profiles because custom modules need to be fetched from somewhere else - pretty much the situation we have right now.
• B The "Namespace spam" scenario: An onslaught of custom modules and themes in CVS.

Either way and in between we don't win.

I'm aware that offering off-site packaging will not remove the temptation to put up custom modules on d. o., I wonder what additional options we have. Full fledged packaging will be one step in the right direction of actually making install profiles work.

#16 @dww, you wrote in response to packaging non d.o. projects: "[...] I'm just worried about the precedent it would set, and the attempted proliferation of this practice into modules and themes, which would start to undermine d.o's role as the central hub for the community, *and* create a support headache for the d.o admins. So, I'd rather find another solution."

I would actually see it the other way round: e. g. if it's not possible to package external dependencies on d. o., most install profiles won't be possible to be hosted on d. o. For instance http://drupal.org/project/managingnews includes two custom themes, 6 feature modules, 3 custom modules. If there is no off-site packaging, we'll be forced to host it somewhere else. I'd like to avoid that.

#17 @crell "As the server distributing the code, we would be liable if we distributed something inappropriately"

I wonder what creative options we have - again, whitelisting, anything else? If really the only way of protecting the DA is not packaging anything off d.o. at all that would be a big blow for install profiles.

Alex nails it --

This is definitely a PITA to build, from a legal, technical, and a security standpoint.

Without this, though, install profiles will leave d.o --

We ran DrupalEd off d.o for exactly this reason -- the target audience for the profile was VERY intimidated by the need to install external libraries, to the point where they would give up on Drupal.

Recently, we have not put much (okay, anything) into keeping DrupalEd current as we sorted out the best way to develop the project moving forward. The ideal would be a solid set of base functionality within an install profile, with featuresets to bring more nuanced functionality to specific sites.

While we can set up the infrastructure to manage this ourselves, off-site, we really don't want to. We would MUCH rather centralize this through d.o, as it brings more attention back to where it belongs: drupal.org, and the community here. Install profiles scattered over the internet is a step toward fragmenting the community; we see that as a bad thing.

Packaging external code is the special sauce that can distinguish Drupal from other projects. It's not going to be easy to build (for both technical and community reasons); but this would make install profiles much more useful than they are now.

It sounds like this is a collision between four factors:

1. Drupal.org allows anyone with a CVS account to check in code that turns d.o. into a "Distributor" for that source code. If it is not legally kosher, d.o. and the DA are legally liable.
2. There is almost certainly not enough manpower to switch to 'review and approve each case' model at present, even if the GPL-only/NO external libs policy were changed.
3. If any external libraries are actually required by the projects/profile/distro, having a manual phase of the install where the user tracks down all the dependent libraries and manually installs them defeats much of the purpose of this system.
4. In D6 (and possibly d7) there is no reusable 'plugin manager' mechanism that would allow profiles to automatically pull down these kinds of components during/before the install process.

Is that a reasonably accurate summary?

@eaton, Yes, I think that about sums it up. Solving #3 by allowing off-site code to be packaged up on d.o runs us right up against #1 and #2.

Solving #4 (allowing post-download fetching of 3rd party dependencies) is, IMO, the best solution as it resolves #3 (at least to some extent) while not introducing problems #1 and #2. It's also an approach already used by other distributors (eg, Linux distributors) so there is precedent for it. (Not that a 2 step process isn't annoying, but it's a common solution to this sort of problem.) A solution to #4 can probably live in contrib, and then just be included by install profiles. (Even if it requires adding keys to info files to work, core caches the entire info file so that shouldn't be a problem.) I think the libraries API wants to be that solution, but I don't know how viable that is.

A forced whitelist (eg, the system actively blocks anything not on a whitelist from being packaged) would be legally OK, IMO, but I foresee the d.o admins getting flooded with "well why isn't MY 3rd part code whitelisted yet?" requests that I suspect they don't want to deal with.

I agree that we want to encourage all non-3rd-party code to live on the d.o mothership and not scattered about the web. Frankly even "feature servers" scare me that they would result in nasty fragmentation.

Yes, it's a PITA, but it's not unprecedented. Many Linux distributions ship without support for, say, mp3 playback or DVD playback because they're not allowed to distribute the necessary code (for copyright, patent, or flat out extortion reasons). Instead, the first time you start a certain program it will prompt you to click a button to download the necessary 3rd party library from another source.

This is a huge thing for driving the reach of Drupal. It lets us provide much more of a "working system" out of the box without forcing people to undertake a lot of (to them) arcane steps to add dependencies. Also, making it possible to provide "extended" drupal downloads will enable the #smallcore vision to really reach its full potential, since it will provide a realistic answer to a lot of questions about including function X in core itself.

IMHO integrating this into the install profile experience via an "get external stuff" contrib hook sounds like the doable win. We should find out if this is possible, and make it so if not, and then document it in a way that will let people really run w/the functionality. To the extent that we need to get some funding together to drive this development on d.o. to package the onboard stuff, I'm happy to kick in.

I am not seeing that much issue with a whitelist.

d.o is a bad place for good install profiles. Even a pure Drupal solution, Acquia Drupal can not be hosted on d.o as it contains an external PHP library.

My advice would be to not waste your time on d.o solutions, when it can be easily achieved on an external site.. I know, external site has different problems..
d.o may use some of the achievements later, but some issues simply can't be solved here..

... Pasqualle, we're talking about was to resolve the exact issue you mention (3rd party code). There's a general consensus that fragmenting Drupal all over the web on external sites is a bad thing, so we're trying to avoid making that happen.

#20 dww:

The situation is not quite as bleak as "we absolutely must have tarballs which include 3rd party code or Drupal is doomed...".

My point was that without solving the problem of custom code and external libraries _install profiles_ won't be much better than they are right now. Neither Drupal nor install profiles will be doomed.

I'd love to not rely on client side packaging. There is no solid implementation of it yet (while drush_make works). Further, I am very sceptic that it is a good practice in the first place (the people it is trying to help are the ones who can get stuck for good when the permissions aren't right).

Again: I'm interested in shooting for the best solution - in an ideal scenario, wouldn't we want to download full fledged packages from d.o.?

I'm seeing only one argument against it here: legal implications. And there has been a solution suggested: whitelisting. It may also be worth exploring further our legal options (saying this without knowing how far Crell or other's research has gone). I don't think we should capitulate yet.

For the record: any movement towards better packaging is a great thing. Also: thanks, 3281d Consulting for funding work on packaging and thanks to dww and others who will be actually be implementing packaging on d. o.

I will bounce this question off of our attorney to make sure we're not being over or under cautious, although I would much rather be over-cautious than under-cautious. :-) I'll report back when I have a summary.

@dww #26: I am full of surprises. The drupal.org maintainers already do a lot of stuff, maintaining a whitelist shouldn't be too hard as long as we have succinct rules about what can be whitelisted and what not.

But again: let's first get d.o only stuff working.

Subscribing.

why exactly can't we just package things and have a 'complain about licensing issues' link, with a take down process?

Because "better to ask forgiveness than permission" is in practice not as good an idea as it sounds, especially where the law is concerned, and as such a takedown process would inevitably end up at least partially on my plate I'd rather be proactive in discouraging people from playing fast and loose with copyright licenses and support issues.

see , what we have here is in my mind similar to what youtube.com has to deal with.

it is simply not possible for them to stop people from uploading copyrighted content to begin with, and if they decided to do what we are proposing here, youtube.com would not exist at all.

i simply don't believe that some magical possible copyright issues are a good enough reason NOT to do something that is incredibly useful to many people.

And when we talk about a take down system we are talking about an automated system. You complain, downloads go away, mails get sent out, a real person checks it out and actions are taken.

We already have the barrier of entry high enough to even be able to commit things, have people agree to a guidelines document, if they fsck up their access is removed.

I agree with Adrian on either crowd sourcing the licensing issues or whitelisting certain domains, or even both.

I agree. A takedown notice is not an automatic expulsion to the gulag. It is a letter that says "Hey hippies - you don't own that. Please remove from your site.". Then we unpublish and investigate at our own leisure.

also linux distributions distribute different open source packages with different licenses.

so when you install ubuntu you get MIT, GPL and BSD licensed software. They don;'t distribute things that are patent encumbered, which is something we don't actually have to worry about.

if the packages that are sourced don't include a license.txt in their tarball, that is hardly our fault.

considering the space we are working in, i think the number of take down notices we will be receiving is so very small it really does not make sense to disallow the entire concept just because.

we have far far far far far far far more liability from theme developers including copyrighted images in their themes, yet we dont seem to be trying to stop people from uploading image files.

OK, so I have a few thoughts on this at this point.

1.) I think it's a mistake to believe that most profiles will not be utilizing some code outside the bounds of d.o... virtually everyone agrees that from a usability standpoint there needs to be a wysiwyg editor available out of the box, if given the ability I would bet that most profiles would implement such a thing (especially if it could be done w/o interaction from administrator to set it up). So for me, it's at least 50/50 that will use outside code... if not more

2.) Is there not already a legal precedent set for simply complying with a "cease & desist"? As Adrian points out, this is really no different than youtube, and what they do has far more liability related to it (in my opinion). Granted, they're a much larger entity, I'm simply pointing out, that what ends up in the packages couldn't end up there if it weren't already downloadable on the net somewhere. Furthermore, while we may be distributing the tarball, I have to imagine that the liability for this could be placed on the maintainer of the profile to some significant extent. What would happen if a module maintainer uploaded code to CVS that falls within these same concerns? How much liability does d.o really have here and how is it different from what's going on currently? I understand we're making it easier to accomplish, so it could more easily happen, however I'm not sure I see how it's much different.

3.) I'll also point out that anything committed to CVS has to be GPL2+ compatible... not to be nitpicky, but in this scenario what's committed is a make file... which is certainly GPL2 if it's committed... what's distributed, is not that make file, but rather the tarball result of that make file, and that is NOT in CVS. I simply point this out because no one else has, and I think it's a potentially important distinction.

4.) And finally I must agree whole-heartedly with Adrian. Our exposure to problems at the moment is MUCH greater from the theming side, especially if it gets easier for themers to upload/maintain their themes. Many of the stock photo companies are incredibly nitpicky about this (and understandably). I think that code just happens to be the demon we identify more clearly with. The images are a MUCH bigger issue, and much harder to police in my mind.

In summary, if someone wanted something off of the site because they own it and it's being redistributed w/o their permission, what's the procedure for making this happen currently? And please explain to me how we're more exposed/liable with the profiles than with themes. The vast majority of code is still going to carry some sort of Open Source license, it just won't all be GPL2.

One other factor I haven't seen mentioned here, is that the namespace issue that alex_b alluded to in #18 could be mitigated by allowing .module, .install, .info, etc. to be committed into contributions/profiles. Currently that is prohibited to prevent users from building their own distros by duplicating modules in CVS, but if we could build and bundle existing modules into profiles, that would leave the possibility of including distro specific modules in contributions/profiles.

@mikey_p : this is a major major major pain point for me.

for aegir, i have an install profile (hostmaster) with a module that only works with that install profile (hosting) with a theme that only works with that module (eldir).

because i can't commit the distro specific components that are designed not to work outside of the install profile to the repository, i am stuck maintaining 3 separate projects with 3 separate issue queues and having to write 3 release nodes every time i make a release.

although that is not the issue we are discussing here. The loosening of these rules, which would be facilitated by employing drush make will allow me to more easily manage my project on d.o

@moshe weitzman +1,000,000 for a "hey hippies" legal notice.

It's not really drupal that's following the "better to ask forgiveness than permission" path: it's the individuals who are creating install profiles. They are on the hook ultimately, and that's as it should be.

The truth is we already let people do things that are illegal. Once I get my CVS account I can upload anything -- including something as nonsensical as copyrighted videos -- and until someone complains it will be there, freely available.

I think drupal aught to continue to enable lots of great things, and including external code sources in install profiles is no bigger legal loophole than giving people the right to check *whatever* into CVS, and could/should have the same remedy, IMHO.

Additionally, if Gerhardt is into whitelisting, that'd be fine. The main reason this matters is WYSIWYG and a few other things around media. The number of sources people want to pull from should ultimately be pretty small.

i'm actually thinking a blacklist is a better idea than a white list.

basically the black list is the end result of the complaints we get. i really don't expect there to be that many, so it'd be less work to maintain a list of things not allowed than a list of things allowed.

If we want to add an extra gate, we could require release nodes for install profiles to be reviewed before being published, which wouldn't be a huge barrier and not eat that much man power.

I agree with the comments on CVS already allows check in of teh nasty, and on takedown to remove.

Also: if someone were to make a '+1,000,000 for a "hey hippies" legal notice' t-shirt, I would wear it.

To me it seems like a whitelist will involve more upfront work, but gradually slow down or flatten out as time goes on - we only have to whitelist tinymce once, regardless if 15,000 install profiles use it, the chances of 15,000 install profiles each requiring a different external library are much slimmer. Once a domain is added, it's barely going to get removed or changed. Whereas a blacklist could keep growing forever. Partly depends on administration though - a whitelist is like the CVS applications process, a blacklist is like the Drupal security team, both have scaling issues.

However - it seems like if we can agree that either a whitelist or a blacklist is a good idea - compared to allowing anything, or allowing nothing, that'd be enough to get started on it - since the implementation is going to be very similar if not the legalities/process about what goes into them.

@Boris Mann, this is for you: http://barthy.spreadshirt.com/us/US/Shop/Article/Index/article/Hey-Hippi...

@mikey_p #40: Agree. Modules and themes in profiles seems like a great way of how we could address the problem that not everybody is going to want to host custom code elsewhere (like I hinted at in #18).

By the way, why are modules or themes not allowed in profiles at the moment? Any pointers to issues?

So let me ask a political question then.

There are lots of modules that bridge 3rd party code. I maintain 2 of them myself. The odds of either one being part of a major install profile are pretty slim, though. That means that they probably won't get whitelisted (by whatever procedure comes up for that), and will generally be downloaded not in an install profile but as plain old vanilla modules.

Which don't get auto-packaging, because they're not install profiles. Which means that they're just as much of a pain to use as they are now; no more, no less. So now 3rd party bridge modules are second class citizens unless they happen to be downloaded as part of an install profile. Fail!

The only way to deal with that, then, is to allow auto-packaging of 3rd party code for all modules, or only blessed modules, or something. And unless we say that we only bless modules that meet some usage threshold or something, that's going to be a lot more modules than I think people realize. It's *not* just about WYSIWYGs.

Whereas if we solve the post-download issue of pulling in 3rd party code, we get a solution for modules *and* install profiles that neatly avoids most legal issues for Drupal.org.

I wonder if we've thought through the implications of allowing automatic aggregation of code from all over the web...

Criterium for whitelisting external code should be license compliance (and maybe size, ie distributing server images is out of scope ;).

All you'd need to do is to create an install profile.

So now 3rd party bridge modules are second class citizens unless they happen to be downloaded as part of an install profile. Fail!

I don't think it's a fail. One step at a time! If we can get install profiles to include third party code, it may not provide the silver bullet solution for all third party software issues, but it is a step towards better Drupal products. I don't think just because a solution for modules isn't answered here in this issue is reason enough to throw it all out.

If we start with install profiles, see how it goes, and it looks like everyone's behaving themselves, and the Drupal world is a better place for it, then we can start to move towards modules being packaged with third party software.

In Crell's case, a natural next step would be to allow for 'packages' - for meta modules like drubb/advanced forum which people might want to plug into a live site, pretty much what features is I guess. Those would need to be a separate project category from modules as such, but also from install profiles.

Oh, that's true... With features and feature servers, there already is a solution for that, albeit not on d.o. I'm pretty sure DevSeed is hoping that there is a central hub for feature servers though. Now if only that hub was at features.drupal.org: http://www.reyero.net/en/node/181 ;)

Also, a whitelist doesn't mean that people stop working on the post-download solution (plugin manager contrib, plugin.php in core) - it just means we have something in place until that's ready to go.

I'm back to my original formulation. In order of priority (IMO), we have:

1. Profile downloads should include Drupal core and contrib modules, not just the .profile file.
2. Profile downloads should be able to include their own 'custom' code, like profile-specific modules and themes, rather than forcing those elements into separate projects.
3. Profile downloads should be able to include (via some mechanism) external GPL'd libraries.
4. Profile downloads should be able to include (via some mechanism) external GPL compatible libraries.

AFAIK, #1 is already planned for implementation, #2 is probably not UNLIKELY but hasn't been committed to yet, #3 is basically a policy issue that conflicts with our 'no external code' rule but could be relaxed if we feel it's important, and #4 is where the legal fuzziness lies.

And it's been proposed that #4 could be handled with a 'HEY, HIPPIES, READ THIS BEFORE YOU ROLL A PROFILE" notice combined with a blacklist to stop people from including stuff that we already know is unkosher.

Does that sound like a fair summary of where we're at?

@Crell

I think you make a good point here. Wysiwygs are just the easiest to pick out. Certainly I don't want to package phpbb with a module that hooks it into drupal... I don't think any of us want that. :-) With that said, I'm actually ok with modules NOT pre-packaging external code, as people who are installing modules by hand need to be able to follow technical instructions (imo). I've already admitted that wysiwygs are just the easiest to pick on, but with that admission sitting there, I'm going to return to it as an example.

Wysiwyg module contains bridging for MANY different wysiwygs... I would be horrified it it prepackaged ALL of those editor in the module download... I think we can all agree with that (I hope), but including a single one in an install profile for the convenience of your users seems like a perfectly good use case to me still.

In short I make a delineation between "modules" that require the end user to have some knowledge, and the ability to follow semi-complex instructions and "profiles" which would be best served as a complete tarball... setup db, click install, follow a wizard.

Returning to my original example, I wouldn't have a huge issue with distributing phpbb in a profile that's already bridging the two together... now that's horrifying because of many different reasons... namely that phpbb is like swiss cheese for security, but that's not really the point (Although I am kind of interested in what the security teams take on a profile distributing code with security issues would be).

Obviously I recognize that there ARE issues to be surmounted here. I'm simply saying that ALL install profiles need to be as easy to install as the default drupal profile is. Anything else is a FAIL. I don't care what the final methodology for that is.

@eaton, in #54, re "Does that sound like a fair summary of where we're at?"

Yes.

With the possible addition that we might need to make Boris a shirt as outlined in #44.

It seems like we could move this forward more efficiently by focusing on #4.

@catch #54: Also, a whitelist doesn't mean that people stop working on the post-download solution (plugin manager contrib, plugin.php in core) - it just means we have something in place until that's ready to go.

I really don't want to go off the plugin manager tangent here, but I'd like to point out that for some use cases client side packaging won't work at all - i. e. it doesn't scale well for managing many Drupal installations.

So while yes, d. o. packaging doesn't mean that people will stop working on the post-download solution (and it's a good thing that they aren't), I wouldn't agree that d. o. profile packaging would be a stop gap solution.

If we are going to automatically download (download and cache?) 3rd party code for distribution via profiles, then I'd favor the whitelist approach personally. Ideally it should be as specific as the to the lib major version and extension, and possibly so we can implement some sort of notification system for profile maintainers when there are updates.

@pwolanin

Absolutely! A serious profile is going to need a LOT of maintenance from someone who's utilizing many modules + 3rd party code. Every update would need testing, and re-integration. So I'm very ++ on the specific version approach, though I don't think the whitelist itself needs to be that specific. Just the make file. Though from a security standpoint I could see wanting to whitelist specific versions...

I haven't thought through how this would work technically, but the most flexible would be a whitelist and a blacklist together - that would let you do things like whitelist google code, but blacklist incompatible projects / insecure releases etc. if they're flagged.

@catch - since google code project can (I think) be under an arbitrary license, I'd always want to limit our whitelist to specific projects and never allow arbitrary downloads from a domain. If nothing else - Imagine I go to a gcode project and attach a file to an bug report - do we want that file to fall into the white list?

+

@pwolanin - maybe for google code, but what about jquery.com?

I haven't read every comment here, so forgive me if this has already been mentioned.

Why not force some of the responsibility to end users, if profiles are packaged from drush make files then can't the external URL (sources) be extracted and shown on the profile page along with warnings listing that this package contains code from an external source (and list the sources). D.O could have a list of trusted sources as well (e.g. tinymce.org) which only give a small warning, while packaged profiles containing untrusted sources (joesblog.com) have larger warnings.

This doesn't solve all the legal questions, but means the contents of packages are transparent.

@catch: Bad example. jQuery doesn't centralize their plugins the way Drupal does, so they're scattered all over the web. If we wanted to actually enforce an auto-packager for pulling in 3rd party jQuery plugins, we'd be white listing dozens of separate web sites, many of which have no standardization between them.

@Crell - Actually no. There is http://plugins.jquery.com/ ? jQuery, much like Drupal, has a centralized repository and then some people choose to host elsewhere. Note also - http://plugins.jquery.com/user - it runs Drupal and Project* modules.

That is true; jQuery plugins are listed on plugins.jquery.com, which is a server running Drupal, and using the project module. They don't seem to use a CVS server, but I don't think that makes any difference about what they are hosting, and which license those files have.

Hm, it looks like they have more of them on the site than they used to. That's good news, although their format is somewhat inconsistent. (Some are .js.txt, some are zips, one I just saw is a rar, etc.) That means they'd still need individual handling in some form.

As Kiam said, though, jQuery plugins don't all have the same license so we can't blanket-whitelist those, as much as that would be nice.

@crell re: comment #29 any word from the lawyer yet?

Aside from potential legal issues in #29, most of the comments here seem favorable to making this happen. Obviously would require some additional human resources, but I think the benefits of keeping things centralized will outweigh the additional fragmentation that could occur if d.o. can't provide the most complete solution possible.

Unless, I'm completely missing something, I say we figure out how to secure the resources to make this happen.

Obviously would require some additional human resources

I am not sure there are enough human resources.
There are some issues open against project without uploaded code, or that have committed files not licensed under GPL license, and those issues still require an action to be taken; I cannot imagine what would happen with the installation profiles including third-party code.

Apart from being a legal issue (and I am not saying that is a minor issue), it is also a problem of human resources. Even with a white-list, there should be somebody that checks the request for adding new items in the list (I don't think that we are going to populate the list only once, and then leaving it as it is); if items are successively added to the list, somebody should check the domain to add in the list, and be sure there are not exceptions for that domain.

This discussion is not defining terminology & applicable goals / use cases, properly yet. I think by doing so, the answer falls out more obviously.

Clarifying the solution space

1) This is really about the packaging, and packaging = "Distributions", not "Install Profiles." The bundle of files that we want to enable people to download is (more properly) called a "Distribution." There's no functional difference between a Distro & an Installation Profile: They're both an assemblage of files + some code that executes to set it all up.

The rest of the open source world calls this a Distribution. We (the Drupal world) already make people learn too much new terminology; let's stop the madness, and start using the term Distribution before we make these things (Distros/profiles) popular. :-)

1a) Distribution contents. Any Distribution can & should be able to contain non-Drupal open source code, and even non-open-source code (ignoring d.o implications for the moment). See point 2) below for expansion of this topic.

2) Distributions need built before they can be listed for download on d.o. Where, and by whom they are built is a separate question from listing them. The build options are:

2a) Built off d.o by those who agree to take responsibility for them. Examples: Acquia Drupal, PressFlow, Open Atrium, etc. These are all built by companies / teams / individuals (a "supplier" here) who can be asked to make assurances to the DA that they will meet technical requirements (update the Distros with security patches promptly), legal requirements (track down and assure license compliance), etc. The supplier builds externally from d.o, and uploads the pre-assembled Distribution to d.o for download by the world.

2b) Built on d.o in some automated fashion. This is the case that everybody's been thinking about so far. Note that there are two sub-cases:
2b1) Automatically built from code that is all checked into d.o CVS. In this case, we have issues around the policy of "What you check into CVS must be released under GPL."
2b2) Automatically built from code that is not all checked into d.o CVS. This case might avoid the CVS-is-GPL issue. Nonetheless, I assert 2b2 is a non-starter because of the many gnarly issues described above, and 2a) above provides a "safer" path (for the DA) to provide the same benefit to downloaders.

3) A post-download "assembler" is dead before it starts. There is no chance that anybody can successfully assemble a capability that will go fetch, install, and assure that all component parts are automagically pulled together post download by the average user.

The issue is that the external component parts can't be relied on to all have a useful, or the same interface for being pulled, and whatever interface they have now can't be relied on to remain the same. Component A might be available via http, but Component B might require Git; does your average novice Drupal user have git installed? No? So do we bundle a git client in the Drupal installer? What license does that have? And how does the one accessible via http handle release version changes? Is the URL release-independent or release-dependent? Which should it be? What happens if their distribution mechanism changes before the Distribution is updated? And what's the method for users to resolve issues if the assembler-after-download fails during the download/install step? Etc. ad nauseum.

#fail before it's out of the gate.

-----

Solution analysis

I assert the solution to this discussion is:

1. Add a Distributions listing capability to d.o. This listing / download enables users to download Distributions that are "files" that live on d.o.
2. Distributions can arrive on d.o in two ways:
   1. Built off-site & uploaded. The DA should create a structure for formal arrangements with such Distribution suppliers, requiring them to (at least) (a) make sure the Distros are license-compliant within themselves, and (b) if they Distro is not license-compliant, the supplier agrees to a hold harmless clause for the DA. (Read "supplier guarantees to assume liability for non-license-compliance".) Suppliers can upload new Distro's to d.o whenever they have a new build available.
   2. Built onsite from CVS - only. D.o adds infrastructure that will automatically build a distro with a given install profile using code from CVS - scenario 2b1) above. No supplier agreement is required here; it has the same level of commitment / support that a module maintainer is expected to provide.
3. We do not solve the 2b2) problem.

The key here is to refocus on Distributions - not on (building) Install Profiles. Doing so opens up the method for how suppliers can solve the problem of including non-Drupal code, such as WYSIWYG editors, but it even opens up non-open-source possibilities. A conceptual example of the latter: a Distro might include a closed-source, binary business analytics processing package, which (when invoked by cron) processes apache logs & shoves post-processing summaries into mysql. This can then include an open-source Drupal module that lets Drupal display the summaries.

-jb

There's no functional difference between a Distro & an Installation Profile. They're both an assemblage of files + some code that executes to set it all up.

Let's not get sidetracked on semantics, but there is a functional difference. Install profiles are the 'code the executes to set it all up', whereas a distribution contains an install profile + Drupal core + Modules + Themes, etc. etc. Right now we have no distributions living on d.o, only install profiles. We don't have any Distributions living on d.o b/c there is no 'assemblage of files' or packaged distribution. If I'm undersanding everything correctly, this issue is about resolving that.

Um, well, RHEL contains code that sets up Linux on a host. It detects installed graphics & ethernet cards, disk types; lets you configure for a desktop or a server; and configures the OS accordingly. And these functions ("Installers," if memory serves me correctly) are no different than what an Install Profile does for Drupal. But the bundle of both (code+installer) is the RHEL Distro.

And the fact that d.o doesn't have Distros listed isn't because there aren't any. Acquia Drupal is a Distro. It's only not listed on d.o because there is (currently) no mechanism to do so.

And semantics is important; people live within a language, and function better if they don't have to expand it. This is all about making it easy for people who don't know Drupal to get their arms around it. We should use language they know - not learn new terms. Please let's not make Drupal newbies learn a new term.

@batsonjay, Oh, ok I thought you were saying that install profile == distribution and that we needed to start calling install profiles distributions. You said "There's no functional difference between a Distro & an Installation Profile." What I was trying to say is that there is a functional difference. Install profile is the script, Distribution is the package. Install profiles need to live on. Distributions need to be added.

#73 @batsonjay - thanks for your clear outline.

I absolutely agree with your Solution space points 1) (Install profile is a technical component of a Distro) and 3).

In response to 2) I'd like to reiterate what I've said further up in #28:

We're giving up before even trying in the face of maybe-daunting legal consequences that we even have solutions (blacklists/whitelists) for.

What software other than operating systems refer to their releases as distributions?

IMHO referring to the packaged, downloadable tarball of a specific install profile as a distribution doesn't offer any more clarity to new users. In fact, I think distribution implies something not true of every packaged install profile to users who are already familiar with the term. I know Dries used this terminology back in 2006 to refer to install profiles like DrupalEd and CivicSpace, but I think there is a simple test to determine what is just a packaged profile and what is a full blown distribution.

"Even if we collaborate and all Drupal distributions are built on the exact same Drupal core, there will still be incompatibilities in terms of documentation, support and vision."- Dries, 2006

While vision will hopefully always differ from profile to profile (otherwise why exist?), the support and documentation are likely what separates a basic packaged profile from a distribution.

At least initially, the Open Media Project and Radio Engage are looking to maintain "starter kits" using what each organization believes are the best practices for the niche of Drupal users they are working with. We have no intention of altering or redocumenting core features or even features of the contrib modules we've included in our profiles.

What separates us from groups trying to maintain distributions like Development Seed and Acquia is we simply DO NOT have the resources to take on the burden of providing documentation or support for anything beyond our profiles and the few custom modules we've written. We can build on the core Drupal 6 or Drupal 7 distribution, but we shouldn't qualify as a unique distribution. If we do, we've probably bitten off more than we can successfully chew.

Using this criteria as the test, Open Atrium would be considered a distribution because they are willing to change core functionality to the point it requires additional, custom documentation. Acquia Drupal is a little different, since (I think) it is simply using specific version of code that is available from Drupal.org's CVS with the exception of the Acquia Agent and Acquia Site Information modules, but the difference there is really defining what is supported.

While any distribution could be driven from an install profile, not all install profiles should be considered distributions.

And i'd like to add that the legal consequences are not proven. The same way that redhat (in your example) ships with libraries on the disc with numerous different licenses, you should be able to ship a distribution with packages of various licenses.

Those packages remain separate packages in their own right, and most will have a license.txt inside them stating which license they are under.

@kreynan - not to be pedantic, but:

AsteriskNow = "... a customized Distribution ... that includes Asterisk, the Asterisk GUI, and all other software..."

MT = "... a distribution of the core (Movable Type -ed) publishing platform licensed under an open source license."

Nexus Professional = "... a distribution of Nexus (Maven-based repository manager -ed) with features that are relevant to large enterprises and organizations..."

etc.

A complete package should be a Distribution, and an Install Profile is adjective describing, or a feature of a Distribution.

It's certainly possible to have Distributions that are lighter weight than others - particularly if they're only containing things from d.o. Your projects only have some profiles & custom modules (which, by definition, must be GPL, so can be checked into d.o). Therefore, your Distribution could fit under my case 2b2) above: (fully qualified name) A built-by-d.o Distribution that uses the Open Media Project Install Profile.

I was going to stay out of this, but since it appears there is a passionate group willing to work to come up with a safe, secure way to get code outside of Drupal.org's CVS into install profile packages, I just want to warn against the YouTube approach adrian suggested (way back) in #34.

YouTube's policy of allowing users to upload anything and then automatically removing it when the video is flagged has been repeatedly abused. Since this has something to do with the space I work in, I've been following this since the McCain campaign complained about YouTubes take down policy in 2008. Since then, the automatic takedown has been used to remove by a climate change denier, fake grassroots organization, and embarrassed police department to have videos removed.

In many cases the person who posted the video was able to get the video reposted, but that creates work for the person to prove their innocence. This would require someone or group on Drupal.org to review each contested case. This delete first ask questions later policy creates even http://arstechnica.com/tech-policy/news/2009/01/what-fair-use-three-strikes-and-youre-out-of-youtube.ars">more problems for users who don't have time to respond and get flagged repeatedly. Given how active discussions around licensing are on Drupal.org and how few people are really qualified to be making judgments on who is and isn't correctly interpreting licenses, I have a hard time imagining how this would work.

I'm not saying there isn't a way the flag approach could work, but just pointing out that Drupal would need a better process than YouTube. I don't think it's a jump to imagine that if DeanSpace where distributed with a YouTube-like takedown from Drupal.org, someone would have claimed a licensing violation to disrupt the distribution.

@batsonjay While I can see where you are coming from, don't you think that distributions are defined by what is not the same from another version they may be confused with?

The only time I'm even aware of which distribution of linux a server is running is when something I thought I knew how to do doesn't work.

This is how I see OA. There are several things that would normally work with the modules they distribute, but customizations in that distribution have caused issues them. Not to pick on Dev Seed, but from a support perspective anyone using the OA distribution who is posting that an issue to FeedAPI about creating feeds would need to disclose they are using the OA distribution to have any hope of solving the problem.

I'm guessing in most cases, a module maintainer is going to redirect that user to the community supporting the specific distribution for support. This is why it's so important to us to differentiate between simple packaged profiles and distributions.

We really can't afford to have Open Media users with Organic Group issues rejected by default because they are using a "distribution" when they try to get support through the normal Drupal channels. I find it hard to believe that module maintainers are going to invest much time trying to determine which install profiles are lightweight and don't interfere with the module's functionality and which profiles are actually the cause of the issue.

@batsonjay @alex_b I agree that "distribution" is a nice word. It's already out in the wild:

http://drupaldistrowatch.com/

And, once again, there's zero, zip, zilch, nada stopping any of us from hosting our own "Distributions." Acquia does it. Open Atrium does it. When we got 1000's of non profits to try this crazy little thing called "CivicSpace" that's how we did it. I still maintain that a dedicated site, branding, language is what most potential users/customers of a distribution really want and that getting this hosted on d.o is more of a matter of convenience and setting community standards. It's important to remember that anyone can roll their own, and probably should if they think they have a good idea.

It seems to me that what we are hashing out here is how much Drupal.org can do to support this. I think we should do a lot (ala Ubuntu Universe), and I agree especially w/alex_b that the way to think of it in terms of what we now call "install profiles" is that these are core technical components of Distributions. Install Profiles have always been a good idea, but also always been undersupported in terms of what you can do with them within the core. Changing the language on the site is easy enough if we get there; sorting out auto-packaging and ways to do more with them is the pre-requisite we're currently addressing. I think.

@krenyan while I think it's good to anticipate malignant use cases, I have a hard time mapping partisan politics and/or police malfesiance to the world of Distribution flagging. This does not seems like an apples to apples comparison. There could be something like this arising from heated commercial competition, but even that seems unlikely, and the reputation costs for getting "found out" as a misuser would be huge. In any case, it's easy enough not to have the function auto-disable the item, but instead flag for qualified review, and we can handle abuse of abuse-reports the normal way.

@kreynen, IMO, whether or not a distribution provides support does not nullify its distribution-ness. There is no reason an issue with a module/theme/core on an OA site is any different from an issue on any other Drupal site, as OA is not 'hacked.' I don't think by distribution we're talking about allowing people to have a hacked version of Drupal Core in their distribution. Is that what you meant by the mention of FeedAPI?

Based on the conversation started by @batsonjay above, install profiles should live on their own in the same way that modules and themes do. Distributions are our goal here, not altering the nature of install profiles. I'm going to 'be bold' here, and change the title of this issue. I also want to re-iterate what @eaton posted in #54, rewording for distributions instead of install profiles:

In order of priority (IMO), we have:

1. Distributions can include Drupal core (at the very least), an install profile, contrib modules, themes, etc.
2. Distributions should be able to include their own 'custom' code, like distro-specific modules and themes, rather than forcing those elements into separate projects.
3. Distributions should be able to include (via some mechanism) external GPL'd libraries.
4. Distributions should be able to include (via some mechanism) external GPL compatible libraries.

I feel like #1 is actually easily attained, since there are no licensing issues, so why can't we just start there and move down the list?

[Edit: re-worded #1 above a bit]

I think that by using the approach outlined in #84, we would end up with a lot of OpenAtrium_new (or similar) distributions. While it's tolerable with modules, Drupal core is 1MB by itself. If you get a bunch of complex/heavy distributions such as OpenAtrium, I think that we are going to end up with some storage issues. I'm not sure how much hard disk space d.o has available, but I think packaging distributions in this way is a very bad idea.

You can bundle modules and themes with install profiles very easily, without needing to include Drupal core. The modules and themes required by a given profile can be included inside that profile's directory. There's no reason that we cannot just say untar this in profiles/ and run the installer (as we do with modules and themes).

@cweagans, maybe an approval process similar to getting a CVS account? For instance, having to actually upload your distribution for approval before you'll even get access permissions to create distributions on d.o.

So from what I can tell... There are no solid legal issues that would prevent this from happening? Technical/infra issues could be resolved?

Main issue seems to revolve around 'do we have the human resources to maintain?'. If that's the case, I think this already serves as a good first step - http://drupal.org/node/596488

There is a huge amount of untapped funding and human resources outside of what the DA can provide.

Why was the title changed? This issue has everything to do with pulling in code from external sites and how we do that.

Packaged install profiles on d.o _are_happening_ and this is not the issue about them.

@q0rban I think everyone is on the same page regarding priorities. The only question is are we going to use the $10K raised with 1 as the only required deliverable to also take a stab at 2-4. Since dww was open to doing the work on the project end, he established Oct 15 as a deadline for coming up with the policies/spec.

Does anyone still think it's possible to agree on how to make 2, 3 or 4 happen before Thursday?

@greggles, please read the responses above from about #73 down. Don't just assume stupidity without actually investigating it! ;) I'll leave the title as is, but I *really* think it's a bad idea to have install profiles be packaged distributions as well. Very confusing. IMO We *need* to use 'distribution' here.

Whoops, @kreynen switched it back... ;)

Setting title back, as dww said...

@dww, thanks for the clarification

Will d.o provide a mechanism for people to automatically package code hosted off-site into a tarball that lives on d.o for download?

Please re-title if you feel the need, however I just want to make sure we're not packaging code along with install profiles; I want to reiterate that I think this is a BAD idea. Install profiles should be just that: an install profile. No more, no less.

What I think we need to determine is can third party code by added to a distribution and live on d.o. Is that correct?

It sounded like you meant to revert the title but forgot, so I'm reverting it for you. However, based on what you just posted, a better title might be "Provide a mechanism for packaged 3rd party code to d.o".

Lets create a new project type called 'distribution', and add the capability to package third party code into that.

ARG.

I just want to make sure we're not packaging code along with install profiles; I want to reiterate that I think this is a BAD idea

at this time, this is something that's going to happen. it's been a topic that has been discussed for awhile, and we've been trying to get the time and funding to make it happen. now all that's in place, and both the project module maintainers and the infra folks are on board with the proposal.

as dww said, the focus of this discussion should be on whether to include third party code in the packages as part of the initial 'phase 1' effort.

if that's not cleared up soon, then that discussion will need to take the form of 'should we implement this in phase two?'.

Don't get me wrong, I don't have any problem with third party code being allowed in install profiles. For that matter, it'd be great for modules and themes too. However, we wouldn't want modules and themes to include drupal core along with them, would we? Forgive my ignorance, but why can't we redirect all that beautiful, positive energy and funding towards distributions instead of install profiles? This living in install profiles is just plain messy.

Ok, I'm done trying to word things in an understandable fashion. :)

Thanks so much for all your work on this everyone. It's exciting!

but why can't we redirect all that beautiful, positive energy and funding towards distributions instead of install profiles?

so much easier said than done. it's been a battle to get things this far. look at the potential 'analysis paralysis' we've got just on the third party code question! no thank you, sir -- i'm not giving up the alignment we've created this far in search of perfection... ;)

the step we've committed to is a vast step forward from our current situation. nothing is written in stone -- if in X months/years it doesn't serve us, then we'll work to improve it yet again.

and please let's not lose site of the fact that this improvement is *retroactive* -- a rare feat in the drupal world. 5.x and 6.x folks will be able to benefit from this change as much as 7.x+

Just a quick note on disk space: that should be the last of our concerns. Storage is cheap and easy. Making things simple for users is hard and valuable. Burn some cheap/easy stuff to get the hard/valuable thing done makes sense, and as hunmunk points out, is already happening.

but why can't we redirect all that beautiful, positive energy and funding towards distributions instead of install profiles?

Because in Drupal, installation profiles can serve as a mechanism for defining what drupal core version, modules, themes, and code make up a given distribution. As Jay mentioned in an earlier thread, this is a terminology issue, not a technology issue.

By creating a mechanism for Drupal.org to 'bundle up' a distribution based on the metadata in an install profile, we achieve a solid distribution mechanism that applies to Drupal 5, 6, and 7 simultaneously. It's a good, solid path forward.

@eaton, Yes, this is *totally* a terminology issue, and a very important one in my opinion.

Keep in mind that you can have multiple drupal install profiles shipped with a specific drupal core.

So you can have drupal (or pressflow) with 5 or 6 different applications (install profiles) that live on top of it, and sites that are instances of those applications.

Another side effect of building the profiles along with drupal core is that every time either one of them is updated, you need to rebundle the entire thing.

so new drupal security release, have to re-package and re-download however many profiles there are.

Has there been any decisions on whether or not this is going to happen?

I am of course ok with postponing this feature and getting something simpler to work first.

Move along, move along...

So that's it, huh? No explanation?

Aaaand I finally talked to our attorney on this front. (His turnaround time was fine; I was the slow one. Sorry folks.)

Redux:

1) Block all 3rd party code in distros: He's cool with this.

2) Whitelist 3rd party code: This is also safe, legally.

3) YouTube strategy, aka "Hey hippies!" strategy: Our attorney says that this does get YouTube in trouble at times. The "ostrich" strategy wouldn't give us legal protection. So, this won't work as it still leaves us potentially sue-able.

4) Red Hat strategy, aka "delayed download": He's uncomfortable with this one, too. So far, this strategy hasn't been tested in court and is still fairly new, and we're not sure exactly how Red Hat and Canonical structure it, or if it would really work. They just haven't been sued yet that we know of. :-) So this is possible, but makes him uncomfortable. Quote: "True, I am being cautious here, but being cautious may well save some headaches down the road."

So it looks like our game plan is/should be:

1) Deploy first with no 3rd party code, period. dww is already doing this.

2) Add some form of 3rd party packaging that relies on a d.o webmaster-managed whitelist. I'd argue that this should be done at the module level, rather than the package level, so that we can build tarballs for modules that include their 3rd party library. The package builder can just pick that up along with the module. That's an implementation detail I leave to dww, though. :-) How we manage and vet that whitelist is also a job for the webmasters to figure out, especially as much 3rd party code is only available as a tarball or zip itself and therefore needs to be unpacked and moved around post-download. (Unless we can negotiate a direct download URL with each 3rd party? Ick.)

especially as much 3rd party code is only available as a tarball or zip itself and therefore needs to be unpacked and moved around post-download.

drush make already supports downloading and unpacking tarballs/zip files from arbitrary URLs into specified directories, so this problem is already basically solved, it's just waiting on the whitelisting stuff.

The white-list idea is not sustainable. How is that the best course of action?

I think the best course of action is to reconsider the following policy:

Drupal.org's package management system automatically adds the GPL license to all packages. If we allow other licenses in CVS, it is going to get messy, and sooner or later we're going to run into licensing issues. Already, we get quite a few questions about Drupal's license. If we are going to add more licenses to the mix, it is going to be harder to audit, or provide answers to such questions. So, not allowing other licenses in CVS is a deliberate choice.

We've also decided against mirroring other projects in our CVS repositories--unless there are good reasons to do so. So when people need a non-GPL library, it is best to instruct them to download that library from that project's website.

It's obvious there is good reason to "mirror other projects in drupal's CVS repositories" given this 100 post issue.

So why does is the Drupal policy still forbidding this?

Think about it, moderating the CVS repository becomes no more difficult. If anything, it will make it easier because there will be less problems like this. The moderators just needs a simple cheat-sheet of GPL-compatible licenses.

The easiest solution is simple to allow GPL-compatible code in the repository.

This will resolve a majority of the problems module and distribution developers are currently having with libraries. After this policy changes, then other solutions can be presented for non-GPL-compatible code.

You are barking up the wrong tree. We are already allowing GPL-compatible code in CVS (under certain circumstances at least). We have however no intention to blow up the CVS repo beyond a reasonable size by allowing everybody to check in the Linux Kernel sources just in case somebody might want to download them from drupal.org together with a module.

(I'd sure be able to make a Drupal module that would require a kernel module. ;)

Gerhard - According to the policy, GPL-compatible code is not allowed. Only libraries with GPL can be included under certain circumstances. Furthermore, you're being hyperbolic. Revoking this policy is not going to "blow up" the CVS repo. If certain libraries are being used by dozens of modules (which I presume you are afraid will happen), then they should be included in Drupal (like jQuery) or something like Libraries API.

What Gerhard is worried about is something that can really happen.
Approving CVS applications, I have seen people including files available from third-party sites, including Mootools, jQuery and other files that could be easily dowloaded from the final user.

"Easily downloaded" does not apply to everyone, and that is the point here. If you want to drive adoption of Drupal you can't expect new users to download 5 libraries just to get something like image handling to work like it does out-of-the-box in Wordpress.

Like I mentioned earlier, the popular libraries like MooTools should be included in a module like Libraries API. This prevents the problem you for see that 20+ modules will have MooTools duplicated in the CVS. The modules that use MooTools can just make a dependency on Libraries API, problem solved!

For modules like Image FUpload, it makes no sense for Libraries API to constantly update the swfupload library because it is only used by one or two modules. In this case, the Image FUpload maintainer should be able to include the GPL-compatible library (MIT License) in their module.

Think about the size increase, the swfupload library is a whopping 291kb uncompressed. MooTools is 101 kb uncompressed. It would make more sense to ban images than libraries, if you are concerned about "blowing up" the CVS.

Also, there is nothing wrong with advertised that everything in CVS is GPL, except for some libraries. The libraries can even have their own special directory like Libraries API is suggesting, to prevent any confusion as to what is under GPL and what is a GPL-compatible library. People who would even care about licensing conflicts would understand that libraries can't be expected to be GPL.

This seems like a no brainer.
- Option A) Drive adoption with new users by making it easy to install and decrease the time it takes to build new site, or
- Option B) Require everything to be GPL so the minority of people using Drupal who even care about everything being GPL won't be confused by non-GPL libraries (confusion which likely won't even happen if a special libraries directory is used).

If done right, installation profiles have the potential to help end-users get sites done faster and accelerate the adoption of Drupal. However, this promise has yet to be realized, and this core Drupal feature is currently being vastly underutilized. Source

I think the way libraries are currently being handled is preventing this goal from happening.

Require everything to be GPL so the minority of people using Drupal who even care about everything being GPL won't be confused by non-GPL libraries (confusion which likely won't even happen if a special libraries directory is used).

Drupal.org currently allows only GPL licensed code in the CVS repository, not to make people who find other licenses confusing happy, but because there are (or there could be) legal problems with mixing code using different licenses.

Drupal.org currently allows only GPL licensed code in the CVS repository, not to make people who find other licenses confusing happy, but because there are (or there could be) legal problems with mixing code using different licenses.

First, lets not forget, many libraries are licensed with GPL. Why aren't those libraries currently allowed in the repository?

Second, what legal problems will come by, for example, using the MIT licensed swfupload library in a libraries directory that is excluded from Drupal's automatic licensing protocol?

Most of the open source licenses used by libraries are designed to be used this way. It makes no sense to claim legal problems will be the result by including them in CVS.

@philbar:

1) "If it comes out of CVS, it's GPL" is a lot easier for someone's legal department to say is OK than "if it comes out of CVS it's probably GPL, but sometimes it's MIT, but some of it is BSD, and there are a few parts in public domain, but all of that could be made GPL if we wanted to because of those specific licenses, but we not the PHP license because that's not GPL compatible even though it's BSD-esque, so..."

2) If I download the Library API, why do I want mootools coming with it? I don't. I never use mootools. Why should I pull down a mass of 3rd party code (potentially with security holes) that I don't need? Will that also pull down tinymce and ckeditor, as well as 4 different flash video wrappers just in case I need them? That makes no sense whatsoever.

@Crell

1) Out of the thousands of Drupal users, how many have a "legal department". Probably less than 10%, most are too small for those kind of resources. Out of those use cases, the chances the legal department wants to know that all code is strictly GPL is probably less than 10%. So you are making an argument that because less than 1% of the people using Drupal will have a tougher time explaining licensing, we should not make it easier for the vast majority of the Drupal users who think it is a pain to download libraries from dozens of websites to get their Drupal site built.

If you were my legal adviser and you tried to explain to me how Drupal uses GPL-compatible libraries by jabbering the following:

if it comes out of CVS it's probably GPL, but sometimes it's MIT, but some of it is BSD, and there are a few parts in public domain, but all of that could be made GPL if we wanted to because of those specific licenses, but we not the PHP license because that's not GPL compatible even though it's BSD-esque, so...

Quite frankly, I would fire you. Your better than that and you know it. Sounding like an idiot to make a point doesn't help. Is it really so hard to say, "If it comes out of CVS it is GPL, unless it is a 3rd party library, in which case it is GPL-compatible." Again, most people using Drupal just want to know "Can I use this to create my website for free?"

2) I never suggested Library API include every library ever used in Drupal. The concern is that certain libraries will be used by dozens of projects. For these libraries, it is best to include it in one module and make it a dependency than to duplicate them in all the projects and "blow up" the CVS. MooTools may or may not be one such library, I'm just using it as an example.

These modules would work exactly the same as the processing sub-modules in the ImageAPI module. You can choose to enable GD2 or ImageMagick. So explain to my why the Wysiwyg module can't include the TinyMCE and CKeditor the same way. Why shouldn't you be able to enable a TinyMCE library sub-module with the Wysiwyg module?

These modules would work exactly the same as the processing sub-modules in the ImageAPI module. You can choose to enable GD2 or ImageMagick. So explain to my why the Wysiwyg module can't include the TinyMCE and CKeditor the same way.

The clear difference is that imageapi_gd.module, and imageapi_imagemagick.module are two Drupal module, while TinyMCE, and CKeditor are not two Drupal modules. The module ImageAPI doesn't come with ImageMagick, which must be installed separately on the server.

@philbar: Do you have a source for those numbers or are you making them up? While most Drupal-using institutions may not have a full army of lawyers, anyone who uses any software professionally *should* care about licensing and legality. Just because some (I will not speculate what percentage) are ignorant of it does not mean we should be.

That also does not address the "it creates a fork of the 3rd party code" problem, which is true regardless of the license involved.

While most Drupal-using institutions may not have a full army of lawyers, anyone who uses any software professionally *should* care about licensing and legality.

Yes! And I agree! The problem I have with what you've been saying is that you've been building strawman arguments to support your decision. The level people are concerned about licensing and legality (of which, I am one) is not to the degree that warrants the need to keep everything in CVS under GPL.

Can you think of one use case that would be crippled if GPL-compatible code is included in the CVS?

And you shouldn't think of this in terms of a "fork". It a "redistribution", because the version of the library has been finalized, and the library is just being used as intended. The jQuery library is included in Drupal, do you consider that is a "fork" or a "redistribution"?

I feel bad about hijacking this thread, so I've created an issue specifically dealing with libraries in CVS:

http://drupal.org/node/682642

Let this issue be turned back to the discussion to include a white-list for the drupal packaging script.

Can you think of one use case that would be crippled if GPL-compatible code is included in the CVS?

I don't know the answer to the question, but I wonder why GNU would be building an OS using only GPL licensed software. If they did such thing, then there must be a reason, and that reason could be the answer to why only GPL code is allowed in Drupal repository.

To get this issue moving forward, here are some URLs of common libraries:

TinyMCE
http://downloads.sourceforge.net/project/tinymce/TinyMCE/3.2.7/tinymce_3...

MooTools
http://mootools.net/download/get/mootools-1.2.4-core-nc.js

jQuery
http://jqueryjs.googlecode.com/files/jquery-1.3.2.min.js

SWFupload
http://swfupload.googlecode.com/files/SWFUpload%20v2.2.0.1%20Core.zip

For libraries hosted on SourceForge or Google Code, the white-list should verify at sub-doman or sub-directory level for security concerns (anyone can start projects at these locations with malicious code). The self-hosted libraries should just be verified at the domain level.

A logical white-list for the above examples would be:

http://downloads.sourceforge.net/project/tinymce/
http://swfupload.googlecode.com/
http://jqueryjs.googlecode.com/
http://mootools.net/

Can someone code a way to check the drupal-org.make file against this type of white-list? Also, how is it going to be kept up-to-date?

jQuery is already included with Drupal, and you cannot load two different versions of the library (considering that Drupal already loads it); Mootools is not compatible with jQuery as it defines its own function $().

jQuery may need to be updated, which requires the library download: http://drupal.org/project/jquery_update
jQuery can also be run in compatibility mode with MooTools: http://drupal.org/project/jquery_compatibility_mode

The problem is that Drupal doesn't run it in compatibility mode, and you cannot replace a version of jQuery with another one without to be sure to not create problems to the scripts used by third-party modules. That is why jQuery Update used a compatibility script.

Don't killer the messenger. I'm just letting you know that people can and do use newer versions of jQuery and MooTools despite jQuery being included in drupal core.

For reference, I add the following issue to Drush.make's issue queue.

#684788: Verify Library URLs against a White-list for drupal-org.make

We still need to figure out where the white-list will be stored, and who will maintain it. Let's keep that discussing going on this issue, since it is regarding Drupal.org infrastructure.

At the technical side, we need to make a decision on how the white-list will be formatted. If we decide who will be the official maintainer of the white-list, we can see what their preference it in the other issue (#684788).

Who will maintain the white-list?

As I suspected...it seems we are having a difficult time finding a maintainer.

I will volunteer to maintain the whitelist.

After talking to dww, hunmonk, crell and dries at DrupalCon SF, I've broken out an issue to create a whitelist:

#779452: Whitelist for external dependencies

(also see "Install Profile Packaging" community initiative page: http://drupal.org/node/779440)

A whitelist content type has been created on d.o:

http://drupal.org/node/779452#comment-3207134

Once we have the white list for profiles, maybe we can extend it to individual modules?

for example, w.r.t: #862192: Allowed to include GPL font with contrib module?

I've communicated with cweagons and philbar and unless they are any objections they will monitor a new component on drupal.org infrastructure queue and then create white-list content types.

We need to finalize the criteria for accepting libraries onto the whitelist. As suggested in #40, it seems logical for 2 criteria:

1) The code must be GPL Compatible
2) The code must be less than... (perhaps 2GB...)

I heard size was less of an issue, but we should get someone from the Drupal Association to give some insight into this issue.

That's really an infrastructure team question, not DA question. I defer to killes and nnewton on that front.

Branching so important parties can decide without reading #140 comments of this issue.

#982634: Decide on a Size Limit for External Libraries

ok... I REALLY don't understand. Why are we discussing committing non-drupal code to cvs? This all seems to stem from Crell's desire to do whitelisting at a module level instead of for distributions in general. I think this is a REALLY bad idea. The beauty of the drush_make solution to distributions is that we don't have to worry about CVS at all... if there's something bad in a particular distribution's release, we just kill that release, and suspend the distro from making new releases until the problem is solved.

Allowing 3rd party code into CVS is just asking for unnecessary headaches IMO, and it's really not related to the original question. As I stated previously in this thread, downloading and installing modules should be the realm of those with a bit of expertise. Installing distributions should be the realm of those who want to follow a wizard and get consistent results. One requires downloading 3rd party code, and manually installing (and understanding what that all means) and the other does not. We shouldn't confuse these two systems, they are not the same, and we're asking for trouble.

Eclipse

@dww

Yeah, this is a misunderstanding on my part of the implications of what Crell said in 107. I just finished a discussion with killes in irc, and he said it was never explicitly handled in this thread because the infra team would NEVER go for per module 3rd party packaging. This is a transparent issue to you all, but it's pretty opaque to non-infra people. My post was simply regarding that because I generally feel that's a really bad idea.

As far as the benefits of drush_make creating full distro releases, I'm about 10000000000% on board there. That's never been the issue for me, the issue I was addressing was modules getting 3rd party packaging per crell's statements. For the record, I wrote Feature Server's distribution handling via drush_make almost a year ago now, so you don't need to explain those benefits... I "get" it.

Eclipse

@EclipseGc: Indeed the Install Profile Packaging page does not describe the functionality of whitelists well. I've added these two sentences:

A whitelist would allow specific external libraries to be added into a distribution. External libraries would not be committed directly to the distribution's code base, but tied in via a drush make manifest.

Hm. This is the first I've heard of the infra team being absolutely against integrate-on-tarball-creation for modules in addition to distros. Admittedly with the increased use of libraries api that may be a non-issue since you couldn't do that anyway, but I was never informed that infra was solidly against it.

Or maybe it's just killes is solidly against it; I don't know. :-)

In any case, yes, we're full-speed-ahead on getting whitelisted 3rd party integrate-on-tarball-creation functionality in place for distributions.

Sub.

I will say that allowing any GPLv3 compatible licensed code on the whitelist sounds like a great practical compromise. The list at http://www.gnu.org/licenses/license-list.html#GPLCompatibleLicenses is cut and dry. It also allows us to say that the whole of any distro downloaded from d.o. is GPLv3, although some code may also be available under other open source licenses.

re: #145 - where is there opposition for using it for modules? I think that's going to be a need at least equal to profiles.

Subscribing.

Subscribing.

@pwolanin

I'm still against packing 3rd party components into modules, but I do wonder if you couldn't get the same sort of thing working by using recursive git clone. Has anyone tried that? It might "just work" today. Of course you'd need a git repo for the 3rd party components as well, but I think it might get you what you want.

This isn't about any tricks with git, nor is it about putting code into module packages. This issue is about third party libraries being packaged with install profiles/distributions. Can we please stay on topic for this issue? Once we get the third party stuff written and deployed for install profiles, we can look at doing the same for modules (pending a decision from killes/infrastructure team people).

Tagging.

subscribe

Subscribe.