Packaging: document whitelist

Agreed. But this is just a drupal.org-specific documentation task, not something for Project* itself. Moving to a more appropriate queue. Also, since this is blocked on other issues, setting status to postponed.

Thanks,
-Derek

Yup I think you moved this and the other 2 to the right spot.

Alex - are you just filing these so they're in the queue (ie. mental note and intending to write it yourself) or will you need Docs team folks to work on this? Just wondering as I'm not sure how many docs team people will actually be up on this stuff enough to manage these issues...(I'd have to do some recon to find helpers).

ok, awesome - lemme know if you need any additional support!

We need to create some guidelines before we start creating a whitelist. I think using this documentation page will be the best location so we don't need to duplicate content. Here is a start:

Including Third-party Libraries

NOTE: This functionality is still being developed. Get Involved!

Due to legal concern, the Drupal Association has decided to restrict external, third-party code inclusion in Drupal distributions hosted on drupal.org. A whitelist has been created to allow approved third-party code to be packaged with install profiles. Below is the criteria used when approving third-party code inclusion:

1) Must be licensed under the GNU General Public License, version 2 or later.
2) ...

Many projects are already included on the whitelist. If one is missing that you think should be there, create an issue in this queue.

I'm not sure the GPL restriction will be true. We need Larry and the legal team to comment on that. So long as it's an open source license, I think it's fine. Our GPL policy is specifically about code that lives in our version control repositories. I believe we can still redistribute non-GPL code that talks to our GPL code. I know not all of the 3rd party code people might want to include is going to be GPL, so it'd be nice to not enforce that restriction if we don't have to.

Are we going with GPL-compatible? Will we prohibit incompatible licenses?

Also, will this be the only criteria?

Actually, I don't remember the details. Someone should re-read #594704: Allow packaged install profiles on d.o to pull in code from other sources + sites and summarize the decision there, since all the stakeholders are already subscribed over there...

New criteria:

1) GPL-compatible.
2) Less than 10mb compressed.
3) Not legally questionable or malicious.

We just need to get the wording right.

sub

adding tags

So, this has been hanging a while out here. Nobody objected to the summary in #10, so I guess we can treat it as agreed on.

To be safe, I just opened #1360456: Finalize the criteria and process for the whitelist for external dependencies packaged with Drupal distributions in the infra queue to get final clarity on the criteria and process. Once that's resolved, we can reopen this and prepare the actual documentation.

Thanks,
-Derek

Also, I just added this to the summary:

Note, there are actually a few different things that need to be documented about this for potentially different audiences:

• Criteria for inclusion on the whitelist (distro maintainer, d.o admins)
• Process for requesting something should be added/removed to/from the whitelist (distro maintainer, d.o admins)
• Details on how to actually maintain the whitelist itself (d.o admins)
• Docs on how to reference 3rd party code when building a distribution (distro maintainer)

Should all of those be handled here, or do we need to split those out into separate issues?

Thanks,
-Derek

This can now basically happen anytime. Although distro maintainers can't make use of this just yet, we can already document the process for getting on the whitelist, and the instructions for the whitelist maintainers, since #779452: Whitelist for external dependencies is now done and deployed (as is #1360460: Populate initial team of 3rd party packaging whitelist maintainers for that matter).

Where in the structure of the docs would we put the instructions for whitelist maintainers? Here's my proposed text:

Maintaining the 3rd party packaging whitelist

Packaging whitelist maintainers have the following responsibilities:

• Subscribe to the packaging whitelist issue queue.
• Ensure that libraries mentioned in the issue queue conform to packaging guidelines*.
• Add a packaging whitelist node (node/add/packaging-whitelist) for the libraries mentioned in the issue queue, if they have met the packaging guidelines and have been RTBC'ed and approved by a module maintainer or a whitelist maintainer.

*Guidelines are currently being finalized in #1360456: Finalize the criteria and process for the whitelist for external dependencies packaged with Drupal distributions; when that issue is complete, the guidelines will be published here.

Adding a packaging whitelist node

The packaging whitelist is comprised of nodes containing the title of the project, and a regex pattern of all potential library download URLs, separated by newline. As an example, for jQuery UI, the regexes are:

^http://jqueryui\.com/download/.+$
^http://jquery-ui\.googlecode\.com/files/jquery\.ui.+$

There are two entries, because jQuery UI can be downloaded either from a packaged file on jqueryui.com, or from the source repository at googlecode.com.

In the above regex, the carat (^) signifies the beginning of the line, all periods (.) that aren't part of the regex formatting need to be escaped by placing a backslash (\) in front of them, the period-plus (.+) signifies "and anything after the preceding string", and the dollar sign ($) means the end of the line. Each pattern should be separated by a newline.

I've posted the above (with slight revision) to http://drupal.org/node/1404994, marking as 'needs copy/style review'. The other thing that should be documented sooner rather than later would be how install profile maintainers can easily add libraries via the drupal.org drush make file... I haven't done this myself, so I'll try looking up the docs on this soon if nobody beats me to the punch.

Now that this is deployed (see #1365536: Switch distribution packaging system to use just drush core and drupalorg_drush), documentation would be quite helpful for the actual implementation in a .make file. I'll try to get this going in Open Parish, and update the docs for install profile maintainers.

Someone suggested this somewhere, but it would also be pretty awesome to have a story for drupal.org, or planet, or somewhere similar, announcing this feature... Because I know a lot of distributions could be fully hosted on Drupal.org now, instead of off-site.

Update: drush make documentation available here: http://drupalcode.org/project/drush_make.git/blob_plain/refs/heads/6.x-2... (specifically, the Libraries section and Patches under modules...).

Okay, of the original issue summary, here are some updates/notes:

Criteria for inclusion on the whitelist (distro maintainer, d.o admins)

The final criteria are still being discussed in #1360456: Finalize the criteria and process for the whitelist for external dependencies packaged with Drupal distributions, but that discussion is wrapping up. Where should these criteria be posted, at the end of the discussion?

Process for requesting something should be added/removed to/from the whitelist (distro maintainer, d.o admins)

Where should this process be posted? Basically, we could just have a simple note on the 'packaging a profile' page saying, "To request changes or additions to the whitelist, please visit the Drupal.org Library Packaging Whitelist project page, and read through the instructions there."

Details on how to actually maintain the whitelist itself (d.o admins)

Discussion is ongoing here: #1404492: Add documentation on how to add whitelist entries. I've posted an initial page of instructions: Maintaining whitelist for external project dependencies, and it's a good start. I think we can check this off the list.

Docs on how to reference 3rd party code when building a distribution (distro maintainer)

I've added an example in the sample make file on Packaging a profile on drupal.org. (Both for 3rd party code, and for applying a patch to a project).

clarifying exactly what whitelist-related docs we need (and the target audience for each)

@geerlingguy: Thanks *so* much for jumping in and trying out the new functionality yesterday. I was so happy to see you had successfully made a new release including a library. ;) Thanks also for diving in on the documentation issues.

To start to answer some of your questions on that:

- Re: criteria: I'm not sure, I'll have to look around a bit to find the best place. This page has two target audiences -- distro maintainers and whitelist maintainers, so it should be "near" where distro maintainers look and linked to from where maintainers look.

- I love the instructions on the packaging whitelist project page. That seems like a great place for those. So yeah, just a simple link from Packaging a profile on drupal.org seems like that's all we need.

- Yes, I'll definitely do a big announcement, but my plan was to do this as a bit of a "soft launch" for the first few days, just to make sure nothing unexpected comes up, first. Plus, I'd really like to get -dev support finished off before making too much of a big stink about this, and that's going to be at least another couple of days before it's ready. Finally, there are some PR constraints on such an announcement -- a group of shops pooled resources to fund a lot of this work, and I'm still waiting for the final list of sponsors to thank. ;)

- Thanks for the issue and docs for maintainers. I replied there. Yes, that's now fixed, so I crossed it off the list in the summary.

- Thanks also for the example of libraries in one of the .make files over at Packaging a profile on drupal.org but I think we'll need more extensive docs than that. Perhaps it's worthy of a whole new section on that page. Or, *gasp* maybe that page should be split into sub-pages, since it's getting a bit massive and I think important info is probably getting lost.

Or, *gasp* maybe that page should be split into sub-pages, since it's getting a bit massive and I think important info is probably getting lost.

Yeah, the first time I read through Packaging a profile on Drupal.org, I was a bit overwhelmed with all the information included. Maybe have a brief overview of the make file and what it should be named, and at a minimum, move all the discussion of automated makefile creation and other more advanced things (e.g. things that a small install profile maintainer probably won't do) off to a child page.

tagging the Distributions issues so the issue triage avoids closing this issue

crossed off docs for whitelist maintainers since that now lives at /node/1404994

Thanks to recent doc edits by myself and geerlingguy, this is now completely done. I just updated the summary with links to where all the relevant docs live.