Last updated April 22, 2012.

Since December 1st 2009 we've had distribution packaging working on drupal.org - yay!

1. External dependencies

Modules can require a user to download external libraries like jQuery UI, CK Editor or Simple Pie and ask her to place them into very specific places in a site's directory. This is a small and acceptable hurdle for modules, but install profiles are geared towards a different audience. A given Drupal application can potentially contain various external libraries in different places in Drupal's directory, and placing them correctly and troubleshooting any misplacements can be difficult for the Drupal novice. Moreover, linking a specific library version hard to an install profile will allow for making sure that a given installation, if not modified, contains a given library version, which facilitates debugging.

Proposed solution

There is consensus that rolling offsite libraries into distributions is a good idea, however there was a heated debate on whether it would be legally advisable to do so. The conclusion was that a whitelist solution would be acceptable. A whitelist would allow specific external libraries to be added into a distribution. External libraries would not be committed directly to the distribution's code base, but tied in via a drush make manifest.

There are 3 classes of things we need to tackle to call this resolved:

Whitelist

#779452: Whitelist for external dependencies
That issue is about actually building the whitelist plumbing on d.o itself.

  • Ensure that the existing packaging whitelist nodes have all the fields we need to do their job.
  • Display the whitelist in human-readable form to distribution maintainers (and potential maintainers) so they know what 3rd-party code is currently allowed.
  • Display the whitelist in machine-readable form (e.g. JSON) so that drush_make can enforce the whitelist during packaging.
  • Create a feature including the whitelist content type, fields, views (html, json, rss) and related code for deployment and ongoing maintenance.

#1406902: Provide taxonomy or field for recording packaging whitelist license details

Packaging

#684788: Verify Library URLs against a White-list for drupal-org.make
#779996: Add support to record and display external dependencies included in release packages
#1447558: Make fails for library archives with no file "extension"

Process

#1360456: Finalize the criteria and process for the whitelist for external dependencies packaged with Drupal distributions
#783186: Packaging: document whitelist
#1360460: Populate initial team of 3rd party packaging whitelist maintainers
#1404492: Add documentation on how to add whitelist entries

Also, once we know what external code is packaged with each distribution release, we can provide views of projects + releases with filters for 3rd-party code so that the Security Team can find all the affected projects when upstream security problems are reported and we need to do security update releases and publish SAs. (TODO: turn this into an issue).

2. Patches

Building a Drupal site almost always requires patching a module that is maintained by somebody other than the site builder. This is no different when building an install profile. Moreover it's not possible to wait while patches are committed upstream before releasing an install profile and packaging its dependencies. Even very obvious fixes to a module can take too long to be committed to their respective module.

drush make (the drush plugin powering profile packaging on d. o.) supports adding patches to modules in the package:

projects[ctools][version] = "1.2"
; http://drupal.org/node/728508
projects[ctools][patch][] = "http://drupal.org/files/issues/ctools_plugin_static_cache.patch"

This features effectively allows for decoupling upstream (core or contrib) fixes from downstream (profile) releases. This is important as it frees profiles from waiting on upstream fixes and it encourages contribution to dependencies.

Proposed solution

#779460: Allow dynamic application of patches to dependencies
#779998: Add support to record patches applied to release packages
#1366482: Fix distribution packaging plugin to record patches and libraries included in distributions
#1365446: Add support to display patches applied to release packages
#783218: Packaging: document dynamic application of patches
#1455614: Packaging script doesn't allow distributions to patch core or specify a git revision
#1433784: Fix how drupalorg_drush handles release history XML and propagates data to the packaging script
#1472744: Change packaging script to use the new JSON metadata from drush make for package contents
#1469714: Fix validation to support drupal-org-core.make files and prevent defining the drupal project in drupal-org.make itself
#1476226: Add support to display patches applied to remote items (libraries)

3. Themes and modules in installation profiles

Most Drupal sites have one or more custom modules containing "glue code" specific to the site's use case. The situation is very similar with themes. This is again, no different with install profiles. Currently, there is no other way of adding a custom module or a custom theme to a packaged profile than committing it to contributions/modules (or /themes) and thus including it like any other module or theme. This practice pollutes an already constrained namespace which should be avoided.

Done!

Proposed solution

Allow modules and themes to be committed to installation profiles, discourage users to commit contribution modules or themes directly.

#779476: Remove commit restrictions for installation profiles
#675162: Second copy of modules directory after drush make run
#783244: Packaging: document custom modules and themes in profiles

4. Development snapshot releases (-dev) in distributions

This item covers changes to allow distributions to package development snapshot releases of the modules and themes they include. This is necessary when upstream project maintainers do not ship official releases that include necessary bug fixes or new features when the distribution is ready for a new release. Currently, -dev releases are prevented from being included in profiles, but the work as outlined here would lift this restriction going forward.

However, packaging from a development snapshot produces instability for the distribution maintainer, since it becomes harder to know exactly what upstream code was included in a given release of the distribution. This can be resolved by the following:

#1314180: Make dev release numbers useful
#1371298: Rewrite .info files inside drush make when checking out from git to include version and project info
#1371306: Add validation to ensure that if a .make file includes a git hash, it also defines a branch
#1371302: Remove restriction on -dev releases in distributions
#1401990: Record a 'rebuild_version' for release nodes that are rebuilt
#1430888: deploy drush 5.0-rc1 and latest drupalorg_drush for allowing -dev snapshots in distribution releases
#1440412: Confusion between drupalorg_drush and d.o packaging script over how to specify 'core'
#1474778: .info file re-writing is broken for core

Other Measures

Workflow and tools for distribution maintainers

#1476026: project_verify_package should optionally validate drupal-org-core.make files
#1476022: project_verify_package should enforce that 'core' matches the version of the release node
#1475956: Remove web UI for verifying and converting drush make files
#1427752: Support drush make includes[] in drupal-org.make files

Deployment-related

#1365572: Remove drupal.org-specific code
#1365536: Switch distribution packaging system to use just drush core and drupalorg_drush
#1365538: Deploy drupalorg_drush and latest drush for distribution packaging system
#1475882: Create jenkins jobs for deploying drush and drupalorg_drush for d.o distribution packaging

We will need to communicate these significant changes to the packaging infrastructure and make sure that profile maintainers are aware of them:

#783288: Packaging: Display message to profile maintainers
+ Write up an explanatory announcement in the style of "Fully packaged Drupal distributions now deployed on drupal.org" http://drupal.org/node/647374

Also:
The Drupal Security Team should have a discussion about our process for SAs when 3rd party code vulnerabilities are published.

See also

Comments

Hi -

In building & packaging Drupal Commons I've noted a couple of other things I'd like to suggest need covered:

  1. Statistics. It's useful to have the normal usage statistics reports that Modules have available. This is somewhat linked to the next item:
  2. Module vs. distribution distinction. Some distributions (Commons included) will often have distro-specific code. For Commons, we put this code into a module (called commons_core for now in our case; this may actually get busted up into multiple modules as we proceed). So we have both a module, and a distribution with similar names, but slightly different purposes. Certainly the distribution probably needs a distinct name/path on d.o - e.g. /project/commons - but does the module embedded in the distro want its own project separate from the distro - especially if the module is mostly ineffective without the rest of the distro? The reason for a module is that the easiest way to track usage may be to actually track a key (designated?) module from the distro - e.g. commons_core. It also may be that it's useful to have patches you can apply to that module (a'la Drush above). These are open questions, so we should probably create some packaging standards / best-practices.
  3. Features? As we proceed, Commons (like OA?) will likely be adding new clumps of functionality in the form of Features. How should we handle that on d.o? I know this is a topic of some discussion in the past; it would be good to figure out how the solution might (or might not) relate to d.o.

I may think of more (and Mike Stefanello may have his own thoughts, too). But these are the first to come to mind.

------------------------------------
Commercially supported Drupal
Co-founder, VP, retired
acquia.com
------------------------------------

Re: features, putting them on d.o. could be really problematic.

I don't know if anyone has discussed this yet, but the big issue would be with namespace conflicts. There could be a lot of distros with an events feature and having to map out a unique namespace for your features module could be an issue. If there are suddenly 15 features all with some variant of event in the name, that is going to become very confusing for people trying to reuse other people's code.

One thought would be to let features simply be part of a distribution, i.e. that nodes are structured in a way where you can:

1) Upload a distribution, module or theme

2) Create a feature that is attached to that item

3) When someone views the original item, you get a list of features associated with it

This would probably be most useful for distros and could solve the issue of people having to map out specific namespaces.