Disparity of Versions: Towards Creating a Sustainable Solution to the Disparity between Drupal Core and Contributed Modules

Hi,
This post is created to promote a rational community discussion regarding the release system we use. It was born out of a conversation at DrupalCampNYC5 in which about 8 of us got together to discuss why we hadn't been launching sites on Drupal 6 yet.

We know this will probably raise some 'ghosts of Christmas past', in which similar conversations have went south. We're not interested in that. No one in this is demanding anything, we just want to have a conversation and garner feedback.

In comments, please avoid anything that isn't giving input. No one needs to see '+1', or 'I disagree' without supporting rhetoric.

That said, a group of us created this article of the past week. We then opened it to a small group for feedback, and later edited some more based on that.

We like to think we all put a lot of thought and care into it. However, it is a conversation starter, not a definitive white paper. We're all open to having our opinions changed. Please let us know what you think.

===============================
Disparity of Versions:
Towards Creating a Sustainable Solution to the Disparity between Drupal Core and Contributed Modules

"This is not about core moving too fast, but about core and contrib moving away from each other."

Define the Problem

Summary:

* The current release of Drupal 6 is not being implemented by much of the community due to the unavailability of certain contributed modules, despite it being a great core release on its own.
* As Drupal core versions, and core grows, the community becomes spread across several versions (There is active development on the full featured release Drupal 5, the new release Drupal6, and the upcoming release Drupal7). This is natural, but should be considered in releasing.
* Part of the conceptual development of a core release is dependent upon contributed modules interaction with the previous release. Drupal 7 is in danger of becoming a 'de-facto fork', because Drupal 6 has yet to have the time to interact with many contributed modules.

There are a set of modules that are not in Drupal Core, but which are critical for fully featured production sites to function. Many critical contributed modules exist in isolation to core, such as CCK, Views, Pathauto, Token, Node Queue, Image, File Field, and more. Critical modules affect core. So long as several of those modules do not have a stable release, Drupal 7 cannot take input from that interaction.

It would be easy for this discussion to descend into finger pointing at certain functionality, modules, and people. THAT is NOT the point. If that is what you want to do, move on, please. Were it not Views or Token, it would be NextBigModule Charlie, or NextBigModule Omega. The issue we are attempting to describe is a flaw in how we determine the viability of a Drupal Core release as it pertains to the current contrib environment at any given time.

Only now are we beginning to see a hint of the first major sites migrating to Drupal 6, and even that is tentative and taken with the understanding of the risks involved in using pre-release versions of several critical modules. Over half a year after the release of Drupal 6, there is a general feeling that these sites are early adopters.

Additionally, many developers have openly stated they plan to wait for a Drupal 7 release, and to jump directly from 5. This has become a commonly accepted practice, and at face value it would seem there is nothing wrong with it. However, anyone who planned on doing that for 4.7 -> 6.0 has been stuck on unsupported, potentially insecure software for close to a year now. You can easily say that they can upgrade to 5, and they could, but either skipping a version is a valid standard option or it isn't. Having this constantly be a judgement call indicates the need for some standardization in our release cycle. This is not a new issue, but, it appears, the first time versions have spread out to encompass wide use of three Drupal's.

The developer community loses cohesion as versions spread out. While many developers are concentrating on upgrades and trying to keep up with development they do not have time to participate in the development of core. While this is a natural process, it is also detrimental to a community that stresses participation.

The reality is that the current release of Drupal 6 does not provide all the features that professional site developers have come to expect when building high feature sites. Drupal 7 is in real danger of becoming a de-facto fork, because it is simply impossible for development work on 7 to be informed by the community's use of Drupal 6.

For various, highly valid reasons, it is not advantageous to inhibit development in any area. The issue is not the pace of development, but rather the disparity of compatible versions of core and contributed modules. There is no pride in a release for the sake of a release. There is only pride in releasing the next best Drupal platform.

Define a metric (how do we measure the problem?)
Summary:

* Rate of Adoption by Deployments: Seven months after the release of Drupal 6, we still don't have a stable release of many "critical" modules preventing many sites from implementing this essential upgrade.
* Rate of Adoption by Developers: Many developers are still hesitant, or even unable, to use Drupal 6 for existing projects.
* Ratio of previous contrib stable to current contrib stable: Currently less than 44% of contributed modules with stable 5.x releases have a 6.x release. 7 is set for code freeze in January.

If we are going to acknowledge a problem, the first aspect is to define it, and the second is to assign a metric so we know both the extent of the issue, and a way to define concrete guidelines to avoid the problem in the future .

However, as the disparity of Drupal core releases and Drupal contrib releases grows deployments of latter releases become inversely proportional. That is to say, the core release with the smallest number of compatible contrib modules is also the least used core release. We need to keep an acceptable Rate of Adoption by Deployments. We need to conceive of release criteria that foster the adoption of new Drupal releases, rather than viewing them with dread.

Similarly, we need to maintain a good Rate of Adoption by Developers. If Drupal is indeed a developer centric community, a release that has a slow rate of adoption among those developers is a red flag of the highest proportion.

Finally, if we accept that core Drupal is benefited by its interaction with the contrib repository, then it follows that a future release of core needs to be dependent upon some/certain modules being compatible with a previous release. A core version that marches ahead adding new features without this input in effect is blazing a trail, not collaborating towards an end.

Possible solutions that are self-correcting and sustainable
Summary:

* There are many potential solutions, not exclusive to each other. We don't have an easy answer.
* Potentially include more things in core, Not a long term solution.
* Create a "Supported" tier of modules, a level between Core and Contributed that directly influences the release cycle of core.

As a community there is a set of things that need to be nurtured and supported, not only by the community, but by the people defining the direction of Drupal. We can't tell a developer what to do. But we can (and do) call a code freeze, thus telling them that if they continue in a direction, it may not be supported later. So, some development guidelines do get traction.

We don't have a good solution. We can suggest some course of action, but REALLY want to have an open conversation. These are merely starting points for a broader, rational, community discussion.

Some have said, "The disparity of versions problem would be solved if this or that module were part of core." Although such an action may have merit on its own basis and should probably happen, we believe it is not a long-term solution, and is actually irrelevant to the problem. There are cases where our fellow developers have decided that they can contribute their work most effectively by keeping a module out of core so they can retain more control over a module's development path -- we respect and see no reason to second-guess their decision.

Adding new modules to core would, by virtue of the inevitable increase of complexity of core, slow down future releases of core, but there will always be a new contributed module that suddenly becomes essential to production sites, thus continuing to create a disparity between core and contrib. That is to say, this is a temporary solution and not a systematic future solution.

One possible solution, following the lead of several other Open Source projects, including many distributions of Linux, would be for the community to create a "Supported" tier of modules, as a level between Drupal Core and Contributed modules. The current work flow states that at a certain point in time, a code freeze for core is in place for a stated amount of time, to give time for contributed modules to catch up with core. There are emails sent to encourage people upgrade in a timely fashion.

The system could change to make this more explicit, defining that a certain critical set of modules must also have a stable release for that version before the final Drupal core is released. Given the track of several Drupal related shops and businesses we are seeing a supported tier of modules naturally develop already. Making that an official community policy would not be out of place.

We are used to a release, followed by a lag until certain contributed modules are ready. This model would lessen the ensuing lag time, as much of the essential contrib functionality would be ready on release. It would also require that some criteria be defined for a core RC to let the small contigent who doesn't require the contrib tier to know when an RC is ready.

The community could choose a method to vet which critical modules will be considered to be in such a 'Supported' tier. What method gets chosen does not matter so much, as long as the method is explicit and intrinsically tied to the release cycle. Recognizing the problem, multiple companies have already stepped up to the plate and publicly offered to do this job of selecting a set of critical modules to support. Their are many vetting methodologies we could use to define such a Supported tier and it is outside of the scope of this paper to define one. We understand that there is a group actively looking into various rating/voting methodologies [ http://groups.drupal.org/module-metrics-and-ranking ] and we look to them to participate in this discussion and inform this part of the debate.

For this to have any real meaning, we would finally need to alter the work flow for future Drupal releases. After the announced release candidate, we would not release a final core version until all (or a large majority of) Supported modules had stable releases. This code freeze may or may not have an associated duration; we need to continue to maintain the flexibility of responding to the demands of a current situation. However, we might further state that if a Supported module is not stable (or perhaps with a release candidate) within a certain time frame, then it will automatically be relegated back to Contributed status until such time as developers decide to give it the needed attention.

Such a methodology could exist side-by-side with other proposed methodologies, such as modules popular amongst the larger community, company-sponsored (or otherwise funded) modules, and those pinging from an enabled Update Status module. In fact, ALL of these methodologies already publicly exist in some form or another. The aspect that is lacking is a community policy acknowledging that there exists an essential group of contributed modules on which the effective deployment of Drupal core in most environments depends.

We, the signatories of this document, believe that the disparity between Drupal Core and its Contributed modules has become prohibitive to the effective deployment of the latest Drupal release, and that steps need to be taken to bring them back into balance, without hindering the pace of development. We believe that this is necessary to the image and future growth of acceptance of Drupal, and for sustaining an exponentially increasing adoption rate of Drupal. We believe that our proposal to create a Supported tier of modules, if adopted by the Drupal community at large, would help to solve the current disparity of versions, while at the same time encouraging the essential fast rate of development of both Core and Contrib.

Aaron Winborn
Eric Goldhagen
Nat Meysenburg
Joe Golden
Kris Vanderwater
Oleg Terenchuk
Matt Corks
Sam Tresler

This document was born out of several conversations had at DrupalCampNYC and a small working group that interacted for a week after that.