TL;DR

  • There are several issues with core development as it stands: not innovating fast enough, insufficient commercial incentive to contribute, contributor burnout, and unpredictable release cycles.
  • We need to think about how to solve these not only for core developers, but also contrib developers, technical site owners, and non-technical site owners. They each have a different set of needs and challenges.
  • In addition to starting from Larry's core conversation on the Drupal release cycle, we evaluated other projects' approach to release management, and attempted to create a framework that could work for our community, which entails:
    • Release new minor versions on a timed schedule: every 6 months, until the final minor release, release a new minor version that includes mostly-non-BC-breaking improvements and additions.
    • Only support the latest minor release of any given major version.
    • Have a single LTS release per major version, which is the final minor release of a major version.
    • Provide both security and regular bug fixes to this LTS release until 3 months after the next major version's (e.g., 9) LTS release is available, and continue providing security fixes until 3 months after the following major version's (e.g., 10) LTS release is available. This adds more work for the security team (e.g., Drupal 6 would be security supported until 8's LTS rather than only until 8.0), so we'll need to figure out how to make it possible.
    • Only branch the next major version of Drupal when there's something big enough completed to justify it, and these will probably continue to release every 2-3 years.

Introduction

In #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x, there has been a lot of talk about Drupal 8 adopting semantic versioning, where version numbers and the way they change convey meaning about the underlying code and what has been modified from one version to the next. While we do certain classes of changes and additions during stable versions now, there's a lot of things we don't touch until we do major versions, in part due to limitations of our current 2-digit version scheme, and in part due to a culture of shifting focus to the next major version right away.

I believe semver is a great framework to adopt. The tricky part is in the details. I propose we look at this problem through the eyes of four personas.

Personas for Drupal core

Persona 1: Drupal core contributors. They'd like to see more frequent releases to increase the velocity of innovation happening in core, and to reduce the time commitment required to get innovations into the next release. Right now, we couple innovation in our admin UIs, markup, APIs, and internal implementations of subsystems to huge releases that re-write half of core. Not only does this result in a multi-year cycle to get new functionality released, but it also makes development of even minor functionality and UX improvements difficult because it requires constant patch rerolls as things shift.

Persona 2: Drupal contrib maintainers. Contrib authors may come across core limitations that either block their porting or development progress altogether, or require sub-optimal workarounds that are difficult to maintain. The faster these get resolved, the better the entire Drupal ecosystem is for it users. On the other hand, contrib maintainers also require a stable core API to work against so that once they release a stable version of their project, they don't need to worry about it breaking with every minor core update. This means we need to be very careful in what we do and do not break between minor releases.

Persona 3: Users without ongoing development resources. A lot of people get their Drupal site up and running and pretty much stop spending time or money on it, either because they don't have the resources or because they don't want to. Other than dealing with the occasional security updates, they don't want to be bothered upgrading themes or modules. Ideally, Drupal would expand the Update Manager feature to automatically upgrade Drupal core from one minor or patch version to another, to allow for both innovation and security releases to be less disruptive to this audience.

Persona 4: Users with ongoing development resources. Some users have maintenance contracts, internal staff, or sufficient free time to improve their site over time, to upgrade Drupal modules, etc. These people are willing to both chase and contribute to the latest functionality if it benefits them, but don't want to do so at the cost of a major, multi-thousand dollar upgrade.

How to balance these audiences' needs?

Let's look at some of the details of the Drupal core release cycle if we adopt semver:

  1. How strictly do we preserve backward compatibility (BC) between minor releases?
  2. How often do we release new minor versions?
  3. How long do we support the previous minor version once a new one is released: i.e., do we provide an overlap window to give people time to update?
  4. Do we mark one or more releases as a "long term support" (LTS) versions of Drupal? If so, when and for how long?
  5. How will contrib authors and security team members be able to cope with more frequent releases of core?
  6. What happens with major versions?

With these four personas in mind, I think the best way to go is with:

The below text in chart form.

  1. Preserve BC fairly strictly. Not 100%, but close. We should only allow for trivial BC breaks (e.g. breaks that affect a very small number of contributed modules or themes), to minimize impact on the personas that cannot easily fend for themselves. This means that the three main approaches to innovation are: (i) providing a backward compatibility layer (e.g., if moving a particular hook to an event, then continuing to fire the hook as well), (ii) providing a separate, alternative implementation (e.g. core could ship with an experimental field UI as a separate module) and (iii) adding completely new systems (e.g. a new entity or field type could be moved into core).
  2. For predictability, release a new minor every 6 months. To start with, I suggest 3 months of code thaw, followed by a 6 week beta period followed by a 6 week RC period. With time, I would like to lengthen code thaw and shorten beta and RC if we find that we are able to do so. Note that even during the code thaw period, every patch committed to an already released major version's branch must not knowingly introduce any release blocking follow ups. It's okay to introduce nice-to-have follow ups that if completed would make things better, but if not completed, the next minor release can still happen on schedule.
  3. No support overlap window for minor releases. Once a new minor is released, drop support for the old one. So long as only trivial BC breaks are allowed and we provide guidelines for people on how to mitigate them, there's no reason for anyone to stay on an old minor. Site owners unwilling to keep up with minor updates should wait for the LTS release. However, the initial (.0) patch release of a new minor version must not itself contain security fixes that haven't been backported and released to the prior minor; this might in some cases require releasing a final patch release for the prior minor on the same day as releasing the new minor (e.g., release an 8.0.6 on the same day as 8.1.0, but from that point on, no longer support 8.0).
  4. Make the last minor release, and only that one, the "Long Term Support" (LTS) release, fully supported until the next major version's LTS plus an additional 3 months to allow for migration time, and security supported until the LTS after that plus 3 months. In other words, site owners can migrate from 8 LTS to 9 LTS and continue receiving all bug fixes, or from 8 LTS to 10 LTS, but during the last couple years that they're on 8, only receive security fixes.
  5. To help ease contrib porting, we could hard freeze the code of a new release (only allow commits that fix release blockers) during the RC period of 8.X.0 (and clearly communicate this) to provide contrib a window to update their code if necessary prior to site owners updating to the new release.
  6. Approximately 1 major version every 2-3 years. For major versions, we would hold off opening the next major branch for committing patches to until that is needed. This might mean, for example, that 9.x isn't opened until after 8.2 or 8.3 is released (i.e., 12 to 18 months after 8.0 is released). This enables patches to Drupal 8 to be committed to Drupal 8 directly, without needing to first be committed to Drupal 9 and then backported to Drupal 8. However, this means that people working on major rearchitecture of Drupal during this time (e.g., #1447712: Evaluate Symfony form component as a replacement for Drupal FAPI and #1843798: [meta] Refactor Render API to be OO) must do so in sandboxes. Integrating multiple such initiatives or proceeding with mass conversions to the new architecture requires 9.x to be opened, so we'll open it when such initiatives are far enough along to warrant that. Once we open that branch, we would also freeze the currently stable major and begin preparing its final (and therefore, LTS) minor release. Therefore, in order to not have too long a time go by with no innovations in core, we would also target releasing the next major version within a year or less of opening its branch.

What is the practical application to these audiences?

Given these set of recommendations, and nearly complete backward compatibility, here is what it means for each of the personas.

Drupal core contributors: Core developers can improve core and use those improvements on real sites faster, so long as they (mostly) preserve BC. More frequent, less backwards-compatibility-breaking releases of Drupal core would allow us to move culturally to a more agile development approach of getting smaller changes out faster, seeing how they do, and making further changes based on real-world data. This move also potentially allows us to build core more sustainably, by giving core developers the option to tie some of their development to billable client work, which can only happen if meaningful changes are released more often than once every three years.

Drupal contrib maintainers:
Contrib maintainers can expect their modules and themes to keep working with new minor releases of core, most times with no changes needed at all, sometimes with new integration code needed for whatever new implementations are added to core, and very rarely with some small fixes needed to accommodate a trivial BC break. There will be at least one release candidate for each minor release to allow contrib authors to test their code in advance of site builders updating their sites.

Users without ongoing development resources: We will need to provide guidelines to these users, such as either stick to LTS releases or:

  • Don't use custom modules; stick to only contrib modules that are adequately maintained. Or, if your site requires custom code, make sure the developers you hire know to limit their code to only accessing APIs marked as stable (for example, if $form arrays were marked as unstable APIs, don't use hook_form_alter()).
  • Since markup of admin pages might change in minor ways, stick to core/contrib admin themes; don't use a custom theme.
  • Other guidelines TBD.

So long as these site owners stick to these guidelines, they should be able to update to new minors as easily as updating to new patch releases.

Users with ongoing development resources: These site owners can use custom modules and themes, as well as contrib projects that they help maintain, to the extent that they're able to keep them up to date with minor core updates. Since only trivial BC breaks will be allowed, this will not require too much of their resources. More of their resources would be needed if they want to use and integrate with whatever new features/implementations are available in the new release, but they can decide which of those to use and when.

How does this approach compare with other projects?

We compared and contrasted release strategies with a number of different projects: WordPress, TYPO3, Joomla!, Firefox, Ubuntu, PHP, and Symfony. All have a substantive release anywhere from every 6 weeks (for Firefox) to 1 year (for PHP), with most at around 6 months. The projects differ in whether or how they provide LTS releases: Ubuntu does so once every two years and supports them for 5 years, while WordPress only provides security support for their latest release, but tries hard to always preserve BC, even in major releases.

The proposed approach attempts to take the best ideas from these projects, and put them into a framework that can work for our community. The approach of mostly BC-preserving but innovative releases every 6 months maps closely with most of the projects studied, providing a single LTS release as the final minor maps closely with Joomla!, and supporting an LTS release until after two additional ones are released maps closely with Ubuntu.

Next Steps

  • Gather feedback on this proposal. If generally positive:
    • Evaluate the current backport policy and determine how it should change for 8.x+ (separate issue).
    • Develop a plan for what infrastructure changes are needed to the Drupal.org packaging system, update status module, Drush, etc. to allow for this move (separate issue).
  • Else, iterate and come up with an amended proposal, then do the above. :-)

I think it's important for us to hash this out now that Drupal 8 has positioned us via its new architecture to support iterative innovation, and hopefully grow the core development team as well.

Questions? Comments? Please provide feedback about this proposal.

Big thank you to catch, alexpott, webchick, Crell, effulgentsia and xjm for helping me create this proposal, and to all the participants on #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x, whose comments have also influenced this proposal.

Files: 

Comments

This is phenomenal, fantastic, and many other superlatives. 100% support from this contrib guy, developer, and client supporter. Hooray for a grown-up release cycle!

Having known and short release cycles will relieve some anxiety from the front end folks. Building a usable UI requires numerous iterations and we really don't get those over large major release cycles. We can run tests, but these aren't the same as having real users undertaking tasks. We wouldn't be compelled to change everything about the UI all at once for fear of missing our window. Big changes often lead to epic bikeshedding and at worst, inaction. It's easier to reach consensus through multiple consecutive small changes.

It has a lot of sense.
Those projects that don't need constant evolution will be happy with the Long Term Support (LTS) version.
Those projects that are pushing the envelope, can contribute back easily to Drupal core, since releases are faster.
And I've seen a lot of good feature requests with the reply. "Nope. Wait until D9, 2017". That's depressing. With a faster release cycle we can be more innovative and use feedback instantly.
Thumbs up

There's one problem here: starting the "security support extended until LTS" with 6.x. Don't try to do that. 6.x has no test suite. Supporting 7.x until 9.x LTS might be harder but supporting 6.x is not harder, it's near impossible.

I'd tend to agree with chx, actually. 6.x core is barely maintained as it is now because of how risky it is to add fixes, and contrib has already moved away from 6.x overall. It's not just that it would take significant resource investment, it's also that the risk of breaking existing sites with fixes is very high. One could argue that it's preferable for sites to be broken than have security vulnerabilities, I guess, but I'm not sure site owners everywhere would agree. We should probably have some overlap for 6.x support, but I don't think we are ready to drag it out all the way to the 8.x LTS.

I do think it is appropriate for 7.x and releases going forward to be supported until the LTS, though.

This decision could help us deliver on Drupal 8 faster too. See my first bullet point.

Next Steps

Today, We could:

  • Create a version for the drupal project called 8.1.x and start triaging issues to that version that aren't going to be ready for launch and that fit within the above's guidelines for non-BC breaking patches.
  • Plan ahead for work that we want to get done during code thaw and get all the bikeshedding done up front.
  • Work on bigger pieces that take longer than the 6 month dev cycle by breaking them up into smaller deliverables and get them in over time.
  • Use the minor cycles to update libraries and handle any fallout that might occur as a result.
  • Survey application devs to find out what they want the platform to do
  • Start theming are releases on different names for "blue" / "water" / blue things

*thumbs up*

I just wanted to note that it might be nice to name releases after rivers.

Yeah I really like this idea. I also agree with @chx in #4, maintaining a codebase like D6 without any tests for another year? Seems almost an unreasonable task for anyone.

@irk, O yes, a naming convention per release is an awesome idea!

There's one problem here: starting the "security support extended until LTS" with 6.x. Don't try to do that.

One alternative to this that was discussed with catch is if we make 8.0 or 8.1 a special release (e.g., "8.0-transitional", name can be bikeshed later if the idea is what we go with), and to support that until the 8 LTS. However, this results in two things:

1) Site owners migrating from 6 will need to do more work: i.e., migrate from 6 to 8.0, and then a year or two later, update from 8.0 to 8.3 (or 8.4 or whatever the LTS one ends up being). This second update will be much easier than the initial migration, but still, it adds some burden to site owners.

2) Contrib maintainers will need to either be able to use semantic versioning there as well (see #1612910: Switch to Semantic Versioning for Drupal contrib extenstions (modules, themes, etc) for discussion on that), or will need to have annoying branching logic in their modules. For example, suppose that between core's 8.0 to 8.1, we change the Views UI settings forms (admin/structure/views/settings) such that we move the "Live Preview Settings" container out of the "Basic" tab, and create a new "Live Preview" tab for it (I'm not saying anyone actually wants to do this, just making up an example). I think this is the kind of minor UI iteration that we want to in principle allow, but it does break BC (trivially, in line with Dries' proposal), because suppose there's a contrib module, Foo, with version 8.x-1.0, that adds a live preview setting by hook_form_alter()ing the 'views_ui_admin_settings_basic' form. Now Drupal 8.1 comes out, and Foo needs to change this to altering the 'views_ui_admin_settings_live_preview' form instead. If Foo releases this as 8.x-1.1, but if Drupal 8.0 is still supported, then we have a problem if a security bug is found in Foo 8.x-1.0. So one approach is to allow Foo to release such a fix as Foo 8.x-1.0.1, which means we need to settle on #1612910: Switch to Semantic Versioning for Drupal contrib extenstions (modules, themes, etc). The other approach is that when Foo updates its code to work for Drupal 8.1, it must continue to also work for Drupal 8.0, which means adding various if/else statements where needed.

Possibly, both of these are a less cumbersome option than extending D6 support though.

As a core and contrib developer I like the idea of possibly allowing BC breaks in minor versions if they amount to useful improvements. Also, mostly as a contrib developer, I am very very much against it. There are too many modules that are useful, but do not receive any or only little love after their initial stable release. BC-breaking changes to core, how trivial they may be, may break modules or render them sufficiently useless for a certain target audience, and there problems may not be fixed for a long time.

Well, we need to define what's acceptable as a BC break and what is not. As effulgentsia nicely lays out, almost any front end work might break form alters. Emphasis on might. I would cautiously say that we allow for this. hook_form_alter is great but we make front end improvement impossible if we don't allow for these kind of rare breaks. And frontend is crucial because Drupal 8 backend is so botched up our only hope is that the frontend will sell it to the CxO people and then the developers will suck it up.

Beyond that, the usual additions I guess are a go. What else?

One possibility we discussed was having a whitelist of "high-sensitivity" forms that we treat as almost an API, because we know they're common targets for form_altering. Most of them, however, particularly in the admin, we treat as changeable if usability can be improved by doing so. The node add/edit form was the main one we could think of to whitelist.

Also, entity forms are now pluggable via the Entity API, so we could ship a totally redesigned entity edit form that is enabled only on new installs but the old one is used on existing installs until you flip a switch somewhere. That would let us try out new and fancier form designs without immediately breaking existing sites and custom modules. People could easily move up to the new form if they wanted to/when they felt comfortable doing so (which could be never, potentially). I don't know how many other forms that would be possible for, but the new entity API does allow for a fair bit of experimentation of that sort.

The point about D6's maintainability is valid. Even if we don't have multiple D8 LTSes, could we extend D6's support period at least part way into D8's lifetime? Even if it's just one or two minors (6-12 months), that's a relatively low risk for us (let's face it, D6 doesn't have that many security holes left after this long) but really good karma for users on D6 who want to move to D8 but can't do so immediately. We can then move to the full "LTS jumping" approach starting with D7/D9. chx, would that be workable do you think?

Interesting read. Questions/comments:

  1. What is the actual difference between a "minor release" in this proposal (e.g. Drupal 8.1.0 or 8.2.0) and a point release in Drupal 7 (e.g. Drupal 7.22 or 7.23)?

    Based on the description in the proposal ("Once a new minor is released, drop support for the old one" and "We should only allow for trivial BC breaks (e.g. breaks that affect a very small number of contributed modules or themes)") it sounds like the same thing to me.

    Some of the comments here suggest that we'd start allowing UI changes that have the potential to break hook_form_alter(), but we already do that in Drupal 7 (example: #606598: Human readable image-style names).

  2. One difference does appear to be the release schedule (once every 6 months, rather than the once-a-month release window in Drupal 7, any of which might contain API additions). However, the point of the current system is to allow flexibility in the case where a release is really needed. For example, @webchick did frequent releases immediately after 7.0 while Drupal 7 was still stabilizing.

    If we find that some API isn't working well after Drupal 8.0 is released and needs a rewrite (a near-certainty based on past experience) I wouldn't want to wait 6 months for that.

  3. The LTS idea is interesting and a good way to formalize the current "this version of Drupal is old so let's stop trying to rewrite it" thing that currently develops (painfully) over time in the issue queue. But I'm concerned that as written it might have the opposite effect as intended - in particular, that it might cause people to have to wait longer for core innovation than they currently do.

    To see why, let's pretend that we already did this now. With Drupal 8 originally intended for a fall 2013 release, the LTS version of Drupal 7 would probably have come out in late 2012 or early 2013. Issues like #606598: Human readable image-style names (June 2013) would have therefore been rejected as too late for Drupal 7.

    Meanwhile, Drupal 8's release cycle stretches on and on, doesn't get released until well into 2014, and by the time contrib catches up people don't really use it on production websites until sometime in 2015.

    Bottom line: Innovations that could have made it onto production websites within months instead get deferred for years.

    To mitigate this, I would suggest that perhaps the LTS release should not coincide with the opening of the next major branch, but rather happen much later. We could use Drupal 7 as a guinea pig for that; it's certainly not in LTS mode now, and practically speaking won't be for a while.

  4. Calling this "semantic versioning" is probably incorrect. If the official policy at http://semver.org were being followed, then something like Drupal 7.20 (a security release with big API changes) would have necessitated Drupal 8.0.0 (or Drupal 9.0.0 if it happened during the Drupal 8 cycle). I'm pretty sure this issue was already raised on one of the other threads.
  5. The impact of this proposal on the security team would be less severe than many other proposals, but still substantial. Currently, the number of people who regularly work on Drupal core security issues can probably fit inside a small car (and hope the car doesn't crash)... it's really a dire situation. Unless people start volunteering in droves (and volunteering to work on backporting patches to older versions of Drupal, which is the last thing most people are interested in) this would probably require money to solve.
  6. Overall: Let's be honest, the reason this discussion is hard for Drupal is that the project has evolved with flexibility as one of its core goals - specifically the idea that everyone should be able to alter everything Drupal core does. That philosophy means that pretty much any substantial change is going to be a "BC break" of some kind.

    There is no code-level magic that will ever change that. It can only change if we decide that Drupal should be less flexible (or at least, less flexible in the kinds of ways it has traditionally been flexible). Certainly other projects have chosen to deliberately limit core flexibility in exchange for faster core innovation... WordPress has done it and has been extraordinarily successful. Backdrop is aiming to move in that direction also (at least a little bit). Drupal could try it too.

    But if we announce some kind of big new release schedule without having that discussion, what will actually happen is that people will get frustrated (either because the changes they wanted to see get into core in the next 6 months didn't get accepted, or because other people's changes did get accepted and wound up breaking their website, or both).

I don't think small API changes in minor core releases will be a large burden on contrib. Websites that follow the minor releases are likely to have ongoing development resources available, and these can be used to upgrade contrib.

It might be necessary to tie contrib releases to certain core releases if they are not compatible with older versions, do we might need to start using 8.0.x-1.1, 8.1.x-1.x-dev etc.

As a contrib developer I might be wary to call a release "8.x-1.0" if it might perhaps be incompatible with Drupal 8.1.

Another question here according #16 how fixes like 'image style itok' correlate with LTS - they looks like secure bug also affects a lot of contrib by breaking api

In general I think the "persona" approach is pretty broken, for the simple reason that it loses sight of who the real customer is.

The real customer isn't persona 1 or 2 -- it's persona 3 and 4.

Persona 3 is the most critical audience, as far as I can see. But it's actually not that different from Persona 4 in the critical details -- having dealt with both 3 and 4 customers for years, I can tell you that 4 can be just as cheap as 3 when it comes to paying for stuff they don't understand why they should need.

I'll give you an example: Right now, I'm being assailed by the account team at my agency for the fact that all the D6 sites we've built since D6 release now "need" to be updated to D7 or D8, at substantial cost to the client (since the themes are not transparently portable and any reasonably sophisticated site won't survive a major-version upgrade intact).

So we're having to tell clients who spend $10K/year on marketing that they're going to have to budget $10K JUST TO KEEP THEIR SITE CURRENT, with no obvious benefits.

It kind of pisses them off, as a rule.

So, persona 3 will ALWAYS want either LESS-FREQUENT release cycles or longer support time frames. Persona 4 is almost always going to want the same thing, because they're liable to be overworked and don't want to have to waste time worrying about the bloody website when they should be spending it working on securing their infrastructure.

One thing I want to emphasize here: The goal is *not* to have API-breaking changes every 6 months. It's to separate "API breaks" from "feature additions". Upgrading from 8.0 to 8.1 or 8.2 should, if all goes according to plan, be no less stable and reliable than upgrading a contrib module that fixed a few bugs and added a new feature or two. If anything, probably more reliable than a typical contrib because there are so many more eyes on it and the test coverage is so much more complete.

So the maintenance cost of upgrading from 8.0.3 to 8.1.0 should be only marginally larger than going from 8.0.2 to 8.0.3 in the typical case. If it's not, then we're not doing it right. :-)

API changes that are mandated by security fixes would stay the same: We really really try not to, but if there's no alternative then security trumps all.

To mitigate this, I would suggest that perhaps the LTS release should not coincide with the opening of the next major branch, but rather happen much later.

There wouldn't be an immediate release, but it would mean that the next release is the LTS one.

This does leave a frozen period where the LTS is out, and 9.0.0 is not released yet. I think the only way to make that work is to have the 9.0.0 release be a lot more focused than major releases previously have been (i.e. we wouldn't need to cram lots of new features in, because they could go into 9.1.0). Or like you say trigger LTS a bit later.

Overall: Let's be honest, the reason this discussion is hard for Drupal is that the project has evolved with flexibility as one of its core goals - specifically the idea that everyone should be able to alter everything Drupal core does. That philosophy means that pretty much any substantial change is going to be a "BC break" of some kind.
There is no code-level magic that will ever change that.

This is less true (although still true for lots of things) for Drupal 8.

For example entity fields can only be accessed by methods on the entity interface and controllers, which are determined by an interface - you can no longer do $node->body[$langcode][0]['value'] to access it, nor set it straight on the object either - which means the internal data structure of a node object is no longer part of the API for accessing entity fields. We also require that querying entities happens via EntityFieldQuery rather than raw SQL, and in fact don't allow any direct interaction with storage at all - not that core has 100% caught up with this, but it's getting there.

This means that a change to the internal structure of nodes no longer becomes an API change. Or if that's exposed somewhere (storage controllers or similar), it's an API change only for implementors of those specific APIs - not any module doing anything with $node as it would be now.

On patch-level vs. minors. If there was an API 'bug' that completely blocks a contrib module port, then we might release an addition or similar for that in a patch-level release, but yes I'd expect there to be a lot less of these, and instead those to go into the minors. In general, waiting for a minor is not going to mean a 6 month wait, on average it'll be a 3 month wait if an issue is fixed in the middle.

Where we find that changes are impossible without API changes, and it's not possible to provide any kind of backwards compatibility layer, that's the sort of change that would get queued up for the next major version - and the more of those in the queue, the more likely that branch gets opened. The main differences compared to what we have now:

1. We don't open 9.x until there's a very good reason to (there was really no reason to open 8.x for about a year or so after 7.x was released)

2. Anything that can safely be committed to 8.x until 9.x is open can go directly into 8.x - no backporting, and no dealing with widespread API changes.

3. If there are changes that can't be safely made against 8.x, then those become the first, high-priority items to resolve for 9.x. For example, we might want to change how form altering works so there are operations on specific elements in the form, and on the form structure as a whole - but not the entire form structure and specific elements all at the same time.

For me that's the important part of this change. There's going to be thousands of issues in the queue once 8.0.0 is released, and many of those can be resolved in 8.x without the need for API changes. Delaying the 9.x branch until a specific set of well thought-through and partially implemented API changes are queued up means that API-agnostic changes can be worked on considerably faster - and it'll be much clearer which changes have to be done in a new major version rather than at any point.

The LTS releases and semantic versioning parts of this are much more about predictability and standardization, but they're details for me rather than the actual meat of the proposal.

Before 7.x opened, a lot of issues were moved to 8.x because they were 'features' or API additions, that were then backported to 7.x anyway 18 months or 2 years later. This gives us some clarity about what actually is and isn't in scope. There is more clarity about that now for 7.x than there was when it opened, but this is partly about documenting some of those changes that have already been made over the past couple of years too.

I am -1 as it stands today.

Reiterating what I said a few times in #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x.

Any proposal that increases work for security team (especially for D6, but even for D7+) without addressing the workload of the security team is a non-starter from my perspective. It's not really the security team as a whole, but specifically people working on core security issues. If we say "The people in MAINTAINERS.txt need to fix security issues in their area" I'm open to that - the answer doesn't have to come from the security team.

As it stands, this proposal does increase work for the security team and doesn't seem to address that.

On security team workload, there's one slight mitigating factor (only slight though).

Right now, 7.x and 6.x security issues are fixed in private. After an SA is released, a public 8.x issue gets opened - https://drupal.org/project/issues/search/drupal?text=&assigned=&submitte... has most of the recent ones I think.

With this change, once 8.0.0 is released, then 6.x, 7.x and 8.x issues would all have to be worked on in private by the security team. However there would be no 9.x branch for the vast majority of the time that 6.x is supported - so very few issues would have to be forward-ported publicly. Also 8.x should have a more stable API after 8.0.0 than before.

In terms of the private workload of the security team, this is worse.

In terms of the overall number of branches that security issues have to get fixed in regardless of public or private, it's only worse for a short period (the period when 9.x is opened but 6.x is still supported - assuming that stays).

Once 6.x is dropped after 9.x is opened/8.x LTS is out, then it ends up about the same as now (7.x and 8.x in private, 9.x in public).

In practice, there's a lot of crossover between the individual people who work on the private 6.x and 7.x issues as members of the security team, and the 8.x issues in public. So the majority of extra workload is having to do SAs for three branches, and having to fix issues in three branches before an SA, instead of two before and one after publicly.

That's a lot of extra work still, just want to be clear on what exactly the extra work is.

I do think 6.x is going to be (and is now) considerably harder to backport fixes to from 7.x/8.x, than 7.x will be from 8.x/9.x - because of both lack of automated testing, and things like PHP4 support, but agreed those are two slightly different conversations. And if we really want to go ahead with an extended 6.x support cycle then it has to be tackled as a real issue in and of itself.

I think this is a great discussion forward, but agree with pfrenssen that we *must* have a plan for contrib versioning with respect to core compatibility.

ad API changes - I think we miss a definition of what's treated "officially" as API - as that's required for people to know what this incorporates or not. Form and render arrays are obvious ones that need clarification here, but moreover we should be clear on whether this entails non-interface public, protected methods in various implementation classes, etc.

I think that the proposal would be difficult for persona 3, the typical build once, but have security support as long as possible site. For those having to upgrade minor releases to a LTS is definitely a big burden as every subtle change might break certain customizations. Imo that's ok though, as long as the LTS schedule is predictable and people can decide to wait for that - so it really should be.
I must say basing the LTS release date on when we branch the next Drupal major version seems to be a bit arbitrary to me. I see the possible shift of focus for core development, but a) the does not mean people stop working with the previous version - not at all, b) the LTS release date stays unpredictable and c) the necessity for branching a new major version does not mean the previous version is ready to go to LTS, if so I see it more as a coincidence or a matter of the time that past meanwhile.
Thus instead, I think we would better of with a fixed LTS release once the major version came out; e.g., define upfront that the minor version 18 months after initial release becomes the LTS release.
That way persona 3 can have a look at the nice and shiny Drupal X and decide to go for it *now* as it knows the extra-burden of minor release API changes will be X times - or not and wait.

Then, imo this proposal requires #1612910: Switch to Semantic Versioning for Drupal contrib extenstions (modules, themes, etc) to work out. Modules need an easy way and consistently (=automatic) marked way to declare releases compatible to a certain minor release version, such that e.g. drush could respect that and could offer commands for automatically updating all patch level updates - but skip update A from module B which only works with Drupal's next minor releases.

The other approach is that when Foo updates its code to work for Drupal 8.1, it must continue to also work for Drupal 8.0, which means adding various if/else statements where needed.

I'd assume there will be situations where this is not possible - e.g. hook parameters got changed but you can the define the function just in one way? (Well without doing conditional functions, but instead of that I'd much rather use use branches.)

Lastly, I think we need at least a small timespan of additional security support for the past minor release, e.g. I could 1 month working out. Updating might not be completely trivial on a rather big site, so *having* to take care of that in only 1 week to be able to roll out the next security updates at the next security release window is imo nothing we can/should require.

And if we really want to go ahead with an extended 6.x support cycle then it has to be tackled as a real issue in and of itself.

#2136029: Decide if and how to extend D6 security support 6-24 months past an 8.0 release

StatusFileSize
new4.64 KB
new2.1 KB

What is the actual difference between a "minor release" in this proposal (e.g. Drupal 8.1.0 or 8.2.0) and a point release in Drupal 7 (e.g. Drupal 7.22 or 7.23)?

I'd like to respond more fully to this soon, but for now, I'm attaching extractions from D7's CHANGELOG.txt. Starting from 7.12 (1 year after 7.0), we began noting changelog entries containing API/data/UI/markup changes and API additions, so the attached files contain all of the entries that were noted as such. I know that prior to 7.12, we made quite a lot of changes without noting it (D7 adoption was still low then anyway), but I don't know how many BC affecting changes were made since 7.12 that weren't marked as such and therefore aren't in these files.

D7 adoption was still low then anyway

Strike that. Looking at https://drupal.org/project/usage/drupal, I see that D7 adoption was already at 300K sites (reported) by Feb. 2012. Pity our CHANGELOG.txt was less detailed for prior point releases.

Would really love someone to answer the points raised in #16 (namely 2. 3. and 6.). People having to actually wait longer for core innovation than they currently do would be a pretty unexpected side effect.

Semantic version is (from http://semver.org/)

Summary

Given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make incompatible API changes,
  • MINOR version when you add functionality in a backwards-compatible manner, and
  • PATCH version when you make backwards-compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

@David_Rothstein:
This sounds exactly like what we're proposing. We are saying that if we make API breaking changes we'll do everything we can to do that in MAJOR releases with only the exception of required security fixes overriding that rule.

We plan to introduce new APIs in MINOR releases

And simple bug fixes (that are BC compliant) in PATCH releases.

With that definition of what semantic versioning is, let's consider @anavarre's concern of slower core innovation.

1. To be clear we're not talking about bug fixes. That can happen as PATCH releases and these can proceed at our current pace (of once a month?).

2. The proposal states that a MAJOR release that introduces new functionality would have a controlled release schedule of about once every six months. Given that we're scheduling a release such as this rather than continuing our current policy of releasing new APIs if everyone who is working on core is OK with that, yes, it's possible that means that core innovation happens slower for those cases.

But on the other hand, the focus of core development shifts from "OH the last version is done so I can't extend it anymore without having to fight a 20 hr a week political battle because adding APIs to core is rare and I need to make my case. So I'll just work on the next version which isn't going to be out for another 3 years." to "I have a great idea on how to make Drupal better so I better get it done in 3 months so it can be ready for the next major release".

Maybe we can even get to a point where we don't ever need to break APIs unless Symfony breaks them. Imagine being on Drupal 9 for 5 years and still being able to advance the state of the art for Drupal. Wouldn't that be nirvana?

IMO projects like the Chromium team, Firefox, are successful solely because of their rapid pace of development. The have a good balance and communication between application and platform developers. We need to evolve our development practices to keep pace with the speed that the web and the world is evolving. We also need to be realistic.

With our current rate of development perhaps a six month cycle for major releases is as fast as we can possibly go while making good progress. It would be cool if we could put together some small teams and try this out. Maybe we will find that short dev cycles (say 3 months) still work well for developers. Maybe we will find that shorter dev cycles don't work for customers.

As we proceed it would be wonderful if we applied our newly mighty UX skills to discovering where the sweet spot for this is.

anavarre: cosmicdreams already covered this a great deal, but I want to add one more point. Right now, we mentally couple "improving the system", "major release", and "big API breaks" as if they have to always come together. That's not true. That's an artifact of our development cycle and architecture. We need to unlearn that association.

Drupal 8 already largely addresses the technical part of that. Adopting non-breaking feature releases, and a period in which that's all you can do (because there is no breaks-OK branches open), helps us make the transition culturally. That's why I advocate for this approach.

This whole discussion presumes that we do not want to throw out half of D8's APIs the moment we can. I do not concur.

By working with D8 longer, we should have a more concrete and data-driven idea of what we want to "throw out" come Drupal 9 rather than just emotional reasons. And since it should be easier to have BC layers in D8, that means we can also experiment with and signal what the APIs will look like in the future so that modules can get a head start on it.

It's a move to gradual and planned evolution rather than periodic "do anything and we'll see what survives the crucible by being sufficiently stubborn" process we have now.

So I didn't see any comments on this bit from greggles:

If we say "The people in MAINTAINERS.txt need to fix security issues in their area" I'm open to that - the answer doesn't have to come from the security team.

The extra burden on the security team is one of the main sticking points of this proposal. Many people from MAINTAINERS.txt are active in this thread. Is this something you'd be willing to take on? Chime in here: #2139769: [Policy, no patch] Block 8.0.0 and possibly minor versions on critical security issues.

One more perspective here to clarify and expand on #35: in general the security team's goals (which I think of as defining the scope of our responsibility) don't include writing code to fix issues. The team's official goals: try to educate people to avoid security issues, receive issue reports and facilitate fixing/releasing, and help with security of d.o. That's it. If a contrib maintainer goes for too long without fixing a bug we generally just say "c'est la vie" and mark the project as unsupported. Obviously we can't do that with core.

Historically the security team has worked to write code and fix bugs, along with the core maintainer, for things in core. I think that worked reasonably well in the past but my sense is that it is no longer as feasible.
1. Core is more complex, which makes it hard for either the ~40 members of the security team or the core maintainer to know it all and be able to work on it all.
2. The role of core maintainer has shifted more toward gatekeeper than code author, so they are less involved in fixing the issues than they used to be. Just like the public core issue queue, I think we need the various system maintainers (and anyone else who is currently doing work in that area) to help write the code.

Basically I bristle when I hear statements like "the security team can't handle the workload" because to some extent we've been doing more than our charter says we should. This is a bit of semantic debate, especially since many of the people who work on core security related issues in public are the same people doing it in the private queue. I just think the more entrenched the perspective of "security issues are fixed by the security team" the more we'll have problems.

In short, I propose folks update (or solidify) their perspective: The Security Team facilitates fixes, but it is ultimately up to the maintainer to write/test/release.

I'll throw in my two cents here.

While the proposals for an LTS version might be interesting for D7 and beyond, keeping D6 around, even in the LTS format, doesn't seem to make any sense to me. I work with predominately small sites, many of which are still in D6 and are facing challenges mustering the funding to do the upgrade to D7 so in theory I should be happy to hear of an LTS version for D6. However, I share the concern expressed by Greggles in 22.

I think that the service the community, and particularly the security group, provides to this project is already remarkable. Asking everybody there, as well as core maintainers and let's not forget contrib module maintainers, to keep an eye on two past versions of Drupal seems to be asking a bit much. The PHP 4 issue for D6 is also important.

There is another issue with D6, although I do not know how much weight to give it, in that I have been a campaign for much of the year convincing site owners to upgrade to D7 on the grounds that support for 6 will stop. I am sure that I am not alone in doing that. The sudden switch to an LTS version might leave certain smaller site owners with a bad taste in their mouths, realizing that they didn't have to fund an upgrade to D7 after all. Again, I am not sure what weight to give that in the bigger picture, but it should be given some consideration.

Drupal 6 will itself be six years old in February, which is already twice as long as Ubuntu LTS (three years). It seems to me that that is long enough.

I do not yet contribute yet (although I hope to start in 2014), so perhaps my voice carries less weight, or my arguments miss certain key elements that should be attributed greater weight, but from my perspective, I think it is too late in the game to propose an LTS version for D6 that would not sunset until D9 is released.

Actually, those are good points, and thank you for sharing them. :) The one about clients feeling swindled is a new one that hasn't been brought up yet, I don't think. You can call 2013 the year you started contributing. :D

One small clarification though is that under this proposal, 6.x would not be supported until Drupal 9.0.0. It would only be supported until there was an LTS release of Drupal 8 (which is currently estimated to be 12-18 months after 8.0.0 vs. what will most likely 3 years+ after 8.0.0 until D9.0.0, if current trends continue). And, it would be supported in basically a "life support only" capacity: security fixes and that's it; not even bug fixes, and certainly not new features.

Therefore, even if this proposal goes through, you are still very well within your rights and being completely forthright to recommend your clients get off of Drupal 6 before Drupal 8 comes out. The wider Drupal community has most certainly moved on from Drupal 6 even now, and so things like support, custom feature development, etc. will only get harder and harder (and/or more and more expensive) to get once Drupal 8 is the "new shiny."

On PHP4 and resourcing around D6, check #2140113: [Policy] Stop supporting PHP 4 and #2140299: [meta] Looking for Drupal 6 security maintainers (which are linked from the broader discussion at #2136029: Decide if and how to extend D6 security support 6-24 months past an 8.0 release). There's basically unanimous support for dropping PHP4 support in D6 early next year, and there are some technical folks offering to continue providing security coverage for D6, as well, since their clients still need it. So while D6 support in contrib is still an open question (and should be discussed over there), I'm cautiously optimistic that the D6 situation is not as dire as it may seem on the surface.

I haven't chimed in yet but have been following this thread. I started actively co-maintaining a module when D7 was in it's infancy and yes most of my client sites are still on D6. As per @RKopacz most of my clients are small shops, when I installed my last D6 site it was right before D7 came out and I wondered then whether to go with D7. However, most of the contrib modules either had not started a D7 port or the state was only developmental. I still maintain those small sites on D6 and do it as a courtesy to the clients, maybe I'm strange and have a different philosophy than others but with clients if I purposely do something with older versions then I will maintain them. Sure it may only be by maintaining security fixes but its better to keep a client rather than loose them. Of course there are conditions as well such as remaining a hosting client on one of my server.

As to contrib, I use to maintain both PHP4 for D6 and PHP5 for D7 and found it awkward to have to support a EOL'd version so I quit maintaining PHP4 per-se, I did find that using the 80/20 rules than with only a few tweaks the module would run fine on both versions. On the other hand converting a module originally written in PHP4 is definitely an advantage however the trade-off is more lines of code than if was originally written for PHP5.

I don't profess to be knowledgeable in any particular language, just know enough and dig enough to do what need's to be done. D6 is dying slowly as it should, D7 has been out now for a couple of years and only recently was drupal.org converted over to D7. D8 I've played with but can't get my head around it yet and I think that is what this discussion is all about.

The old saying is if it ain't broke don't fixit applies to clients in general. Their view is why should I have to pay for something that's working. There will always be clients like that but eventually they will have to change anyway which will more than likely be due to changes in technologies. PHP4 support being one of them, it's dead and if a lot of people are like me this discussion makes sense, although I'd rather not have it.

I tend to look at discussions like this from both sides. The developers want the new "shinny stuff" and if one of your clients is not on drupal then the "shinny" wins out (as long as contrib has the extra stuuf you need, even if it's only a developmental release). For clients that have used drupal 7, no worrys, major ones any way. Clients on D6, they have issues, especially if they're on PHP4.

Sure you could always find a hosting company that still offers PHP4, those companies still offer it because they know that clients usually DO NOT MOVE................ It's a given fact, no one likes change and even then they REALLY don't want the hassle. As for the actual hosting company, they too don't like change though most of them are more concerned about the security of theirs servers rather than pissing off a client. I ran duel versioning on PHP4/PHP5 for about a year after 4 EOL'd maybe up to 15 months then pulled the plug on 4.

All my D6 clients now are now using PHP 5.4.20 so I believe as others in this post that yes it may cause more workload for the security team but if there are enough volunteers for #2140299: [meta] Looking for Drupal 6 security maintainers it can be done without the teams workload increasing drastically. I also see this issue cropping up for D7 once D9 enters alpha so bike shedding this issue will crop up again in a few years if the infrastructure is not in palce to handle that transition as well. A good example is contrib, look at how many modules were abandoned when D6 came out.

Drupal 6 will itself be six years old in February, which is already twice as long as Ubuntu LTS (three years). It seems to me that that is long enough.

Prior to Drupal 7, new major releases (and I'm considering each 4.x a major release, despite the misleading version numbers at that time) came a few months to 1 year apart, so the EOLing of the X-2 version came in way less than 6 years, so in that sense, it's true that D6's 6+ year lifetime is unprecedented for Drupal.

However, if we want to compare to Ubuntu, then:
- Ubuntu LTS are for 5 years, not 3.
- Ubuntu LTS are usable on day 1, whereas without CCK, Views, and other contrib modules, Drupal 6 wasn't usable for most sites until >1 year after the 6.0 release, making D6's 6 year anniversary on Feb. 2014 more like 5 years as far as site owners are concerned.
- Ubuntu LTS come predictably every 2 years, meaning the 5 year support gives you a 1 year overlap to upgrade to the X+2 version, which is roughly what's being proposed in terms of extending D6 support.

Note: Drupal isn't Ubuntu, so we don't need to do something just cause they do it, so the above is just informational.

@effulgentsia: Ubuntu 10.04 LTS Desktop was only three years. The five year support LTS across server and desktop started with 12.04.

And if we are saying that five years is a reasonable useful life, that means that 6 has run its useful life.

I think that an LTS for Drupal 7, starting now, while it is still the current recommended version, with the opportunity for developers and site owners to plan for that, makes more sense than for 6, which up until this discussion started, was preparing to sunset in a matter of months.

I think the current policy should be to encourage site owners in 6 to upgrade to 7, and then plan their technology strategies taking into consideration that 7 might have an LTS version as an option. That also gives the security group time to think of the best way to support Drupal and also solicit additional volunteers.

No D6 LTS please. We weren't expecting one, all my D6 clients have been warned about the impending loss of support, many have upgraded and the remainder probably never will anyway. Just let it die when 8.0 is released, as promised.

When D6 was released, it was understood that it would only be supported until D8 was released (for that matter, similar could be said of D7). Given the delays experienced by both D7 and D8, D6 has already lasted longer than was ever intended. I think it is commendable that the Drupal community is even considering supporting D6 beyond the release of D8, but I don't think there should be any significant pressure to do so. Speaking as someone who is still mostly supporting D6 sites, I knew what I was getting into when I developed them. If you choose to support my D6 sites a little longer, then I'll be grateful for the gift. However, it would be a gift. If doing so is going to put any undue pressure on the security team, then forget about it. I'll live.

As for the proposal moving forward, I am very excited by the prospect of a six month release cycle, both as a user and as a occasional contributor. So, yes please!

effulgentsia:

What is the actual difference between a "minor release" in this proposal (e.g. Drupal 8.1.0 or 8.2.0) and a point release in Drupal 7 (e.g. Drupal 7.22 or 7.23)?

I'd like to respond more fully to this soon, but for now, I'm attaching extractions from D7's CHANGELOG.txt.

Thanks for that list! And I'm still very interested in hearing the rest of your response :) So far I still have pretty much all of the questions I asked in #16 (and this is the biggest one)...

Would it also make sense to split this into multiple issues? (Especially because some of those issues already exist.) There are about 4 or 5 different things being proposed here, and they aren't really related to each other, which makes the discussion difficult to follow (and in some cases discussion that took place elsewhere has already been repeated here).

For example, catch mentioned in #21 that his favorite part of this proposal is the idea of not opening Drupal 9 up for development until well after 8.0 is released, and I agree that is potentially interesting. But it's something that could be done entirely on its own without the other proposals being made here.

While not encouraging interminable use of D6 but being respectful of the challenges faced by clients as described by @ctmattice1, not to mention that drupal.org has only recently updated to Drupal 7 and not without hiccups it does not make sense for Drupal 6 to be dropped immediately when Drupal 8 is released.

Providing only support for security fixes to D6 for a short time will likely be better for the Drupal community and clients using Drupal.

Some may see this as an opportunity to transition through to Drupal 8 to take advantage of the improvements it offers.

Just curious, for those who are advocating dropping D6 on Drupal 8.0(.0) shipping: How do you propose to handle the 200,000+ active Drupal 6 sites (20% of all Drupal sites) (which are only the ones that we know about) when we know that about 50% of security issues for D7 apply to D6 as well? It seems at best irresponsible to not provide security fixes for 20% of our user base until there's a viable Drupal 8 release for them to move to (which will most definitely not be Drupal 8.0(.0)), and at worst could lead to tarnishing of Drupal's reputation as a secure CMS if a really terrible 0-day vulnerability came out for D7 which was not subsequently patched immediately in D6.

#46 reads as if we're breaking an agreement with our users. The opposite is true. There's already been a longer life to Drupal 6 than people should reasonably have expected when it was released. If a wormable vulnerability is found in Drupal 6 on the day after Drupal 6's EOL and it gets widely exploited, sure that would be bad for Drupal's reputation. But...what do you propose we do about it? Saying that D6 is supported doesn't make it a reality.

In an ideal world I agree that supporting Drupal 6 for some period of overlap with Drupal 8 is a good idea. Getting contribs ready to release on the day D8 is released would also help solve that problem. But in spite of efforts from various people ( ;) ) we don't live in an ideal world (yet).

Sorry, I didn't mean to imply we were breaking an agreement with our users. I was trying to simply point out reality. Saying to everyone, "Ok, time to upgrade your sites to Drupal 7!" also doesn't make it so. :)

If you built your site in the last couple years, then it's on D7 (or should be). If its an older site then either you've already been planing your D7 upgrade, or you've decided you're OK with using an unsupported CMS for a while. I still have an old D5 site kicking around, but I know that's a gamble and if it goes down, it goes down.

We shouldn't forget that we already have a perfectly good, thoroughly supported and extremely stable version for D6 users to move to. It's true D7 isn't the latest and greatest, but if you're still on D6 then that might not be the biggest consideration for you anyway. And with this plan, D7 will continue to be supported for many more years, so moving to D7 won't be in vain.

Perhaps I'm naive, but I find it highly unlikely that—after the last six years of real world use and development—a security vulnerability capable of taking down all/most of the D6 sites will surface within a year of its retirement. And if such a vulnerability were to surface, I find it extremely unlikely that our reputation will be tarnished. It's a six year old unsupported CMS.

The thing that worries me more than stopping support for D6 is over taxing our volunteer security team and core maintainers. If we loose them, because the job become too overwhelming, then forget D6—we'll be wondering what to do about D7 and D8. From what I'm hearing, supporting this proposal from D7 on is totally doable, but maintaining D6 support would be difficult, since it lacks testing support and such. Given that, it seems like the potential benefit isn't worth the cost.

@David.

Yes my favourite thing about this is opening 9.x when we need to, not just because we can. I quite publicly suggested that 8.x should not be opened until 7.x was critical bug free, about three weeks before the branch was opened at DrupalCon Chicago. Opening the branch when there was still significant work to do on Drupal 7, and the bulk of Drupal 8 changes had months of planning ahead, made it much harder to resolve numerous Drupal 7 issues for some time for multiple reasons (committer access, superficial changes, backporting), despite no major 8.x changes for about a year afterwards.

The other thing that I think would be a major benefit, although I know that others are less enthusiastic about this, is that it potentially allows us to concentrate particular types of changes to core at particular times.

For example, with semantic versioning, there's no reason that a major new feature (even Views or WYSIYWYG) couldn't go into to an 8.3 release - assuming the bulk of the API groundwork (and realistically some of the implementation as well to ensure the APIs are enough even if that's in sandbox) was done in time for 8.0.0.

One of the most frustrating things about core development is that despite the release being 'ready when it's ready', we have several arbitrary deadlines (feature freeze, code freeze) that mark a cut off point beyond which certain changes aren't allowed. There's nothing stopping us adding those changes right up through beta and even release candidate in some cases, and in practice we do add some of these changes (configuration translation this release, drag and drop in 6.x) when either we're hopeful they won't completely destroy the readiness of the release, or Dries thinks it's worth risking even if they do. However the reason we have those deadlines is because we know that in the process of adding features and changing APIs we'll have built up a tonne of technical debt, which needs time to resolve.

However that current situation is at least partially self-defeating:

1. Having strict deadlines encourages things to go in with known technical debt before whichever deadline is in place, with everything possible (performance, UX) pushed as late as it can be - since those changes aren't deadline bound but instead block the eventual release. That's how you end up with 130 known critical issues.

2. All the big changes, and lots of little ones, tend to happen around the same time, resulting in lots of conflicts.

3. We're not good at sticking to the deadlines (for good reasons but still...), so the period to resolve the technical debt gets squeezed, or realistically pushed back along with the chance of releasing sooner rather than later.

4. It's hard to define what exactly is and isn't affected by different deadlines - since the boundary between features, enhancements, bug fixes, UX, API changes and refactoring aren't always clear - especially when it comes to administrative or framework-level changes. As a result there are often misunderstandings in both directions as to what is and isn't allowed at any one time.

5. There are usually peaks of activity just before each deadline, then troughs of exhaustion and disinterest afterwards, which often leads to people working unsustainably on getting certain things in, then being too knackered to follow-up on them once they are, or certain changes missing out with no chance of making it in for another 3-4 years which is extremely demoralising.

Having a clear route to add features after the X.0.0 release would mean that while feature development might happen before that, commit could either happen before if it's really ready, or postponed until six months later i the next minor release, or the next one if it's not. This should help to relieve some of the pressure off issues such as #1741498: Add a responsive preview bar to Drupal core where the choice is currently 8.0 (and an exception to 'feature freeze') or 9.0 2-4 years later, or not at all, or a 'random' inclusion into a point release which otherwise might just have been a security update.

So for example, 'code freeze' could happen at beta/RC for the major release, and 'feature freeze' might only ever happen during RCs (possibly short betas before them) for the major and minor releases - reversing the current situation where the only time to legitimately add features is also when APIs are at their most unstable.

This doesn't help with major framework-level features like CMI that need a year (or two) in core to convert everything to them. However it should benefit that kind of work too. There'll be lead up time before 9.x is opened with a stable 8.x API to develop the feature against (apart from conflicts with other large projects also slated for 9.x), then a period where people know that several large API changes are all going to go in within the space of 3-6 months - because those are the reason we opened 9.x so we could actually commit them.

This is actually exactly what we're doing already, by ripping out the upgrade path, but making it clear that we'll release 8.x without migrate finished (although conceivably it might be finished in time anyway, but it doesn't have to be).

Where semantic versioning, six month minor release cycles etc. comes in is only really after both of these - how to communicate that a certain point release is 'the migration release' (or adding a new admin feature or both), what to do about encouraging people to skip major versions if there's no way to migrate when the x+2 version is released etc.

As a core developer/committer I really only care about making the core development process more sustainable. For site building as long as it doesn't make anything significantly worse than it already is that's great - if things also get more sustainable/predictable for site builders then that's a happy side effect.

I do think it makes sense to discuss what the minimum set of changes we could make to allow some of these things happen is, and which parts of the proposal are optional - i.e. can we adopt everything else then figure out the exact timing of 6.x end of life separately? Can we agree to open 9.x much later (relatively to previous releases) after 8.0 without adopting semantic versioning and an LTS cycle?

However I'm not sure it's possible to split the conversation out, since so many of these issues are interconnected - at least depending on your viewpoint, and plenty of people's priorities aren't the same as mine.

How do you propose to handle the 200,000+ active Drupal 6 sites (20% of all Drupal sites) (which are only the ones that we know about) when we know that about 50% of security issues for D7 apply to D6 as well? It seems at best irresponsible to not provide security fixes for 20% of our user base until there's a viable Drupal 8 release for them to move to (which will most definitely not be Drupal 8.0(.0)), and at worst could lead to tarnishing of Drupal's reputation as a secure CMS if a really terrible 0-day vulnerability came out for D7 which was not subsequently patched immediately in D6.

This concerns me as well. While I can appreciate thoughts that having a D6 LTS would make a bunch of consultants potentially look like Chicken Littles, I feel the greater concern is the impact on Drupal were there to be a major security exploit that takes down or defaces thousands of D6 sites -- or even just one higher profile site. We can say, "But it's old and unsupported!" until we're blue in the face, and the response would be, "Why wasn't it supported? Does Drupal leave you holding the bag?" It could be catastrophic for Drupal's image, especially if the issue was known and fixed on D7.

The reality is that having a large amount of Drupal sites left with no easy update path and no community support represents a pretty big risk to Drupal's perception -- a risk that might not have been such a big deal a few years ago, when Drupal was more obscure and only tech-minded folks and orgs were willing to take it on (live by the sword, die by the sword). But now we have a successful product that requires little maintenance once built, but significant effort to upgrade, and therefore an abundance of site owners with little tech expertise and insufficient resources to take on an upgrade.

LTS may be a hard thing for the community to entertain, but in terms of image and branding, not providing support carries risks that could leave Drupal's image severely tarnished (a la PHP Nuke).

As for how to make LTS work, I am not sure how to jump start it with D6, which is short notice and a sudden burden relatively speaking. One alternate option might be as an independent project run by a subset of the community with vested interest in it. (There have been some quiet discussions about this, but I don't know particulars, or how exactly that would work. Perhaps as a semi-contrib style of effort?) If there's interest to make that happen, it would be a shame for the Drupal community not to support that with update release support.

Would it also make sense to split this into multiple issues? (Especially because some of those issues already exist.) There are about 4 or 5 different things being proposed here, and they aren't really related to each other, which makes the discussion difficult to follow (and in some cases discussion that took place elsewhere has already been repeated here).

Well, most of the conversation here has been about extending D6 support, so yes, let's please at least move that to #2136029: Decide if and how to extend D6 security support 6-24 months past an 8.0 release. I updated the summary there to include people's comments from this issue. If anyone feels I missed an important comment from you, please edit that summary accordingly.

As to the rest (opening the D9 branch later, changing the current monthly "point" release into a monthly "patch" release and a semiannual "minor" release, clarifying the scope of what's a "patch" and what's a "minor", and allowing more into a "minor" than what we've done in practice in a D7 "point"), I think those are interrelated enough to warrant keeping in this one issue for now; at least until we get more conversation about each of those to see how separable they are.

Let me ask it this way: excluding the question of D6's support period, is there anything else in this proposal that anyone objects to? Does it work for anyone? Any major problems that haven't been noted and/or addressed? Again, ignoring the question of Drupal 6 for the moment...

Ignoring D6, I think this represents an immense improvement. It might actually remove enough frustration that I'm motivated to try my hand at core contributions again. And as a Drupal user, I'm very excited about the prospect of new features coming a couple times a year.

As a developer / site builder, having a fully featured, edge-case stable contrib would be amazing - I still end up having to run -dev versions of a lot of modules and we are moving towards the end of D7's public life. BC will enable contrib to flourish in this respect.

A larger amount of smaller releases will make updates more manageable, and less costly. That in itself will be a boon to the community.

In other words, +1.

@jstoller, new features get added to Drupal 7 core three to four times per year. You are excited about the prospect of it happening less often in the future? :)

I don't mean to be facetious, but this is why I raised some of the issues I raised earlier.

... I think those are interrelated enough to warrant keeping in this one issue for now; at least until we get more conversation about each of those to see how separable they are.

So here's a summary of the various sub-issues contained within this proposal. I've interspersed the concerns (mostly from my comment in #16 and responses to it, but also what some other people wrote separately; however it's still pretty biased towards the ones I wrote which is why it's here rather than in the issue summary :)

  1. Extended security support for Drupal 6 (separate issue already exists: #2136029: Decide if and how to extend D6 security support 6-24 months past an 8.0 release).

    Concerns: all sorts of concerns by various people.

  2. Extended security support for Drupal 7 and later (separate issue already exists: #2139769: [Policy, no patch] Block 8.0.0 and possibly minor versions on critical security issues.).

    Concerns: #16.5, greggles' comment in #22, all sorts of security team concern.

  3. Introduce semantic versioning to better communicate what's in each release (separate issue already exists: #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x).

    Concerns:: #16.4, but it's a relatively minor concern and was already discussed on the other thread (where the conclusion seemed to be to ignore the "official" semantic versioning rules on occasion). I think there are some concerns about the extent of the required implementation effort also.

  4. Don't open Drupal 9 up for development until well after 8.0 is released.

    Concerns: None? It does mean less time to do "full-scale" development of Drupal 9 than previous releases, but so far I'm not sure anyone has objected.

  5. Start allowing some kinds of features into Drupal 8 (after the 8.0 release) that wouldn't be allowed in Drupal 7 today.

    Concerns: #16.1 and #16.6. More of a request for specific examples than a direct concern, although the concern could be stated as "there might not be any examples" :) I still don't think I've come across a good, meaningful example in this thread... Let's be specific if we want to propose this; otherwise people will think "yay new features" and be very disappointed later on.

  6. Limit new features in Drupal 8 to one release per 6 months.

    Concerns: #16.2

  7. Declare an official point in time after which new features will no longer be added to Drupal 8 (and separately for Drupal 7, I suppose). After that only bugfixes are allowed, and later on only security fixes. This is called "LTS", although if point #2 above doesn't happen it may need a different name.

    Concerns: #16.3, but so far no one has really objected to dealing with that by having the LTS phase simply start much later than originally stated in the proposal (i.e. more in line with the current situation of Drupal 7, which is not yet particularly close to this phase).

  8. Get rid of the weird thing we currently do where we have a "feature freeze" that starts very early in the development cycle (way before a major version of Drupal is released) but then allow some of those exact same features in after the X.0 release anyway. Instead prevent features from going in only during the final release candidate phase (and also introduce a mini-release-candidate phase for minor releases and prevent new features then too). This is part of what @catch was talking about in #50.

    Concerns: None?

Overall, it's a lot of separate ideas that are only somewhat loosely related, although I agree the later ones are more related.

I would suggest that at least the first three be left for separate issues. First because the separate issues already exist for those three, second because they have a lot of heavy concerns that have been raised about them that shouldn't block the rest of this proposal, and third because they also require "real work" to implement (unlike the others which are "just" policy changes).

As far as the sorts of things that would be allowed in an 8.x release, my Prague presentation included a couple of examples, just from REST/Serializer:

http://www.garfieldtech.com/presentations/dcprague2013-drupal-dev-cycle/#/4

It's reveal.js, so hit down arrow to see the rest of that chapter. Also go right one to see the comparison to Drupal 7. Adding whole new UIs or improving inline documentation would not have been OK in Drupal 7, but should be in Drupal 8, IMO.

Adding whole new UIs or improving inline documentation would not have been OK in Drupal 7

Actually, according to What patches may be backported to a stable release of Drupal core?, both a whole new UI (in the form of a feature addition / opt-in module) and admin-facing string changes are allowed into D7 point releases. Perhaps a question then is why did they not happen, or not happen to the extent some people desired? That policy has also evolved a bunch in the last 3 years (has gotten looser with time), so it could be that people have been under the impression that things were stricter than they've really been, and therefore didn't work on things they otherwise would have.

Despite the above, I do think there are good examples of things that aren't allowed even in our current D7 backport policy but that should be allowed in D8, and I'll share them when I'm able to write them up coherently.

Re #16.3 / #56.7:

Point 6 of the issue summary says:

Therefore, in order to not have too long a time go by with no innovations in core, we would also target releasing the next major version within a year or less of opening its branch.

I think part of the concern of #16.3 is a skepticism that we can achieve a <1 year timeframe from opening a major branch to releasing it. Certainly the 3 years of D7 development and what will be 3+ years of D8 development provide reason for that skepticism, even if you subtract an initial year from each in which we potentially could have delayed opening the corresponding branch without impacting the final release date.

I don't recall if this has been discussed in the comments so far, but in various in-person discussions about this, Crell, alexpott, catch, and others have suggested that what could allow us to move from the 2+ year cycle (from beginning of ramped up active committing to release) for 7/8 to a 1 year cycle for 9 is the ability to reign in scope more, partially because there's less expected need to rewrite everything, and partially because there'll be room to get things into 9.1, 9.2, etc.

Let's suppose, however, that you're not convinced by this, and that the better point to trigger an LTS of the stable major is the alpha1 of the unstable major rather than the opening of its branch. For D7, that was January 2010 (one year before release) and for D8, that was May 2013 (will likely be more than one year before release, but hopefully not by much, and even if by much, we really, really need to reign in D9 better than we did D8).

So let's assume that the first point release of D6 following January 2010 (6.16 released in March 2010) and the first point release of D7 following May 2013 (7.23 released August 2013) had been made LTS (in the sense of only bug fixes, no changes/additions, following that).

Looking at the D6 changelog, I don't see anything for >6.16 that's beyond the scope of bug fixing, except perhaps "Improved logging for login failures", but please correct me if I'm mistaken.

Looking at the D7 changelog, nothing yet has been released in D7 since 7.23 except a security fix, but that could just be due to it only being a few months since then, so we haven't accumulated interesting examples. But even looking at 7.23 itself, the only items flagged as a change or addition are:

- Added a drupal_check_memory_limit() API function to allow the memory limit to
  be checked consistently (API addition).
- Added protection to cache_clear_all() to ensure that non-cache tables cannot
  be truncated (API addition: a new isValidBin() method has been added to the
  default database cache implementation).
- Added a drupal_array_diff_assoc_recursive() function to allow associative
  arrays to be compared recursively (API addition).
- Fixed the default ordering of CSS files for sites using right-to-left
  languages, to consistently place the right-to-left override file immediately
  after the CSS it is overriding (API change: https://drupal.org/node/2058463).
- Fixed drupal_render() to always return an empty string when there is no
  output, rather than sometimes returning NULL (minor API change).
- Changed the password reset form to pre-fill the username when requested via a
  URL query parameter, and used this in the error message that appears after a
  failed login attempt (minor data structure and behavior change).
- Added human-readable labels to image styles, in addition to the existing
  machine-readable name (API change: https://drupal.org/node/2058503).

Of those, if those first 3 additions were in the service of solving something important, I don't think it would be inappropriate for comparable things to be added to D8 as patch releases of the LTS minor. If we were following semver strictly, we shouldn't do that, but the proposal isn't to follow it strictly, only to the extent that it makes sense. None of those are BC breaks, so we wouldn't be compromising the spirit of the LTS. The next 2 items are bug fixes, so we'd need to evaluate the benefit of the fix vs. the risk of the edge case break. The last 2 items I would argue should be off the table for adding after the LTS. I think if comparable things like that don't make it into D8 after its LTS is released, it wouldn't be that much of a loss.

However, we never added the Entity API module to Drupal core during the Drupal 7 development cycle. That, to me, stands out as an example of how this new proposal would change things.

@laura_s Windows XP is going down in April, and I am quite sure there will be hundreds of thousands of XP users who do not want to see XP sunset yet. But it is in the nature of software to become obsolete after a period of time, and I think that people understand this.

I also believe that it runs deeper than a bunch of web developers being embarrassed because they sold a client on something that they didn't need to. I feel protective of the community's time, particularly the security group. In principle, I would prefer the community time (and by the way it is always more time than you think) be spent to support the quicker release of a stable version of a wider number of modules for Drupal 8, than to keep 6 on life support.

Believe me, I am saying all this knowing that my clients currently running on 6 would probably love to see an LTS version of 6 so they don't have to upgrade. Selling a 6 to 7 upgrade to a small organization is an extremely hard sell.

In response to @crell 's question, yes apart from the debate around 6, I think the whole concept makes a lot of sense going forward.

As far as the sorts of things that would be allowed in an 8.x release, my Prague presentation included a couple of examples, just from REST/Serializer

Thanks! I think those are some good examples, but also agree with @effulgentsia that they could (theoretically at least) happen in Drupal 7 already.

Actually, according to What patches may be backported to a stable release of Drupal core?, both a whole new UI (in the form of a feature addition / opt-in module) and admin-facing string changes are allowed into D7 point releases. Perhaps a question then is why did they not happen, or not happen to the extent some people desired? That policy has also evolved a bunch in the last 3 years (has gotten looser with time), so it could be that people have been under the impression that things were stricter than they've really been, and therefore didn't work on things they otherwise would have.

Yeah, I think that's one reason. Another might be that it's simply harder to get patches in to an already-stable version of core (more rigorous reviews are required) since we really don't want "followup" major/critical bugs to be introduced on already-live sites.

#365615: Language detection not working correctly for most Chinese readers (and add a user interface for all browser language mappings) is an example of a whole new UI that might still make it into Drupal 7 (the issue is partly to fix a bug, but about 80% new feature).

I think part of the concern of #16.3 is a skepticism that we can achieve a <1 year timeframe from opening a major branch to releasing it.

A bit, but it's not just about releasing it, it's also about when it achieves widespread adoption such that it is reasonable for people on the old branch who want a new feature to upgrade/migrate to the new branch to get it. For example:

Let's suppose, however, that you're not convinced by this, and that the better point to trigger an LTS of the stable major is the alpha1 of the unstable major rather than the opening of its branch. For D7, that was January 2010 (one year before release) and for D8, that was May 2013

Even May 2013 would still be a very long time before Drupal 7 usage peaks and Drupal 8 becomes viable (quite possible 2 years before, although we can't predict the future). That's a long time without being able to get new features.

The last 2 items I would argue should be off the table for adding after the LTS. I think if comparable things like that don't make it into D8 after its LTS is released, it wouldn't be that much of a loss.

The world wouldn't end, but talk to the people who pushed #606598: Human readable image-style names into Drupal 7 - they seemed to want it pretty badly :) #365615: Language detection not working correctly for most Chinese readers (and add a user interface for all browser language mappings) has had a lot of people pushing for it also. I don't know if there are a lot of others though.

However, we never added the Entity API module to Drupal core during the Drupal 7 development cycle. That, to me, stands out as an example of how this new proposal would change things.

I think that's a good example to look at. First, if people wanted to propose adding the Entity module to Drupal 7 core (as-is, as a standalone module) in principle there's no reason it couldn't happen... I don't know if anyone's really interested in that though. I suspect that what people are more interested in would be improvements to the entity API itself, so instead of:

<?php
$wrapper
= entity_metadata_wrapper('node', $node);
$value = $wrapper->field_whatever->value();
?>

You could do something more like:

<?php
$value
= $node->field_whatever->value();
?>

Problem with that is that nodes are currently stdClass() objects in Drupal 7 and you'd have to change them to real classed objects to fix this. That's a change that would break people's code.

Of course for this particular feature the improved entity API is already in Drupal 8 so it won't be a problem there anyway, but my general point is that if the reason a feature can't get into Drupal 7 is because it would break lots of people's code, I don't see anything in the proposal that would change that for Drupal 8.

Yeah, I think that's one reason. Another might be that it's simply harder to get patches in to an already-stable version of core (more rigorous reviews are required) since we really don't want "followup" major/critical bugs to be introduced on already-live sites.

That's the main benefit of patch releases monthly vs. minor releases every six months. The new features can get into a branch months (and if we have a month or so beta/rc period) at least weeks before the minor release.

That should allow for two things that currently don't happen now:

1. Sites that are waiting for a new feature can do full integration of the current minor branch - with the features they were waiting for, but also other things that might break. Currently if I have core patches that have been worked on for a specific client site , I tend to have those patches individually applied until the new release actually comes out - which might or might not include those patches and could be weeks or months away. A minor release in beta or rc state would encourage people to test the entire release before it actually gets rolled out. Doesn't need every site to do it, just enough to discover the worst issues. There's also a lot more sites doing automated testing themselves now than there was when Drupal 7 was released.

2. We could run the automated branch and/or patch tests for contributed modules against both the current minor and the next minor branches of core, so any new feature or API addition which accidentally breaks an API used by a contrib module should get flagged up by failing tests as well.

That gives the potential for automated testing of both custom and contributed code with the next minor branch. It would be possible to do release candidates of non-security Drupal 7 releases to try to encourage a similar thing within the current framework, but that also means a longer time between releases if the rc period was going to be meaningful - i.e. at least two months.

Neither of these are any guarantee against not breaking sites, but the opportunity is there, and sites that don't like getting broken can help themselves by having automated tests themselves, and pulling the minor branch early to test them on.

Start allowing some kinds of features into Drupal 8 (after the 8.0 release) that wouldn't be allowed in Drupal 7 today.

Concerns: #16.1 and #16.6. More of a request for specific examples than a direct concern, although the concern could be stated as "there might not be any examples"

@David_Rothstein: sorry it's taken me so long to respond to this concern. It's actually hard to find great specific examples, because:
1) The specific things we should work on in 8.1 and 8.2 should be driven by real world feedback of 8.0, which we don't have yet.
2) I don't have great memory, or a handy list, of all the things we didn't allow to be backported to 7.x.

But, here's one issue that we did allow into 7.14 (16 months after 7.0, so the equivalent in this proposal would have made it 8.3, though I wonder if other aspects of this proposal would have made it take less than a year to get done): #1164812-83: Improve UX for machine names for fields.. Notice that a couple comments later, you (in my opinion, correctly) questioned that commit as something that's higher risk than we normally allow. This is one of relatively few UX-improving patches that we allowed into 7.x after releasing 7.0.

What I see in this proposal is the opportunity to do many more improvements like this (and even bigger than this). Yes, a couple modules might break in very minor and easy-to-fix ways as a result of these kinds of changes, but that's where scheduling them on a 6 month cycle, and with their own mini RC period, rather than in a random point release, helps.

the experience of a d6-d7 upgrade has 2 organisations I work with regret ever going near drupal - so plus 1 for longer LTS and more BC

the comment at #61 cracked me up - XP was released in 2001!!!

love your work and all - but pleaassssssssse stop breaking everything every 2 years!

@Erich Schulz I think you missed my point. The point is that there will always be people who want a piece of software to continue indefinitely. Even if they extended XP for 20 years, at the end of that 20 year period there would still be people who say, "Keep it going!", but that might not be a good reason to keep it going.

Microsoft also paid the salaries of employees who maintained XP. For Drupal, it is all volunteers. That is a big difference.

I think that site owners tend to forget the amazing features and functions that they get for a much lower cost than they would, had there not been these open source projects. A dynamic website does cost something, and to say they are unhappy because they have to pay something to upgrade it now, after 3 or 4 years of good service at a minimal cost monthly cost, is a bit disingenuous.

A group of Drupal developers could always opt to maintain D6 for a yearly fee. Then site owners could choose between maintaining their 6 site for the annual fee or upgrading to 7. My guess is that it would be overall cheaper to upgrade.

I have two concerns. There are still major changes to the Drupal 8 APIs that we all agree on as necessary (think #2016629: Refactor bootstrap to better utilize the kernel for example) and if push D9 out into the far future these might not be doable. This, in itself, raises another concern: I really would like to see a canonical definition of what can go into 8.x.y and what needs 8.(x+1).0 and what needs 9.0.

Somewhat related to #64's point, here's an example of an issue that was recently bumped that I think would benefit from such a versioning scheme: #1170538: Rename core .gitignore file to example.gitignore and add explanatory comments

I was never comfortable rolling this back myself (obviously this would be David's call these days), because by the time it became clear this was not a good idea, we were 5 or 6 point releases and ~100K sites into D7. Without a really clear way of signaling "HI, SITE BUILDERS. THIS MIGHT BREAK YOUR STUFF, TEST CAREFULLY." (which "7.10" doesn't really do, but which "7.1.0" does do a bit better), stuff like that tends to just sit around forever and people are forced to do work arounds for 6+ years. :\

@chx: the bootstrap issue comes down to how we address what are public APIs or not (which needs its own discussion), and whether bc layers are possible. For that issue I'd expect the main API changes from #2016629: Refactor bootstrap to better utilize the kernel to affect drush and custom scripts that call drupal_bootstrap(), but not a lot else.

To put a change like that into 8.1, we could keep drupal_bootstrap() and the constants, but have them all do the same thing (or rough equivalents to what they do now if we still have some kind of phase in there). Assuming there's a bc layer and we don't knowingly break lots of code, that seems fine to me. Note I also have a much more limited patch at #2023495: Make bootstrap three steps which would make the current bootstrap phases more honest.

Two things that would get pushed to 9.x:

1. Removing the bc layer if one gets added.

2. Making the change at all, if it's not possible to have a bc layer for the main changes (since a small bc break for some edge cases might be fine in 8.1).

Both #1 and #2 mean there's identified technical debt which is not possible to resolve without opening the 9.x branch. Sufficient changes of that type, with viable patches, should trigger opening the 9.x branch (alongside assessing the state of 8.x).

So there's really a few kinds of changes:

1. 100% internal refactor/tweak or addition with no API change (goes into 8.x and 7.x now)

2. API change with a transparent bc layer (100% OK for 8.x, sometimes goes into 7.x)

3. API change but only for edge cases or rarely used APIs where no bc layer is possible (OK for 8.x until we find out it's not, very rarely goes into 7.x)

3a. API change with a bc layer, where the bc layer has a significant cost/complexity (could go into 8.x, but is going to be high priority to remove in 9.x, unlikely to go into 7.x)

4. API change where the new system can be added in parallel to an old one, but the old one should be removed/converted in the next release (adding the state system to 7.x alongside variables?, lock and queue maybe).

5. API change which affects lots of modules, and can't have a bc layer (9.x only)

Anything from 3a to 5 means we're identifying real technical debt in 8.x, which can only be addressed by opening the 9.x branch. However in the meantime, everything from 1-4 is potentially getting into 8.x, which means it's in 9.x too of course.

Let's look at the commits to 8.x vs. 7.x each year for the past three years, and I added 6.x and 7.x for 2010 for comparison:

I'm using 7th Jan as the start date, i.e.:

git log --pretty=oneline --since=01/07/2011 --before 01/07/2012| wc -l
     754

This means that 2013 is actually only 11 months instead of 12, but it's good enough IMO to see what happened:

            6.x         7.x      8.x
2010       177         2487     -
2011        -           754     1175
2012        -           637     3383
2013        -           238     3961

There are also over 700 open issues tagged as needing backport from 8.x to 7.x (across both branches). It'd be interesting to compare this monthly, in a graph, and pull out a few of the major 8.x commits, when new committers were added, core office hours started etc. to see how that affected both 8.x and 7.x momentum, but not right this minute!

Conclusions I'd draw from those numbers:

1. There were significantly more patches committed in 2010 than 2011 overall, that shows that we lost a tonne of momentum in core development (using quantity of patches as a measure) when 7.x was released and/or 8.x was opened. There's various reasons for that between burnout getting 7.x out the door, committer availability, big changes in planning phases etc. but numerically it's very obvious.

2. Drupal 7 was under significant development for 18-24 months after it was released, but has also dropped down over 50% in 2013.

3. More patches were committed to Drupal 7 in the two years after release, than were committed to 8.x in the first 9-12 months that it was open - this is despite 99.9% of Drupal 7 patches having to go into 8.x first then sometimes non-trival work involved to backport them.

This also doesn't take into account backportable patches that either never got moved to 7.x (probably true for a lot of smaller docs and test improvements), or were moved back but not tagged correctly then never got committed. Those patches would have got into 7.x if 8.x hadn't been open yet. So the numbers are really only going to get higher for 7.x committable stuff, compared to the raw numbers of what actually went in.

@David

I think that's a good example to look at. First, if people wanted to propose adding the Entity module to Drupal 7 core (as-is, as a standalone module) in principle there's no reason it couldn't happen... I don't know if anyone's really interested in that though. I suspect that what people are more interested in would be improvements to the entity API itself, so instead of:

I don't think there's any reason that Entity API module couldn't have been added to 7.x, but I don't think it would make sense now, since we have a completely different entity API in 8.x. I do think stuff like entity_save() and entity_delete() would have been worth adding to 7.x to round the existing API out, as well as moving the internals of $entity_foo() into controllers that could be swapped out.

One thing that's been missed in this discussion in terms of features getting into 7.x or not, is that the backport policy requires that we get things into 8.x first (unless they're completely irrelevant to 8.x like modules that have been removed).

If we went back to the first year after 7.x was opened, didn't open 8.x for 18 months, and aimed to get big new features into 7.x, one obvious example would be Views.

Views in core could have been pretty much a copy and paste from contrib to core for 7.x. For 8.x however, it's been refactored significantly to take into account new APIs such as plugins, PSR-0, CMI, the new entity/field APIs and Twig. Some of that happened prior to the initial commit, some happened afterwards.

Backporting Views from 8.x to 7.x (if Views didn't exist as a contrib module beforehand) would be a near impossible task now - because it' s not just a new feature, it's a new feature using a completely different set of APIs. Would be interested to hear from people who worked on VDC as to whether they think Views would have ended up in the same position as it is now, had it originally gone into 7.x say 2-3 months before 8.x opened up. One big difference would be the pressure of dealing with bug reports against a branch that's being used, compared to one that's in development - not that this isn't true for Views in contrib for 7.x but it puts all that focus into the core queue, and Views doesn't follow the core backport policy.

So it's not just about the kinds of changes that can go in, it's also about the logistics of getting changes into two active branches with different APIs. The same goes for various Spark changes, render (and other) caching improvements, and new systems like state, settings, phpunit all of which could or have been implemented for 7.x to at least some extent, but where there's absolutely no effort to get them committed to 7.x core now. This mainly comes down to 'opening 9.x later' as opposed to semantic versioning, but it's a real pressure on both new features and bug fixes in terms of the work involved, once the new branch is open. Even before the branches diverge much there's still process overhead - we don't even use cherry-pick yet.

There are also over 700 open issues tagged as needing backport from 8.x to 7.x (across both branches).

Do we consider this a problem? If so, why aren't these being backported and how can we reduce this number both for backport to 7.x and backport to 8.x?

- backport policy: By delaying branching of 9.x, you keep the number of these issues at 0 for longer.
- committable change policy: Probably already looser than people think it is for 7.x, being loosened for 8.x.
- RTBC queue: There are 7.x issues that have been committed to 8.x and RTBC for 7.x for over four months. People aren't going to bother backporting if they don't think their patches will get committed. Is there any way those of us without commit access can help? (I'm not criticising any individuals, more reflecting on the priority that the Drupal project gives to 7.x development.) The branching delay will help here too, as core committers won't need to review 8.x and 9.x versions.
- Tag cleanup: Lots of those issues shouldn't have that tag as they don't apply to 8.x
- Issue priority: Most of these are normal or below, so maybe it's OK just to live with them rather than risk breaking existing sites.

@ianthomasuk, it was definitely a problem the first six months or so after 8.x was initially opened. I personally got very, very annoyed at the difficulty getting critical bug fixes into 7.x and 8.x. This was both due to Dries being the sole person able to commit any patch at all for 8.x (until webchick was given 8.x commit access 4-5 months later for backportable patches), as well as lots of relatively superficial but wide-ranging changes to 8.x breaking 7.x bugfixes that needed to be applied there first.

As far as whether it's a problem now I'm less sure - but that's mainly because I only pay attention to bugs that either apply to 8.x too and happen to need backport, or which I'm being paid to work on for client sites experiencing the bug. I imagine this is the same for a lot of people who work on the 8.x version of patches, so unless someone is actually experiencing the bug in 7.x and has time set aside to fix it, they're going to sit there.

Looking through the major/critical bug reports against 7.x, there's a lot of them that should probably be triaged into 'needs more info', 'cannot reproduce', moved to the relevant contrib module etc. Thresholds against 8.x ensure the major and critical bug lists are quite well triaged (and did so for 7.x issues too when thresholds applied to both branches), but otherwise issues posted to the core queues tend to drift unless someone is actively working on them.

Very few people look over the issue queue as a whole because it's too high traffic. Triaging both the RTBC queue in terms of re-tests via the bot and re-reviewing to ensure things are really RTBC, as well as the high priority issues as a whole is the easiest thing for non-committers to do in terms of ensuring important issues get attention and committed - at least in terms of volume rather than any one particular issue.

Status:Active» Fixed

Thanks everyone for all of your helpful input. What I'm hearing largely is that concerns are mainly centered around extending LTS support for Drupal 6, and whether or not this policy will actually slow down the rate of innovations in core relative to the monthly release window that Drupal 7 does today.

In terms of security, I think various concerns raised here are very real. For example, we do have challenges with the velocity around core security fixes, and often times this is due to worries about making changes to Drupal 6, given its lack of test harness and continued support for PHP 4, which was EOLed around 5 years ago. However, those are all existing problems, and we must fix these even if we decided not to proceed with this proposal. We are making an attempt to do so in:

I also understand the volunteer burden concerns around Drupal 6 LTS, and they are all very valid, but the fact is we need to put the needs of Drupal's users first. I would encourage module maintainers who do not wish to support D6 versions of their modules beyond Drupal 8's release to announce this on their project pages and to seek out co-maintainers to handle those specific branches; as we can see from the above issue for core, there are still many folks with a vested interest in maintaining Drupal 6, but they need to be asked to volunteer.

In terms of the pace of innovation, I feel arguments have been made that while policy-wise there might not be huge differences between what we technically can allow in Drupal 7 vs. Drupal 8, that both the approach and mind-set will certainly be different. It will be different because we'll explicitly target innovations to 8.x releases instead 9.x as we won't open the 9 branch for a while, we'll have more QA around minor releases (e.g. release candidates), etc. While the differences might be negligible on paper, the whole process and mind-set change won't be.

I've discussed this proposal with many folks from outside the core issue queue over the past few weeks, including large-scale Drupal users, smaller Drupal shops, and individual Drupal developers at in-person events. All were overwhelmingly in favor, as are most of the core developers who are represented here. It's also made the rounds on Twitter without huge concerns being raised.

So although there are valid concerns, and we need to address those concerns, I'm comfortable saying that we should move forward with this proposal as-is. The last time Drupal's release cycle was revisited was back in the 4.x days. This modernizes it for the current state of Drupal and its users. Even after today, nothing is ever set in stone and we'll continue to experiment and iterate based on what we learn. A sincere thanks to everyone who has helped with this. Yay!

Wow, this is amazing news! Huzzah! Thanks, Dries and everyone else involved in this effort. :-)

I'd like to echo dsnopek's sentiments and also thank Dries and everyone involved in this proposal, particularly with the extension of D6 LTS.

I've moved the majority of my clients from D6 to D7 but there are a few particularly large and complicated sites that will be costly to upgrade and one of the responsibilities I assume for my clients is to help them make decisions that will give them the best value for their money. I've been tinkering with D8 alpha's for a while now and my hope is to migrate these larger complicated sites right into D8 but I won't know if that will work until D8 is released and the dust settles.

The other concern I have regarding D6 LTS is one of perception. We developers aren't always the best communicators and I have seen some of the "sky is falling, you must upgrade to D7 immediately" messages that have been circulating regarding the impending demise of D6. I've been called into various conversations about this particular topic and my message has been "yes, we will have to upgrade your site in the near future but the issue has not been addressed officially and until we know when D8 will be released and when D6 support will be dropped we should prepare but not panic". I think once this proposal has been ratified, a cohesive, official message about D6 LTS would be helpful to calm those D6 site owners who have been rattled.

Thanks again for all of the work on this everyone!

Andrew

Status:Fixed» Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Please see #2139769-6: [Policy, no patch] Block 8.0.0 and possibly minor versions on critical security issues. which is a proposal from Michael Hess, the security team lead, on how we might help address some of the security team burden concerns brought up in this issue.

I've just been into Drupal for about a Year now and I am possibly the less indicated person to give an opinion, but I'd love to contribute my two cents.

On the first post they talk about "insufficient commercial incentive to contribute, contributor burnout" and later on "persona 3 and 4 are the important user base".

Wordpress success is primarily based on the fact that it is heavily oriented to"persona 3", which is the key to obtain "financial support".

If there was an economic ecosystem to allow "persona 3" implementators to fairly fund the efforts made by personas 1 and 2, we would probably have a less "burnouted contributors" base.

This might sound like a crazy idea, but imagine this situation:

- I am a drupal implementation company and use SQL Server driver all over my projects, so I decide to monthly tip €0.5 to the driver's project in Drupal.org. For a little 7,000 installations this could mean a potential resource of €3,500/month to keep the module up to date, or maybe I decide to make a on-time tip to the maintainers, depending on how active the module is and how mucho funding they have already got.

- There is an issue or feature request for a module or core, so people crowdfund the issue with small tips. The maintainers (or even third parties) are incentivated to work on the issue and propose patches, once these are commited and tested, the bounties are fairly distributed among the people that participated in solving the issue, this might even include non-contributors as long as the patch has been properly proposed and tested.

- All this funding is done through Drupal.org that is in charge of making sure it is fairly distributed and that things don't get out of hand. For issues and feature requests, Drupal will retain the funds until it is resolved, and fairly distribute it among people that contributed.

Of course to make anything like this work:

- Drupal.org should keep a % of all transactions, to cover costs and fees, and would act as a Big Brother: money always leads to discussions.

- Rules as to how is this funding is going to be distributed should be made clear.
Commits are properly attributed and this information is shown on each user's page, as well as the amount of found received.

- Everything should be fully transparent, so that people can fairly put the resources they can in the issues,modules and even contributors that most need them.

This model will "empower" low-budget implementators (you know Drupal is one of the most expensive CMS's to implement) but properly transmit resources to the people that are making Drupal possible so to keep it going properly.

Without a proper and sustained economic model, there is little point in talking about other things.

There is much more underlying complexity under this idea and this speech is probably totally against what Open Source should be, but I just wanted to "fume" my feelings as to why there is so much abandonement in contributed modules. There are so many uncommited "fixed" bugs....

Due to Drupal's complexity there is a big gap between what "personas 1 and 2" are able to accomplish with ease when compared to "personas 3 and 4", but it is exactly this gap that makes it technologically the best out there.

Hey,

i try to figure out this thread and especially the decission. Their are good reasons for every possible decission. Can someone answer me (and possible other not so involed people):

A: Has their been a decission yet?

B: If yes, what is it? How long after the D8-release will D6 be supported?

C: If no, in what (time-)frame the decission will be made?

Thanks for all the work.

I think adding release cycles that include feature updates for Drupal 8 will keep the community engaged with Drupal 8 longer, rather than looking immediately toward Drupal 9 after 8's first stable release.

A: Has their been a decission yet?

B: If yes, what is it? How long after the D8-release will D6 be supported?

C: If no, in what (time-)frame the decission will be made?

The decision was made in #73 to essentially adopt what's in the issue summary with respect to how to manage 8.y.z versions. However, with regards to D6 security support, that is still being discussed in #2136029: Decide if and how to extend D6 security support 6-24 months past an 8.0 release. As far as I know, there has not yet been an official policy change to extend D6 support beyond the 8.0 release, nor yet a timeframe announced for making that decision. It's something Dries and others want to do (see Dries's comment in #73), but we still need to resolve that issue before that can be committed to and announced.