Problem/Motivation

1. Our current two digit versioning scheme (e.g., 7.1, 7.2, ..., 7.22) doesn't make it clear how big/interesting/risky/urgent a particular update is. For example:

  • Drupal 7.17 fixed some bugs, added new features, added new hooks, and more.
  • 7.18, 7.19, and 7.20 were security only fixes.
  • 7.21 fixed a regression introduced by 7.20, but was not itself a security fix.
  • 7.22 fixed and added a bunch of stuff, including a BC break.

While this information is readily available in the release notes, it's not at all communicated in the version number itself.

2. Even with the loosened rules for what can be backported to an already released major version, we are still very conservative about backwards compatibility and non-trivial additions, which means many improvements that people would be excited to work on won't make their way into a Drupal version that people actually use until the next major version of core is released (and adopted). Up until Drupal 6, major releases were rarely if ever more than 1 year apart. But D7 took 3 years, and D8 is looking to be at least that long as well. One of the things discussed in DrupalCon Portland's Making core development sustainable core conversation was that a lot of employers/clients don't want to fund work that will take more than a year before it's usable by the business, and similarly, something that far out isn't always that attractive for people to work on in their free time either.

Proposed resolution

Semantic versioning specification

Switch to a three digit versioning scheme, following the Semantic versioning specification as closely as possible. The specification leaves a couple things open for the project to decide. Here's how we propose Drupal apply them:

Software using Semantic Versioning MUST declare a public API. This API could be declared in the code itself or exist strictly in documentation. However it is done, it should be precise and comprehensive...Major version X (X.y.z | X > 0) MUST be incremented if any backwards incompatible changes are introduced to the public API.

What is public API

For purposes of this, we propose defining "the public API" as:

  1. Methods (and their signatures) that are defined in interfaces.
  2. Hooks (and their signatures).
  3. Theme hooks and their variables that are defined in the 'variables' or 'render element' entry of the corresponding hook_theme() implementation.
  4. Form and render element types and their properties that are defined in the corresponding hook_element_info() implementation.

What is not public API

While precise, it is not "comprehensive". Examples of things not included:

  • Public methods of classes without interfaces. For example, most plugin managers, and other services intended more for internal rather than external use.
  • Protected methods of base classes. However, we know that some base classes will be commonly extended by contrib modules.
  • The complete detail of $form within hook_form_FORM_ID_alter() implementations.
  • Additional variables (those not defined by the hook_theme() entry) added to $variables within a hook_preprocess_THEME_HOOK() implementation, even though later running preprocess functions might make use of those additional variables.
  • The order in which hooks are executed.
  • Database and configuration schema.
  • Settings available in settings.php.

Changes to the above are all potential BC breaks, and we may want to consider carefully if and when to allow them, but for purposes of SemVer, we propose limiting the definition of "the public API" to the earlier list, so as to provide the most flexibility for what we later want to allow into 8.x. Note that the specification says "should be comprehensive", not "must be comprehensive".

API breaks to resolve critical issues

Also, if a change to even the limited public API is needed to resolve a critical issue, then we will allow it, as per our current policy, without necessarily raising the major version. That would be a clear violation of "Major version X MUST be incremented". However, given what's in the definition of the public API, it's unlikely for that to be needed.

Minor version Y (x.Y.z | x > 0) MUST be incremented if new, backwards compatible functionality is introduced to the public API. It MUST be incremented if any public API functionality is marked as deprecated. It MAY be incremented if substantial new functionality or improvements are introduced within the private code. It MAY include patch level changes.

If we choose to follow the "MUST" portion of this, it means that if a security fix requires an API addition, then we still need to increment the Y version. Which means, the version number alone wouldn't convey meaning as to whether something includes a security fix. Similarly, a benefit to increasing Y is that you can (but don't have to) release small fixes to Y-1 via the Z number. However, if such a small fix is not itself a security fix (e.g., 7.21), then again, the version number alone doesn't tell you whether it's a security fix.

Release cycle

This would allow us to make a radical shift in the release cycle, starting with Drupal 8.0.0. Crell gave a talk on this at DrupalCon https://prague2013.drupal.org/session/future-friendly-evolution-and-drup...

This is catch's slightly different suggestion after lots of discussion both before and after the talk - so not a 1-1 summary of that talk but the principles are similar.

The primary change is that we would not open Drupal 9 immediately after Drupal 8 is released. Instead we would continue to add features and API additions to Drupal 8 - since semantic versioning would allow us the minor backwards compatibility breaks this requires. We'd then line up the changes that we really want to do in Drupal 9, and can only do there (a major refactor of form API is an example of a change that's been discussed for 9.x, and could not go into an 8.x.x release). Once those changes really need a 9.x core branch open (i.e. to commit an initial patch and start on conversions), then and only then, would we open the 9.x branch. 9.x primarily focuses on changes that actually require BC breaks, since new features/API additions can go into later 9.x.x releases anyway - doesn't matter if they miss the boat.

Another big change is that instead of supporting 3 major Drupal versions for security (i.e. direct security team support for 6.x and 7.x at the moment, then 8.x forward ports in the public queue), we'd support a particular x.y LTS for a fixed period. The security team supports LTS releases plus the latest stable, it does not support older, non-LTS releases at all.

A possible timeline might look like this, the exact time between releases, what exactly LTS and non-LTS releases mean and how long they're supported for is all up for discussion, but this particular schedule came up as one which might reduce the impact on the security team and core branch maintainer workload compared to some other possibilities.

Year 0 Month 0 - Drupal 8.0.0 (LTS) released.

Year 0 Month 6 - Drupal 8.1.0 released (with migrate upgrade path, this gives us a chance to refine it before the next LTS if we have to, without supporting that code forever).

Year 1 Month 0 - Drupal 8.2.0 (LTS) released.

Year 1 Month 6 - Drupal 8.3.0 released.

Year 1 Month X - Drupal 9 branch opens (NOTE COMPLETELY ARBITRARY DATE THIS COULD BE EARLIER OR LATER). Now Drupal 8 patches start getting committed to 9.x first).

Year 2 Month 0 - Drupal 8.4.0 LTS released

Year 2 Month 6 - Drupal 8.5.0 released

Year 3 Month x - Drupal 9.0.0 LTS released (when it's ready NOTE COMPLETELY arbitrary date again). Drupal 8 stays on 8.5.0 (not sure about this bit)

Year 3 Month 6 - Drupal 9.1.0

Year 4 Month 0 - Drupal 9.2.0 LTS is released, and Drupal 8.0.x is dropped. If you're still on 8.0.x, you can update to 8.2.x and have LTS support for another year - should be a relatively small upgrade that lots of other sites will already be running.

....

Year 5 Month X - Drupal 10 branch opens)

This means:

1 LTS release per year, each LTS release is usable for four years.

Implications for the security team

Security team supports a maximum of 5 branches at any time (up from 3).

However most of the time it only has to deal with two major API versions at a time - there'd be a shorter window when both 8.x.0 is supported and 10.x is under development compared to now with Drupal 6 and Drupal 8 when that window is 3+ years. Since 8.0.0, 8.2.0, 8.4.0 will have smaller changes, backporting security fixes, while it might not always be a cherry pick, ought to be more predictable than from 8.x to 7.x to 6.x

Implications for site owners

Actual sites know exactly how long they can stay on an LTS release - so there is no more guessing when support will be dropped since it doesn't depend on when the latest version comes out. Additionally the required upgrade will not be to 9.x, but to a later 8.x release - a much smaller jump - so in practice if you started on 8.0.0, you might get 5+ years on some kind of 8.x.x branch.

The only exception for this would be those sites which are built on the last x.y.z LTS release - since when that's dropped the next step is straight to the next major version, but you still get four years, unlike if you launched a Drupal 6.x site in 2014 when support would immediately be dropped.

Implications for contributed module authors

(fill me in)

Implications for core contributors

(fill me in)

Implications for core maintainers

(fill me in)

Arguments against

  1. Per above, this proposal doesn't achieve the goal of conveying how urgent an update is based on its version number. It might, however, help to convey how big/interesting/risky it is.
  2. The key benefit of the Y digit is that you can then support (release fixes for) multiple Y versions. That allows you to be less conservative about what you include in minor releases, which is one of this issue's key goals. However, our security team is currently stretched too thin already, and based on the comments in this issue, is unable to support security fixes for multiple minor versions (see #19, #22, #65). Without that benefit, there's concern that implementing this proposal would involve a lot of infrastructure work to d.o., project module, update manager, etc. with insufficient benefit.

Remaining tasks

  1. Decide if this switch is worth doing
  2. If so, figure out what would need to happen to get it done

Comments

donquixote’s picture

I had some thoughts in a similar direction.
There was more than one place where I talked about that, so here is one of them:
http://groups.drupal.org/node/98564
"Drupal core release management: Exceptions to the feature freeze?"

The idea: Introduce new features or flex points, without breaking the API.
Especially: New hooks usually won't hurt any existing code, except for theoretical nameclashes. And if we had a more rigid naming scheme and/or PHP 5.3 namespaces (that would be D8), we won't even have to think about nameclashes anymore.
On the other hand: New hooks can be totally backwards compatible, but they can become obstacles for future code changes. So, we do need to be careful.

The X.Y.Z scheme could make this easier: X.Y.Z = maintenance, X.Y = backwards-compatible new features and API extensions (new hooks), X = new branch that breaks the API.
Each X.Y would be a branch then.

If we don't want this for D7, here is one dirty trick:
Get D8 out asap (when this discussion has finished), with very few changes, and then start with that new scheme. Yeah, that is not going to happen..
Another idea would be some kind of ping-pong release cycle: D8 would introduce some backwards-compatible changes, and only D9 would be allowed to break the API.
No, I like none of these ideas.

I'd say, too late for that thing for D7, so let's just be one step more permissive in D7 than in D6, especially when it is about new flex points (hooks).

catch’s picture

I would like to see a smoother process for backports between Drupal 8 and 7 than we've had from 7 to 6. A classic example is #1017672: D6 to D7 update process permanently deletes comment bodies and other data, and throws fatal SQL errors which is a critical, data-loss issue against Drupal 7 because we forgot that an issue had to be backported to 6 and 5 after it was committed to 7. It's going to be a lot less risky with the test bot, although that doesn't mean it's risk free.

Having said that, I don't have any suggestions for how to do this - well except perhaps keeping webchick on as a permanent core maintainer for both branches ;). We should also consider opening dedicated backport issues alongside the main development issue versions, rather than moving issues down between versions - where things often get lost with followups etc.

There's already been some indications that new hooks could be backported, I'd like to see non bc-breaking performance fixes get backported as well - mainly because I spent a good chunk of the Drupal 7 cycle trying to un-fork things in Pressflow from Drupal 7, and it'd be a shame if relatively small performance fixes have to be maintained as a patch set and a fork again. We could also backport tests where the functionality is more or less the same.

But yeah it's a good question to ask (less sure on the numbering, we should only be backporting things that are completely invisible unless you need it IMO, and this could result in divergence unless there was some kind of timetable for featurish patches - like mini code-thaws, which sounds horrible).

donquixote’s picture

Another idea: Allow to replace slow or controversial algorithms with contrib code.

This could be how it works:

<?php
function menu_rebuild() {
  $replacement_function = drupal_get_function_replacement('menu_rebuild');
  if ($replacement_function) {
    $replacement_function();
  }
  else {
    // original algorithm.
  }
}
?>

Replacements would be declared via hook_function_replacements() (or should we say "hook_contrib_replacements").
We would need some way to allow the user to sort different candidates..

This way, contrib could become a testing ground to rewrite stinky parts of core.
Core would no longer need to accept heavy and risky patches to improve performance. Instead, it would simply open up yet another algorithm to be replaced with contrib code.
This can easily be done for any branch of core, with minimal risk.

It would not even cause namespace problems, because we actually don't introduce any further hooks except one.

catch’s picture

That's very close to the approach pressflow has taken in D6, for path_alias caching at least. At one point chx wrote a variable_function() helper that was a similar idea too.

This would need to be done case by case, but while it's a little bit ugly, it would allow so many things to be done via contrib that can currently only happen via maintaining patches or forking. Although I'm hopeful that with automated testing, serious performance issues found in D7 we'll actually be able to get the fixes into core still.

donquixote’s picture

I'm hopeful that with automated testing, serious performance issues found in D7 we'll actually be able to get the fixes into core still.

I think we can have both. Let the solutions live in contrib, until they are mature enough.
In contrib, these innovations be tested by a large audience, whereas in the core issue queue, only few people will try a patch (and give feedback) before it is rtbc'd.

I imagine this approach could shorten the typical discussions about core improvements a lot. People can benefit from suggested improvements much earlier (by installing the contrib module), and core developers waste less time on it.

effulgentsia’s picture

Version: 7.x-dev » 8.x-dev
effulgentsia’s picture

Title: Re-introduce X.Y.Z version numbering for core? » Re-introduce Major.Minor.Bugfix version numbering for core?
effulgentsia’s picture

Issue summary: View changes

Updated issue summary.

effulgentsia’s picture

Title: Re-introduce Major.Minor.Bugfix version numbering for core? » Re-introduce Major.Minor.Bugfix version numbering for core? [policy, no patch]
effulgentsia’s picture

RobLoach’s picture

I put up #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc), and didn't know this was already an issue. Close the other as a duplicate?

http://semver.org

effulgentsia’s picture

Hm. That issue seems to be discussing impact to contrib version numbering, whereas this issue is more geared towards the ability for core to ship minor releases that are less conservative than we've traditionally allowed (e.g., pull in latest jQuery and other libraries). Given that, I think it makes sense to keep them both active for now, but maybe at some point closing one of them as a duplicate will make sense.

webchick’s picture

webchick’s picture

...And #1172214: Allow major/minor/revision numbers in release versions where this was marked "Closed (as designed)" :(

EDIT: Oh, right. Because project_release can already do it, technically-speaking.

webchick’s picture

Bottom line is we'd need a bunch of testing with Update Status to make sure it didn't choke if suddenly the core numbering system changed again.

dww’s picture

FYI I just posted #1612910-23: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) in which I address both core and contrib. It's a bit confusing having both of these threads open, but until one is marked a duplicate, I figured I should post here to leave a pointer.

Summary: I'm (tentatively) in favor of bringing back 3 digits for core (although I have some lingering concerns) and strongly opposed to *only* having strict semver for contrib.

Cheers,
-Derek

patcon’s picture

Oops. Posted something related to this in the other thread before reading @dww's final post, indicating core discussion should happen here:
http://drupal.org/node/1612910#comment-6980978

Basically, I'm coming to feel that using semver for all of core together is pretty impractical, and would seem to require that the component modules that make up core should be versioned separately to really do this properly.

Why is trying to semver all of core together impractical? Because how could contrib possibly use semver when we're treating the whole core api as a single unit? What each semver-using contrib maintainer really wants to do is depend on a certain version of specific core modules. So for example, admin_menu might want to depend only on specific versions of -- say -- menu module and system module and some other pieces, and then just lock itself to those. It doesn't care if dblog/book/dashboard make an api-breaking major version bumps, because admin_menu doesn't use those apis (???). Won't Drupal core likely have a tough time moving quickly and fluidly if all core modules keep breaking all their apis and bumping major version in lockstep?

Sooo... how crazy/impossible is this to discuss? :)

joachim’s picture

> Then critical bug-fixes can be released as 8.0.Z, and non-fully-BC improvements can be released as 8.Y.0.

I don't see what this proposal as it stands actually improves.

Currently, we'd have a release history like this:

8.0: big API change
8.1: bugfixes
8.2: API improvements
8.3: bugfixes
8.4: security update

With the new system, the same history would be:

8.0.0: big API change
8.0.1: bugfixes
8.1.0: API improvements
8.1.1: bugfixes
8.1.2: security update

So if you don't want the upheaval up API change on your site, you know to stay on the 8.0.z branch. But that's only fine until the next security update comes along -- at that point, you need to bite the bullet and update to 8.1.2. The only advantage is that you were able to sit on 8.0.z for longer and delay the API change -- which is arguably not an advantage at all, as you now have to make a bigger change to your site in one go rather than incremental ones.

So it seems to me that users gain nothing from this, unless Drupal core is going to backport security fixes to the minor branches -- in other words, release a 8.0.2. at the same time as 8.1.2 with the same patches.

Version numbers will be more complicated for end-users to understand, when really the information that is being conveyed is best read in the release notes.

donquixote’s picture

So it seems to me that users gain nothing from this, unless Drupal core is going to backport security fixes to the minor branches -- in other words, release a 8.0.2. at the same time as 8.1.2 with the same patches.

I think the idea is that this is easy enough with git.
If not, we indeed gain nothing.

dww’s picture

I agree we'd need a clear policy for what the security team and core maintainers are prepared to support. A few possible approaches:

Note: this list has been moved into the issue summary

A) Support every minor branch for any supported major version

We always support security updates on all minor version branches. Even with Git as a powerful tool, probably still too much work for the current sec team and core maintainers to keep up with. We'd probably need to reorganize some things and get more people involved in this work to be able to pull this off.

B) Always support (for sec updates) the X.0.Z branch and the latest X.Y.Z branch

We always maintain X.0.Z as a stable, bug-fix and security-fix only branch.
We always support security updates for the latest minor version branch.
We might optionally support security updates on other minor version branches on a case-by-case basis.

I think the sec team could probably pull this off without a major influx of volunteers and restructuring.

End users would know that if they just want a stable, functioning Drupal core, they can stick with X.0.Z, and update Z with high confidence that it's not going to break anything (with the usual proviso about always testing before deploying a change). If they want/need to upgrade above X.0 to X.Y, they realize they're going to have to keep up with minor versions and possibly rev minor again if there are security updates.

C) Always support the current and previous minor branch for security updates

The same work as option B, but not as good for end users. I don't think this is worth doing, but since someone is probably going to propose it, I wanted to include it for completeness and to express that I think B is better.

D) Only support security fixes on the latest Y minor branch for a given X major branch

Basically the status quo. Sec team and core maintainers could pull this off.

Not obvious it's worth going through all the trouble to switch back to 3 digits if we're only going to do this. Agreed with joachim's concerns in #17.

---

A would be best if it were possible, just in terms of end-user flexibility and satisfaction. But, we'd need a real plan for it and a lot of people to commit to making it work. Short of that, I think B is do-able. I don't think this is worth doing if we're only going to support D. I'll ping the rest of the sec team to ask them to ponder this and comment here.

Cheers,
-Derek

donquixote’s picture

Let's for one moment assume we go for option (A).
What would be the most practical and efficient way to pull this off with git?

Branches and tags:
X.0 will be a parent of X.1
X.1 will be a parent of X.2
Tags X.Y.Z on the respective branch X.Y
etc.
Every commit on X.2 will be merged up to X.3, X.4, X.5 etc.

Bug fix:
- Identify the oldest X.* branch that is affected.
- Commit on the oldest X.* branch.
- Merge upwards into all younger branches within X.*.
- Testing needs to cover all affected branches.
- Sometimes we need corrections on a patch for a specific branch. E.g. a bug fix on X.2 applies nicely to X.3 and X.4, but breaks on X.5, or just doesn't do the job. So we need a second patch for X.5.

Note that this is the opposite of what we do now: Instead of backporting bug fixes, we merge them forward.
(This only applies between minor branches. We don't change the backport policy for major branches.)

donquixote’s picture

I think this is quite a challenge for our current patch + testing infrastructure.
We might consider things like personal branches instead of patches, where commits can be posted in issue comments, etc.
That's a lot of work to do..

greggles’s picture

I appreciate very much the goals of this idea. They would be great in a lot of ways.

On the other hand, I guess that even after 8.1.0 is released that 8.0.Z would need to get security fixes. I don't think that burden can be carried by the security team in private, at least not with our current workload/workers ratio (btw, join sec team).

Basically I'm in favor of dww's solution D with an amendment that the public queue could handle backports to other 8.Y.Z values. That feels like it could get messy.

patcon’s picture

Issue tags: +Composer

I'd just like to point out that we would be straying from semver because we can't always assume that a bugfix or security fix will always be a patch release. If it breaks an API, semver technically considers it a major version bump.

http://semver.org/
> Bug fixes not affecting the API increment the patch version, backwards compatible API additions/changes increment the minor version, and backwards incompatible API changes increment the major version.

Where we're bumping up against convention here (conventional use of semver) is that we take it as the community's responsibility to re-roll security fixes, instead of allowing developers on each version to cherry pick. Again, as you can see in this link, it's much more intuitive to do and share when the community uses a package manager. Instead of pulling in the official "rails" version, one person cherry-picked the security update, applied it to the old version of rails that they were stuck at, and pointed the package manager's file (like composer.json) to build from this custom branch. Anyone else can point to or fork that patched repo and do the same, without any load on the rails team
http://blog.steveklabnik.com/posts/2012-10-04-run-rails-with-custom-patches

so this might be an area where we can't take the load off the core maintainers and sec team until we adopt new practices like composer

Sorry if it sounds like I'm talking gibberish or about things that can't possibly apply to the Drupal world, but I don't believe it's impossible to get there :)

pwolanin’s picture

Issue tags: -Composer

I'd agree with #17, that this is a very bad idea. It's already horribly confusing for people that contrib modules may have several active branches for one core version. Now what?

My 8.x-2.x branch of contrib is combatible with core 8.5.x? How can we explain that coherently?

Or assume I add more to the contrib version - 8.5.x-2.x branch? But what if it's really compatible with 8.0.x through 8.5.x?

In short - let's continue backport features and non-breaking enhancements. If anything needs to change, let's provide a way for modules to declare a core compatibility in .info that's more than a simple major version (unless that's already supported?) and I think that would be enough to account for modules that require a new future?

greggles’s picture

Quick clarification of #22 -

Basically I'm in favor of dww's solution D

I'm not really for or against this idea in general. But, if it does happen, I'm in favor of D with the amendment.

joachim’s picture

> let's provide a way for modules to declare a core compatibility in .info that's more than a simple major version (unless that's already supported?)

AFAIK that's doable on 7.x -- certainly I've seen several contrib modules declare a minimum of, say, core 7.2 in their .info file.

joachim’s picture

I was thinking that dww's option B (B) Always support (for sec updates) the X.0.Z branch and the latest X.Y.Z branch) was sounding pretty good, but I've thought about an unintended consequence this could have.

When we start with this, all is fine. Sites built on 8.0.z stay on the 8.0.z branch, to avoid API change. This is great for early adopters of the 8 branch.

But by the time we're on 8.4.z (say), new sites will be faced with a dilemma: start on 8.0.z and get security patches without API change for the whole D8 lifetime, or start on 8.4.z and effectively get on the API change escalator (because once 8.5.z comes out, 8.4.z will no longer get security updates).

I suspect we'll see a lot of enterprise sites and larger sites opt to play safe and go with 8.0.z even when 8.BIG.z is available, which could skew the usage of core versions. We'll get fewer users of the latest version, and thus fewer bug reports / bug fixes / etc.

RobLoach’s picture

My 8.x-2.x branch of contrib is combatible with core 8.5.x? How can we explain that coherently?

Modules define their own version dependencies in their .info file:
dependencies[] = system (>=8.5)

This could also be applied to contrib module dependencies over at #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) .

I suspect we'll see a lot of enterprise sites and larger sites opt to play safe and go with 8.0.z even when 8.BIG.z is available, which could skew the usage of core versions. We'll get fewer users of the latest version, and thus fewer bug reports / bug fixes / etc.

There arn't any Enterprise sites currently using 8.x, and we still get a whole bunch of bug reports. Sites playing safe is completely up to them. They can update to the latest to get both feature and bug fixes, or they could just stand still on the bug fixes. We can choose to support 8.0.x while the 8.6.x series is coming out, or not. My point is that git makes it extremely easy to push these bug fixes between branches for security updates. If we decide to do this, git makes it easy.

One thing we could do is provide security releases for the past X feature fix releases. Once 8.2.x is out, we stop security releases for 8.0.x. When 8.3.x comes out, we stop security releases for the 8.1.x series. Of course, this is completely up to catch and our willingness to provide this level of support.

greggles’s picture

Re #28's response to:

How can we explain that coherently?

I think that question was asked from the perspective of how we might explain it to people who don't read info files.

dww’s picture

Re: #27:

One thing we could do is provide security releases for the past X feature fix releases. Once 8.2.x is out, we stop security releases for 8.0.x. When 8.3.x comes out, we stop security releases for the 8.1.x series. Of course, this is completely up to catch and our willingness to provide this level of support.

That's what I called option C in #19.

My point is that git makes it extremely easy to push these bug fixes between branches for security updates.

Not always. There could be security holes discovered in new APIs that were added in a later branch (in which case we'd *not* want to try to merge that fix to a previous branch). There might be security fixes that have to work differently in different branches due to different APIs, etc. It's not just as easy as git merge or git cherry-pick in all cases. And, in spite of a ton of automated tools to help, any new release requires a fair bit of human effort to test, prepare release notes, announce, etc.

greggles’s picture

My point is that git makes it extremely easy to push these bug fixes between branches for security updates.

The difference between theory and practice is that in theory there is no difference between theory and practice while in practice... ;)

greggles’s picture

Issue summary: View changes

Updated issue summary.

dww’s picture

Right, what greggles said. ;)

I just updated the issue summary to reflect the current state of this proposal and to include the options for security-supported minor branches. I hope that helps. Others are encouraged to update the summary if they have an alternate proposal for what branches are going to be supported. I think it's premature to hash out implementation and next steps, since it's still not obvious this is worth doing. But, if anyone wants to start fleshing out that section, knock yourselves out. ;)

Cheers,
-Derek

dww’s picture

Issue summary: View changes

updated the summary with the current proposal, links to semver.org, and the current 4 options of what minor branches would be supported by the security team and core maintainers.

dww’s picture

Issue summary: View changes

fixed a typo, fleshed actual next steps, added more context and links to some of the options

RobLoach’s picture

I understand, and am completely with you guys. Knowing what we want to accomplish, and then understanding how we should accomplish that is key. Currently, we're implementing Option C, but only the latest feature release (Is that B?). If we were to compare the current release system to the new versioning system, it might look something like this:

Release type Old system New system
Stable 8.0 8.0.0
Security 8.1 8.0.1
Bug fix 8.2 8.1.0
Security 8.3 8.1.1
Security 8.4 8.1.2
Bug fix 8.5 8.2.0
Bug fix 8.6 8.3.0
Security 8.7 8.3.1

 

This is just an example of how the current release workflow would look like in the new versioning system. We could do more security releases in the older bug fix branches, as dww outlined in #19, but this is just an example of what it might look like. Semantic Versioning doesn't complicate the release workflow, it just allows us to accomplish more than what our current system allows.

This might be Option E? Keep the releases the same, but adopt the x.y.z standard. Only support the latest bug release with security updates. Might be the same as B.

Thanks,
Rob

joachim’s picture

> Once 8.2.x is out, we stop security releases for 8.0.x. When 8.3.x comes out, we stop security releases for the 8.1.x series.

Option C doesn't seem very useful to me.

There is no way for a site to remain on something stable, with no API changes, and a guarantee of ongoing security patches. When 8.0.z stops being maintained, then you have to bite the bullet and have the API change. All we've achieved is that you can delay that moment for a few weeks or months. So the result is that you effectively have API change as frequently as if you kept up to date with the latest 8.y.z, just you're a few weeks or months out of date.

dww’s picture

Currently we're implementing D, not C. I think (hope) we all agree that C is the worst of the 4 options.

Fundamentally our choice is about the number of minor branches we support: 1 (D), 2 (C or B), or all (A).

If we're going to do more than 1 and less than all, B makes much more sense than C and is the equivalent amount of work.

If we stick with D, it's not hugely obvious that it's worth a switch, but I'm still tentatively in favor, since I support the goal of letting the numbers be more self-documenting about what's being included in a given release. Agreed that semver itself isn't more work to maintain, but it will require (a lot of?) work to achieve, and opens us to the possibility of (a lot?) more branches to maintain.

greggles’s picture

There was a time period where we did two security releases at the same time - one that had the security updates plus all bugfixes/features since the last release and one with the security fixes only. We stopped that....in October of 2012 because it was a lot of work and caused confusion. So, we had been doing roughly the same as C for a little while but are now doing something more like a variation of D.

RobLoach’s picture

This would be great for Drush components, considering they are Drupal-version agnostic. A Drush component does not care what version of Drupal is installed. In some cases, it doesn't even care what version of Drush is installed. Cache Audit, for example, is tagged as a 7.x Drush component, when it could actually work just fine on Drush 4.x, 5.x and 6.x.

dww’s picture

@Rob Loach: This issue has nothing to do with drush. Again, please see #322626: META: Package and version non-modules for download instead. This issue is about the versioning scheme for core.

Thanks,
-Derek

RobLoach’s picture

Great! Thanks for the link, dww. Didn't see that one.

klonos’s picture

Sorry for being lame, but I had no reply in #1875516: (dev) version numbers and dates missing from modules listed in the "Available updates" page (regression) since December. Does anyone else see this?

pwolanin’s picture

So, I think having a middle digit to flag no-BC-breaking api additions, major features, etc would be useful to flag those clearly if we think that's going to be important to D8 and beyond.

However, I personally would consider it that we should only move to such a system if we are very, very, explicit and set as firm policy that when 8.1.0 comes out, that's the only forward path from the last 8.0.x release, and that no further 8.0.x releases will be made or supported (e.g. 7.19 to 7.22 might be in this world like 7.0.19 to 7.1.2).

I'm not sure that's a policy that will be understood by the community (or users), so for that reason I'm still thinking this is a bad idea.

pounard’s picture

This kind of policy is used by tons of open source projects, I think that the users you are speaking of are power users (especially if they install their sites themselves), integrators and sysadmins, I think they can understand that.

joachim’s picture

So basically, what's proposed in #41 is option D from #19. For 8.y.z, there is only one y that is currently being supported.

This means that the y is for information only -- nothing changes about the actual release cycle.

pwolanin’s picture

@joachim - yes, option D is the only one we *know* is is feasible.

I disagree with dww, and I think the sec team could certainly NOT pull off option B without a major influx of volunteers and restructuring.

RobLoach’s picture

The documentation about the versioning schema is out there, and the benefits from it are clear, and the majority of open source projects already use it. Our current strange versioning system also already mimics what semantic versioning accomplishes.

The next Drupal 8 alpha release would be 8.0.0-alpha.3 ... The tag would be 8.0.0-alpha.3. A minor version release could be 8.1.0, a new security release from that would be 8.1.1. Whether the security team could handle it or not is irrelevant, the fact is that it's possible, documented, and already used everywhere.

effulgentsia’s picture

The documentation about the versioning schema is out there, and the benefits from it are clear

Strict semver, as proposed in the issue summary, where BC is enforced for minor number increments as well as patch number increments, has some benefits, so I'm not opposed to us proceeding with that (option D). However, it doesn't achieve my original goal for this issue, which was to allow minor API breaks (like reorganizing a form, which breaks hook_form_alter() implementations of that form) on a faster schedule than once every 3 years.

the majority of open source projects already use it

Perhaps. But PHP is not one of them. They allow small BC breaks in minor updates.

Whether the security team could handle it or not is irrelevant

I think it's very relevant. If all we can achieve is option D, then that requires us to use semver strictly, not allow any BC breaks in minor updates, and keep Drupal core evolving much slower between major releases than other projects like Wordpress. However, if we were able/willing to allow small BC breaks in minor updates, as PHP does, and figure out an acceptable security policy related to that, that would be a different situation. If we make a move to strict semver now, how does that affect our ability to later move to a process / versioning scheme that lets us do sweeping API changes on a multi-year cycle and small API changes on a less-than-a-year cycle?

RobLoach’s picture

Strict semver, as proposed in the issue summary

I don't necessarily mean strict semver, just the syntax. I proposed how this would align with the current release system in #33. Should this be updated in the issue summary? Thanks.

effulgentsia’s picture

I don't necessarily mean strict semver, just the syntax.

I don't think they can be so easily decoupled. To the extent that semver gains popularity (as it already has been), if we use a 3-point syntax, people will expect it to follow semver rules. Given that, what if we adopt a 4-point syntax (Major.Semimajor.Minor.Patch)? Within that, minor and patch would follow semver rules. But if we at some point choose to allow for a small BC break without wanting to call that a 9.0, then we can increment the Semimajor number. Maybe we'll decide to never do that during the 8.x lifetime, but maybe we will.

From the http://semver.org/ FAQ:

Semantic Versioning, and the insistence on a well defined public API can keep everyone and everything running smoothly.

But in Drupal 8, the reality is we don't have a well defined public API. We can consider methods defined in an interface to be our public API, but we also have all the internal details of $form open for any module to hook_form_alter() (and the same for nearly every other alter hook). So I think splitting Major into two parts would help acknowledge this reality. Even in the 7.x cycle, we occasionally knowingly made small BC breaks when needed to fix a critical bug, because with all of Drupal's hooks, it's simply too hard to not make a BC break. So having a semimajor component in the version number could at least help us communicate that when we do it.

Crell’s picture

As a data point, Symfony also had some API changes within the 2.x series prior to 2.3. They were largely in smaller packages, but even some more mainline things like flash messages changed in 2.1.

I'm not suggesting we should break things regularly within the 8.x series, but at least one other project isn't treating every possible API change as major-verison-incrementing event; at least not all the time. (Post 2.3, maintaining BC is priority #1 so don't expect that to happen any more until 3.0.)

catch’s picture

@Crell I haven't seen this mentioned recently, so perhaps it was only up to 2.3, but it was my understanding that there's several components (forms being one) that were still considered unstable after 2.3 with the potential for API changes.

Crell’s picture

I think that was up to 2.3 only, but I can't speak for the Symfony core team.

In any event, I'm +1 on switching to semver, having feature-adding non-breaking "dot y" releases, and saying that some things are not part of the API so if they have to change, they have to change, sorry. (For some definition of "some things" that we've not worked out yet.)

larowlan’s picture

First of all 100% behind this.
From my point of view what I like most about this proposal is that we can continue to make api changes.
8.0 becomes 8.0.0 and we guarantee no API changes in the 8.0.x series.
In the 8.0.x series we mark functions as deprecated as well as the version number in which they'll be removed eg @deprecated functions in 8.0.x will be removed in 8.2.0. This means we can continue moving procedural code into services/OO. We retain the old procedural code (as a wrapper) for two minor versions and then dump it. Parts of core will ship in 8.0.0 as a hybrid frankenstein of OO and procedural but we can continue clean up after the release is out, without needing to wait until 9.x. This keeps the momentum around clean-up moving without the rush on features that a new major branch brings.
There were plenty of core conversations in Portland along these lines and one particular quote from an audience member who worked at a large University who said it was hard to justify the budget for a major upgrade (eg 6.x to 7.x) but they could easily justify incremental minor upgrades.
In my opinion this issue should be categorized as critical.
We need to have this discussion finalized before 8.0 comes out - that makes it a release blocker.
Who agrees with me?

kim.pepper’s picture

I'm 100% behind @larowlan on this.

We want to be able to remove deprecated functions between minor versions (e.g. 2 minor versions) and not have to wait until a full major release.

xtfer’s picture

I agree. Semantic versioning is good. We should do this.

greggles’s picture

I don't think anyone is opposed to this in theory (maybe some folks are). I think the breakdown is that it will mean a fair bit of work both one time (making update module and the packaging system on d.o and the release system on d.o and our git validation and our documentation work with it) and on an ongoing basis (for the security team).

So, if you are in favor of this, you should also say "...and I'm willing/able to help with X piece." or "...and It hink we can do this like X to reduce the workload."

larowlan’s picture

Priority: Normal » Critical

I am
a) in favour of it
b) willing to help with infrastructure related issues
c) willing to help with back/forward porting of patches

In terms of D9 we're circling around a testing initiative and that hinges heavily on continued cleanup of existing code to enable more phpunit conversions which I feel this policy complements.

So being bold and bumping, feel free to downgrade if I'm out of place.

pwolanin’s picture

Priority: Critical » Major

@larowlan - This sounds like a way to make Drupal even less stable and hard to build a site with if you really are proposing to significantly change APIs or make large BC breaks within a release series.

I don't see how this is critical - we could certainly release 8.0 without making this change.

dww’s picture

I've been offline for a few weeks and just now catching up. To re-iterate and summarize what I said in #19:

What the security team and core maintainers are willing to support is absolutely relevant to this discussion. If we go through a ton of work to make a switch that leaves us with the same exact situation we're in now (only 1 branch of core supported for each major release) we haven't done anything but add a digit to make us feel better. The only reason to add another digit is if we're going to use it to start supporting more branches.

So, how many branches of core "we" are willing (and able) to maintain is the key question this issue resolves around. Hence, greggles in #55 is totally right-on.

Cheers,
-Derek

RobLoach’s picture

The syntax could actually align exactly with Drupal core's current even/odd release pattern:

Release type Old release pattern New release pattern
Alpha 8.0-alpha1 8.0.0-alpha.1
Beta 8.0-beta1 8.0.0-beta.1
Major 8.0 8.0.0
Security 8.1 8.0.1
Bug fix 8.2 8.1.0
Security 8.3 8.1.1
Bug fix 8.4 8.2.0
Security 8.5 8.2.1
Bug fix 8.6 8.3.0
Major 9.0 9.0.0

The workflow and maintenance of the release cycle stays the same, just the tags have a different syntax. I like this because it aligns with a standardized versioning syntax that most other open source projects use. It eases the confusion between knowing what is a bug fix release versus a security release. I can help with the education and making sure the Project module works using the tagging syntax.

effulgentsia’s picture

If we go through a ton of work to make a switch that leaves us with the same exact situation we're in now (only 1 branch of core supported for each major release) we haven't done anything but add a digit to make us feel better. The only reason to add another digit is if we're going to use it to start supporting more branches.

#59 mentions "standardized versioning syntax that most other open source projects use" and "eases the confusion between knowing what is a bug fix release versus a security release" as something a little more tangible than "make us feel better". But whether that alone is enough to justify the work is questionable.

One more benefit to add to that list is that we can decide during the 8.x lifetime whether we want to support security fixes on more than one minor. We can release 8.0.0 without changing our security policy, but maybe at around 8.6.0, decide that we have the ability and desire to start supporting multiple (but not necessarily all) minors, and that doing so allows us to be less conservative and introduce more innovations into our minors. Whereas, if we release 8.0 based on our current security resource limitations, we limit 8.x innovation for its entire lifetime.

dww’s picture

I don't understand your table, unless by "bug fix" you mean "BC API changing fix and new features". If it's really just fixing bugs, why rev the minor version? According to my understanding of semver, if you're just fixing bugs, that's a patch-level change.

Unless you don't mean semver at all, but "Drupal-modified 3 digit version number semantics". ;) I already asked this question, but maybe it was in another thread -- are we doing the standards-compliant world any favors adopting the outward appearance of an agreed-upon standard (3 digit version numbers ala semver) but then changing what we mean by it?

And even if we *did* adopt the semantics that semver means, there will always be cases where a given "bug fix" could easily be classified as either a minor or patch-level change depending on your POV.

Meta: this whole thread is a good example of why even semver is confusing, arbitrary and ambiguous. You can clearly define what each digit is supposed to mean (either the standard semver, or the modified Drupalism proposed here), but then in practice, it's not always so clear cut what a given change actually entails and what semantics you're trying to guarantee with your version numbers...

I'm not an obstructionist. I just want us to be extremely clear about what we're actually doing if we invest a lot of effort to change this (again). So far, my impression of this thread is "Standards are great! Everyone's using them! But, they don't exactly match our reality. So let's bend the standards to match our reality." That makes me nervous...

Thanks,
-Derek

dww’s picture

Cross post with effulgentsia. My point is that 3 digits aren't necessarily more "self-documenting" for security-only fixes if that's just our Drupal-specific interpretation of what our 3 digits means. Sure, we can document it that way, and it'll be a little better than what we have now, but I don't think it's as clear-cut a win as it's being made out to be.

If we introduce this in 8.0.0, what happens to 7.x? We'll still be supporting both for years. Do we leave 7.x at 2 digits and have 8.y.z and 9.y.z as 3 digits? Is that actually any better for our user-base?

In some ways it's cleaner to (re)introduce 3 digits at the start of a major version, but not necessarily required. We could potentially do this at any time, including in the middle of 7.x. It might be a little messy, and it'll require some work for the Update Manager not to break, but it's at least possible. And in some ways, introducing this across all supported versions of core simultaneously seems better to me than just starting 8.0.0 like that even if we have no immediate intention of using the additional flexibility. Don't quote me on this final paragraph, I'm just thinking out loud for now. ;)

effulgentsia’s picture

Assigned: Unassigned » effulgentsia

I'm working on an issue summary update to try to incorporate the comments since #45.

effulgentsia’s picture

Issue summary: View changes

added ids for minor branch support policy choices

effulgentsia’s picture

Issue summary: View changes

Updated summary to clarify specifics about how to integrate SemVer specification

effulgentsia’s picture

Issue summary: View changes

Bolded key phrases for easier scanning

effulgentsia’s picture

Title: Re-introduce Major.Minor.Bugfix version numbering for core? [policy, no patch] » [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x
Assigned: effulgentsia » Unassigned

Done updating the issue summary. Please update it further if I missed something important or included something irrelevant or wrong.

effulgentsia’s picture

Issue summary: View changes

Updated issue summary.

pwolanin’s picture

I think the comments from #19 should be in the summary, so I'll try to add them later - we don't have any reason to believe that our capacity to support security releases will be greater than today within the next year or more, so supporting more than one minor version on this scheme is something that would be left to the future or done inconsistently, and hence not clearly any gain over today given the level of work involved.

pwolanin’s picture

Issue summary: View changes

Updated issue summary.

effulgentsia’s picture

I think the comments from #19 should be in the summary, so I'll try to add them later

The issue summary already included the basic conclusion of that in "Arguments against", point 2. I just now just added links to #19, #22, and #65 to that. Feel free to add more links or info though.

Crell’s picture

This is a major part of my Prague Core Conversation: https://prague2013.drupal.org/session/future-friendly-evolution-and-drup...

Let's talk more there.

pwolanin’s picture

@Crell - are you also volunteering for the security team?

Crell’s picture

The potential impact on the security team is something I will be discussing, yes. I encourage members of the sec team who are able to attend to do so.

greggles’s picture

Maybe we could collaborate on your ideas online since most of the security team won't be at Prague much less able to attend your core conversation.

Either that, or feel free to propose anything that doesn't increase the burden on the security team ;)

Crell’s picture

At the moment, the main thrust is on the benefits to customers and developers and architecture to not having the massive boom-bust cycle of core development, and culturally retraining ourselves to innovate without breaking APIs daily. greggles, if you have some time either this week or the week of the 16th I'd love to talk about how to better flesh out the "without making the Sec team go on strike" section. :-)

greggles’s picture

I think the rough-consensus of the team is pretty clear in this issue already. I can't (or maybe won't is a more accurate word) speak for the team without getting their feedback on an idea first, so a real-time meeting seems suboptimal.

effulgentsia’s picture

@Crell: Here's some things I think would be good for that conversation, in relation to security team impact:

  1. Assuming we went with semver, what kinds of things could we add to 8.Y.0 that would allow us to stop supporting security fixes on 8.y<Y.z?
  2. An alternative to "a new minor at least once a year" could be "a new major every year". Where, a new major could break some APIs, but not feel compelled to break all of them, because we could get out of the "get it in to this version or wait 3 years" mindset. What are the relative pros/cons of these approaches? What are examples of other large open source projects that use these differing approaches, and how has that worked for them?
  3. If we went with "a new major every year" approach, would we (as in Drupal users) be ok with still having security releases only supported for 2 majors at any given time? What would be required to be ok with that?
  4. Who are the people / organizations (I don't necessarily mean specific names, but personas) who would most benefit from having more than 2 branches (whether that's >2 majors or 2 majors with >1 minor per major) security supported at a time? What would it take for them to contribute resources to the sec team to enable that?
donquixote’s picture

Just speaking from my own experience:
There are sites that are created once, with a medium or large effort, but then run on minimal maintenance (security only) for e.g. 2 or 3 years, because neither the client nor the contractor has the time or resources.
It may not even be a money thing (although it usually is), but it can be a question of focus and availability.

Dustin@PI’s picture

This question seems really important to the future of Drupal. Any chance it could answered at one of the core conversations at Prague?

Not knowing whether we will have to wait 3 years for D9 to add features, or if there'll be D8 minor releases with the chance to add new (backwards compatible) features effects how people treat the current API freeze.

(PS I won't be there but I vote for adding minor releases).

marcingy’s picture

Backwards compatible features are already allowed and have happened during the D7 release cycle already.

Dustin@PI’s picture

@marcingy: I know that there have been features in the form of new hooks and small api additions--but I was under the impression that there were no string changes (translation freeze), so no new features that would require UI change, am I incorrect?

If so, that makes some form of semantic versioning even more important to me--as that would mean I had new features to my users without realising it.

marcingy’s picture

I see what you mean, but just because we had semantic version doesn't mean that translation strings will be allow to change any more than now. It really depends what is consider ok for a minor and how often will a major version be released. E.g for drupal 7 we have had some features added and some security releases so rather than 7.23 we might be on say 7.7.0 (this is made up). Major release cycles in drupal currently have no fixed time table drupal 8 as per drupal 7 will be released when it is ready. So what I am trying to say is simply by changing how the numbering works doesn't mean that further reaching features will hit the release branch any faster.

Xano’s picture

Changing the numbering system may, however, more accurately reflect what we do or want to do. We have already added some minor features to Drupal 7, but our version numbers did not reflect that. Switching to a versioning system that does reflect what we do, will make it easier for people to understand the impact of a new version.

@marcingy: I know that there have been features in the form of new hooks and small api additions--but I was under the impression that there were no string changes (translation freeze), so no new features that would require UI change, am I incorrect?

In most cases a change means breaking backwards compatibility (BC), which is not allowed after a certain point in time (July 1st, 2013 for Drupal 8) and certainly not after the first stable release (Drupal 8.0 or 8.0.0, for instance), with the exception of changes that fix security bugs. Changing a string means that all translations of the old string will no longer apply and that sites that are upgraded will show the new, untranslated string (because it takes a while for translations to be updated).

klonos’s picture

I only found out about Backdrop today and it seems to promise what to me seems a more logic release cycle (more or less what is being discussed here): http://backdropcms.org/#release-cycles

patcon’s picture

From summary - Arguments against:

1. Per above, this proposal doesn't achieve the goal of conveying how urgent an update is based on its version number. It might, however, help to convey how big/interesting/risky it is.

Security release info doesn't need to be addressed in the versioning scheme. I don't know of any other place that does. In fact, this would be a great opportunity be better open source citizens and start managing our vulnerabilities via the Open Source Vulnerabilities Database (OSVDB), which has a Drupal section already. In Ruby-land, a nice approach is to use bundle-audit gem, which checks against the OSVDB and warns accordingly. Bundler-audit uses a second git repo that scrapes from the OSVDB API, since it's rate-limited. But perhaps with the PHP/composer community support, we could set up a proxy service and work on building an analogous composer-audit tool. That would seem forward-thinking, and we wouldn't be going it alone.

2. The key benefit of the Y digit is that you can then support (release fixes for) multiple Y versions. [...] However, our security team is currently stretched too thin already,

Why would we support multiple Y releases with security patches if we're truly following semver? Everyone should be able to ride with the most recent Y, so security releases only need to be supported for the HEAD of a major X release. This is because nothing that we release with Y bumps should break anything that contrib is tying itself to. If I understand correctly, #2 is no longer an issue if, as I tried to make the case for in #1, there is no need to try to keep security release metadata in the versioning scheme.

Note:If you think I'm rehashing anything critical, perhaps just call me out, and I'll contact you on IRC and we can hash it out before updating this convoluted thread :)

effulgentsia’s picture

Why would we support multiple Y releases with security patches if we're truly following semver? Everyone should be able to ride with the most recent Y, so security releases only need to be supported for the HEAD of a major X release. This is because nothing that we release with Y bumps should break anything that contrib is tying itself to.

See the proposed resolution portion of the issue summary, the items I'm suggesting we exclude from our definition of the public API as far as semver is concerned. The reality is that even Drupal 8 is not yet fully encapsulated into interfaces decoupled from implementations, which leads to the 2nd problem/motivation in the issue summary. So I don't believe that we can simultaneously achieve "more interesting/innovative minor releases than what we've done for D7" and "everyone can update to the recent Y without worrying about any BC break". See also #46 and #49 about how PHP and Symfony also don't maintain strict BC in minor updates; however they do provide security releases for multiple minors to compensate, something we're not yet willing to do with our current sec team resources.

greggles’s picture

In fact, this would be a great opportunity be better open source citizens and start managing our vulnerabilities via the Open Source Vulnerabilities Database (OSVDB), which has a Drupal section already.

Could you propose that with more details at https://drupal.org/project/issues/securitydrupalorg ? I don't see a reason to couple this concept to the semantic versioning issue.

In Ruby-land, a nice approach is to use bundle-audit gem, which checks against the OSVDB and warns accordingly.

You mean bundler-audit? This sounds like our update.module, right? Are there benefits to bundler-audit beyond update.module?

patcon’s picture

patcon’s picture

Issue summary: View changes

Updated issue summary.

catch’s picture

Issue summary: View changes

Updated issue summary.

RobLoach’s picture

greggles’s picture

Larry's plan pushes packaging to the DA (makes sense to me to some extent, afaik they're already doing that). I wonder whether this is a high priority task for them, though.

The first mention of security is 24 minutes in. The mention of semantic versioning says something like "If we find security issues we fix them the same way we do now." Seems fine to me if what you mean is "only support the latest 2 branches for security updates".

At 29 minutes in Larry covers security in more detail and the plan deviates from the first proposal. At this point Larry suggests cherry picking should work fine and that it's "not that big of a deal" and a solvable problem and suggests the DA should solve this. This I can't support. I don't know how you reconcile the plan with the mission of the DA (not to work on code). Even if it were possible in their mission, adding paid people to a volunteer team isn't simple 1+1 = 2 math. We've discussed how to handle adding funding to the security team for years and haven't agreed on a plan that people feel comfortable with.

joachim’s picture

My experience over at Flag module, where we currently have 4 branches active at varying levels of maintenance (2 each for D7 and D6), is that cherry-picking works, until it doesn't. If your branches haven't changed much, then you're fine. If your bugfix affects code that's been cleaned up, or refactored, or had extra features added, then you're backporting patches. Or waiting and hoping for the community to do it...

coltrane’s picture

crell mentions the DA getting companies to pay members of the security team, a distinction that I think would avoid the DA getting involved with Drupal code.

The DA has collective weight and authority that could assist with getting members of the Security Team paid for some duties, but I do agree with greggles point that adding paid people to an otherwise volunteer team isn't simple. That complexity shouldn't mean this option isn't rejected tho, it just needs a thorough understanding and discussion before implementing, if at all.

Dustin@PI’s picture

Another issue that we'll hit is contrib modules being dependant on particular versions of Drupal for new API additions ... example "my_contrib" module version 8.2.0 requires Core 8.1.0 or higher, but "my_contrib" module version 8.1.0 supports core between version 8.0.0 but less then 8.2.0. The composer syntax supports something like this, but there's no support in D8--except for custom module dependencies (but maybe we could add support in D8.2.0 : D )

PS: I really hope we do this in a way that is similar to what's described in Crell's discussion/Sun's recent changes to the description of this issue. This issue/policy would get my +1 .. I would love to see this moving to fixed and starting to tackle the problems and challenges being raised.

Xano’s picture

@Dustin@PI: Drupal 7 already introduced support for versioned dependencies. Or did I miss your point?

Dustin@PI’s picture

@Xano: Thanks for letting me know ... I did not know that functionality was there.

But I was referring to the dependency on Core .. so we'll have to extend that functionality to the Core version as well (i.e. "my_contrib" only works with core greater then 8.1.0 due to using an API that got added in 8.1.0). Currently the core dependency is limited to the major version ("modules cannot specify the minor version of a branch of Drupal").

But the awesome thing is that we could add this in Drupal 8.1.0 and be fine.

Xano’s picture

Modules can specify a dependency on a minor version of core by adding a dependency on System. Effectively, this gives us the same possibilities as an actual versioned dependency on core, but it's just not as pretty.

patcon’s picture

Dustin@Pl

[...] but "my_contrib" module version 8.1.0 supports core between version 8.0.0 but less then 8.2.0.

Just wanted to point out that this would be very unlikely if "my_contrib" were using "public API's" (whatever that means for Drupal), since minor version bumps shouldn't break anything. If a ceiling constraint is required, then someone probably messed up :)

Xano’s picture

Stable Drupal core branches break BC on rare occasions. IIRC Drupal 6 once did that because of a security issue and Drupal 7 did it earlier this year to remedy a serious performance issue, with the possibility to use the old API instead.

catch’s picture

Versioned dependencies against core is the least of our worries with this I think. It's something that shouldn't be necessary for every module, but we do need to support it and it's very fixable in core if we decide to do this as a whole.

The DA can't direct Drupal core development, but it also shouldn't obstruct it - so packaging and other d.o requirements to be able to do this will need to block the 8.0.0 release (or at very minimum the first point release afterwards) and we'll need to inform the DA that that's the case.

The main challenge here is the number of branches that need to be supported at once - both for the security team, for branch committers, and people working on backports of patches that don't cherry pick.

For the couple of weeks before Drupalcon and during many discussions at Drupalcon I'd been trying to think how that might work. It wasn't brought up that much in Larry's talk (which apart from frequency of minor releases I agree with on most points now), but I don't think that means we can't make it manageable.

Couple of things:

1. While fixes to the latest development release are fixed in the public issue queue, it's usually a consideration when working on fixes for the stable branches. This proposal means we'd almost certainly not open 9.x up until 12-18 months after 8.0.0 is released so that's one potentially highly divergent branch taken out of things at least to start with.

2. We don't need to support every minor branch during the 8.x cycle. The issue summary currently has a possible schedule of 6 month releases, with every other release being LTS (means a maximum of 5 branches at any one time, but at least two of them minor rather than major versions).

We could also do something like:

Year 0 month 0 8.0.x (supported for 1 year)

Year 0 month 6 8.1.x supported for 1 year)

Year 1 month 0 8.2.x (LTS 4 years)

Year 1 month 6 (8.3.x supported for 1 year)

(Drupal 9.x branch opens)

Year 2 month 0 (8.4.x) - (last Drupal 8 release, also LTS 4 years).

Year 2 month x Drupal 9.0.0

For maintainers and the security team, there'd be a maximum of 3 8.x branches to support at any one time, and two for most of the cycle. There'd only be one 7.x branch, and with 9.x we'd start the same thing again. It does mean 4-5 branches to support instead of two, but I don't think it's double the work at all - the minor branches I'd expect to be an extra 10% or 20% work compared to a major branch.

That'd means a year after 8.0.0 is released, sites that are waiting for contrib and/or specifically want to stay on LTS know there's a version that's going to be around for four years. Others can follow the minor releases until the last one, then are able to stay on that for a good chunk of time as well. For the first year you either need to upgrade to the LTS when it comes out, or you're planning to upgrade for each minor anyway. So only supporting one or two minor branches will have some impact on sites + contrib, but not necessarily a bad one at all - just needs to be communicated up front.

3. We can also do things like test patches (and contributed modules) against each supported minor branch + HEAD (and add a private test bot for the security team that does the same). That'd mean it'll be obvious which patches cherry-pick and which need backporting - since patches won't apply or will fail tests.

I also think even if we release 8.0.0, we don't need to guarantee when 8.1.x comes out, so if technology and/or resource issues aren't solved we can simply refuse to do the additional work of supporting two minor branches until they are.

patcon’s picture

@Xano

Stable Drupal core branches break BC on rare occasions. [...] Drupal 7 did it earlier this year [...] with the possibility to use the old API instead.

Not to nitpick, but I don't understand the above :)

If you can use the old API, then it's backward compat, right? I believe the situation you're describing is semver-compat, no?

@catch I have some questions, but I'll catch you on IRC to clarify before coming back here.

ianthomas_uk’s picture

@catch #95 Why six monthly? If you've got LTS releases after a year and after the 9.x fork, then the security team can just say they will support a version until 8.(x+2) is released (which is what you're proposing, except fixed at 6 months). If that's too fast for a company, then they should use LTS.

@patcon #96 See https://drupal.org/drupal-7.20-release-notes - The "backwards compatibility" was a setting to disable the security patch... (or at least part of it).

ianthomas_uk’s picture

Where would bug fixes be applied?
- All branches that are maintained for security fixes
- The latest 8.x branch only
- Only as part of an 8.x.0 release, reserving .y releases for security fixes only

Pasqualle’s picture

My only concern is, what would 8.1, 8.2, ... mean for contrib developers.
If the api change breaks my contrib module, then for me Drupal 8.1 is as much work as Drupal 9 would be, as I would have to maintain different versions of my module for Drupal 8.0 and Drupal 8.1. Porting is easy, maintaining is hard.

As I see contrib devs currently only maintain 1 stable version of their modules for D7. D6 modules are not maintained any more. From different point releases 1 is the "upcoming" great feature release and 1 is the stable release. Any other release is just garbage mostly.
As I see having even 1 stable release is just too much work in contrib space, requiring more releases for different core 8.1, 8.2 compatibility could easily crash the Drupal contrib.

If an api change affects most of the contrib modules then it should not be Drupal 8.1 it should be called Drupal 9, even if it is a small change.

Pasqualle’s picture

Drupal 7.20 (released 7 months ago) is the latest security release. According to latest usage statistics, 36,82% of the Drupal sites are using version 7.19 and below. To me it means many Drupal owners do not care about how urgent the core update is, adding a number to the version string will not change this trend.

Xano’s picture

APIs will not be changed in minor updates, unless very severe problems require us to do so. The only things we can do is introduce API additions. These will not break anything, not even contributed modules.

Crell’s picture

Pasqualle: You're missing the point. 8.1, 8.2, etc. should NOT break APIs. They should add functionality *without* breaking APIs. (Contrib modules that want to rely on a new feature in 8.2 would, of course, only be compatible with 8.2 and later, not 8.1, but we already have support for that and have for years. Non-issue.)

Yes, this is a rather alien concept to Drupal developers. We've brainwashed ourselves to think that new features only happen by breaking APIs. I want to change that expectation.

Crell’s picture

Issue summary: View changes

Updated issue summary.

xjm’s picture

Issue summary: View changes

Updated issue summary.

xjm’s picture

Issue summary: View changes

Updated issue summary.

xjm’s picture

Issue summary: View changes

Updated issue summary.

chx’s picture

> Protected methods of base classes.

In Prague we discussed having base classes deal with the dependency injection and that we will make it best practice to extend these. You can't make methods that everyone is expected to rely on heavily non-public-API.

tim.plunkett’s picture

I partially agree with #103. We have this special subset of base classes that are like "super base classes": ControllerBase, FormBase, PluginBase. Their protected methods *are* an API.

That said, I don't think we need to extend that to every base class ever.

Crell’s picture

We could consider tagging protected methods with @api for those that we expect to be an API for subclasses (eg, the ones in ControllerBase, FormBase, etc.), and not tagging those that are protected as a "just in case" but aren't really intended for subclasses to use most of the time. That provides a clear signaling mechanism to module developers for what to expect.

catch’s picture

The issue summary has a list of 'public API' vs. 'non-public API' examples which is supposed to show things that definitely wouldn't vs. might change in a minor release. Would be good to flesh it out with more examples, but it looks reasonable to me as it is at the moment.

cosmicdreams’s picture

Why can't we stay on version 8 longer? We have an architecture that allows for a lot of non-breaking evolution while at the same time contrib space can do a lot more that could eventually be brought into core without breaking APIs.

The whole point of this new versioning system is to be able to iterate faster, to deliver the results of our work faster, so that we can grow our architecture faster, enable more innovation, and encourage more developers to take part in the process.

I think a good next step would be to create a Drupal 8.0.0 version so we can assign issues to that bucket that either block or enhance it's release. Everything else can stay in Drupal 8.x (or Future in other words).

Let Drupal 9 occur naturally when we're ready to remove everything we deprecate along the way.

Owen Barton’s picture

#95 feels like a good balance of enabling more iterative improvement, whilst keeping a handle on upgrade burdens for site owners. [edit: discussing security team overhead separately for now].

Owen Barton’s picture

Issue summary: View changes

Updated issue summary.

effulgentsia’s picture

FYI: Dries posted a proposal in #2135189: Proposal to manage the Drupal core release cycle. Not closing this as a duplicate yet though, as there may be some scope to this issue not covered by that. If/when we decide there isn't, then lets close it.

Crell’s picture

Issue summary: View changes
Status: Active » Fixed

The other issue implies semver for core rather directly and was accepted, so I think we can mark this fixed. The policy decision is "Yes".

greggles’s picture

The other issue had a followup proposed which seems kinda important:

Develop a plan for what infrastructure changes are needed to the Drupal.org packaging system, update status module, Drush, etc. to allow for this move (separate issue).

Do those issues exist?

catch’s picture

Not that I've seen, no.

webchick’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

clemens.tolboom’s picture

Nice ... but:

  1. where is the change record?
  2. why is there a 8.1.x branch too?
  3. where is the doc?
clemens.tolboom’s picture

Xano’s picture

drumm’s picture