New module update numbering scheme for contrib modules

Personally this strikes me as way too complicated and also a bit brittle (more than 10 updates in a minor version, while not likely to happen, definitely could)...

I think a better approach would be to add in the ability to the update system for developers to associate meta data with their updates. I've no idea how this would work, but maybe something like:

module_update_47_info() {
  return array(
    'version' => 1.4,
    'description' => t('Adds a new column to the foo table so that we can track bar.'),
  );
}

module_update_47() {
  // Do the update stuff.
}

Then in the UI we could see something like:

foo.module:
[ Update 47: Adds a new column to the foo table so that we can track bar. [V] ]

i'm in favor of a convention, and have been using one myself for my own contribs. it's similar to core, and sort of like what's proposed here, but i think more simple and easy to understand/use:

3 parts:
1 digit for core compatibility (jah be praised, only need 1 digit for that anymore)
1 digit for major release version (e.g. is this the 5.x-1.* (1) or 5.x-2.* (2) series of your module?)
2 digits for counting (10 is way too low of a limit, but 100 should be more than enough).

so, for example:

signup_update_5200()

this is the first update to get your DB ready to run signup 5.x-2.*

also, my current convention says that the 2nd digit should be 0 for initial porting to the new core API. so, when porting to 6.x, you'd start with:

hook_update_6000()

i.e. everything that's running 6.x of your module needs these updates. only updates that were specific to the 6.x-1.* series would need to happen as hook_update_61XX(), so that, for example, people could jump directly from 5.x-2.* to 6.x-2.*, and they'd get 60XX and 62XX, but not 61XX.

make sense?

that said, i'm not inherently opposed to the kind of meta data stuff webchick is talking about, but i think that's a totally separate issue. the reason core has these updates numbered in the 1000s is to handle CVS branching. if everything is totally sequential, and then we decide we need to update the schema to fix a security problem in 5.1 core, we're screwed. however, since all the 6.x updates are 1000 updates away, we've still got room to make some updates specific to 5.x that don't break everything when you eventually upgrade to 6.x.

the exact same problem exists in contrib (only, it's even more of a concern, since you can have N branches of your module that are all compatibile with 5.x core). that's the real reason we need a convention like this for contrib. the self-documenting clarity about what version a given set of updates is for is just a bonus.

thanks,
-derek

dww's approach sounds sane. It provides the advantages of sun's without requiring too much thinking to know what the next number should be. :)

And yes, my comment was off-topic. I hadn't looked at this issue closely enough to see that it was a documentation issue and not a core issue; please disregard.

Structured codes (compound keys) are generally a bad idea: in the future, you renumber things, and everything breaks down.

Can we just have it simple? Major API + 3 digit sequence?

@kbahey: That's how it works for core; because each core version is a major version.. 5, 6, 7, 8... so update hooks are (from now on) numbered 6000, 6001, 6002.... until Drupal 7.x when it becomes 7000, 7001, 7002....

However, in contrib, Project module 5.x-1.x and 5.x-2.x could be *completely different* schema-wise. Therefore, you need the extra number to tell which version it's associated with:

5100, 5101, 5102... for 5.x-1.x. ... which is all most contributed modules will ever use.
5200, 5201, 5202... for 5.x-2.x.

Pardon me; I don't want to confuse this... 5.x schema numbers should NOT change. So please amend the last part of my reply to:

However, in contrib, Project module 6.x-1.x and 6.x-2.x could be *completely different* schema-wise. Therefore, you need the extra number to tell which version it's associated with:

6100, 6101, 6102... for 6.x-1.x. ... which is all most contributed modules will ever use.
6200, 6201, 6202... for 6.x-2.x.

cool, thanks. however, you've got some typos:

+ * - mymodule_update_6100()
+ *   - This is the first update to get the database ready to run mymodule 5.x-1.*.
+ * - mymodule_update_6200()
+ *   - This is the first update to get the database ready to run mymodule 5.x-2.*. Users can directly update from 5.x-2.* to 6.x-2.* and they get all 60XX and 62XX updates, but not 61XX.,

update 6100 == 6.x-1.*, not 5.x-1.*...

"Users can directly update from 5.x-2.* to 6.x-2.* and they get all 60XX and 62XX updates, but not 61XX."

Is this true? How would Drupal not hit all of the updates higher than the current number..?

Ah, got it. :) Duh, sorry. :)

Now, two comments:

1. I'd link somewhere off to http://drupal.org/node/93999 for a "for more information" link about contrib version numbers... there might be a better page, but that seemed to be the best from what I could find.

2. I would reserve X0XX to core, and have contrib modules use only the X1XX, X2XX, etc. namespace. The text as it reads now seems to imply that for my first initial port from 5.x-1.x to 6.x-1.x I would number it 6000, but for any subsequent updates it would be 6101, 6102... which seems confusing.

Ah, got it. :) Duh, sorry. :)

Now, two comments:

1. I'd link somewhere off to http://drupal.org/node/93999 for a "for more information" link about contrib version numbers... there might be a better page, but that seemed to be the best from what I could find.

2. I would reserve X0XX to core, and have contrib modules use only the X1XX, X2XX, etc. namespace. The text as it reads now seems to imply that for my first initial port from 5.x-1.x to 6.x-1.x I would number it 6000, but for any subsequent updates it would be 6101, 6102... which seems confusing.

@webchick: "I would reserve X0XX to core, and have contrib modules use only the X1XX, X2XX, etc. namespace. The text as it reads now seems to imply that for my first initial port from 5.x-1.x to 6.x-1.x I would number it 6000, but for any subsequent updates it would be 6101, 6102... which seems confusing."

that misses the point that i made in #2 which sun clarified in #13.

here's a scenario that describes in more detail the case i'm worried about:

• updates 600X live in HEAD while you're porting to 6.x core.
• when you branch DRUPAL-6, you've got 600X in foo.install already.
• say you create a DRUPAL-6--2 branch and start working on 6.x-2.*. you start adding 620X updates for new features.
• meanwhile, you discover some critical security problem that can only be solved by means of a schema change (could happen). on DRUPAL-6 branch, that'd have to be update 6100.
• however, sites that are already running 6.x-2.0 have schema_version == 6201 in {system} already, so they're not going to run 6100 again. furthermore, update 6100 only lives on DRUPAL-6 branch. so, on DRUPAL-6--2 branch, you'd have to add, say 6202 for this security update. plus, for all you know, the schema change would have to be different on this branch, since updates 6200 and 6201 might have already changed something related to what you'd try to do for the security problem. in fact, maybe the security problem doesn't exist in DRUPAL-6--2 due to the changes you already made there, and you *only* need the new update on DRUPAL-6 branch for 6.x-1.* users
• now, someone who hadn't updated to 6.x yet decides to use this module. they look at the release history and decide they want the new features from 6.x-2.* branch. so, they need to upgrade from 5.x-1.* directly to 6.x-2.*. they want updates 600X (initial changes for migrating to 6.x that are shared by all versions), and 620X (updates specific to 6.x-2.* releases), but not update 6100, since that'd screw up their schema.

does that all make sense? it's kind of a weird case, but it certainly could happen, and the possibility of something like this is exactly why we need branch-specific update numbers in the first place. end users don't have to think about this crap, only module maintainers, since the schema_version in {system} will automagically do the right thing so long as maintainers understand and follow the convention.

IMHO #16 is now RTBC. however, if folks think it needs further clarification based on this explaination, feel free to set this back to CNW. perhaps this little scenario belongs in a handbook page somewhere, too.

Committed to HEAD (finally) with slight modifications (I reflowed the looooong line.)

As I wrote on the devel list:

What other software has the very first digit go up and the feature set go down? I'm surprised to hear you have this expectation, and I'm curious how wide-spread it is.

One example out of thousands: PHP went from 4.4.* to 5.0.* and no one thought "0 is smaller than 4, this must be an older feature set -- where can I find 5.4.*?"

That seems reasonable to me.

Furthermore: people advocating the "keep your major versions the same" approach keep referring to "API/feature-set" compatibility. Porting to core by definition involves sweeping API changes. I don't see how anyone can possibly claim that 5.x-2.x and 6.x-2.x of any contrib "have the same API". Modules that themselves provide an API, perhaps, but even then it doesn't really matter, because the plugin I wrote that works with your 5.x-2.x API module won't work in 6.x, either. I've got to port to D6, too, so we might as well all start over at 6.x-1.* once we do that.

How will this pan out in 2012 with Drupal 10? (Or maybe the mayan calendar reboot will happen and it'll all be moot...)

@aaron: As I wrote on the devel list:

"B) Nothing about this prevents Drupal from going to version 10. At that point, we go past 9000 and start numbering things 10000. No problem, except a 1 sentence change to the docs. We don't need a delimiter for this."

Here's my reply to dww on the list:

Here are my thoughts. 6.x just tells me what version of drupal it works on. I don't take that number into account as to what features that module has. The module version number tells me what version the module is on. Let's use CiviCRM as an example. It has it's own version numbers because it is not dependent on Drupal. Their next version is 2.1. I don't expect it to be 1.1 for D6 and 2.1 for D5.

Let's use Pear as an example of a module for PHP. I don't expect pear to be version 1.x for PHP5 and 5.x for PHP4.

More text I sent to devel...

On Aug 7, 2008, at 9:57 AM, Morbus Iff wrote:

I want the latest version of the *module*, not the latest version of Drupal core and then the module.

That's the point of embedding the core version in the version strings of contrib in the first place. What you want is fallacy. You can't possibly not care what version of core you're using, since Drupal is inherently non-compatible across versions of core. In Drupal contrib, the world revolves around core. I just chose to embed that indisputable fact into the version strings themselves.

Re views: There's no 5.x-2.*, nor a 6.x-1.*. So "Views 2" is synonymous with "D6 views". It's irrelevant. Panels would be a better example, since at least there's a 5.x-1.* and a 5.x-2.* (though no 6.x-* yet).

OG is a good counter-example. Moshe is at 5.x-7.3 now, and (thank god) he chose to call the first 6.x compatible version 6.x-1.0. Otherwise, at his current rate, he would have been at 7.x-21.0 in 2 years.

Part of the difference is how Earl vs. Moshe chose to make use of major versions in their own development workflows. Roughly speaking, every coherent set of new features and bug fixes resulted in a new official release of views (5.x-1.5 -> 5.x-1.6), and resulted in a new branch/major version for OG (5.x-6.* -> 5.x-7.0). Given those development styles, it makes perfect sense that Earl puts a lot more weight into his major versions, and wanted to preserve them, while Moshe does/did not.

The main point is this: no matter what we do, the version strings alone will confuse people. No matter what convention anyone follows, some people will think "ahh, just as I expected" and others will think "WTF? Where did xxxxxx go?" Hence the need for clearly stated intentions in your release notes and on your project pages.

Plus, the tools can't and shouldn't enforce a certain development/release process on all developers (I'm sure you're all saying "amen" at that). I'm trying to document what I think are reasonable practices and try to make sure that the tools don't get in the way of people doing sane things, but ultimately, it's totally fine with me that Earl and Moshe are using the tools in completely different ways.

I don't think telling Earl: "No, you *MUST* call it views 6.x-1.0" is any more helpful than telling Moshe: "No, you *MUST* call it OG 6.x-8.0".

Can we please forget about trying to force one model on both of them?

Thanks,
-Derek

I don't see how anyone can possibly claim that 5.x-2.x and 6.x-2.x of any contrib "have the same API"

This is probably the best example I know of: http://cvs.drupal.org/viewvc.py/drupal/contributions/modules/porterstemmer/ - afaik, the D6 port changed one line. I'd imagine a lot of filter modules like bbcode etc. have very few changes as well.

@earnie, you are now on the docs team. Feel free to edit that page so that it makes sense to someone coming into it new like you.

Note: the purpose of this convention is not just to "help you keep track of what changes go with which versions". The point of the convention is to make it possible to safely have DB updates on different branches of development. See #64212: upgrade system needs to support branches. This isn't about convenience and readability, it's about correctness.

Other than that, I agree with Earnie. I don't see how adding 2 digits helps "reduce confusion" at all. IMHO, all of the energy spent on this topic really boils down to a 2 sentence change to the docs, something like:

If your contributed module reaches major version 10 for some reason, you should move to a 5 digit numbering scheme: 1 for core version, 2 for your major version, 2 for a sequence number. Once you do this, you must ensure that all future updates always use the 5 digit convention, even if you reset your major version for a later version of core.

It could use a little polish, but that's basically all we have to say.

@dww: Your post #17 was illuminating, but there's one situation that I haven't been able to figure out: if I maintain a module for D5 and D6 in parallel, and I want to make the same database change for both versions — if I name this update 5117 and 6105 and someone updates the D5 version and then later upgrades to D6, then they're in trouble.

It seems like sticking to equal (low, old-style D5) numbers is the better way as long as the versions are maintained in synch, or am I missing something?

In the scenario you exposed in #17, you conveniently assumed that your user would "upgrade from 5.x-1.* directly to 6.x-2.*", but the same issue arises when he decides to go to 6.x-1.* first and then to 6.x-2.*...

This is one of the reasons why structured numbering schemes are bad.

I just had this in a module that has a 5.x and 6.x version.

So, I did this:

function foo_update_2() {
 ... some changes to the database schema
}

Both the 5.x branch and 6.x branch have the same function, with the same number (2).

If someone is at update 1, and runs the update while on 5.x before upgrading to 6.x, or only runs the update after they upgrade to 6.x, the result is a good schema, regardless.

If we go for 5xxx and 6xxx then we can either do an update twice or miss one, and each can have dire consequences.

Down with structured numbering.

I am aware of that, and object to it as well. However, the issue in core is not the same in contrib. Core is under the scrutiny of a lot of eyes, gets a lot more testing, and we don't go and add hook_update_X() to stable versions that overlap with the current development version and clash with it in the future (at least not very often). So, I can reluctantly live with it in core.

In contrib, it is totally different, given the parallel updates for 2 or more stable versions, and hence more probability for errors. A sequential numbering scheme is far safer than the structured numbers here, since we can never the same update twice on a particular site.

Hope this is clearer.

It seems that those of us in contrib who stuck to low, old-style hook_update_N() numbers are better off...

@salvis "It seems that those of us in contrib who stuck to low, old-style hook_update_N() numbers are better off..."

I agree, and I am sticking with this, because it allows updates to span versions and hence less probability of missing updates, or running them twice.

Simplicity is a virtue ...

@salvis: "It seems that those of us in contrib who stuck to low, old-style hook_update_N() numbers are better off..."

Until you run into the problem that this numbering scheme solves. :(

Frankly, the root of the problem here is that we were lazy in core when we first discovered this problem (#64212: upgrade system needs to support branches) and we never fully thought through the potential problems since core's own schema was a special (trivial) case of what update.php has to potentially deal with. The appeal of a "solution" that didn't change the hook_update_N() API was so great that we just went with that. Then we tried to apply this scheme in contrib to avoid the same problem, and yet it appears that by solving 1 problem we introduced another.

<claim class="bold unsubstantiated">
I need to think through the case that's currently failing more, but part of me thinks that it's being caused by people committing hook_update_N() implementations to the wrong branches at the wrong times.
</claim>

Anyway, I'm not yet going to agree that going to old-style N numbering is a good idea.

Either way, we should probably bump #64212 to 7.x-dev and propose an API change that will solve all cases of this problem in the future, based on our experiences here.

@dww: I need to think through the case that's currently failing more, but part of me thinks that it's being caused by people committing hook_update_N() implementations to the wrong branches at the wrong times.

Not at all. I'm maintaining a D5 version in the DRUPAL-5 branch and a D6 version in HEAD, and I want to keep their databases synchronized. IOW, I always create a hook_update_N+1() function for both versions at the same time. They have the same N+1, they do the same thing, and it is essential that they're run exactly once, either in the context of the D5 version (if the site is still on D5) or in the context of the D6 version (if the site has been upgraded to D6 before running the update). If the site is upgraded afterwards, the hook_update_N+1() must NOT run again!

The issue that was addressed with the structured numbering involves maintaining versions with dissimilar schemas in parallel, either under different Drupal versions or within the same version, but that is probably the rarer case within contrib.

No, the issue that was addressed with the structured numbering involves the case where an older branch needs *ANY* schema changes at all. I don't think that's particularly rare.

/me sighs -- much confusion abounds.

@earnie: The problem isn't that 5200 is still in the .install file in your example. The problem is that a user *already* ran 5200 on their site when they upgrade from 5.x-1.3 to 5.x-2.0. Now, they want to upgrade to 6.x-2.0, and update.php is going to think it needs to run 6200, but that's not what we want. You're saying the solution to this is "just don't have 6200, they already get that from 5200". However, what about users at 6.x-1.2 who need to upgrade to 6.x-2.0? They're screwed if there is no update 6200. Does that make sense?

@sun: Yes, the more different possible branch + schema scenarios we dream up in here that break one "solution" or another, the more I agree that without support from update.php in core, there's no complete solution to all cases that doesn't involve making some of the updates protect themselves from trying to do the same schema change twice.

@dww: No, the issue that was addressed with the structured numbering involves the case where an older branch needs *ANY* schema changes at all. I don't think that's particularly rare.

This issue only occurs when you fork your module, i.e. when you actively maintain more than one branch with different schemas (like 5.x-1.3 going to 5.x-2.0 with a big schema change, and continuing on to 5.x-1.4 with a smaller schema change (the *ANY* that you mention)). I agree that it's not particularly rare, but it's not the common case.

1. I'd guess that the large majority of maintainers would want their users to upgrade to 5.x-2.x rather than having to indefinitely maintain a competing, incompatible 5.x-1.x branch in parallel with the 5.x-2.x branch.
2. If the schemas of 5.x-1.x and 5.x-2.x evolve independently, then the task of providing an upgrade path from each 5.x-1.x to the then-current 5.x-2.x soon becomes overwhelmingly complex — it can't possibly be solved by a clever numbering scheme, but it will always need inside knowledge in the form of module-specific logic.
3. Maintaining 5.x-1.x and 6.x-1.x with identical schemas (evolving in lock-step) is a very common case in contrib — this is the one that needs to be addressed first and foremost, in contrib.
4. #3 is probably the most desirable way (from the community pov and maintainer pov) to maintain a module, but at some point the maintainer may decide to leave 5.x-1.x behind. At that point (but not sooner!) it would be a good idea to skip some numbers in the N+1 numbering sequence of 6.x-1.x for emergency surgery in 5.x-1.x, but when that becomes necessary, all the later update_N() functions need to be adjusted to check whether the emergency surgery was applied or not. This cannot be automated.

#3 should not be encumbered by attempting to solve #2, which is unsolvable in a generic way.

Note that I'm talking about contrib only here, core is a different matter, but even core resists evolving the 6.x schema from the day the 7.x version was opened up, because maintaining upgrade paths is difficult when the old schema evolves asynchronously.

@earnie: "If the upgrade sequence 5200 hasn't been run they would upgrade when they move to 6.x-2.0 would they not?"

Not if their {system} table thinks that schema_version for this module is already 6105 from when they installed 6.x-1.2. update.php just looks at the current schema number in the system table (C), and tries to run any hook_update_N() it can find where N > C.

@salvis: Yes, there are different use-cases in contrib. But, no, not everyone just has one version of their module, and that's not necessarily a bad thing. Some number (I don't have hard stats) of maintainers are doing exactly what the release system was designed to support (which we could never do at all previously): having both a stable, bug-fix only branch that most people depend on, and a new feature development branch where they can add experimental stuff without a) only doing their new features as they port to a new version of core nor b) adding instability to the version most of their end-users are depending on. This is desirable for both maintainers and end users. The bummer is that core's update.php is not a sufficiently powerful tool on its own to handle all the possible schema migration paths that might result from this situation.

Given this thread is recommended in Writing .install files, but contains debate and a potential later rejection of the renumbering proposal after its initial implementation, before finally coming to a conclusion that apparently we probably should stick with this, it is somewhat unhelpful that the issue summary at the top doesn't actually summarize the issue, but rather is the original proposal—which wasn't the accepted one. (Rather, #2 was accepted.) I've read the thread, and don't think I'm the best person to update the summary... but I suggest someone probably should.

Edit: Well, just realized that the writing .install files version I was reading was the Drupal 5.x branch version, so perhaps this isn't such an issue. :)