Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
By dww on
now that the debate around version numbers for core has been resolved (it'll be 2-digit version numbers), i'm turning my attention back to the new system for releasing drupal contributions.
there are a couple of main issues in the queue with a series of patches that are getting us close to a working, final implementation:
http://drupal.org/node/83339
http://drupal.org/node/84706
however, the patches are all from before DruaplCon (where there were some good discussions about all this, and some suggestions for changes), and before the switch to 2-digits for core. in writing and thinking about the system over the past few weeks, i've hit a few new complications that i'm unsure how best to resolve. any input on the following would be appreciated...
release nodes that are pointing to the HEAD
in the new system, releases will be real nodes (http://drupal.org/node/83339). if you enable the cvs.module, each release node has a cvs tag/branch that is associated with it (http://drupal.org/node/84706). release nodes that point to a cvs branch (not a tag) get nightly snapshots packaged up, just like the current "releases" we get in contrib. if enabled, cvs.module will form_alter() the release node form to rip out the fields to specify the version #, and add a drop-down to select what cvs tag to use. the only choices would be tags on the project that have no releases associated with them. once the tag is selected, the cvs.module figures out the version # from the tag name. so far, so good...
however, things get a little tricky for the trunk. :( my current plan is that if you select 'HEAD' then you *do* get to specify a version #. so, if project maintainers wanted to have nightly snapshots off the trunk, they'd get to indicate what version that should be.
furthermore, you should be able to edit a release node that's got 'HEAD' for the branch and move it to a different branch. an example should help explain all this:
say, you've got a release node pointing to the trunk for some module that hasn't been fully ported to the 5.x API yet. the version number for these snapshot releases is currently "5.x-0.0-dev" or something. eventually, it's done being ported, and the maintainer creates the "DRUPAL-5" branch. it'd be nice to just change the existing "5.x-0.0-dev" release node to start doing snapshots off the DRUPAL-5 branch, instead. why? because the version field in issues are really stored as the node id (nid) of the associated release node (that's basically how it works now, only it's a release id (rid), not a nid).
it'd suck for all the issues that were marked against "5.x-0.0-dev" (e.g. issues currently marked "cvs") to just automatically jump to being labled 6.x-0.0-dev if you changed the version number on the existing HEAD node. it'd be better that once created, the version number on a release node basically has to stay fixed. but, in the special case of the tag pointing to 'HEAD', it could be moved to point to a more appropraite branch, once the branch was created... then, they could create a new release node pointing to HEAD, with "6.x-0.0-dev" as the version, and that would show up as a different option in the version field in the issue queue.
is that all reasonable? does anyone have a better suggestion for how to handle this mess?
releases for projects that represent subdirectories of other projects
this thread: http://groups.drupal.org/node/1515 about potentially splitting up CCK field modules into their own projects is what got me thinking about this problem. there are a handful of projects that point to subdirectories of other projects, as opposed to being stand-alone projects. eCommerce has a few, and i guess location, event, potentially CCK, and possibly others, too. the motivation is usually to have separate CVS access control and more easily separate issue queues...
unfortunately, these beasts sure complicate things for the new release system. :( i could make it so that the subprojects are forbidden to have release nodes associated with them and say that the releases are all controlled from the parent project. maybe that's the only legit solution... it's certainly closest to what we have now (where the packages are solely based on the directory in CVS, without any real knowledge of the associated project nodes). however, since the use-case for subprojects is to have separate CVS access control for the subdirectory, presumably, that means a different maintainer, too. so, now the parent project maintainer has to coordinate w/ any/all sub-project maintainers about when things should be tagged for new official releases...
is that acceptable? i guess if they don't want to be under the control of official releases of the parent project, they have no buisness living in a subdirectory... if they decided to live as a subdir of the main project, they decided they want to be packaged with the main project, which gives the parent maintainer the control over branching, tagging, and releasing the code, even if another maintainer has been granted commit permissions and a separate issue queue to deal with.
complications because of 2 vs. 3 digits for core
as i wrote in the table of pros and cons in the debate around the new version number conventions for core, switing to 2 digits will complicate things in the new release system. in particular:
- the current implementation of the new release system stores each element of the version string in a separate DB column, since a) that makes it much easier to validate certain things (e.g. sequential releases along each branch), and b) it'll be significantly faster to do certain DB queries we'll need to perform. however, now we have to handle the case that a certain field could be defined or not. for example, to represent the older releases of core, we need to use the major, minor and patch fields. for 5.0 and beyond, we'll just use major and patch. so, the code has to be able to know if a given field isn't in use. seems like i'm going to have to switch to leaving fields NULL in the DB, instead of using 0 as a default, since there are nowreal cases where a field could be missing, but where 0 is a legitimate value. :( sadly, storing NULLs in the DB is a pain, since db_query() doesn't let you use NULL values... ugh. alternatively, i can use -1 as the default, but some folks feared that'd seem a little weirder in the DB since -1 tends to imply some kind of error.
- currently, if you use the latest patches from the issues linked to above, each project gets to define a formatting string to construct the appropriate human-readable version string from these fields (e.g. for naming the tarballs, and for displaying in the issue queue). each field is represented by a '%field_name' that gets replaced by the actual value (e.g. "%major" is replaced by the version_major field of the release node). once again, the fact that %minor or %super_minor might now be missing just screws up everything i had planned. ;) previously, the delimiters (like the '.' between each number in "4.7.3") were handled directly in the formatting string, like so:
%major.%minor.%patch. what if %minor isn't used? we'd end up with "5..0". so, i'm thinking of putting the delimiters directly into the %keyword thing, along with additional logic about only using the delimiter between %keywords that get expanded. so, now the format string would look like%major%minor%patch, and the code would know that if we had {4, 7 and 3} as the values, it should create "4.7.3", but if {5, NULL, and 0} were the values, we'd only need 1 delimiter and we'd end up with "5.0".
any insightful, constructive comments about any of this would be most appreciated. ;)
thanks!
-derek
Comments
Hey derek, since I have a
Hey derek, since I have a couple of the projects that live in subdirectories, I thought I'd chime in on that issue. It makes sense to me for them to be controlled by the parent module. I wouldn't branch them until the parent had branched, anyway. But does this make more work for the maintainer of the parent? I'd like to avoid that. I'm not sure I really understand how the new system will work, so I don't know exactly what the maintainer of the parent and the maintainer of the sub would have to do.
I'd be OK with moving everything out of subdirectories, too. It makes some sense to keep related modules packaged together, but I don't think it would be all that onerous to just move them to separate directories and tell people they need to pick up the downloads separately. Users get confused about what is included and what is not anyway.
sub-projects need coordination on branches, tags and releases
probably the best (if slightly dated) summary of the new system and how it will work is the handout (http://drupal.org/files/new_release.pdf) i gave out at the drupalcon session on this (http://drupal.org/node/86694). that'd be a good thing to read if you want to understand this stuff better.
the bottom line is that if projects point to subdirectories of other projects, that was a conscious decision to have a separate issue queue and project page for something that gets packaged with the parent project. honestly, this seems weird in many respects, the more i think about it. the parent project's tarball includes files/code that aren't supposed to be reported in that same issue queue. how are end users supposed to know this? furthermore, if you care about the subproject, and you go to the subproject's home page, there's no download link. confusing.
sure, downloading a set of modules all at once is nicer than having to get a bunch separately. however, i think there are probably better solutions to that problem (e.g. install profiles, maybe other things).
however, if folks really want to be included in the parent tarball, and they've gone down the road of a project pointing into a subdirectory of a parent project, they clearly want to be included in the same tarball (that's the only possible reason to stay in the same directory as the parent).
so, if they want to be in the parent's tarball, then the 2 projects have decided to be released together. therefore, the child project won't have any releases of its own, and all releases will be handled by the parent project. the parent project's maintainer therefore has to know when it's a good idea to add release tags, create release nodes, etc, since those will effect both the parent and the children...
does that all make sense? again, if you read the handout i linked above, you should have a pretty solid handle on how the new system will work...
thanks for the input, karen,
-derek
___________________
3281d Consulting
special case < 5?
Hi Derek,
starting with the last issue (since it's the easiest). Depending on your point of view, the 2-digit version does just mean that %minor always == 0. So why not just store it that way in the DB and special case the display when %major < 5 so in a couple years when all code is at 5.0 or greater you can just delete the special case.
---
Work: BioRAFT
i think storing in the DB as NULL is the cleanest
after thinking about this more and more, 0 is a valid value for most of these numbers, and we definitely need to keep 0 and not-used separate. if we just hack weird special cases for this, the code is going to be ugly, and practically useless for any sites other than drupal.org. :(
so, even though it's slightly more of a pain in the ass when inserting/updating the SQL, that small, localized pain will lead to much less pain and hacks in the rest of the code. i think i'm just going to take a stab at this and see how it goes...
___________________
3281d Consulting
version format string: crazy idea?
to deal with the format strings to handle variables that may or may not exist (#2 from the last point about 2 vs. 3 digits for core in the original post), i had a potentially useful, potentially horrifying thought... ;)
first, i should clarify that the version format string is only going to be touched by project administrators (e.g. killes, dries, etc) not by average project maintainers (everyone who owns a project node). so, this degree of complication only has to be mastered by people we can assume are clueful. ;)
instead of trying to hard-code logic about the delimiters, or not deal with them at all, i had inspiration from the new way
t()handles the placeholder variables -- the first character can be different to have specific behavior/meaning. here's my current proposal:%foo == print $foo with a '.' in the front
#foo == print $foo with a '-' in the front
!foo == print unmodified $foo
in all cases, if $foo isn't defined, the entire thing will be replaced with '' (an empty string).
the logic of these 3 values being:
t()this will be quite easy to handle in the code, and i think it would solve all the problems that 2 vs. 3 digits create.
for example, using this system, the format string for core would be:
!major%minor%patch#extraa few examples:
"4.7.0-RC-1"
"4.7.3"
"5.0-BETA-1"
see how powerful and easy that is? ;)
for contrib modules/themes in the new system, the format string would be:
!super_major%super_minor.x#minor%patch#extraagain, some examples:
"4.7.x-0.3"
"5.x-0.1-dev"
make sense? am i nuts? ;)
-derek
___________________
3281d Consulting
Hmmm...
Rather than remembering placeholder mappings, what if we switched to just using one of the placeholder styles as variable substitution? And leave the rest as just a string?
For example:
4.7.x-0.3 = "%super_major.%super_minor.x-%patch%extra"
5.x-0.1-dev = "%super_major-x-%minor.%patch-%extra"
This would retain consistency with things like the user mail variables and such.
Another thing I proposed was just not worrying about this for 4.7.x and below and only making the new release system possible for 5.x and above, as it seems like this would greatly simplify the logic. Derek's feeling on that was that developers of 4.7 modules are the ones who mostly need this system, so it needs to take that into account. However, if the 4.7 release cycle was any indication, almost the second that it was released (and actually even a few weeks before), almost no one developed 4.6 modules anymore and started coding everything for 4.7. If 5.0 goes the same way, then I worry that this could be a lot of wasted effort for nothing. :(
1 format string for the project, + why we need to support old #s
1) this is a single format string for the entire project, across all versions. so, it has to be able to handle if %super_minor is defined or not (to handle legacy releases and new releases with the same formatting string). the format string is a property of the project, not the release, and is how we can translate the version #s of each release into a human-readable string for issues, titles, tarball names, etc. so, if we used
%super_major.%super_minor.x-%minor.%patch%extra(note: you left out the "%minor." part in your example above), then in a 5.x compatible module, where $super_minor was NULL, you'd end up with the string "5..x-0.3". those two periods next to each other is the complication i hoped to avoid by putting the delimiter in front of the value of the variable only if that variable is being used. see what i mean? so long as the delimiters are string literals in the format string, to optionally include them correctly with variables that may be NULL, we'd have to add weird logic to know how much of the format string's literals to consume if a variable isn't there. yuck.2) it'll actually be impossible (or at least, way more work) to keep the project code functioning at all if it's trying to handle both releases-as-nodes and current-releases-not-as-nodes at the same time. whatever minor complications this crap to handle 5.x vs. 4.7.x will cause is nothing compared to trying to handle both kinds of releases simultaneously. :( so, unless we just declare "sorry, you can no longer find or download 4.7.x or 4.6.x versions of any modules" (which is obviously completely ridiculous), we need to be able to handle both kinds of version numbers with the new system...
___________________
3281d Consulting
contrib's 2 digits should be major.patch, just like core
after a brief discussion w/ chx and kjartan on IRC, we all converged on calling the 2-digits of core's version number "major.patch", just like 5.x and beyond in core. so, the above format string for contrib should really be:
!super_major%super_minor.x#major%patch#extraand the examples should be:
"4.7.x-0.3"
"5.x-0.1-dev"
...
___________________
3281d Consulting
-1 to subprojects
I think that overall trying to track sub-projects adds too much complexity here. The thread started with an "issue queue being to big".
I think it was rightly pointed out that components could solve this problem without the user having to know. I'm already preplexed as to why component is a required field for an issue, given that I can't currently find a way to filter by it. (I use the project module for non-drupal issue tracking by the way).
Imagine developing views of issues by component so the groups of maintainers could organize their issues as they worked on it. A problem gets logged by a user, which turns out to be for the wrong module, the component field just gets set to the right value. I'd though previously that this would best be done by making compenents into taxonomy terms, but I may be wrong here.
As for CVS access control, I link the existing arrangement with one flat structure for contrib modules works well. It takes less sysadmin intervention. I do think a naming convention for ckk could help a great deal. If you're writing an extension for CCK, have everone agree to follow the convention CKK_MYMODULENAME as a convention. It bundles the projects on the project list and also helps with the grouping in module settings pages, etc.
trying to distill some useful ideas
I think there are some useful suggestions here- can there be meta projects/meta releases? For example could there be a project that is a bundle of (for example) OG plus several OG-add on modules? This project itself need not have an issue queue, but could represent a way to let end users downlod a recommended "set" of related modules. The maintainer of this project could then make relases to represent/include new releases of one or more of the bundled modules.
I realize that this could bring clarity or chaos, so maybe there should be an additional premission requred to make such a meta project.
---
Work: BioRAFT
potentially good ideas but this is totally out of scope
maybe we'll want some kind of meta-project thing, or maybe we'll come up with other ways for folks to easily download a set of related modules. at some point relatively soon, we'll certainly want to create a new kind of project node on d.o for installation profiles (or whatever we're going to call them -- distributions, packages, etc, etc). those will certainly act like meta-projects, in some respects.
however, all of this is outside the scope of the new release system (at least, the initial push to get it live). once it's in place, there are *TONS* of things we could build on top of it. but, i don't want to delay the basics to worry too much about the bells and whistles. ;)
that said, if you're really inspired to think this through a little more, please submit an issue about it (http://drupal.org/node/add/project_issue/3281/feature).
thanks,
-derek
___________________
3281d Consulting
only 4 subprojects -- who cares? ;) other thoughts/replies
1) turns out there are only 4 subprojects on drupal.org:
http://drupal.org/node/43378 /modules/ecommerce/contrib/worldpay/
http://drupal.org/node/67060 /modules/event/contrib/event_views/
http://drupal.org/node/67375 /modules/location/contrib/location_views/
http://drupal.org/node/75541 /modules/ecommerce/contrib/inventorymangement/
2) this is basically evil. if the only goal is filtering the issue queue, there are other means. if the goal is separate CVS ACLs, just put it in its own directory. if the goal is having them packaged together, we just need better ways to handle dependencies and groups of related modules.
given that there are only 4, i'm prepared to completely punt on the problem and not really worry about it at all. i can easily disable releases for those 4 projects during the conversion on drupal.org (there are already a few other projects that shouldn't have releases enabled, anyway). other than that, i'll help the ~5 people this will effect to figure out a way they can survive, or i'll help them move their code into a separate directory (which is exactly what OG does for all the OG-related modules -- they all live in their own directories, with their own project nodes).
3) you can currently filter by component using the "advanced search" interface for a given project. for example, the signup module's advanced search page is here:
http://drupal.org/project/issues/search/signup
4) you should be able to filter by the component in the standard query interface. that's a goal of mine. see http://drupal.org/node/65336
5) views integration for project_issues is discussed here: http://drupal.org/node/76725
6) see http://drupal.org/node/49690 about cck_foo.module as a convention (and why, in spite of it seeming like a great idea (which i advocated for months ago), it's in fact a bad idea and not going to happen).
___________________
3281d Consulting
all complications accounted for
just wanted to record that the issues raised by this forum post are now basically all either resolved, or there are issues created for the remaining items.
re: release nodes that are pointing to the TRUNK:
re: releases for projects that represent subdirectories of other projects:
the drupal.org conversion script just disables releases for these 4 subprojects. otherwise, this problem just has to be solved via documentation for now. eventually, we should probably do away with these subprojects completely.
re: complications because of 2 vs. 3 digits for core:
basically, the version format string stuff i talked about is what i implemented. the only difference is that now the "super_major" and "super_minor" are all handled via an "API compatibility" taxonomy, so it's much better and more flexible. but, the basics (e.g. for core) are as i described above. we now "store" things in the DB that we're not using as NULL, which, while taking a little bit more code in a few places, is actually much cleaner and better in the long term...
if this were an issue, i'd mark it as "fixed". ;)
thanks,
-derek
___________________
3281d Consulting