Clean up faulty branches

Ugh, this is a growing problem: see http://drupal.org/node/197445

Here's the full list of faulty projects at DRUPAL-5--1:

mysql> select n.nid, n.title, p.uri from node n INNER JOIN project_projects p ON n.nid = p.nid INNER JOIN cvs_tags c ON n.nid = c.nid WHERE c.tag = 'DRUPAL-5--1';
+--------+-----------------------------+------------------------+
| nid    | title                       | uri                    |
+--------+-----------------------------+------------------------+
|   6018 | API                         | api                    | 
|  35189 | SMTP Authentication Support | smtp                   | 
|  80756 | fastpath_fscache            | fastpath_fscache       | 
| 133830 | Mass Contact                | mass_contact           | 
| 134607 | Theme Settings              | themesettings          | 
| 158301 | PHPIDS                      | phpids                 | 
| 164570 | Widont                      | widont                 | 
| 167734 | CCK Address Extensions      | cck_address_extensions | 
| 188206 | Courier MTA                 | couriermta             | 
+--------+-----------------------------+------------------------+
9 rows in set (0.01 sec)

API can be explained by drumm having super-user CVS powers. So, I'm assuming he just got confused by the change in conventions and created the wrong branch. That's unfortunate, but at least understandable. All I can think of for the others is that once that tag existed, they were able to do "cvs co -r DRUPAL-5--1 contributions/modules", they got something since the tag was already there (in api module), and then CVS let them add new directories or files with the same tag. :(

CVS archeology on smtp module supports this theory:

revision 1.2.2.1
date: 2007-02-15 19:37:03 +0000;  author: oadaeh;  state: dead;  lines: +0 -4;  
commitid: 1bac475431264567;
file smtp.info was added on branch DRUPAL-5--1 on 2007-12-03 16:39:03 +0000

Same with themesettings:

revision 1.5.2.1
date: 2007-06-16 09:35:37 +0000;  author: johnalbin;  state: dead;  lines: +0 -7
;  commitid: 113e46931d884567;
file themesettings.info was added on branch DRUPAL-5--1 on 2007-07-10 05:47:53 +0000

Sadly, it's pretty hard to rename a branch in CVS. :( The only way I know how is to directly edit the ,v files in the repository itself, which is a little scary, but it certainly works.

It's going to create havoc in the d.o-specific CVS + Project* integration if we try to support both conventions. It's already confusing enough that we had to switch at all, and having both in D5 is making a bad situation worse. :( Therefore, I propose the following action items:

A) We should disable the CVS super user functionality in our repository. Even the core committers should go through the usual access control stuff, to help catch errors like this. Seems like this is safer for preventing accidents, too. All of them have 'administer nodes' perms on d.o, so if they need to commit anywhere, they can grant themselves CVS access.

B) I'll make backups of the CVS repository directories for these 9 modules, and then have a little script go through to edit the ,v files to rename "DRUPAL-5--1" branches to "DRUPAL-5". This will unfortunately confuse existing CVS workspaces checked out with these directories, but that seems like a necessary evil to cleaning this up. Anyone with a copy of one of these modules checked out will see something like this when they run cvs update:

cvs update: Updating .
cvs update: CHANGELOG.txt is no longer in the repository
cvs update: INSTALL.txt is no longer in the repository
cvs update: README.txt is no longer in the repository
...

and cvs status would tell them something like:

cvs status: signup.info is no longer in the repository
===================================================================
File: signup.info       Status: Entry Invalid

   Working revision:    1.1.2.1 Tue Dec  4 03:09:10 2007
   Repository revision: No revision control file
   Sticky Tag:      DRUPAL-5--1 - MISSING from RCS file!
   Sticky Date:     (none)
   Sticky Options:  (none)

The solution is simply to run "cvs update -r DRUPAL-5" and all will be solved...

C) I'll run a simple query to fix the records in {cvs_tags} for these projects.

D) I'll send another note to the devel list about this point, that the change in convention is for DRUPAL-6 and beyond, that "DRUPAL-5--1" is an invalid branch that should never have been created in the first place.

E) Someone should update the CVS docs to encourage people to create new branches with cvs tag, not via checking out a copy of the repo from their preferred tag and trying to add their files to that. :(

I just think we're going to get more and more pain and confusion if we let this continue...

Any thoughts, complaints, alternate suggestions, etc?

-Derek

Ok, I just wrote up some magic PHP and query code that generated emails to all project maintainers and people with CVS access to the effected projects, informing them about the problem and pointing them to this issue. The email said I do the move later this week if there are no objections, so we'll see what develops in here. ;) Stay tuned.

For reference, here's a copy of the email:

From: dww <noreply@drupal.org>
Subject: Faulty DRUPAL-5--1 branch for xxxxxxx contrib module

You're getting this email since you're either the module maintainer or have CVS access to the contributed module mentioned in the subject.  The module has a "DRUPAL-5--1" branch.  This doesn't match our branching conventions, is supposed to be disallowed by our CVS repository, and is not supported by various parts of the release system.   The "--1" part of the branch name was added only for DRUPAL-6--1 and beyond.  For legacy reasons, the 1.* series for 5.x and earlier just uses the same branch as core, e.g. DRUPAL-5.  For more background, see the following:

http://drupal.org/handbook/cvs/branches-and-tags/contributions#stable
http://drupal.org/node/147493

There are a handful of other modules with the same faulty tag, and it's starting to cause problems and confusion.  Therefore, I'm proposing to rename these branches in our CVS repository from "DRUPAL-5--1" to "DRUPAL-5".  For more about the problem, how I think we got to this point, and my proposal for the rename, see:

http://drupal.org/node/152832

Unfortunately, there's no painless, transparent way to rename branches like this.  So, existing CVS checkouts of these modules from the DRUPAL-5--1 branch will be confused after the rename, requiring you as maintainers (and anyone else who deployed your module via CVS) to run "cvs update -r DRUPAL-5" in the directory to reset the workspace's "sticky tag" to point to the correct branch. 

The drupal.org CVS team has agreed this is the least evil solution to the problem, so please speak-up now if you have any objections or concerns.  Otherwise, the rename will probably happen later this week.  Instead of replying via email, please use the issue queue:

http://drupal.org/node/152832#comment-form

Thanks,
-Derek (dww)

Nah, it'll be better in the long run and simpler for the CVS history if my script just renames, instead of having you manually do it. Thanks for the offer, though.

In case it wasn't clear (and I could see how some of my language above could be misread), I certainly didn't mean to imply any of you did something wrong maliciously or to intentionally cause me grief. ;) It's just a confusing situation, and somehow the safety checks I added weren't bullet proof, and accidents happened. Apologies if it seemed like I was accusing anyone of active wrong-doing.

Cheers,
-Derek

@oadaeh:

A) Thanks for the offer to help cleanup your own modules. Before I say "no thanks, that won't be necessary", lemme look closer. Meanwhile, please don't take any action on this until you hear back from me. Thanks.

B) Wow, what a complicated system for having a *LOT* of directories on your hard-drive. ;) It seems this is semi-common, and why once a single DRUPAL-5--1 branch was created, it was propagated in the wild. Out of curiosity, didn't it bother you that your DRUPAL-5--1 checkout only had 10 modules in it? ;) Didn't you ever wonder where all those 5.x-1.x-dev releases were coming from? Your script should be checking out just "DRUPAL-5" as the primary branch for 5.x contrib work. In reality, only a relatively small minority of projects use any of the higher branches. You can see http://drupal.org/node/97084 for a complete list. In fact, that page is what first alerted me to the problems in our repository, and why this issue exists in the first place.

C) official releases (from tags) are definitely supported. E.g. the DRUPAL-5--1-1 tag is perfect for the 5.x-1.1 release. All I'm talking about is the DRUPAL-5--1 branch.

D) However, that doesn't mean it's particularly helpful to check out all of these official release tags like the inner loop of your script is doing. You can never commit changes to a tag. Only a branch.

@JohnAlbin: Hrm, upon closer inspection I see that you had a DRUPAL-5 branch already, then added a DRUPAL-5--1 branch later. :( That complicates what my script hoped to accomplish. :( I'll have to ponder this a little more and get back to you. Maybe I'll just delete the existing DRUPAL-5 branch on that project and rename the DRUPAL-5--1 to DRUPAL-5, since it seems like you didn't really make many revisions on DRUPAL-5 itself. Anyway, stay tuned.

@drumm: Yes, I'm sympathetic to the confusion, but it was widely agreed that the best solution going forward was to always require the --N part. I tried every way I knew how to make this clear:

the very first sentence of the post where I proposed this change: http://drupal.org/node/147493
the current handbook page about our contrib branches: http://drupal.org/handbook/cvs/branches-and-tags/contributions#stable
my email to the devel list about it: http://lists.drupal.org/archives/development/2007-07/msg00556.html Subject: "Changes to CVS tagging/branching conventions for contrib (MUST READ)"

Not sure what else I could have possibly done to inform everyone about it. Suggestions welcome for next time (though, I sure hope there are no more changes!)... ;)

Thanks,
-Derek

Brief update: drumm and I were both on IRC at the same time, and the rename for API module was easy, so that's now done. I'll be working on the others soon, at least figuring out what's the best solution for each one.

Another update:

A) I inspected the other directories with DRUPAL-5--1 branches. Here are the results:
api: trivial, now fixed.
smtp: trivial, now fixed.
fastpath_fscache: will be simple, not yet fixed.
mass_contact: major branch confusion :( will be difficult to resolve.
themesettings: looks like we can probably should just remove the DRUPAL-5--1 branch... there's only 1 minor commit to it.
phpids: massive confusion, will be difficult to resolve.
widont: trival, now fixed.
cck_address_extensions: much confusion in cck_address_mexico and cck_address_belgium, will be hard to resolve.
couriermta: will be simple, not yet fixed.

B) I just figured out a way to prevent this problem in the future: http://drupal.org/node/198278 yay! ;)

whoops, sorry, wrong issue.

http://drupal.org/node/198278 is now fixed, committed, and deployed on cvs.d.o. Yay!

in other news, i fixed fastpath_fscache and couriermta.

@JohnAlbin: Upon further inspection -- yes please. ;) You should re-add the files from the end of your DRUPAL-5--1 branch back onto the DRUPAL-5 branch, and remove them from the DRUPAL-5--1 branch. Then, I'll just remove the DRUPAL-5--1 branch entirely, instead of trying any rename funny business. That should take care of themesettings.

That leaves us with the 3 where the maintainers were really confused and made a big mess of their branches:
cck_address_extensions
mass_contact
phpids

I'd like to clean these up, too, but it'll take more time. :(

I think my next priority is to nip the "DRUPAL-6" branch projects in the bud:
http://drupal.org/project/google_analytics (budda, hass)
http://drupal.org/project/kasahorow (Suuch)
http://drupal.org/project/remove_upload_enclosures (mfer)

And then, get to all the "branches" that should be tags:
DRUPAL-5--1-4
DRUPAL-5--1-0-BETA1
DRUPAL-5--1-0
DRUPAL-5--0-1
DRUPAL-4-7--2-0

And finally, to the ones that are just totally bogus:
DRUPAL-5-1
DRUPAL-5-0
DRUPAL-4-7-2

@hass: That email was auto-generated. I haven't had a chance to individually inspect each of the faulty projects yet. See above in this same issue!! (since you like to use exclamation points so much). ;) If you already cleaned up your own mess, then, no, I won't rename the DRUPAL-6 branch, I'll just remove it. It just depends on exactly what kind of a mess each maintainer made to figure out how to best resolve it...

@JohnAlbin: Thanks. Yes, I can subvert my own script. ;) http://drupal.org/cvs?commit=90508 -- those files are all removed from the DRUPAL-5--1 branch. I then deleted the extra 5.x-1.x-dev release node you had created which was pointing to DRUPAL-5--1. Finally, I removed the DRUPAL-5--1 branch entirely from the themesettings directory. So, it seems that themesettings is totally happy now. In fact, the packaging script just ran and fixed up the error which used to appear on the 5.x-1.x-dev release node pointing to DRUPAL-5 (http://drupal.org/node/136089):

Packaging error messages
ERROR: contributions/modules/themesettings does not exist after cvs export -r DRUPAL-5

Now, it's looking fine:

Nightly development snapshot from CVS branch: DRUPAL-5
Download: themesettings-5.x-1.x-dev.tar.gz
Size: 14.22 KB
md5_file hash: 895ab9e56d00a1662f9741cd2771086e
First released: April 13, 2007 - 09:19
Last updated: December 6, 2007 - 16:10
...

Thanks for your help in here,
-Derek

p.s. I guess I should assign this task to myself... ;)

@hass: Yup, upon further inspection, your DRUPAL-6 branch in google_analytics was no longer in use at all, and could be safely removed entirely. Now gone.

Still have to inspect the other 2 DRUPAL-6 projects and/or hear back from their owners.

Yay, DRUPAL-6 is now totally cleansed from the contrib repo (and therefore gone from http://drupal.org/node/97084)!!

google_analytics: DRUPAL-6 branch removed

remove_upload_enclosures: DRUPAL-6 branch renamed to DRUPAL-6--1 (I found mfer in IRC)

http://drupal.org/project/kasahorow: DRUPAL-6 branch renamed to DRUPAL-6--1 (was a trivial rename, so I just went ahead with it)

In other news:

image_exact: DRUPAL-5--0-1 branch converted to a tag (I found josh_k in IRC)

so, we're down to the following branches that should be tags:
DRUPAL-5--1-4: addtofavorites (thierry_gd)
DRUPAL-5--1-0-BETA1: dataview (surge_martin)
DRUPAL-5--1-0: filmforge (tatien, anarcat)
DRUPAL-4-7--2-0: MyDrupal (taherk)

and the following totally bogus identifiers:
DRUPAL-5-1: MyDrupal_Impact (taherk), node_browser (kristy)
DRUPAL-5-0: teamspeak (Cainan)
DRUPAL-4-7-2: MyDrupal_Impact (taherk)

therefore, we just need to make sure the above branches-which-should-be-tags can just be safely converted to tags, and then, we need to figure out what to do about these other identifiers (are they supposed to be branches or tags? are they in use at all? should they be renamed or removed? etc).

AjK just offered to help, so hopefully he can claim some of these...

surge_martin replied via direct email about dataview. in that one, we need to rename the DRUPAL-5--1-0-BETA1 branch to DRUPAL-5, add a true DRUPAL-5--1-0-BETA1 tag on an earlier revision of the files, and fix up the release nodes accordingly. should be fairly straight-forward.

@AjK: any news on your investigations on any of the others?

@rconstantine: yeah, cck_address_extensions is a big mess in CVS. please don't add any new release nodes or tags/branches until we get it sorted out. are you ever on IRC? i notice your profile doesn't include the "IRC nick" field. it might be faster/easier to just sort this out in real time, instead of the issue queue. when I'm on IRC, i'm still "dww"...

Ok, spoke to oadaeh and bdragon in IRC at some length about the mess in mass_contact. To fix this project's particular mess, we'd have to lose history one way or another, since he and the previous maintainer were both committing to DRUPAL-5 and DRUPAL-5--1. plus, some of the release tags are on HEAD, others on branches, etc. it's simply impossible to just rename the tags or branches to fix this. :(

We decided that the least evil option was to just clean out everything that's there and start over from scratch. first, I made complete mysqldumps of {cvs_files} and {cvs_messages}, just to be safe. then, i moved all the records in {cvs_files} and {cvs_messages} related to this module into new {cvs_files_attic} and {cvs_messages_attic} tables, and removed all the records from the live tables. Then, I removed all the contents of the contributions/modules/mass_contact directory in the contrib repo.

oadaeh is now going to take the release tarballs for this project, start with the original 5.x-1.0 release, commit that as the initial revisions of his files, tag it, make a DRUPAL-5 branch, commit 5.x-1.1 to the branch, tag that, etc.

it's a drag to lose the original commits, but many of them were bogus, on the wrong branch, etc. this seems like the least evil option, and oadaeh is up for doing the work to get back a sane module with the right branches and tags going forward...

Yay, under my watchful eye, oadaeh just finished cleaning up mass_contact. ;) So, we're down to just these two for DRUPAL-5--1:

cck_address_extensions
phpids

(and of course, the 7 other random CVS ids listed in http://drupal.org/node/152832#comment-651079 (#25)).

@hass: If you want to be helpful, read the issue. If you did, you'd see why it's still active. Thanks.

Can y'all please leave this issue alone with this unrelated topic? This is about cleaning up junk that was put in the repository before I had locked down all possible ways to break the conventions. This is not the place for a support debate about how to name your tags. Thanks.

Yeah, I'm not really interested in trying to fix this anymore. If someone really wants to try, let me know, but at this point, I don't care.