Change {versioncontrol_labels}.name collation to be case sensitive on mysql

I seem to be having the same problem with my project. I created a tag 6.x-3.x-dev on my project bugbits and I pushed the tag back up to origin.

git checkout 6.x-3.x
git tag 6.x-3.x-dev
git push origin 6.x-3.x-dev

Everything went ok but I don't see the tag under the releases tab under my project. I created the tag about 36 hours ago. From my understanding there is a bit of a lag, but I would suspect it wouldn't be any longer then 24 hours. Anyone have any ideas? Is my project foobared? It's an old one that's been around since Drupal 4.6 days.

Thanks

Trevors-MacBook:bugbits tmckeown$ git push --tags origin
Everything up-to-date

I don't think that is it.

6.x-3.x-dev is not a valid tag for our system.

What you want is a BRANCH named 6.x-3.x -- you can then turn that into the 6.x-3.x-dev release.

I looked at the fb project and 6.x-3.0-rc9 should be a valid tag, but is not showing up as available for a release. This may be a bug. Checking with dww.

6.x-3.0-RC9 is not valid, since the tags and branches are case-sensitive.

6.x-3.0-rc9 should work. However, there's no record of that tag in your repository in the {versioncontrol_labels} table from VersionControl API. That's why it's not showing up as an option at http://drupal.org/node/add/project-release/178857

There are (at least) 2 reasonable explanations for this:

A) The parse worker job might have died and didn't parse all your branches and tags properly. This occasionally happens. At this point, the way to recover is to just push another change (since any push triggers a full repository parse). So, I'd try removing the RC9 tag (since it's bogus) and pushing that change, and see what happens.

B) There might be a bug in vcapi and/or vcapi_git where it gets confused with case collisions. MySQL isn't case sensitive, and that might be nailing us. E.g. vcapi saw RC9, and when it parsed the rc9 tag, it ignored it since it might have thought it was already there (or something). I thought this was fixed at #1073526: Deleting tag via git does not remove the label from {versioncontrol_labels} but maybe not completely, or maybe there's another bug entirely.

Anyway, let's start with A and see if that clears it up.

Cheers,
-Derek

Does adding tracking as noted in #1081308: Include "-u" option in new branch instructions fix your problem?

Re: "When I read about how to use git, it's usually "git is great and easy," followed by an incredibly complex set of instructions with parameters that baffle me."

Couldn't agree more! ;) I always laugh at the people who say "But Git is so much easier to learn and use than CVS -- CVS is so cryptic!". Git is at least as cryptic, only there are now at least 3 ways to do anything... But I digress.

Re: "I don't know what "{versioncontrol_labels} table from VersionControl API" is."

It's a database table on drupal.org that holds information about all the labels (aka branches and tags) in all our Git repositories. It's provided by http://drupal.org/project/versioncontrol and http://drupal.org/project/versioncontrol_git. It's what http://drupal.org/project/versioncontrol_project talks to to figure out the available labels to give you at http://drupal.org/node/add/project-release/178857

So, your tag is in Git, but VersionControl API doesn't know about it, so versioncontrol_project doesn't know about it, which is why it's not on your node/add/project-release page. Therefore, the bug is somewhere between Git itself and this DB table. Either the worker that parses your repo is barfing, or the underlying code is wrong. Probably need someone more familiar with the internals of vcapi and vc_git to look closely, since I unfortunately have almost no time this week to debug something like this. I'll try to ping some folks in IRC...

Cheers,
-Derek

I don't see how "moving" to github would help. First, you're always free to mirror it over there, and doing so is as trivial as adding another 'push-url' line to your git repository's config. Second, the data is all still in git. The only question is whether it's available in our packager - and since the packager is what makes it easy for people to actually work with it on real sites, and you can't get our packager on github...not a lot of point to that, is there?

There are some hiccups with the parsing. We're working on improving the parser so less of these things happen. We know that we parse perfectly in every case when parsing a fresh, new repository - the problem tends to happen when we've got new commits coming in to an already-parsed repo on a push. Given that we do that, though, the easiest solution is just reparsing a whole repo if something goes sideways. Which...is exactly the immediate-term plan. It's just that there's a bit of skunkworks involved in making that work perfectly, which I'm already working on in a topic branch. Once I've got that wrapped, I can pop out some plugins, write some tests that ensure release nodes line up with the reparsed repo, and ipso facto, we can recover gracefully from any errors. Possibly even provide a button on repos that queues this job up so you can do it yourself if something looks out of whack. In any case, my expectation is that by Monday, given a quick ping on IRC I'll be able to pull the switch and fix these problems for you.

This isn't one of those indeterminate "we're working on it" things, btw. There's that topic branch I already pointed mentioned; we've got a major refactor of the whole repo parsing system written and very nearly ready to roll in. And more generally speaking, we're aware that these issues exist, and they're my #1 git priority right now. So please, bear with us.

Nicely said, Sam.

Furthermore, Dave, you can still have people clone from Git, checkout your release tag, and test it. I suspect many of the people that will give you useful feedback on your code would be able to figure out the Git instructions tab, but just in case, here's what you can tell them to do:

git clone --branch 6.x-3.x http://git.drupal.org/project/fb.git
cd fb
git checkout 6.x-3.0-rc9

Cheers,
-Derek

Tagging this for my reference

@Dave: We all agree there's an issue here that needs solving. No one is writing you off... Please see sdboyer's nice explanation at #11 where he both describes the problem and explains what the Git Team is doing to resolve it.

Thanks,
-Derek

Great to see progress is being made here; definitely keep us up to date with any news. I'm still getting used to this git thing, so I screwed up my latest release and I don't seem to be able to make a new release with my last committed branch. Thanks for all the hard work you guys are doing.

Sorry, I'm late here.

B) There might be a bug in vcapi and/or vcapi_git where it gets confused with case collisions. MySQL isn't case sensitive, and that might be nailing us. E.g. vcapi saw RC9, and when it parsed the rc9 tag, it ignored it since it might have thought it was already there (or something). I thought this was fixed at #1073526: Deleting tag via git does not remove the label from {versioncontrol_labels} but maybe not completely, or maybe there's another bug entirely.

Well, not really sure if it is a bug. After parsing it locally I see the error about casing:

User warning: Duplicate entry '21-6.x-3.0-rc9-3' for key 'repo_id_name_type' query: INSERT INTO versioncontrol_labels (repo_id, name, type) VALUES (21, '6.x-3.0-rc9', 3) in _db_query() (line 135 of includes/database.mysqli.inc).

So, in the parser there seems to not be any problem, but we are not making an special case when we got any number of branches/tags names equal, but with different case, so since mysql can not handle it, only the first one(6.x-3.0-RC9) is inserted, and the others are not(6.x-3.0-rc9).

Just to mention that the issue #1073526: Deleting tag via git does not remove the label from {versioncontrol_labels} is about deleting the tag when it is removed from the git repository, but both 6.x-3.0-RC9 and 6.x-3.0-rc9 are still there, so the issue is really unrelated.

@dww, @sdboyer: is there something still to do here? (IMHO this is not a bug, but leaving status for now, I think this is about use, maybe we should open/take-this one issue for preventing pushing tags with bad casing?)

@Dave Cohen: You probably want to delete the 6.x-3.x-RC9 tag. BTW you have a set of commits that are on a tag(6.x-3.0-rc10) but not on a brach, that's unusual and probably you want to avoid it, please do a FF of 6.x-3.x to your 6.x-3.0-rc10 and the push 6.x-3.x. An image helps here to see the status of the repository(I really advice to use one graphical viewer to see non-trivial histories like this one), this is taken with gitk --all on the repo: http://i.imgur.com/BJVtN.png

We could certainly set the {versioncontrol_label} table's collation to 'utf8_bin' if we wanted it to be case-sensitive. Not sure if that's a win, but given that:
- Git labels are case-sensitive
- we have no restrictions on anything in Git (just restrictions on what you can make releases out of)
- we're trying to accurately capture the Git data in our DB
... seems we need to be able to handle case collisions like this.

fyi, nearly done with the plumbing for this. Sorry for the delay...but it resulted in more tests & such goodness, so there's a bright side :)

I didn't realize we were using a non case-sensitive collation for {versioncontrol_labels}. Yeah...really need to do that.

{versioncontrol_labels} is defaulting to utf8_general_ci like nearly all the other tables on d.o (only 8 out of 218 are utf8_bin). We'd need a DB update (probably in vcapi itself) to set it to utf8_bin if we wanted to force it to be case-sensitive.

Yeah. I skipped this in the deployment just now because I checked it locally and found that it was case insensitive, and then thought that was Git. Stupid me, that was on case-insensitive HFS+, and Git just going with what the filesystem tells it. Stupid stupid stupid osx.

We'll have to roll this in, but a patch would be great since I'm not sure how to set collations in schema/update functions.

We still need the fix to the collation, but @Dave Cohen - your rc9 tag is at least showing up now.

It makes sense if you're used to thinking in distributed terms, but not intuitive if you're coming from a centralized VCS where you're always mirroring between your local and a remote.

This is in our docs, though. And FYI, you also used an unnecessarily verbose format for it - this would have cut it:

git push origin :6.x-3.0-RC9

So, I'm having a similar problems with http://drupal.org/project/tablefield - I created a tag yesterday "7.x-2.0-beta" which shows up locally and on the repository viewer page. However, it is not available when I click "Add new release" link.

Any way you might be able to help? Much appreciated!

@Kevin Hankens - '7.x-2.0-beta' is not a valid version string. '7.x-2.0-beta1' would be.

Interesting. Thanks, I'll give that a shot!

{versioncontrol_labels} is defaulting to utf8_general_ci like nearly all the other tables on d.o (only 8 out of 218 are utf8_bin). We'd need a DB update (probably in vcapi itself) to set it to utf8_bin if we wanted to force it to be case-sensitive.

@dww: It would be great if you point me to any of the module that have done that change.

My quick review to schema api support for character set/collation, makes me think that it is not standarized on schema api, since it seems like it is not supported on all database engines.

I see two ways in schema api, changing schema table array(not by fields):

• Using 'mysql_suffix'.
• Using 'collation' on the schema array(that seems to be mysql specific key).

I do not see the collate handling by fields on schema api functions, but at least sqlite and mysql seems to support it(not sure about postgres).

better title

tagging

@marvil07: I don't know of a module that does this off the top of my head, and it sounds like you've already done more research into how schema API tries to handle this than I have. ;) Unfortunately, I have no special knowledge to share here on the Drupal Way(tm) to accomplish this. If you need help I can try to track down an answer, but you'll probably be at least as successful at that as I would be. ;) I suggest trying to ping some of the DB layer maintainers (chx, crell, damz, etc) for advice. Good luck!

So, after talking with chx about this, I end up thinking that we can do two things(to get a per field collate, not per table):

• a custom db_query() per DB engine with a previous check for db_type

.

• using mysql_type(and do evil setting it to a full mysql sql field spec string, like VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_bin) in schema field spec, but on a conditional checking for db_type, so we can avoid problems on supporting other non-mysql databases.

So, I will be chewing this a little more, but any of those should solve the problem.

BTW the real query on mysql we want to generate is:

ALTER TABLE `versioncontrol_labels` CHANGE `name` `name` VARCHAR( 255 ) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT ''

and in sqlite is probably better to use api functions, but there is not a mysql_type equivalent I am aware of.

I am attaching the patch I committed.

It tricks schema api using mysql_type as mentioned in the last comment.

Both, the schema array change and the hook_update_N pass through a validation on db_type global, so it is only used for mysql. Postgres do not support collate now, and sqlite3 is for d7(I need to pay more attention :-p).

Just marked #1170984: Tag not recognized as a candidate for creating a release if there is another tag with different letter case duplicate with this, but I see the title here is misleading. We want the .name to be case *sensitive* on mysql. mysql is case insensitive (it ignores case) by default. ;)

Was this ever deployed? Moshe just pinged me in #drupal-infrastructure about an identical bug. He had created a 7.x-5.0-RC3 tag, couldn't create a release node, deleted it and added 7.x-5.0-rc3, and still couldn't add it. The record was still in the {versioncontrol_labels} table as RC3 not rc3.

Argh, nope, never deployed:

mysql> SELECT table_name, table_collation FROM information_schema.tables WHERE table_name = 'versioncontrol_labels';
+-----------------------+-----------------+
| table_name            | table_collation |
+-----------------------+-----------------+
| versioncontrol_labels | utf8_general_ci | 
+-----------------------+-----------------+
1 row in set (0.03 sec)

Sorry I missed this when this issue was marked as 'fixed' and not tagged for deployment.

So, is anything going to break if we upgrade vc_git on d.o? Has there been another release since this bug was fixed that we could target for deployment?

I do not think this was deployed.

BTW the way used to fix collation was field specific(instead of table specific), so the query to figure out it should look like:

SELECT COLUMN_NAME, COLLATION_NAME
FROM information_schema.COLUMNS
WHERE TABLE_SCHEMA = 'drupal_vcs' AND TABLE_NAME = 'versioncontrol_labels' AND COLUMN_NAME = 'name';

(changing drupal_vcs with the real drupal.org database name)

I guess this patch would work fine. In the other side, about deploying last dev, I guess we need to coordinate this well with related modules like versioncontrol_git and versioncontrol_project.

@sdboyer: What do you think?

It has just been deployed!