Migrate drupal.org issues to GitLab issues

Nice work on this! Quite the lift. Read through the code and it makes sense to me, some thoughts:

• Do we maybe want to add offset/page_size as cli params? Might make it easier to recover/resume if there is an error.
• I like how you broke down the labels into individual functions. Makes it clean and easier to manage!
• Do we want to store the gitlab issue id on the project_issue content type?

Coming here from a #gitlab Slack meeting thread about issue IDs. When you run git blame and find a commit, you see things like “Issue #12345: whatever”. It’s going to very seriously harm DX if I have to check both drupal.org/node/12345 and GitLab issue 12345 to figure out which one is actually it.

According to Moshe, GitLab allows us to set the ID when creating new issues programmatically. I strongly believe we should use the d.o issue NID to specify the gitlab issue ID during the migration. Then, after all the issues are in GitLab, we know that we can always go to GitLab/$project/issues/ID and there won’t be any collisions.

I’m 💯 for the redirects. I was hoping to have both.

That’s really too bad that you lose who created the issues if you set the ids. What an sad limitation of GL. Maybe there’s a work-around?

Apparently I’m having trouble explaining my concern. I believe it’s going to be confusing and annoying to have to both check two places (the d.o redirect and the GL link), and not really know if a given ID refers to one or the other. I guess we’ll have to start checking dates in commits, too, and memorize the date the migration ran live. There won’t be d.o redirects for new issues, so if I see “12345” in a post-GL commit, I can’t use the “easier URLs”, anyway.

Or we could change conventions and ask that issues created on GL be called something else than issue, e.g. "ticket". That way any "issue #xx" would be on d.o. and "ticket #yy" would be on GL.

Redirects are great, but I personally feel that these are not enough and that without other connection, this will be very fragile. Imagine that we somehow lose one or more redirects (which I think can happen if these will be editable on d.o. as other content redirects), then there will be no way for users to find that particular issue in Gitlab.

I think there are many places where issue IDs are mentioned just as a plain text (like #123456, without square brackets) and for less experienced contributors it could be hard to find the referenced issue.

I am also wondering if keeping the ID would help directly in GitlabUI, because we still need to resolve/convert strings [#XXX] to the new Gitlab issue IDs, so there would have to be a conversion in place in addition to the redirects (if I understand that correctly). In case we keep the IDs the same, wouldn't we match the Gitlab method of referencing and would it be possible to omit the conversion?

So I agree with @dww and prefer to keep IDs, so we can hopefully find some workaround for that issue with authors (as I think that loosing the information about the issue author is a no-go).

fgm: indeed, I asked/proposed that in the Slack meeting. Sorry I didn’t mention it here. It’s still a bit of cognitive load, but it’s less than checking dates, for sure. But then we have to re-train everyone on commit message conventions…

The redirect controller could forward unknown issue ids to the gitlab url, without any mapping. That way there is one url to check.

Looking at the code of the MR here, redirects seems to be created as a standard redirects via a redirect module.

Another point - we also have links comment/XX, which are redirecting to the concrete comment in the issue. Will these redirects being updated to reference the migrated comments?

@moshe re:

The redirect controller could forward unknown issue ids to the gitlab url, without any mapping. That way there is one url to check.

All kinds of flaws with that proposal:

1. That only works if we assume that Drupal Core is the only project for which this matters. The GL URLs need to include a project name. "12345" could be from any project.
2. If we don't preserve the IDs, "12345" could be both a valid d.o legacy issue ID, and a rolled over new GitLab ID. So d.o/i/12345 will redirect me to the GL issue for that old NID, but I still might end up on some irrelevant issue if I really needed to be at GL/$project/issues/12345 instead. The only ways for me to know are to compare dates, or re-train every Drupal contributor on commit message conventions (which we should do for other reasons, but that's out of scope here 😅).
3. It sounds like there isn't a redirect controller, just a bunch of redirect content being generated, so there's not (currently) a way to implement your idea, even if it could work.

Meanwhile, @poker10's concerns about the longevity of all that redirect content is a great one. Makes me think that instead of just generating redirects directly, we should have a redirect controller, and a {drupalorg_gitlab_issue_map} table (or whatever) with all the legacy d.o NIDs -> project + GL IID values. Then no one can break the redirects via the UI. And we'll have a canonical remap table to use going forward in case we need it for other things. I can't imagine the cost of having such a table in our DB is too great for all the potential benefits that would come from doing it that way. It'd probably a more efficient DB-storage than storing all the redirects separately, in fact.

If there's no way to work around GL's limitation that we can't programatically set both IID + Author to what we want, how about we create a new field called "Original Author" or something? So the formal Author on migrated GL issues would be the "admin" user, but we can at least know the regular d.o user that created the legacy issue? Going forward, "Original Author" wouldn't be set on new GL issues, or we automatically set it to the "Author", or whatever. Sort of a PITA, but IMHO less painful than having issue IDs colliding.

Makes me think that instead of just generating redirects directly, we should have a redirect controller, and a {drupalorg_gitlab_issue_map} table (or whatever) with all the legacy d.o NIDs -> project + GL IID values. Then no one can break the redirects via the UI. And we'll have a canonical remap table to use going forward in case we need it for other things.

In my experience, the risk of a redirect controller breaking in code updates is higher. Unless we get better at test-driven-development for Drupal.org, it will break at some point and go unnoticed. Using the common redirect module ensures we have less code to maintain and upgrade. And I don’t expect people to be editing these redirects in the UI.

And I don’t expect people to be editing these redirects in the UI.

It's not about what we expect people to do, it's about preventing the possibility of changing these, either by accident or malice. This is about data integrity.

It's like the map tables from Drupal migrations. Even if you don't intend to write any code to consume it, the cost of generating that table as part of this migration is almost nil. The possibility that it will really come in handy exists.

If, 3 years from now, we want to check if all the redirects still exist, the table would let us. If we decide we prefer a custom controller for some reason(s), we could.

I have mentioned this on Slack, but will post it here too.

Not sure what is the exact status of the MR (I have not reviewed it), but from what I remember, there were some open concerns about keeping the issues IDs, how issue metadata should be migrated (#3254602: Using GitLab labels for issues on Drupal projects), and similar. @fjgarlin when you will have time to work on this again, it will be great to update the current status (what is done, what decisions are needed and what work is needed).

Given that this will be a one-time migration (for each project), without an option to revert (unlike the GitlabCI migration), I think it would be great, if we can see at least one "demo" project migrated first. We will be able to evaluate and test, if everything went correctly, if all references to issues, comments, etc, are kept and working, how the meta info are migrated, etc. Some feedback could be collected this way before the real migrations via opt-in process will start.

Issues are currently the most important part of our collaboration. Each commit references to an issue (ideally), there are multiple references to the issues IDs, etc. We need to be very careful to ensure, that no history / connections will be lost after migration. This is main reason why I think a first example demo project migration would be a good idea.

Thanks!

I quickly skimmed the MR and tried to find if we are migrating also issues followers and have not found anything. Is there a plan to keep who is following which issue, so that issue notifications are kept as they were on d.o.?

What about existing forks/merge requests, which are associated with d.o. issues? Will these be migrated too?

Thanks for your huge efforts with the GitLab migration @fjgarlin. Since NID is mentioned here, I just want to share something I observed:

• https://www.drupal.org/node/3292161
  Does not redirect to the issue below
• https://www.drupal.org/i/3292161
  Redirects to the issue

https://www.drupal.org/project/drupalorg/issues/3292161

... maybe this is intentional, or maybe not? I thought I'd let you know.

Personally, I agree that keeping old NIDs as GitLab IDs seems more beneficial than keeping authors of issues. Great we can solve it at least this way.

Looking at my earlier comment #22 (and #23) - is there a decision whether we want to keep/migrate also the comments redirects? Seems like there are at least two places in 11.x branch (1, 2), where such a link is used. Not sure how many are used in issues' comments itself (can we check it in the database?).

Also have another question - currently the redirects on i/NID and links on node/NID are universal and are working even without knowledge of the specific project, in which the issue is created. I suppose that if we move an issue to a different GitLab project after the migration, the redirect from node/NID would still work to a correct (new) GitLab project?

And speaking about this (but this question is not related to the migration itself), it seems like that for all new issues created in the GitLab after the migration, we will loose the ability to find such issue just by entering the ID somewhere (without knowledge of the specific project)? Presumably, this is also because issues in GitLab have separate numbering in each project?

Thanks!

Once this is ready, I am curious if we can see a migrated test project somewhere (even in the staging/test env), to help with testing before doing the first production migration of the DA projects? That could probably help identify issues before doing first irreversible production migrations.

And should this issue #3254602: Using GitLab labels for issues on Drupal projects be a blocker too, since we probably need that decided before the first migration? Thanks!

For labels, we do need to decide whether each label created for migration is project-specific or global (set in the /project & /sandbox groups). While there’s always the ability to change things later, it’s easier to avoid wanting to do bulk updates in GitLab. I’ve updated #3254602: Using GitLab labels for issues on Drupal projects to propose making issue status the only global label, the rest would be project-specific, and I’ve also made some notes on which ones might be scoped labels.

I ran into a couple timeouts. These have been very rare.

Created Gitlab issue #2310049                                                                                                                                                                    [ok]
* ERROR: cURL error 28: Operation timed out after 120138 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)                                             [error]
WD drupalorg_gitlab: * ERROR: cURL error 28: Operation timed out after 120138 milliseconds with 0 bytes received (see https://curl.haxx.se/libcurl/c/libcurl-errors.html)                        [error]

Created Gitlab issue #2301745                                                                                                                                                                    [ok]

We should log the file & line number, so we can track down what exactly timed out. If possible, the URL path being requested would be good to log too. If convenient, we can log the issue nid too, so we have a good list in logs, without needing to review console output.

120 seconds seems like a good timeout, I don’t think we need to adjust that. Depending on what’s going on, we might look into retries or some other error handling. Ideally this is something that happens in staging only.

GitLab’s https://git.drupalcode.org/ghost is the equivalent of Drupal’s assigning content to the anonymous user. It can be impersonated: https://git.code-staging.devdrupal.org/project/drupalorg_crosssite/-/iss...

To test on staging, I manually associated anonymous with it:

INSERT INTO versioncontrol_gitlab_users (gitlab_user_id, picture_timestamp, uid) VALUES (46370,0,0);
UPDATE users SET git_username = 'ghost' WHERE uid = 0;

-- Find some anonymous comments
SELECT c.cid FROM comment c INNER JOIN field_data_field_project fdf_p ON fdf_p.entity_id = c.nid INNER JOIN field_data_field_project_machine_name fdf_pmn ON fdf_pmn.entity_id = fdf_p.field_project_target_id WHERE c.uid = 0 AND fdf_pmn.field_project_machine_name_value = 'drupalorg_crosssite';

I don’t want to do that in production, I’m not sure if there might be side effects. For migration, we can hard-code the ghost username for uid 0.

While we are there, $fallback_to_admin looks like it is no longer used, since we’re ensuring everyone has a GitLab account. That logic can be removed.

Comment created date
The migrated comments come in with a current timestamp. I expect that’s something we can’t forge. (If we could forge it by posting as an admin, that trade-off isn’t worth it to me. Knowing who said what is more important.)

Let’s add the comment created time to the header like

Migrated from comment #12 posted 13 July 2012 at 11:10 UTC

Weird comment numbers
https://git.code-staging.devdrupal.org/project/drupalorg_crosssite/-/iss... has some high comments numbers, like “Migrated from comment #14256” which corresponds to “11.0” https://www.drupal.org/project/drupalorg_crosssite/issues/1549374#commen...

These comments were created during the D6 to D7 upgrade where we also added nodechanges and required that every revision have a comment. The comments got a decimal to slot them into the comment thread without changing the surrounding numbers.

nodechanges_get_comment_thread_id() is where this obscure workaround to make these numbers landed.

Note issue summary updates
If we can, let’s add something like this when the issue summary was updated.

> Updated issue summary

While it is impractical to import the full issue summary revision history, adding the context that it was updated will help maintainers make crediting decisions when the commenter didn’t add any context for what they did.

#3300073: Update user edit form with identity portal integration has an edge case where the files posted in #3,4,5 aren’t on the node’s list of files for some reason. So we’ll have to call file_load() in this case to get the file details.

Created Gitlab issue #3300073                                                                                                                                                                    [ok]

* Adding 7 followers to the issue.                                                                                                                                                               [ok]
* Created note 646089                                                                                                                                                                            [ok]
Undefined offset: 6666480 drupalorg_gitlab.module:969                                                                                                                                               [notice]
Undefined offset: 6666481 drupalorg_gitlab.module:969                                                                                                                                               [notice]
* Created note 646090                                                                                                                                                                            [ok]
Undefined offset: 6666483 drupalorg_gitlab.module:969                                                                                                                                               [notice]
Undefined offset: 6666484 drupalorg_gitlab.module:969                                                                                                                                               [notice]
* Created note 646091                                                                                                                                                                            [ok]
Undefined offset: 6666519 drupalorg_gitlab.module:969                                                                                                                                               [notice]
Undefined offset: 6666520 drupalorg_gitlab.module:969                                                                                                                                               [notice]

While it is impractical to import the full issue summary revision history, adding the context that it was updated will help maintainers make crediting decisions when the commenter didn’t add any context for what they did.

That is really unfortunate.

What about moving forward though? Are git lab issue summary diffs available?

Empty comments
https://www.drupal.org/project/drupalorg_crosssite/issues/2981570#commen...
only has the issue summary update, and was short-circuited to not migrate. Let’s insert “Updated issue summary” earlier, so this does get a migrated comment. While these are not really substantive comments, I think we should keep them. At the very least, will save a few people asking about gaps in the comment numbers, and may influence maintainers’ crediting decisions.

Image with absolute URL
https://www.drupal.org/project/drupalorg_crosssite/issues/2981570#commen...
has an image with an absolute URL that gets the domain prefix https://www.staging.devdrupal.orghttps://www…

_drupalorg_gitlab_transform_relative_urls() can have the form element handling removed, there won’t be any working forms migrated.

For detecting absolute URLs, let’s do something like is_null(parse_url($value, PHP_URL_SCHEME)) rather than trying to match http/etc, there might be mailto/ftp/gopher/etc.

If the URL starts with a single /, then we can add the domain prefix as we’re doing now.

Theoretically someone could have posted a directory-relative URL, like href="3548544" but that should be very rare, if it happens at all.

Will comment URL https://www.drupal.org/project/drupalorg_crosssite/issues/1549374#commen... of #11 comment redirect to the appropriate new comment URL https://git.code-staging.devdrupal.org/project/drupalorg_crosssite/-/iss... ?
Maybe it's possible to add markup to the migration message and add an ID attribute to the comment ID (id="comment-8058371" for `#14256` of the Migrated from comment #14256. message)?

Re ID attributes - trying this out quickly, and looking at the documentation, I don’t see that GitLab markdown allows custom IDs at all https://docs.gitlab.com/user/markdown/#heading-ids-and-links. I believe the fragment is not passed to the server, so we couldn’t do any special redirecting server-side.

There have been a couple requests for labels in Slack.

Different colors for components, tags, and versions
I think we’d have to do a static variable in _drupalorg_gitlab_get_labels() to keep track of what we’ve added. Then the API call after Get the project labels and add descriptions and colors. could be replaced with getting the values back out.

Distinguishing “deprecated” components
The available component values are a multi-value text field on the project node. The component of an issue is a text field on that issue. So issues can have components that the project currently does not allow. To preserve the issue data, we do want to keep migrating components as they are on the issue. We could add a (deprecated) suffix to the label to distinguish that the project no longer wanted to use the component.

I pushed a few changes around tracking each project’s status:

• Remove access to legacy issue queue configuration fields once a project is migrated
• Static cache project migration status check, since that will be called multiple times per page load
• Disable legacy issue queue as migration starts, this hides the issue cockpit block for that project
• Clear block cache for project page sidebar updates
• Do not require legacy issue queue to be enabled to show the sidebar link to GitLab issues

Since the migration isn’t quick, we might need to track if a migration is in-flight and show the sidebar link to GitLab issues if it has started. Otherwise, there might not be a link to any issues. We can handle that later, as we’re doing the smaller migrations first.

Thank you for fixing it so quick. For info, here is the slack thread in the new #gitlab-issues-feedback channel