As a new system for contribution records is being built (see https://www.drupal.org/project/infrastructure/issues/3322116), we will need changes on D7 www.drupal.org to adapt and communicate with it.
The following tasks are part of the bigger plan detailed here: https://docs.google.com/spreadsheets/d/1UVM8Bs6VdoJFo2-DQGjTRo5yeecMWMvp...
✅ Migrate existing credits to the new system
✅ Provide links on issues to the new system related record
✅ Trigger webhook for comments to update contributors on new system
✅ Trigger webhook for issue status change to update the contribution record status on the new system
✅ Hide any form or reference to the old attribution system (comment form on issues)
✅ Change to user profiles to read from the new system
✅ Changes to organizations to read from the new system
✅ Changes to marketplace to read from the new system
✅ Improve UX to add individual contribution records and bulk credits
I will use this issue/branch to put all the D7 changes that are required.
✅ First phase: Send data to new.drupal.org:
- MR: https://git.drupalcode.org/project/drupalorg/-/merge_requests/337
- Settings on D7:
-- Set drupalorg_credit_migration_token to the same value as the one set in D10
-- Set contribution_records_base_url to https://new.drupal.org
-- Set drupalorg_contribution_record_delay_microseconds if we want to throttle requests to D10
-- Setup queue drupalorg_issue_events
- Settings on D10:
-- Allow 'contribution_records' in drupalorg_allowed_content_types
-- Allow /contribution-record* paths on Fastly
-- Allow /drupalorg-api/* paths on Fastly (I think this is done)
-- Setup environment variable so this line works: $config['drupalorg.settings']['credit_migration_token'] = getenv('DRUPALORG_CREDIT_MIGRATION_TOKEN');
-- Setup queue contribution_records_import_queue_worker
* [D7 www.drupal.org]: drush drupalorg-contribution-records-sync --raw-import=2 (when the queue runs, the record will be queued to be created in the new system, so the D7 push operation will be quick and should not timeout).
* [D7 www.drupal.org]: drush drupalorg-contribution-records-sync --node-type=sa --raw-import=2 - same but for SA.
* [D7 www.drupal.org]: drush queue-run drupalorg_issue_events
* [D10 new.drupal.org]: drush queue:run contribution_records_import_queue_worker
Compare the data from the current system and the new system. See data integrity checks.
- Merge MR351 / MR364 / more to come...
- drupalorg_credit_system defaults to legacy, so everything will continue to be the same in D7 once merged.
- Test new system (reading credits):
-- drush vset drupalorg_credit_system "modern"
-- drush cc views
-- Visit user pages, org pages, owner tools pages, and re-run marketplace calculations
- Revert to old system (D7):
-- drush vset drupalorg_credit_system "legacy"
-- drush cc views
-- Visit user pages, org pages, owner tools pages, and re-run marketplace calculations
- D7 Make the new system the canonical source of credit information
-- drupalorg_credit_system needs to be set to "modern"
-- Then drush vset drupalorg_credit_system_data_source "modern" (or "legacy" to revert)
- D10
-- Enable permission "edit any contribution_record content" for "authenticated users"
-- Remove integrity checks cron job on prod. Add the following to `helm/drupalorg/prod/values.yaml` (watch indentation) in the "helm-deploy-tree" repo.
migrationJobsStatus:
contributionRecords: false
Steps:
BEFORE
- Communicate change and 20-30 min outage.
- Set red banner top message with the information?
DEPLOY
- Merge last D7 changes: https://git.drupalcode.org/project/drupalorg/-/merge_requests/392/diffs
- Set D7 flag to "modern"
- Run integrity checks one last time
- Merge D10 changes: https://gitlab.com/drupal-infrastructure/sites/drupalorg/-/merge_request...
- Merge helm-deploy-tree MR: https://gitlab.com/drupal-infrastructure/helm-deploy-tree/-/merge_reques...
AFTER
- Clean all that work up (that can happen after)
-- D7: Remove uneeded contributors block: https://git.drupalcode.org/project/drupalorg/-/merge_requests/397/diffs
-- ✅ D10: Remove temp message: https://git.drupalcode.org/project/contribution_records/-/merge_requests/15 + site update.
-- 🚧 D10: Integrity check job in the module: https://git.drupalcode.org/project/contribution_records/-/merge_requests...
-- D7: any code using "drupalorg_credit_system_data_source" if/else
| Comment | File | Size | Author |
|---|---|---|---|
| #153 | Screenshot 2025-09-04 at 10.01.54.png | 37.95 KB | fjgarlin |
Show commandsStart within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
changes, plain diff MR !397compare changes, plain diff MR !395compare changes, plain diff MR !392compare changes, plain diff MR !388changes, plain diff MR !386changes, plain diff MR !384changes, plain diff MR !382changes, plain diff MR !378changes, plain diff MR !376changes, plain diff MR !375changes, plain diff MR !374changes, plain diff MR !373changes, plain diff MR !372changes, plain diff MR !371changes, plain diff MR !370changes, plain diff MR !365changes, plain diff MR !367changes, plain diff MR !366changes, plain diff MR !364changes, plain diff MR !351changes, plain diff MR !362changes, plain diff MR !361changes, plain diff MR !360changes, plain diff MR !355changes, plain diff MR !354changes, plain diff MR !353changes, plain diff MR !350changes, plain diff MR !336changes, plain diff MR !337changes, plain diff MR !332changes, plain diff MR !273changes, plain diff MR !147
Comments
Comment #3
fjgarlin commentedComment #4
fjgarlin commentedComment #5
fjgarlin commentedComment #6
fjgarlin commentedComment #7
fjgarlin commentedComment #8
fjgarlin commentedComment #9
fjgarlin commentedComment #10
fjgarlin commentedComment #11
fjgarlin commentedComment #12
fjgarlin commentedComment #13
fjgarlin commentedComment #14
fjgarlin commentedUpdate: functionality-wise, everything is there. We're starting to review/improve and see if there is anything missing.
The D7 changes can be seen in the MR for this issue. The whole plan (D7 + D9 new system) is in the google doc linked in the issue description.
Comment #18
fjgarlin commentedThe new MR contains the code from the old one as it was 2 years behind and rebase didn't work straightaway.
I will need to deploy this again to a D7 site and test.
Comment #19
fjgarlin commentedUpdated deployment instructions.
Comment #20
fjgarlin commentedComment #21
fjgarlin commentedUpdated deployment commands.
Comment #22
fjgarlin commentedUpdated deployment instructions.
Comment #23
fjgarlin commentedComment #24
fjgarlin commentedComment #25
fjgarlin commentedUpdated deployment information, queues to set up, etc.
Comment #29
fjgarlin commentedWith such a complex MR, it was easier to redo the whole thing than rebasing. New MR: https://git.drupalcode.org/project/drupalorg/-/merge_requests/332
Comment #30
fjgarlin commentedI will try to split the MR into two:
- One for pushing data to D10
- One for reading that data from D10
Comment #34
fjgarlin commentedFirst part: Send data to new.drupal.org:
- MR: https://git.drupalcode.org/project/drupalorg/-/merge_requests/337
- Settings on D7:
-- Set
drupalorg_tokento the same value as the one set in D10-- Setup queue
drupalorg_issue_events- Settings on D10:
-- Allow 'contribution_records' in
drupalorg_allowed_content_types-- Allow
/contribution-record*paths on Fastly-- Setup environment variable so this line works:
$config['drupalorg.settings']['token'] = getenv('DRUPALORG_TOKEN');-- Setup queue
contribution_records_import_queue_workerComment #35
fjgarlin commentedComment #36
fjgarlin commentedComment #37
fjgarlin commentedComment #38
fjgarlin commentedSimplified deployment instructions.
Comment #39
fjgarlin commentedComment #40
fjgarlin commentedAdded a configurable way to throttle the requests to the new system, in case we need it.
Comment #41
fjgarlin commentedThe second MR https://git.drupalcode.org/project/drupalorg/-/merge_requests/336 is mostly ready, but it relies on MR337 being merged first together with the deployment steps for that part.
After that, and when this is merged, we should be able to switch back and forth between the "legacy" and "modern" credit system with the following commands:
Use modern system:
Switch back to legacy system:
A third part, and once the new system is stable and in use, will be to clean up the code from the legacy system and remove the switch, but we will go one step at a time.
Comment #42
fjgarlin commentedUpdated the IS as we updated some of the token and headers names on both D7 and D10.
Comment #46
fjgarlin commentedSA nodes can now be sent to the new site to create the corresponding Contribution Credit.
MR: https://git.drupalcode.org/project/drupalorg/-/merge_requests/350/diffs
Once deployed:
-
drush drupalorg-contribution-records-sync --node-type=sa --raw-import=2-
drush queue-run drupalorg_issue_events(This might not be needed as it is the same queue as with the previous step)Comment #51
fjgarlin commentedAfter the first full import run, we've had around ~1% missing on the new site. The reason for most of these (if not all) was that the issue didn't even have a comment, not even the automated comment #1 which is always linked to the author.
We will send the author with no attribution (as none can be calculated) when there are no comments on the issue. If the author makes a comment with attribution, then this last one will be taken instead.
MR to fix this situation: https://git.drupalcode.org/project/drupalorg/-/merge_requests/353/diffs
After that, we will need to re-run some of the above commands with selected NIDs.
Comment #54
fjgarlin commentedEdge case for when there are no contributors: https://git.drupalcode.org/project/drupalorg/-/merge_requests/354/diffs
Comment #57
fjgarlin commentedMore edge cases:
- Issues with no title
- Issues with no comments and the users who created them no longer have an account
MR for that: https://git.drupalcode.org/project/drupalorg/-/merge_requests/355/diffs
Comment #69
fjgarlin commentedThe first phase is almost complete.
We have been running data integrity checks between the D7 system and the new system to make sure that all records and all attributions are set correctly.
The next phase means merging https://git.drupalcode.org/project/drupalorg/-/merge_requests/351/diffs. That will leave things as they are now but we will be able to switch between the current D7 system or the new system just by setting a flag. I've updated the issue description with the steps needed for this.
Comment #74
fjgarlin commentedThe above merge is huge help for me going forwards. There will be more smaller MRs but won't need to do huge rebases and we won't need to re-review thousands of lines of code.
Comment #83
fjgarlin commentedComment #88
fjgarlin commentedMR367 merged.
Comment #89
fjgarlin commentedThe data integrity phase is now done. We have https://git.drupalcode.org/project/contribution_records/-/blob/1.0.x/scr... to compare D7 credits vs new system credits.
This is a HUGE achievement as there were more than 5 million records to migrate here. Over the last two weeks we were fixing edge cases in the code and data quality/consistency on D7.
We are now focusing on query performance so we can start querying the new system for reading credits.
Comment #94
fjgarlin commentedWe missed something really small in an
ifcondition in the last MR. This one fixes it: https://git.drupalcode.org/project/drupalorg/-/merge_requests/370/diffsIt also adds some warning messages if the data could not be retrieved and it won't cache information in those cases.
Comment #125
fjgarlin commentedWe are now reading credits from the new system.
Comment #138
fjgarlin commentedD10 deployment steps added. I need to link the actual MRs.
Comment #139
fjgarlin commentedAdded the missing MRs.
Comment #143
fjgarlin commentedAdded a quick cleanup MR to the steps.
Comment #144
fjgarlin commentedComment #145
fjgarlin commentedAdded more details to deployment steps.
Comment #146
dieterholvoet commentedChanges to this contribution record don't seem to be saving for me, once I refresh the page everything is unchecked again. I am noticing a lot of console errors in the likes of
Uncaught TypeError: $(...).once is not a functionComment #147
fjgarlin commentedReplied to the above via slack. Needed to be logged in and click on "Save". Can't reproduce the JS error on chrome/firefox logged in/out.
Comment #152
nicxvan commentedIs there a way to see number comments and commits by each person like the old form?
Comment #153
fjgarlin commentedNumber of comments yes. Number of commits no. We can also see if files were uploaded and reactions in MRs (eg: thumbs up)
Comment #154
fjgarlin commentedAll the planned steps for "before" and "deploy" were done. The "after" clean up tasks will take place in the following days.
Comment #157
fjgarlin commentedRemove uneeded contributors block: https://git.drupalcode.org/project/drupalorg/-/merge_requests/397/diffs
Comment #158
itamair commented@fjgarlin I also don’t see / find anymore the very below section (and select) that was allowing to merge a MR directly from the specific issue drupal.org page.
Is this intentiona? and are we now supposed to use a “more raw & traditional Git merge operation forcing the intended commit message with the new crediting format instead?
Where could we find some context on the removal of that MR merging operational option? (in any … )
thx ...
Comment #160
fjgarlin commented@itamair - more and more actions related to code are being moved to GitLab. Merging MRs has been available for years on the GitLab UI and it is now the only place where MRs can be merged.
As the MR widget was heavily linked with the table of credits, it was removed so the merging workflow goes to GitLab. Issues for projects will be migrated soon.
Comment #161
drummThis has been deployed for awhile and the immediate followups & cleanups have been triaged & fixed. 🎉
Any future followups should be their own issues.