I think Project Dependency is ready to go on Drupal.org.
Project Dependency replaces the old project_info module which was hotpatched onto d.o about 2 years ago.
There are many reasons we've wanted dependency detection and enumeration, but the most important immediate one is that testbots can download module dependencies. Many, many projects have not been able to run tests (Views, Commerce) because they have external dependencies. Now that information can be delivered to the testbots, and the testbots can download the dependencies.
Deployment:
- Deploy Project Dependency
- Deploy the latest Project Issue File Test
- Deploy Project module *or* just the patch in #1200486: Invoke hook_project_release_create_package() when a package is being created
- Configure Project Dependency and build the git checkouts for all the projects. This is a straightforward drush command. The complete setup description is in in the README.md in the project folder.
Risks
- Performance: at hook_project_release_create_package() time: It takes a second or two to create the dependencies. It feels really fast when you do it by hand. It only happens when a new package is rolled, so I don't expect it to be hugely expensive.
- Performance: At test dependency generation time: Each test to be sent to qa.drupal.org requires searching the module's tables for dependencies. This seems quite fast, and is only encountered when preparing a test. I don't expect it to be a problem.
- I don't know of any impacts, performance or otherwise, on any other subsystems.
The issue where this was developed: #102102: Parse project .info files: present module list and dependency information.
Comments
Comment #1
rfayComment #2
boombatower CreditAttribution: boombatower commentedsub
Comment #3
drummWe avoid deploying individual patches. BZR does track them, but still would end up annoying to keep tracking. There have been very few commits to project lately.
If possible, please make non-dev releases for pift and project_dependency and mention the releases here. Currently, we have pift 2.4.
So we run
drush project-dependency-process-all
? Do we need to stop pift or anything while this runs?Comment #4
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedSubscribe. Looking forward to having this deployed.
Comment #5
rfay#3, will do on creating non-dev. And it sounds like we'll be OK with project being dev.
I don't think there's any reason to stop pift while the dependency building is running, as the result of not finding dependencies will just be that it thinks there are none... which is what happens now.
Comment #6
clemens.tolboomSubscribe and ...
I was wondering whether this is building graph.inc (D7) structures. As I'm coding http://drupal.org/project/graphapi and #1217648: drupal_depth_first_search (graph.inc) allows for circular graphs maybe I can help visualize stuff :-)
Comment #7
rfay@clemens.tolboom - that would be a great thing. It certainly creates graph structures, but only by database pointers.
Comment #8
rfayI actually jumped the gun just a bit on this one too. I think we need a demonstration system for PIFT/PIFR/Testbot that people can look at. That's dependent on #1227778: Set up dev sites for qa.drupal.org at this point, I believe.
So what needs to be done:
1. Set up a d.o dev
2. Set up a qa dev
3. Point scratchtestbot or another testbot at the qa dev
4. Publicize the results and let people take a gander
Comment #9
Anonymous (not verified) CreditAttribution: Anonymous commentedSubscribing and...
I'm building the Project Browser module, and would like to see project dependencies exposed to the Project Browser module. This would seem to be just what we need for that. Great job on this, rfay :)
-Leighton Whiting
Comment #10
boombatower CreditAttribution: boombatower commentedWell, this doesn’t' expose the data (unless it was massively changed since I wrote it). Doing so would require this revert back to the original design, remove the deviations introduced recently, and be integrated with project module and output information via update.xml or somesuch.
Comment #11
boombatower CreditAttribution: boombatower commentedJust scanned this again as I didn't remember seeing any mention, but I may have missed it. The original plan was to flush out all current project_info data and do a complete re-parse on staging and push over data to d.o. Is that being taken into consideration? Since there is now a scheme change I would assume it is require were as before it was simply to remove corrupt data that we had noticed.
Comment #12
rfay@boombatower, I guess since it only takes a few hours to reparse, it might as well be done on d.o. I'm open though. The database tables are new, with new names, so we'll just drop the old tables. I do think we should provide the pre-checked-out git repositories as per the instructions.
Comment #13
drummComplex deployment like grabbing data from staging makes me nervous about keeping everything in sync. I'm sure there may be a place for it, but I'd rather keep development and production separate.
Comment #14
boombatower CreditAttribution: boombatower commented@drumm: keeping the two separate is definitely a good idea. My only concern was the time requirements to parse, but assume we are fine letting d.o production run this stuff for a few hours then that definitely seems like the best route.
Comment #15
Anonymous (not verified) CreditAttribution: Anonymous commented@boombatower #10 - I wasn't implying this was exposing the data, simply making it available. I plan to expose this in the Project Browser module so that users can download and install whole packages or at least get the dependencies installed automatically when they select a project.
-Leighton Whiting
Comment #16
plachsubscribe
Comment #17
marvil07 CreditAttribution: marvil07 commentedsub
Comment #18
rfayOK, I'd like to see how far we can get with this this week.
I believe we can do this best in stages.
0. Review project dependency and implement any requested changes. I suspect we can get DamZ to do this. Anybody else is welcome in there, of course.
1. Deploy project_dependency module on d.o, without deploying project or pift. The git repositories are refreshed and I have a tarball available, so we don't have to do any cloning of repositories.
2. Build the project dependency databases using drush, per the instructions.
3. Deploy the project dev version (See OP) so that project dependency gets updated when there are new releases.
4. Demonstrate that existing system with a demo testbot
5. Deploy the new PIFT and testbots.
If we can do step 1 and start step 2 today we'll be in great position to test step 3 and perhaps on Friday's code sprint we can get confident enough to just deploy it all and have dependencies for contrib testing! If not, we'll at least be on the right track.
Comment #19
jthorson CreditAttribution: jthorson commentedUnrelated to project dependency, but in the interest of keeping this all in the same place ...
Coder Work:
1. Set up a testbot with the 'Coder' plugin and related configuration/updates. (I believe advisory coder is working, but client certification has been less stable than I'd like ... so it would be good to soak this on a dedicated testbot for a while, to ensure it's stable.)
2. Update the Coder environment on qa.drupal.org to enable coder reviews on a specific branch.
3. Test the 'requeue', 'patch in issue queue', and 'repository commit' methods of triggering new tests, and validate that 'Coder' results are returned in the test summary message as well. Add new nids to the Branch(s) configuration field for the environment, and repeat with more tests as desired.
To disable the Coder environment, set the 'branch' field back to 133700000 ... don't delete & re-create the environment, as that causes other issues (as we found out!)
Comment #20
rfayCreated a stable release of project_dependency: 6.x-1.0-alpha1
Comment #21
boombatower CreditAttribution: boombatower commentedCurious, is the plan to rollout the coder reviews on sandboxes only or more then that? They are far to inconsistent to roll out for everything considering they will mark the test as failed (has thoughts on this). I assume you intend to only roll them out for sandboxes as that is all you have mentioned, just want to be sure.
Comment #22
rfay@boombatower because of #830838: Allow configurable "Advisory" reviews the coder reviews are "advisory" and don't cause a fail. So the idea is to roll them out everywhere I think. But sandboxes are a big use case and why @jthorson got into this :-)
Comment #23
Damien Tournoud CreditAttribution: Damien Tournoud commentedWe are going to do ahead and do steps 0-3 of the plan outlined in #18 today.
Project and Project Dependencies have been merged in bzr. I asked Randy to make sure we quote the parameters properly in the various
exec()
calls inside Project Dependency. Once this is done we will go ahead with deployment.Comment #24
Damien Tournoud CreditAttribution: Damien Tournoud commentedThe code has been deployed today. There are two issues:
* The build job finished with an OOM error; we probably want to add a memory_limit(-1) to the drush command, and could you separate it into a "clean" operation and another operation that only rebuilds releases that are not in the components table?
* The on-packaging parsing of the releases still doesn't seem to work
Comment #25
jthorson CreditAttribution: jthorson commentedForgive me some pure *late-night speculation* without any testing (or even knowledge) to back it up ... but I have a theory which may explain why new releases aren't getting parsed.
CAVEAT: I don't know whether the initial condition which is actually valid: ->package_releases() (line 150 of /release/package-release-nodes.php in Project module) gets invoked with an empty $project_id variable. I don't know if this is valid for the failed release parsing issue, but based on the various "if(!empty($project_id))" checks in that file, I suspect it is an expected scenario.
If this happens, then:
- $project_node never gets assigned (line 164)
-
module_invoke_all('project_release_create_package', $project_node, $release_node)
gets invoked with $project_node = null-
project_dependency_project_release_create_package($project_node, $release_node)
gets invoked with $project_node = null (Line 19 of project_dependency.module)-
project_dependency_process_release($project_node->project['uri'], $release_node->project_release['version']);
gets invoked with a first argument of NULL, and might throw a 'attempt to get invalid property of object' php warning (Line 21 of project_dependency.module)-
project_dependency_process_release($shortname, $version)
is invoked with a $shortname of NULL (line 354 of project_dependency.drupal.inc)- The SQL query gets run with pid = NULL, and returns no values (line 357 of project_dependency.drupal.inc)
-
$release = db_fetch_array(db_query...
is therefore empty (line 358)-
$node = node_load($release['nid']);
results in FALSE-
if (empty($node))
evaluates to TRUE (line 362)- project_dependency_process_release() returns FALSE instead of processing the release.
Solution? Add this just before the module_invoke_all() call at line 248 of /release/package-release-nodes.php in Project module:
Of course, this implies a new dependency on whatever module supplies the versioncontrol_release_labels table, which doesn't appear to exist right now ... but I'll leave that one up to the London folks. :P
Comment #26
Damien Tournoud CreditAttribution: Damien Tournoud commentedThanks jthorson, I think the diagnostic is right. I posted a patch in #1200486-14: Invoke hook_project_release_create_package() when a package is being created.
Comment #27
Damien Tournoud CreditAttribution: Damien Tournoud commentedI have deployed the project fix, and we are currently rebuilding the missing dependencies (we basically rebuild all the releases that do not have components, which means that we are going to reparse non-module projects, just to be on the safe side).
Comment #28
boombatower CreditAttribution: boombatower commentedAren't we rebuilding all of the releases? since much of the previous is corrupt or invalid? or am I misunderstanding?
Comment #29
Damien Tournoud CreditAttribution: Damien Tournoud commented@boombatower: I don't know of any corrupt dependencies. The whole dependency stack has been rebuilt, and everything looks to be humming perfectly. For me, everything is ok.
Comment #30
boombatower CreditAttribution: boombatower commentedRight, the previous data was bad, so if it was rebuilt should be fine....that was my concern.
Comment #31
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedGreat, fixed.
Does that mean that automated testing should work for my project with test_dependencies and dependencies? (Because it didn't #1201540: Get automated tests on drupal.org running). Or is something left to do or wait for?
It fails with
Class 'FeedsMapperTestCase' not found
.That one is added in the Feeds module (a dependency) as:
files[] = tests/feeds_mapper.test
But I can't see it trying to checkout feeds.git anyway.
Comment #32
Damien Tournoud CreditAttribution: Damien Tournoud commented@Niklas Fiekas: we have only deployed the part that extract project dependencies. We still need to deploy the part that uses this information for testing.
Comment #33
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedOk, thanks :)
Is there an issue I can subscribe to?
Comment #34
Damien Tournoud CreditAttribution: Damien Tournoud commented@Niklas Fiekas: you are at the proper place. See the plan in #18. We did 0 to 3.
Comment #35
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedAh, I wondered why it was fixed and didn't notice you set it back to active in #32.
Comment #36
rfayReally all we have left to do before deployment is to make sure that there are no flames with QA or the testbot when the new PIFT goes live. We have an environment set up to test this, so should happen soon.
Comment #37
rfayContinuing the plan in #18 I think we're ready to deploy PIFT and the latest Project Dependency. Last night I put them all through their paces in an environment which sadly went up in flames today, but I tested several different scenarios with a full d.o/qa/testbot setup. Wish I could test more, but it's gone.
Please deploy
project_dependency 6.x-1.0-alpha4
project_issue_file_test 6.x-2.5
Thanks,
-Randy
Comment #38
webchickTagging.
Is there anything else required here? Should it be moved to RTBC?
Comment #39
rfayI think it's RTBC. DamZ has agreed to deploy it when he has time, hopefully this week.
Comment #40
webchickOk, let's do this then. :) w00t!
Comment #41
Damien Tournoud CreditAttribution: Damien Tournoud commentedThe drupal.org part is done. Back to needs work for the test bots.
Comment #42
rfayIMO the first-level fallout from the dependency stuff is worked out in
Project Dependency
6.x-1.0-alpha56.x-1.0-alpha6Project Issue File Test 6.x-2.6
There were
twothree primary issues:1. PIFT was creating a batch that specified a dependency, but was not then providing the dependency batch. This seemed to happen only with ctools, and it was a result of PIFT being too aggressive with its attempt to be recursive on dependencies (and PD is already doing that)
2. Ubercart and Commerce were failing because component dependencies were keyed by component... so there could only be one :-) That meant Ubercart never got the rules dependency.
3. Media module 7.x-2.x-dev had a dependency on file_entity, which used to be part of the release (in 7.x-1.x) but is now a separate project. The dependency search looked and found that it was right there in another version of media and accepted that as a dependency... causing the testbot to try to clone it twice, etc. See #1297392: Ambiguity when a component becomes its own project
I've been successful running a refreshed rfay.redesign.devdrupal.org + qa.scratch + scratchtestbot with all the modules that gave trouble when we turned this on, and things are looking good.
Comment #43
jthorson CreditAttribution: jthorson commentedYou've listed a PIFR release above ... I'm assuming this should be Project Issue File Test 6.x-2.6.
Great job tracking all this down!
Comment #44
rfayWhoops! Edited #42 - thanks. Even the link was wrong.
Comment #45
jthorson CreditAttribution: jthorson commentedProject versions in #42 have been deployed!
PIFT/PIFR communication seems to be stable ... most tests resulted in 'Postponed' on the PIFR side of things (due to branch test failures) - more testing and confirmation needed, but at least it's not freezing us out any more!
Comment #46
jthorson CreditAttribution: jthorson commentedList of issue nids which likely could use some cleanup (after manual test interruption) available at http://pastie.textmate.org/2636553
Comment #47
rfayI'll set those IDs for retest a few at a time. Don't want to overwhelm the system.
Thanks (DamZ?) for deploying!
Comment #48
rfayI think all those are retested. Calling FIXED!
Comment #49
webchickHOORAY!! :D
Comment #51
drummThis was deployed awhile ago.