Deploy Project Dependency, Project and PIFT on d.o

sub

We avoid deploying individual patches. BZR does track them, but still would end up annoying to keep tracking. There have been very few commits to project lately.

If possible, please make non-dev releases for pift and project_dependency and mention the releases here. Currently, we have pift 2.4.

So we run drush project-dependency-process-all? Do we need to stop pift or anything while this runs?

Subscribe. Looking forward to having this deployed.

Subscribe and ...

I was wondering whether this is building graph.inc (D7) structures. As I'm coding http://drupal.org/project/graphapi and #1217648: drupal_depth_first_search (graph.inc) allows for circular graphs maybe I can help visualize stuff :-)

Well, this doesn’t' expose the data (unless it was massively changed since I wrote it). Doing so would require this revert back to the original design, remove the deviations introduced recently, and be integrated with project module and output information via update.xml or somesuch.

Just scanned this again as I didn't remember seeing any mention, but I may have missed it. The original plan was to flush out all current project_info data and do a complete re-parse on staging and push over data to d.o. Is that being taken into consideration? Since there is now a scheme change I would assume it is require were as before it was simply to remove corrupt data that we had noticed.

Complex deployment like grabbing data from staging makes me nervous about keeping everything in sync. I'm sure there may be a place for it, but I'd rather keep development and production separate.

@drumm: keeping the two separate is definitely a good idea. My only concern was the time requirements to parse, but assume we are fine letting d.o production run this stuff for a few hours then that definitely seems like the best route.

subscribe

sub

Unrelated to project dependency, but in the interest of keeping this all in the same place ...

Coder Work:

1. Set up a testbot with the 'Coder' plugin and related configuration/updates. (I believe advisory coder is working, but client certification has been less stable than I'd like ... so it would be good to soak this on a dedicated testbot for a while, to ensure it's stable.)

2. Update the Coder environment on qa.drupal.org to enable coder reviews on a specific branch.

• Under pifr/environment/edit/2:
  • Update the 'Branch(s)' field (inside the 'conditions' fieldset) with the branch nid of the project branch you want to use for testing. Currently, it's set to 133700000, which I assume is just a dummy branch nid. Clearing the field will cause Coder to be run on ALL patches/branches/tests ... so don't leave it blank quite yet.
  • Update the 'Arguments' field (inside the 'Review' fieldset) with "advisory:1" to tell PIFR that Coder reviews should not be considered when deciding the pass/fail status of a patch/branch/project.

3. Test the 'requeue', 'patch in issue queue', and 'repository commit' methods of triggering new tests, and validate that 'Coder' results are returned in the test summary message as well. Add new nids to the Branch(s) configuration field for the environment, and repeat with more tests as desired.

To disable the Coder environment, set the 'branch' field back to 133700000 ... don't delete & re-create the environment, as that causes other issues (as we found out!)

Curious, is the plan to rollout the coder reviews on sandboxes only or more then that? They are far to inconsistent to roll out for everything considering they will mark the test as failed (has thoughts on this). I assume you intend to only roll them out for sandboxes as that is all you have mentioned, just want to be sure.

We are going to do ahead and do steps 0-3 of the plan outlined in #18 today.

Project and Project Dependencies have been merged in bzr. I asked Randy to make sure we quote the parameters properly in the various exec() calls inside Project Dependency. Once this is done we will go ahead with deployment.

The code has been deployed today. There are two issues:

* The build job finished with an OOM error; we probably want to add a memory_limit(-1) to the drush command, and could you separate it into a "clean" operation and another operation that only rebuilds releases that are not in the components table?
* The on-packaging parsing of the releases still doesn't seem to work

Forgive me some pure *late-night speculation* without any testing (or even knowledge) to back it up ... but I have a theory which may explain why new releases aren't getting parsed.

CAVEAT: I don't know whether the initial condition which is actually valid: ->package_releases() (line 150 of /release/package-release-nodes.php in Project module) gets invoked with an empty $project_id variable. I don't know if this is valid for the failed release parsing issue, but based on the various "if(!empty($project_id))" checks in that file, I suspect it is an expected scenario.

If this happens, then:

- $project_node never gets assigned (line 164)

- module_invoke_all('project_release_create_package', $project_node, $release_node) gets invoked with $project_node = null

- project_dependency_project_release_create_package($project_node, $release_node) gets invoked with $project_node = null (Line 19 of project_dependency.module)

- project_dependency_process_release($project_node->project['uri'], $release_node->project_release['version']); gets invoked with a first argument of NULL, and might throw a 'attempt to get invalid property of object' php warning (Line 21 of project_dependency.module)

- project_dependency_process_release($shortname, $version) is invoked with a $shortname of NULL (line 354 of project_dependency.drupal.inc)

- The SQL query gets run with pid = NULL, and returns no values (line 357 of project_dependency.drupal.inc)

- $release = db_fetch_array(db_query... is therefore empty (line 358)

- $node = node_load($release['nid']); results in FALSE

- if (empty($node)) evaluates to TRUE (line 362)

- project_dependency_process_release() returns FALSE instead of processing the release.

Solution? Add this just before the module_invoke_all() call at line 248 of /release/package-release-nodes.php in Project module:

if (empty($project_node)) {
  $project_nid = db_result(db_query("Select project_nid from {versioncontrol_release_labels} where release_nid = %d", $release_node->nid));
  $project_node = node_load($project_nid);
}

Of course, this implies a new dependency on whatever module supplies the versioncontrol_release_labels table, which doesn't appear to exist right now ... but I'll leave that one up to the London folks. :P

Thanks jthorson, I think the diagnostic is right. I posted a patch in #1200486-14: Invoke hook_project_release_create_package() when a package is being created.

I have deployed the project fix, and we are currently rebuilding the missing dependencies (we basically rebuild all the releases that do not have components, which means that we are going to reparse non-module projects, just to be on the safe side).

Aren't we rebuilding all of the releases? since much of the previous is corrupt or invalid? or am I misunderstanding?

@boombatower: I don't know of any corrupt dependencies. The whole dependency stack has been rebuilt, and everything looks to be humming perfectly. For me, everything is ok.

Right, the previous data was bad, so if it was rebuilt should be fine....that was my concern.

Great, fixed.
Does that mean that automated testing should work for my project with test_dependencies and dependencies? (Because it didn't #1201540: Get automated tests on drupal.org running). Or is something left to do or wait for?

It fails with Class 'FeedsMapperTestCase' not found.
That one is added in the Feeds module (a dependency) as: files[] = tests/feeds_mapper.test
But I can't see it trying to checkout feeds.git anyway.

@Niklas Fiekas: we have only deployed the part that extract project dependencies. We still need to deploy the part that uses this information for testing.

Ok, thanks :)
Is there an issue I can subscribe to?

@Niklas Fiekas: you are at the proper place. See the plan in #18. We did 0 to 3.

Ah, I wondered why it was fixed and didn't notice you set it back to active in #32.

Tagging.

Is there anything else required here? Should it be moved to RTBC?

Ok, let's do this then. :) w00t!

The drupal.org part is done. Back to needs work for the test bots.

You've listed a PIFR release above ... I'm assuming this should be Project Issue File Test 6.x-2.6.

Great job tracking all this down!

Project versions in #42 have been deployed!

PIFT/PIFR communication seems to be stable ... most tests resulted in 'Postponed' on the PIFR side of things (due to branch test failures) - more testing and confirmation needed, but at least it's not freezing us out any more!

List of issue nids which likely could use some cleanup (after manual test interruption) available at http://pastie.textmate.org/2636553

HOORAY!! :D

This was deployed awhile ago.