Specify dependency overrides on the Automated Testing tab

Another use case is to test integration with optional dependencies, that are not test helpers or the devel module, but modules that enhance the functionality of the module but are not required.

The following issue may be related to this one:
#698932: "test_dependencies" for dependencies / integration tests

Testing on drupal.org using the Automated testing tab should offer some benefits over testing locally. One key advantage that drupal.org has is that all the versions of a project are available. It should be possible using the Automated testing tab to test a project against a range of versions of the dependencies to determine which versions of other projects are required. The module .info file allows specifying version requirements but currently the process for developers determining the version requirements is a slow process of downloading each version to a local installation and testing. So I do not think very many projects are using the capability of specifying version requirements. If we could provide a simpler means of testing against different versions this would allow better requirements in the project .info file and more robust Drupal installations.

Attached is a mockup of the form on the Automated Testing tab that provides the ability to specify dependency overrides. The textfield for the dependencies would use autocomplete and pop up a select box with the available versions that are compatible with that particular version of Drupal. These dependencies would be passed into the functions that determine the default dependencies based on the information in the module .info file. The dependency overrides would be stored with the test so that any retests would use those overrides. This is necessary both to document the test and to avoid having to constantly require entering the overrides every time a test is run.

Taking this approach would solve several other problems with automated testing on drupal.org. The dependency overrides would allow specification of additional dependencies that may be required by testing but are not required by the project itself (#698932: "test_dependencies" for dependencies / integration tests). Currently the default version selected does not respect the version information in the project .info file (#1351744: Respect versioned dependencies and soft dependencies). Dependency overrides would provide a work around to force a version consistent with the module .info file until that problem can be addressed. The result would be a much more flexible approach to automated testing.

With this change, local testing would be against the project versions installed locally while testing under the Automated Testing tab would allow testing against a range of versions of dependent projects. This is similar to developing a theme for one browser until it works for that browser and then uploading it and testing it against a range of browsers (except that in this case there is no IE6).

I am starting to code this and would like any suggestions on improving this plan.

Looks like a great addition ... and since the dependencies array is already part of a branch test, this should not require much in the way of PIFT reconfiguration.

"The new Automated Testing tab on the project pages is a terrific feature. It's even smart enough to take the module's dependencies from its .info files."

That isn't a feature of the testing tab, but the dependency parser. Anyway why/how would this be better than adding them to .info file as proposed in #698932: "test_dependencies" for dependencies / integration tests. That way they are in code and clean to parse instead of in a d.o specific data field?

@boombatower, the idea would be to override the dependency generation found in the .info files. This would give the author more flexibility, and also would provide a little less brittleness when a problem is found in Project Dependency, etc.

"override" isn't exactly what the description says, "which would add to or override dependencies gained from the .info files." nor how it was described to be used.

Anyway, as mentioned in the other issue adding information like this that is only stored on d.o is a bad idea as in asking for problems, but if we want an admin tool to forcibly inject them...that's one thing, but a common use field based, based on everything in the two issues, seems like a foolish idea.

I don't see this being used for storage of dependencies ... only for on-demand and non-persistent injection of dependencies into a single test run. In this light, it provides some additional flexibility which can by users to work around project_dependency and soft dependency issues in advance of someone manually triggering the Jenkins job or coding the soft-dependency solution.

Because we already pass the dependency array over to qa.d.o, the amount of work needed to implement this should be rather trivial relative to implementing the full soft dependency feature; which makes it a quick win, and worth pursuin despite the fact that it could be rendered obsolete once the soft dependencies are in.

The soft dependencies is trivial as well and actually was in the original patch, but tis history. Anyway as long as we don't use this as the main and "recommended" way to do this that seems reasonable. Realistically if we are worried about waiting for -dev release to rebuild to parse these...with conduit we can just move this stuff into a testbot job or hell as I've mention before the entire release building in general....trival to do on the bot....do what it does not...add a timestamp to .info file and run tarball command.

Here is a first cut at adding dependency overrides. I added the ability to override project dependencies to the Test Additional Branches on the Automated Testing tab. The first use case is specifying projects that are not automatically included in the test. All the dependencies of the projects will be automatically retrieved (specifying components can be used when a component module that is not used pulls in a bunch of other projects). This could be used for dependencies required for testing that are not needed otherwise (soft dependencies) or where a project that is required is left out (possibly due to a bug).

The second use case is specifying the version of a project. Once a project is entered, all the available versions are retrieved and appear in the select box (see attached screenshot). This gives an easy way to test against many different versions of a project so that useful information about the requirements of a project can be obtained. This also allows easy testing against the dev branch of another project for potential problems before the project creates a new release. It also allows overriding the version chosen by the default version selection due to bugs in the version selection process or when the selected version causes problems.

The data for the overrides is stored in a new database table keyed to the release nid. That should allow the overrides to be retained for re-tests so that they do not have to be re-entered every time a test is run (unless a different project version is required - then the test should be deleted an a new test queued).

The patch includes changes to project_dependency and project_issue_file_test.

The testing I have done so far is outputting the dependencies that are generated by the changes in the patch and that is just a few cases. The sandbox is queueing tests but they do not run so I have not tried running tests with the new dependencies.

Given that we are looking into replacing the current testing system with another that has a ton of new options available I think we should hold off designing interfaces and such until we have a clear view on all the options that we want to present. This seems like something that would overlap.

But if this is something really holding people back in the here and now then it might be acceptable to make an exception.

@boombatower, I disagree. The "just wait until the new thing arrives" rarely works out. Things get deferred for ages. Here we have @trobey pursuing this, willing to make it happen.

Does the "another that has a ton of new options" have dependency generation and already have this feature?

Based on what I read this has little to do with dependency generation...and more to just be able to add dependencies or override them. The "other" would allow for this in it's built in options. I agree if we have momentum go for it, but be aware we may go a totally different direction later.

While we will be working on the next evolution of the testing infrastructure, in whatever form that evolution eventually takes, that work will be implemented in parallel with the existing PIFR/PIFT infrastructure. The 'existing' will still be around for a significant period of time. (I'd suggest at least until D8 code freeze, and more likely D8 launch ... so we're probably looking at up to a full year.)

When contrib authors add a new dependency to a module, they currently need to wait up to 12 hours or hunt down a testing infra maintainer before they can test the code introduced with that change. As this is a significant pain point, and the proposal here would provide a workaround for dealing with it, then I'd say that it's certainly worth pursuing.

One of the purposes of the overrides is to allow testing against the development branches of a project. This is going to be very important when D8 approaches code freeze and the contributed projects try to get D8 versions out. For example, as a project like Views makes changes, other projects that depend on Views will need to test against the dev branch to make sure these changes do not break their modules. As bugs are fixed in contributed projects, projects that depend on those contributed projects can test them before a formal release is issued. By making the process quick and easy (instead of repeatedly downloading different versions to a local test environment), projects are more likely to catch problems before formal releases are made. This will make D8 more robust and make adoption quicker as the contributed projects are released quicker.

What would seem like the proper fix the to problem of "dev releases are only rebuilt (dependencies parsed) once every 12 hours" is to trigger the dependency part at least on commit. Then all this goes away since it is still awkward to have to enter the dependencies in two places for seemingly no reason when we already have all the parsing stuff built.

boombatower, I do not think it is even close to say that "All this goes away." The issue you are referring to is just one of many issues and what you are talking about will not address any of the other issues.

My particular project that started all this is uc_dropdown_attributes. If you queue this for 7.x-1.x you get:

Array
(
    [1424926] => Array
        (
            [uri] => ubercart
            [version] => 7.x-3.0
            [tag] => 7.x-3.0
        )

)

Views is missing as is Roles and Entity and the test fails. This is a bug and I have submitted a fix elsewhere to correct it but it still has not been committed. Updating the dependencies in the .info file will not fix this no matter how long you wait. It probably is still a good idea to make the updates take effect faster than a day but it is a separate issue except that this patch will provide a work-around until that is done.

Returning to this bug which boombatower's suggestion does not address, the dependency overrides patch provides a work around that allows uc_dropdown_attributes to be tested. It would still be a good idea to implement the patch I have submitted to fix that bug but until that is done there is a way to test uc_dropdown_attributes. It also means that getting these bugs in the dependencies that are automatically generated is no longer a critical issue and more testing of the patches can be done which can avoid breaking the testing environment.

But what if Views has fixed a bug that is breaking uc_dropdown_attributes and it has been checked into the dev branch and they want me to test it before they make a formal release? This type of situation is common whn a new version of Drupal is released and all the contributed modules are rushing to get versions working for that version of Drupal. I cannot test against the dev branch because the automated dependency generation ends up with Views 7.x-3.0. I could download the dev branch and test it locally but if I am in the middle of something this is a real pain. With the dependency overrides I can enter views for the project and it loads all the compatible versions. I select 7.x-3.x and queue the test. Then I should get for the dependencies:

Array
(
    [1424926] => Array
        (
            [uri] => ubercart
            [version] => 7.x-3.x
            [tag] => 7.x-3.x
        )

)

With a couple clicks I should be able to test the bug fix in Views and respond to that issue that it is okay to go ahead with the release.

Another case is when Views is issuing a new beta release then all the dependent projects can easily test against the dev branch and update their project so that when the new beta release of Views comes out their project already works. And it just takes a few clicks so more contributed projects are likely to do it. Right now, without downloading and testing locally, we release and break and then scramble to fix. Fixing stuff in a critical time like that is more likely to be a kludge and introduce future problems. Does that seem like a good way to manage the release process?

Another case is when I have a project and I would like to provide guidance about what versions of a dependent project are okay to use. We have this capability in the .info files and it is not used very often because it is a pain to download and test each version. If it just took a couple clicks of the mouse it would be more likely that projects would use this capability. This would increase robustness since people would not be using outdated projects and filing issues when the problem is that they need to update their website.

Project Dependency is still necessary. I do not want to try to track through dependencies by hand for projects or figure out the correct versions to use. Most projects should be able to test just using the automatic dependencies. We still need to fix bugs in Project Dependency. But we will not have people breathing down our necks to get the bugs fixed if there is a way to override the automatically generated dependencies. And reading the source code in Project Dependency there are heuristics that are used so the automated dependencies are not going to work in every conceivable situation. So if automated generation of dependencies are going to have situations where they do not work then we need to provide a work around.

I have set up my sandbox so that other people can try out the dependency overrides (hopefully that is okay). When you queue the test this has been modified to instead output the dependencies. For my project the URL is

http://trobey-drupal.redesign.devdrupal.org/project/uc_dropdown_attribut...

If you queue a test for 7.x-1.x you get the following:

Array
(
    [1424926] => Array
        (
            [uri] => ubercart
            [version] => 7.x-3.0
            [tag] => 7.x-3.0
        )

)

Because of a bug there are several projects that are not found so currently I cannot test uc_dropdown_attributes on the Automated Testing tab. I have submitted a patch to fix this but it has not been implemented yet. If you use the dependency overrides and enter ubercart with the default version (automatically generated by Project Dependency) you get:

Array
(
    [1424926] => Array
        (
            [uri] => ubercart
            [version] => 7.x-3.0
            [tag] => 7.x-3.0
        )

    [1485738] => Array
        (
            [uri] => rules
            [version] => 7.x-2.1
            [tag] => 7.x-2.1
        )

    [1337360] => Array
        (
            [uri] => entity
            [version] => 7.x-1.0-rc1
            [tag] => 7.x-1.0-rc1
        )

    [1451068] => Array
        (
            [uri] => views
            [version] => 7.x-3.3
            [tag] => 7.x-3.3
        )

    [1507412] => Array
        (
            [uri] => ctools
            [version] => 7.x-1.0
            [tag] => 7.x-1.0
        )

    [1108406] => Array
        (
            [uri] => google_analytics
            [version] => 7.x-1.2
            [tag] => 7.x-1.2
        )

)

Now I could test uc_dropdown_attributes. Google Analytics of these are actually not needed and if I entered 'uc_cart, uc_attribute' into the Components textfield I am back to getting the original output which would not work because of the bug. But I could enter Rules, Entity, Ctools, etc. and get the testing to work.

If I wanted to test against Ubercart 7.x-3.x I could do that. Instead of Default I select the 7.x-3.x (the compatible versions are generated when I enter the project although it may take a second). Now I get:

Array
(
    [696792] => Array
        (
            [uri] => ubercart
            [version] => 7.x-3.x-dev
            [tag] => 7.x-3.x
        )

    [1485738] => Array
        (
            [uri] => rules
            [version] => 7.x-2.1
            [tag] => 7.x-2.1
        )

    [1337360] => Array
        (
            [uri] => entity
            [version] => 7.x-1.0-rc1
            [tag] => 7.x-1.0-rc1
        )

    [1451068] => Array
        (
            [uri] => views
            [version] => 7.x-3.3
            [tag] => 7.x-3.3
        )

    [1507412] => Array
        (
            [uri] => ctools
            [version] => 7.x-1.0
            [tag] => 7.x-1.0
        )

    [1108406] => Array
        (
            [uri] => google_analytics
            [version] => 7.x-1.2
            [tag] => 7.x-1.2
        )

)

Now I am picking up the dev banch of Ubercart and I can test against that with just a few clicks.

You can try your project and see what dependencies are generated and what you can do with the dependency override patch.

In addition to specifying version dependencies, I would also like to be able to specify patches to be applied to versions of modules. For instance, tests for Mailhandler 6.x-2.x fail because of a bug in Feeds. So in the tests for Mailhandler, I'd like to be able to specify that a patch be applied to the latest Feeds dev release.

An obvious existing solution to this issue would be to use Drush Make files, but I have no idea whether that's practical or not.

Alright, so I didn't read everything in #18 and #19 so correct me if I missed something. Two things, 1) when I said "Then all this goes away" I meant for 95+% of users, 2) I still completely agree being able to override is a good thing, but it definitely should not be endorsed as it was described in the workflow as being a workaround for waiting for -dev release to be rebuilt.

Additionally, the new system supports drush make files which I think would drastically improve this process.

My other point was that the new system provides a much more comprehensive set of override features which would require a much different interface which is what I meant by this going away (I was not clear). So just the interface not the override ability (although the feature will be done completely differently in code).

The new system supports #20 quite easily, I plan to even allow such overrides per-issue quite easily. All of this is supported by the hub and works and simply needs an interface for entering it on d.o.

I rewrote the patch to only change files in project_dependency and posted it under that project in #1591550: Dependency overrides.

So this is a dup of #1591550: Dependency overrides then, marking dup. Thanks, @trobey.