Parse project .info files: present module list and dependency information

FYI: you can already browse projects by category (which is now the default at http://drupal.org/project/Modules, in fact). when you do, there's a whole category for CCK and related modules: http://drupal.org/project/Modules/category/88. similarly, there are categories for Views (http://drupal.org/project/Modules/category/89) and Organic Groups (http://drupal.org/project/Modules/category/90) 3 of the biggest sets of modules that depend on each other in the way you describe...

also, many projects that currently depend on others are fairly good about documenting this and adding links in the project descriptions themselves.

however, it's true that a more formalized dependency system would be nice, if nothing else to standardize this info and visually format it consistantly. we've already got this in the .info files in 5.x core, in fact, so it makes some sense to do that to project nodes, too.

however, i'd be slightly worried about people not keeping these 2 sources of information in sync (and having the project.module parse the .info files and use them directly seems a little scary). also, in the future, you'll be able to specific version-specific dependencies (e.g. og-5.x-1.0 depends on views 5.x-1.0 or later, but og-5.x-1.2 requires views 5.x-1.1 (or something)). to continue that example, og 4.7.x-* does not depend on views at all, but 5.x-* does. it's not obvious if/how we can represent that via project-level dependencies as you're suggesting.

i'm not marking as "won't fix", i'm just sharing my ideas (and concerns), too. clearly, this needs more thought if anything's going to come of it.

thanks,
-derek

I did expect that there would be a few concerns (and many of your concerns were thoughts of mine as well). I think that parsing the .info file would be an option, and a lot less difficult than having the maintainers keep up with it seperately.

I think that having the ability to mark a module as a "library" module would be a good course of action though, because it doesn't make sense to list a module on the Modules download page if the module doesn't really do anything... IMO anyways. (Maybe there should be a way to specify a "library" or API module in the .info file). One thing to note, though, is that some modules provide both functionality, and a library for other modules to use... maybe once there is a way to specify library modules, developers should be encouraged to split off library functionality other functionality (or maybe there should be a "library" project type on drupal.org).

I realize that there are categories right now, but even still, there are a lot of projects to wade through. I'd like for things to be a little less cluttered, but I realize that it's going to add a level of complexity, so I'm not really sure if it's worth it or not.

I think all of these would be useful to have. Dependancies can be taken from the .info file.

well, one of the main reasons I like out .ini format is that the info can there can be used safely by PHP code like project module. i think project should reuse most of the info in the .ini, to be honest. some disagree, which is fine. dependencies are a good place to start. not sure what the UI should look like. maybe just state the dependencies on the project page for now.

if we put code on d.o to parse .info files from drupal modules, it needs to go into project_d_o_hacks.module, not project.module, or else we undo a year's worth of effort on my part to try to make project.module useful for other sites.

the potential for supporting project dependencies and relationships in project.module itself is a reasonable (if low priority) feature reuqest. however, hard-coding .info file logic is out of the question. we need a separate module for that.

See also: http://groups.drupal.org/node/11998

This would be a useful thing to have.

Since dependencies are at the module rather than the project level, I think what we need is actually a project_info module that parses and stores from .info files:

* the association of modules and their titles and descriptions with projects (projects being identified by nid), tracked by version, and
* the dependencies of modules on each other, again tracked by version

From this, we could present e.g. in blocks that appear on a project's page:

1. The projects that this project depends on.

2. The projects that depend on this project.

3. The modules included in this project (with their titles and descriptions).

I'll have a go at sketching this in as a new module.

I'm changing the issue title to reflect this specific aim. A "library" categorization of modules looks to me like a separate feature request (possibly just a new vocabulary or term).

Versioning information is going to be tricky.

In the first place, dependencies vary between core compatibility versions. module_x's Drupal 5 version will have different dependencies than its Drupal 6 version. So we can't say module_x has an overall set of dependencies. Rather, we need to distinguish between dependencies by version.

But dependencies may also vary within a core compatibility version. E.g., a core 6.x version of a given module may exist in multiple releases. Each of these may have different dependencies.

And we know version information only on one side of this dependency. That is, we know that the 6.2 stable release of module_x is dependent on module_y, but we don't know what version of module_y is required (6.1, 6.2, etc.).

Possibly we need to track dependencies only by Drupal core compatibility--e.g., assign module_x a single set of dependencies for Drupal 6.

It's not going to be straightforward deciding how to display this information. But first things first--we need to capture it before displaying.

@nedjo: It's great you're going to be working on this. If you haven't yet, please read http://groups.drupal.org/node/11998#comment-39642. Thanks!

Here's a first very rough draft, completely untested.

Some quick notes:

* Creates a new module, project_info, which is in an info directory in project.

* We capture information on modules and their dependencies by calling a processing function from the packaging script.

* Modules are stored in a new table, project_info, with their associated release node ID and the various data in the .info file (name, description, etc.). Looking again at this table, it may be too normalized--it may be better to store e.g. the core compatibility tid and the project node nid here too, to reduce the need to join on other tables.

* Modules themselves have a release associated with them, but for dependencies all we have is a module name. So we store dependencies as a relationship between a module ID (the ID field of the project_info table) and the dependency's module name.

A lot still to do. I'll pick this up later in the week. Comments meantime or suggestions for how I could test would be great.

subscribing, i guess we probably want to bump this forward to 6.x right?

Subscribe

I've been working on this stuff recently, but from the perspective of an outside system.

http://groups.drupal.org/node/21295

Basically I am building a directory structure of easily mirror-able meta-information, that can be parsed on the client side.
This is specifically with the goal of writing something like apt-get for Drupal.

I'm already working on automating these dependencies in my code, but I'd very happily support this getting into the generated info files itself (I already use the values from the .info files where available)

Something to keep in mind, is that modules provided by a package might change between releases, so this shouldn't be tied to the project, but to the releases (I know it makes it more complex, but that's how it works)

Same case for dependencies. New dependencies might be added or removed.

I'm going to be working on this since I need it for project_issue_file_test (PIFT).

Initial patch to confirm direction.

This should store the list of modules.

This should store dependencies to the highest dev release in the same drupal core API version...later we can add support for the advanced style .info files supported by D7.

I'll add some sort of views support for this.

Module list view done, got to head out...I'll get it and dependency view done before posted.

With views!

Added API.

Now I will attempt to integrate this with pifr.

PIFT integration was simple:

I'll work on deploying on d6.d.o and testing the .info file parsing on release build, and I suppose I need to figure out some sort of script to run on all projects as we talked about.

The view will need to be tailored to a particular release for a project since the current argument is just pid, but that is minor since we don't need to display them...we really need the parsing for pift.

This patch is update to a new schema that keeps track of the specific "module" that is depended on in addition to the release.

After quick chat with dww:

Plan: copy package-release-nodes.php as parse-legacy-info.php and hack it to run in tmp and not touch db.

Preferably come up with hack to temporarily store dependencies and parse in second round without having to re-checkout everything.

More up-to-date, still needs work to finish #24.

Seems to work well, I'll wait for d6.d.o to come back up for final testing.

Cleaned version.

Through testing on d6.d.o fixed this so far, plus requires #424372: :: in .info files causes Fatal error.

Seems to be working well.

Latest patch against PIFT.

Ass patch file.

This is ready to go, works great. @see http://d6.drupal.org/node/350371

Wow! It's great to see this longstanding need close to completed. Thanks the great work.

I browsed through d6.drupal.org and found what look to be a couple of remaining issues.

1. Some releases show the release's module itself as a dependency. E.g. Views http://d6.drupal.org/node/621428. This is probably happening because the release includes modules that depend on each other, e.g., views_ui on views.

2. Some dependencies are missing. E.g. CCK fieldgroup tabs, http://d6.drupal.org/node/553732. Correctly shows tabs as a dependency but misses fieldgroup. At first glance it looks like we're only getting dependencies shown where the module name matches the project shortname.

How hard would it be to capture and display the human readable module names, and also use the project node titles? Doing so would make the blocks more user friendly.

It might also be useful to capture and offer the .info description to offer a sense of what each included module does. Of course this could be done in a follow up patch.

In many cases a project will include multiple modules, some of them less needed or even fairly obscure. In these cases, listing the combined dependencies of all modules in a release could give the mistaken impression that all these other modules must be downloaded and installed before the release can be used.

Should we consider listing modules and dependencies in a single block? Maybe a horizontal one in the content area? It might look sort of like the admin/build/modules page table:

Includes the following modules

Personally, I would rather hold of on the UI stuff. If we want to just hide the blocks that is fine. I need this functionality to do contrib testing (which we really want in place a soon as possible so ppl write tests as they port their modules).

I think your ideas make some sense, but I would rather get the architecture in place before tweaking the UI.

Agreed that the UI can wait. But it's worth thinking about now to make sure the data structure is correct.

Currently the patch captures two types of data: modules (in 'project_info_module') and dependencies (in 'project_info_dependency'). The latter tracks the relationship between releases and the modules that are required by them.

This won't give us enough information to associate dependencies with a particular module. To do so, we would need 'project_info_dependency' instead to track a dependency relationship between two modules by project_info_module.module_id.

For the specifically, I am not sure what that gets us. I mean we can display it, but I am not sure it is all that useful.

This looks to be the source of the problem I noted above. We're only recognizing dependencies that have a module name that matches a project shortname, so dependencies will be saved only for modules that match their project's name.

Another more minor issue is that, at the time we're processing a given release, we don't yet have full data for other releases, so dependencies we determine may not be up to date.

Beyond this, though, dependency is fundamentally a relationship between two modules. Starting with my original patch (sorry!) [edit: on second glance, no, I was indeed tracking by module], in the proposed schema we're capturing not this original relationship but the more general relationship between a release and the modules that it requires. This release-module data may be useful in some cases, but not all, and in particular I'm realizing now that it doesn't give us the level of information we need to provide the most useful information on dependencies to site admins evaluating a release or a project--e.g., the table I suggested in #33.

Rather than tossing out the module-to-module dependency data and storing just the generalized project-to-release info, we should capture and store the actual dependency relationship--from which we can easily determine the release-module dependency any time we want.

Proposed schema:

We could determine the releases a given release depends on with a query something like the following:

(though we'd need an extra join or two plus where clauses to get the recommended stable release for the correct core version).

[edit: fixed error in schema--module name is not unique. And corrected reference to schema in #10.]

Rough, untested function for saving dependency data per the schema in #37:

To return to a key issue I noted in #8: modules indicate their dependencies on other modules, but we don't know the specific versions required.

So we have to either (a) make an inference or (b) add a way for modules to specify their dependencies' versions.

If we're making an inference, probably our best guess is that what's required is the current recommended stable release for the given core api version. But of course, this won't always be the case. E.g. http://drupal.org/project/apachesolr_views requires Views 3 (6.x-3).

The alternative is to introduce a way for modules to indicate the version they require. For e.g. a reliable testing system, I'm pretty sure we need this.

The most straightforward way would be an API change:

but it's obviously far too late for that for d6, and probably for d7 too.

But we could add an optional and supplemental:

and parse this along with the other data. That looks like the only way we'd get reliable dependency information.

That's the thing...the code was intended as placeholder code until we implement the Drupal 7.x style .info files which allow versions to be specified. Then having the relationship to a release makes more sense, not to mention the entire point I am writing this feature, testing. Testing needs the releases as it need something to checkout that it can test against.

For 6.x I thing we should just make an inference and we can choose to add an interface or workaround if we want. The focus is on getting Drupal 7.x contrib testing working. Thus I would like to design this with that in mind instead of adding things to 6.x.

You are correct in #37 that it only loads dependencies where they match the module shortname. That was an oversight when writing the code, but the design works fine. All I need to do is a run a query to determine which project contains he module first and then load that project (as I thought about, just failed to code).

As for the display part I am not sure what we gain, and such. I think we can store more module information and such, but the key is we want this functionality NOW. So apparent from holding up the patch I would really like to move this forward, and deploy this weekend, if possible.

I guess what I am hitting at. There are ton of things to make this 100%, currently with the one issue fixed this should be a 90-95% solution. That gets us a start, then people can play with contrib testing (hopefully encourage them to write tests as they port) and we can work out the bugs in contrib testing while we worry about how to make this 100%. I mean there are a lot of things to consider, and it may be possible to integrate with the Drupal 7 code that already does this, but I'd like to spend more time discussing and figuring that out.

I'll take a look at the code samples and fix/merge stuff. I think your schema makes sense and provides us with more detail. I'm just concerned that we will continue and delay this too long.

The code in #38 appears to be too simple for our needs. The query to determine which module_id the module depends on does nothing to pick a release other then that the release is in the same drupal core compatibility. Which means we are doing them with no respect to order (version) or any kind nor stable/dev.

I am working on changing the code to point to modules, but the code from #38 won't work. For now I will choose the dev release for the latest stable in the core compatibility tid. That will suite 90%+ of modules and we can use D7 .info format later to gain 100%.

Views integration is not complete, but I believe I have workable code using module => module relationships. I need to run it on d6.drupal.org to see how it does, but its looking good.

I am not sure I about views integrations. I pulled that out of the patch so we can commit the structure part and then decide on displaying information.

This will also require a slight teak to pift patch.

I should actually role patch. :)

Updated both patches.

Updated after some testing. Appears to work, I'll be running whole thing once more to double check.

Latest patch looks for on project_info side. Ran entire thing on d6.d.o last night and today I check the results with:

Great work.

Yes, this looks like the best approach to take at this time. Eventually we'll probably want to point to the recommended release rather than a dev one, but as an interim step, to meet the testing needs, dev makes sense.

Is this just the same as:

?

Because some D7 .info dependency data will have versions specified, we need to strip that out as an interim step until it's supported. We can just use whatever's before the first space.

When adding dependency version support, some of the needed code is in core's module.inc.

Shouldn't be the same.

for instance: views 3.x-dev, views 2.x-dev, views 2.7...thus it will stop on views 2.x-dev, not views 3.x-dev.

I added

for other bit.

Updated pift patch after some testing.

This seems to work as well as intended.