Packagist for Drupal Projects

I'd love to use composer and its format for our update system, but that would require changes to d.o. Those are fairly stalled right now for various reasons and won't unstall until well after Drupal 8 freeze. :-(

Stalled or not, this is really cool Haehnchen! Props on the awesome proof-of-concept :)

PS, I think the site's down though!

Very nice :-) . Well done, Haehnchen!

OK, so I'm going to try to start playing around with this strategy of porting updates.drupal.org data into composer-recognized formats. Any thoughts on taking the approach that packagist takes, and giving each package its own endpoint?

ie https://packagist.org/p/kimpepper/drupal.json

Thinking that this way, we don't need to pull everything from d.o infra in bulk, but just hit the updates.drupal.org urls when the respective drupal-packagist url is hit.
http://updates.drupal.org/release-history/drupal/7.x

What do you mean my "give each package its own endpoint"?

Currently, all the packages are in one json file URI, right?
Example: http://drugist.espend.de/p/packages-7x.json

I was saying each project should get it's own json file. (Sorry, I guess I mangled my use of "endpoint".)

If I understand you correctly, one JSON file per project makes sense. With however many thousands of projects we have, in all of their various tagged releases, one JSON file for the entire repository would be... ridiculously big. :-)

@Haehnchen for the record, I'm probably going to pull some of the logic out of your project -- specifically the bits for pulling and parsing project data from updates.drupal.org -- and put it into its own lib on Packagist called something like `updates-drupal-org-parser`. Just trying to get into the composer spirit, since maybe others could use it (myself included). I'll credit you as the author, of course! :)

Note: for the D7 version of d.o, there's now a drush command that generates the release history feeds used by the Update Manager. See #1800190: Update project-release-create-history.php to D7 in Drush for more. This code is automatically invoked by the packaging script whenever new releases are generated for a given project. In principle, seems like it'd be easy to have both:

http://updates.drupal.org/release-history/drupal/7.x

and

http://updates.drupal.org/packages/drupal/packages.json (or whatever)

Since update manager only ever cares about a given core major version, the release history XML feeds are currently segregated by core major version. Does that make sense in this world, too?

If people want this stuff directly on d.o (and they should!) I'd encourage you all to take a look at that drush command and consider uploading a patch that generates the appropriate .json files for you needs.

Cheers,
-Derek

I came across this issue after reading about Wordpress Packagist (http://wpackagist.org/) - looks like the Wordpress community are going this way too. It makes a lot of sense to me, and an awesome POC.

Does this work with the current version numbering?

I.e. can I specify "drupal/views": "7.x-3.*" ?

That's part of the discussion we've been having on #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) and Allow Plugins and/or Script to modify versions before packaged is parsed/fetched.

If it doesn't, how do you think we should get around the version number problem?

Thanks!

The examples given in the issue summary (and on the github repo) show that wild card version numbering works, so the scenario you mention would be possible.

EDIT: To expand on that, composer matches various patterns against tag names in the repo - see https://packagist.org/about

New versions of your package are automatically fetched from tags you create in your VCS repository.

so as long as your tag names are parseable to match the rules in https://getcomposer.org/doc/01-basic-usage.md#package-versions then any naming convention(s) can be used.

You can specify a pattern with a * wildcard. 1.0.* is the equivalent of >=1.0,<1.1.

tsphethean,

But the examples given, are not Drupal module tags.

Most modules (i.e. Views) are not tagged with a "X.Y.Z" syntax that is required by Composer.

For instance, in Views, you have "7.x-3.7" which is not a Composer compatible version number.

How does this Issue get around the module version numbering?

As an example, what if I depend View "7.x-3.6" ? How would I specify that in Composer?

Thanks!

@davidwbarratt - ah sorry I must have had version number blindness! I've not tried it, but I *think* you can use non "X.Y.Z" syntax with composer, so in the package composer.json you could just use the Drupal naming convention for the version number. The comparison logic should still be able to sort on that and work out the most recent version, but I'd have to do a POC to check.

Alternatively, and this might be the best "bridge" scenario between sem ver and Drupal versioning, you can generate branch aliases in composer.json - see https://getcomposer.org/doc/articles/aliases.md

{
    "extra": {
        "branch-alias": {
            "dev-master": "1.0.x-dev"
        }
    }
}

As a result, anyone can now require 1.0.* and it will happily install dev-master.

I'm not sure if the same applies with tags though...

I guess the other aspect with Drupal (and this might be better in the other issue) is the split between the D6,7,8 versions which we do by branch/tag names. I guess in a Sem Ver land, we'd use something like:

"require": {
        "drupal": ">7.0.*"
    },

to indicate which Drupal version a particular version of a module worked with, but I'm a bit hazy on how mutliple versions of the same module would work from a version number perspective!

Here's something contributed by winmillwill. It's a packagist for Drupal modules and themes. They are figured out by this parser

http://static.drupal-packagist.org/v0.1.0

Add

 "repositories": [
    {
      "type": "composer",
      "url": "http://static.drupal-packagist.org/v0.1.0/"
    }
  }

To your composer.json... then do a composer show drupal/drupal and more... My composer file looks something like this

 "require": {
    "drupal/admin_menu": "~7.3.0-rc4",
    "drupal/ctools": "~7.1.4",
    "drupal/date": "~7.2.7",
    "drupal/drupal": "~7.28",
    "drupal/features": "~7.2.0",
    "drupal/globalredirect": "~7.1.5",
    "drupal/migrate": "~7.2.5",
    "drupal/module_filter": "~7.2.0-alpha2",
    "drupal/omega": "~7.4.2",
    "drupal/pathauto": "~7.1.2",
    "drupal/panels": "7.3.4",
    "drupal/strongarm": "~7.2.0",
    "drupal/token": "~7.1.5",
    "drupal/views": "~7.3.8",
    "guzzle/guzzle": "3.*",
    "pokermania/pokernetwork": "4.6.48",
    "winmillwill/settings_compile": "~2.0"
  },
  "require-dev": {
    "behat/mink-zombie-driver": "*",
    "behat/mink-extension": "~1.2",
    "behat/behat": "2.5.2",
    "drupal/devel": "~7.1.3",
    "drupal/devel_generate": "~7.1.3",
    "drupal/devel_node_access": "~7.1.3",
    "drupal/devel_themer": "7.1.*@dev",
    "drupal/diff": "~7.3.2",
    "drupal/mail_logger": "7.1.*@dev",
    "drupal/drupal-extension": "0.1.5",
    "phpunit/phpunit": "*"
  }

I'll add that for now the json updates are being produced on manual run and not on cron from the looks of things, but it looks promising.

As a side note, @Mike Pirog pointed me to http://hubdrop.io , which mirrors Drupal project repositories on GitHub at https://github.com/drupalprojects . As long as we add composer.json files to the projects, and add them straight to packagist, we'd have yet another method of installing modules via Composer.

The third-party services are great work, although +1 to having them officially supported. Being able to file minor bug reports is a good motivation for doing so: drupal-packagist service even seems to support installers-style subfolders, which is great, although buggy: it put mothership and/or zen in a drush/ subfolder, not a themes/ one, and downloaded Drupal core to vendors/drupal/drupal.

One odd thing I did notice is that drupal-packagist.org seems over-generous in their dependencies: downloading mothership, views and ctools got me advanced_help, i18n, panels and variable, none of which are requirements as far as I'm aware. Is this by design? Also the Drupal core download: if I haven't selected a version of Drupal core, then I'm not sure what I can safely do with that, not knowing if an existing database's schema matches it.

Again, though, great work by everyone involved. Here's to our glorious composer future!

@jp.stacey any packages that have the wrong type... file to and issue on github underhttps://github.com/winmillwill/drupal-parse-composer. Also as far as moving packages around... will also built drupal tangler for this. With that said as a work around for the drush issue... you can use installers to place the package where you want because its not of the libraries type. Also ive noticed the samething about the dependencies. Maybe we should open an issue about that as well. It is v0.1.0 afterall.

@generalredneck thanks: there seems to already be an open issue for this so I'll add my 2p-worth there.

Given drupal-packagist.org gets the modules/ subfolder right (i.e. not the composer vendor/ default), it's weird that it puts zen in drush/ and not themes/ . It would be a shame to have to hardwire the installers cruft in composer.json to work around that, when it's so close to being there anyway!

But as I say, +1 for this being a key drupal.org offering in future if possible, and bugfixed as another Drupal project.

Interesting work being done here. I am also looking into how we can use Composer as an alternative to Drush Make.

We are currently as the stage were have documentation of how Composer could actually be used to replace Drush Make as well as a prototype of Composer repository for Drupal projects similar to http://static.drupal-packagist.org/.

@generalredneck: Perhaps there is room for collaboration here?

If anyone would like to discus how Composer can be used for Drupal projects I would love to talk at DrupalCon Amsterdam.

I've done a lot of work to make managing Drupal 7 core, sites, modules, themes, etc. with Composer more of a reality.

I based a lot of my thinking on https://github.com/tstoeckler/drupal-core . Which seems to be the best way to manage Driupal 8 in composer, however there were some challenges to getting this to work in Drupal 7.

First thing I needed was the ability to install dependencies in their proper location. I know that Composer Installers will do this, however, I needed it to work with Drupal Sites. I wondered if it would be better to just have a custom type, but that was swiftly rejected. Because of this I created Composer Custom Installer which allows you to install anything, anywhere (based on the "type").

I set out on a quest to alter Drupal 7's module/theme/site path. It doesn't seem like you can do this in Drupal since the path is hard-coded. Sadly, you can do this with WordPress, and it's rather easy to setup WordPress to use composer.

I then took a look at Drupal Tangler, but I found that it made too many assumptions. Because of this, I created Drupal Structure, which makes fewer assumptions and provides some configuration options.

I hope this helps! I was able to create a Drupal 7 Platform for all of my sites and manage everything with composer:
https://github.com/davidbarratt/drupal7/blob/master/composer.json

Great issue. I'm a little bit overwhelmed right now, and are not sure where the current progress is and where to start from here.

The Slides by kasperg from DrupalHagen provide a great overview. @kasperg, Is there already any progress you are using in daily development?

In my first tests, from what I can see http://static.drupal-packagist.org/v0.2.0/ looks broken right now, or am I missing something? The sources on http://drupal-packages.kasper.reload.dk/ look outdated.

We are sitting in a BoF on Drupalcamp Berlin and just created the drupal-composer Organisation for joining forces. Our first goal would be to have a super awesome drupal packagist server: https://github.com/drupal-composer/drupal-packagist/issues/1

@derhasi: No - we are not using Composer for daily development yet. My presentation from Drupalhagen and the drupal-composer-project are results of a process to convince myself - and others - that is it feasiable.

I think it makes a lot of sense to join forces. Current information is very sparse and efforts to improve the situation are often on a one-off basis. How about creating a Composer group on groups.drupal.org? GitHub is great for code but perhaps not as good for ongoing communication.

Cross-posting #2350639: Rewrite 8.x-1.x, as this seems like it could kill Composer Manager for D8 (which is good).

@kasperg, yes you might be right. So I created https://groups.drupal.org/composer just now. Anybody is welcome to join the group. I simply created it, as from my perspecitve there is a need of joining efforts and sharing ressources.

webflo did some experiments with the original packagist code (see https://github.com/webflo/packagist on top of https://github.com/winmillwill/packagist).

The packagist fork from winmillwill is very good. I deployed it with a few modifications to http://drupal-packagist.webflo.io/. The repository contains over 8800 modules and themes Drupal 7 and 8 and is continuously updated.

@webflo,

Is your fork somewhere on GitHub?

Thanks!

@davidwbarratt. Yes https://github.com/webflo/packagist

I'm currently testing webflo's packagist. It works out good so far with some hiccups. Besides a custom installer this seems to be quite a good starting point for more heavy testing.

I'm currently not sure about the state of drupal-packagist.org. Does it belong to winmillwill (whois information is projected by whoisguard) and does it run on winmillwill's code?

I added "drupal/flexslider": "7.2.0-alpha3", to my require in composer.

I get this message:

drupal/flexslider 7.2.0-alpha3 requires drupal/libraries >=7.2.x-dev -> no matching package found.

There seems to be somthing wrong with the libraries module. I added "drupal/libraries": "7.2.2", but I still get the error

- drupal/masonry 7.1.1 requires drupal/libraries >=7.2.x-dev -> no matching package found.

You have to add "minimum-stability": "dev" to your composer.json, as you must allow composer to download dev releases. Here's a minimal example for that https://gist.github.com/derhasi/a74a1998e5cfa65f105d without a custom installer. With custom installer you can have a look at https://github.com/derhasi/drupal-composer-project/blob/master/composer.... .

For issues on webflo's packagist, you can directly post to https://github.com/webflo/packagist/issues, so we can have an organized view on separate issues without blowing up this kind-of-meta-issue.

Thx derhasi, I did use minimum stability before, but then I got all alpha releases. "prefer-stable": true solves this.

I need a package that is missing. Is there a guide of some kind to contribute to add this package to packagist ?

Also another package has a broken source link. Can I update it in some way ?

@eMuse_be, simply post an issue to https://github.com/webflo/packagist/issues for each problem. Currently there might be some hiccups in retrieving project data, so any feedback in that issue queue will help to sort those out. I will have a look there too.

@derhasi That's clear, I will post there. Some info about contributing here (or somewhere) would be nice though :)

In order to feed your Packagist with almost all modules you could use the following Google Search query which should display pretty much all available modules in Drupal.org:

https://www.google.com.br/webhp?sourceid=chrome-instant&ion=1&espv=2&ie=...

To help parsing these results you can also use Google Search API which can respond in JSON format:
https://developers.google.com/custom-search/json-api/v1/using_rest

Good luck and awesome idea of yous!
By the way, how could I help you with this mass importation??

We shouldn't use something as fickle as Google to determine the full list of modules. We can just ask someone who can query the drupal.org database to provide us with an up-to-date list.

For some inspiration see the Gotta Download Them All sandbox. This was made by Webchick and Greggles to allow people to download all modules. It contains a list of all modules (but this is 2 years out of date) and the readme has instructions on how to update this list.

Used https://www.drupal.org/project/project_module/index and releases.tsv for the initial import.

Ohh that's nice, I didn't know we had that! It even has a filter and everything!

I've been meaning to get my D8 modules on Packagist, and succeeded in getting Honeypot there, but am having trouble with getting Simple Mail and Wysiwyg Linebreaks (I'm getting the error "The package name was not found in the composer.json, make sure there is a name present.").

See:

• #2357499: Add a composer.json file
• #2420357: Add composer.json and put module on Packagist
• #2420355: Add composer.json and put module on Packagist

I haven't caught up with the rest of the comments in this issue, but is what's being proposed a separate Packagist-style repository just for Drupal modules?

Make sure you have a branch named master with a composer.json file containing the package name.

@Xano - in Honeypot, the master branch doesn't contain a composer.json. And if I do a git clone on any of the three projects, it clones the default branch (8.x-1.x in all three cases), which contains the composer.json. Do you know if Packagist has changed to require it in the master branch or something? It seems like it used to work, at least back when I added Honeypot :(

@pfrenssen I suggested that approach because I think it's better to have a great amount of projects instead of just a few, even though it's not a full list. But I'm more then happy to have commented that out because you came with a better approach and then @webflo presented us with this amazing list I have never heard of!! And it's really up-to-date because a colleague of mine has published a new module just a few hours ago and it's there! :D Awesome!

@geerlingguy: I don't know if it changed, but current behavior is that it requires a master branch with a basic composer file that at the very least contains the package name. I do not believe anything else is required. Every branch can then have its own composer file that is used for requirements and metadata.

I added a summary to this issue, so it get's clear that there already is a Drupal Packagist instance available that holds all Drupal projects.

@geerlingguy, IMHO, I would not publish Drupal modules to packagist.org right now. If we really want to use Composer for Drupal it does not make much sense to spam packagist with all projects from drupal.org. One reason is, that drupal.org currently does not support webhooks or similar to notify packagist of changes. In addition we should still figure out a naming convention for Projects form drupal.org: see https://www.drupal.org/node/2401519 for that.

I was just looking around and found:

http://drupal-packagist.webflo.io/explore/ <-- broken link

[Not sure if this is a correct place to post comment.]

@no_angel, currently the site is running again. We still try to figure out why that happened.

It would be best to post issues to the github projects: either drupal-parse-composer (for wrong/missing packages) or drupal-packagist (for general tasks).

I think we could mark this issue as fixed or move it to the infrastructure structure queue.

drupal.org is working on hosting an official packagist server. i think this issue can be closed imo.

FYI, the best composer+drupal docs that i know of are at https://bojanz.wordpress.com/

The related meta issue is #2551607: [meta] Path forward for Composer support.