Problem/Motivation

Five years ago some of us bravely put forth adding metadata files to Drupal modules in #80952: Add .info files to modules to store meta information. Today, tons of problems are trying to be solved by .info files in a Drupal specific way. It'd would hopefully lower a lot of core developer workload and provide a better solution if we could consider an external standard like composer.json

Proposed resolution

With Drupal 8's growing use of NIH solutions I thought I should propose another solution for this problem developed by the php community, the composer.json file. It has places for I believe all of the features requested to be added to .info files, appears to support all the fields we add as part of our packaging, has some nice features like PSR-0 that could help solve other problems we haven't encountered yet, and has a nice "extra" field we could stash our custom things like CSS files in themes, and best of all is not a Drupalism!

Key ideas:
Composer.json is a format outlined in this schema:
https://github.com/composer/composer/blob/master/res/composer-schema.json

Composer is also a PHP PSR-0 application for resolving dependencies generally used as a phar archive:
https://github.com/composer/composer

Packagist is a solr + symfony application serving http://packagist.org/ much like our project.module
https://github.com/composer/packagist

Pluses

  • Its could mean engadgement of more people in the php community outside of d.o(adding drupal support for you code might be as easy as droping a line in your composer.json)
  • NIH
  • Composer is a very featureful package management system based on npm and suse
  • Tests we don't have to maintain! http://travis-ci.org/#!/composer/composer
  • Likely better drush support through things like the "repositories" entry and being able to reference libraries outside the drupal world for downloading.
  • It would deprecate, or at least significantly improve the maintainability of project.module.
  • A lot of people know json

Minuses

  • The composer resolver does contain a lot of code
  • Its JSON and CMI at this point has chosen to go with XML. Configuration is not definition though so this should be a minimal overhead. drupal.org could also probably automate a lot of the management if we're smart.
  • JSON has no commenting like

    , // or /* */.

  • JSON is very strict?(maybe a good thing?)

Related Issues

Remaining tasks

Code review of composer.json

User interface changes

Not really a UI thing other then implementing new features we'll be able to support.

API changes

How a .info file would be translated to a composer.json file:

name = Really Neat Widget
Composer's name is the canonical name of the package. This means that we move the package definition in for it, prefixed by drupal/. The human-readable name is moved to title, as sun mentions below:
"name": "drupal/reallyneatwidget",
"title": "Really Neat Widget"
description = Provides a really neat widget for your site's sidebar.
"description": "Provides a really neat widget for your site's sidebar."
core = 7.x
This is moved to "require". See note about dependencies below...
"require": {
  "drupal/drupal": "7.*"
}
stylesheets[all][] = node.css
composer.json files can hold "extra" meta-information, which would be used for stylesheets.
"extra": {
  "drupal": {
    "stylesheets": {
      "all": ["node.css"]
    }
  }
}
scripts[] = somescript.js
"extra": {
  "drupal": {
    "scripts": ["somescript.js"]
  }
}
files[] = example.test
Since #1541674: Remove the registry, we no longer need to define files[]. Simply removing the entry will fix it for a composer.json. If you're looking to do this in a Drupal 7 composer.json file though, Composer supports autoloading straight up files:
{
  "autoload": {
    "files": ["example.test"]
  }
}
dependencies[] = taxonomy
dependencies[] = exampleapi (>7.x-1.5)
Dependencies are moved over to "require".
"require": {
  "drupal/taxonomy": "7.*",
  "drupal/exampleapi": ">1.5"
}

Having Drupal packages declare their Drupal-version dependency within the version number impedes our ability to move fully towards this architecture. #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) would allow this. One intermediate solution around this would be to use the "extra" options for Drupal-specific dependencies:

"extra": {
  "drupal": {
    "dependencies": {
      "taxonomy": "7.*",
      "exampleapi": ">7.x-1.5"
    }
  }
}
package = Views
"extra": {
  "drupal": {
    "package": "Views"
  }
}
php = 5.3
"require": {
  "php": ">=5.3"
}
version = 7.x-1.5
"version": "1.5"
Due to Drupal's versioning system, we have issues moving to this version architecture. #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) would fix that. An intermediate solution could be...
"version": "1.5",
"extra": {
  "drupal": {
    "version": "7.x-1.5"
  }
}
configure = admin/config/content/example
"extra": {
  "drupal": {
    "configure": "admin/config/content/example"
  }
}
required = TRUE
"extra": {
  "drupal": {
    "required": true
  }
}
hidden = TRUE
"extra": {
  "drupal": {
    "hidden": true
  }
}
project = "views"
"extra": {
  "drupal": {
    "project": "views"
  }
}

As you can see, most of it would be moved to the "extra" information of composer.json. One of the issues is the Drupal version being part of the packaged version information. #1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) and #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x would help, but there are ways we can get around it in the mean time, as I outlined above. Here's an example of system.info as a composer.json file:

{
  "name": "drupal/system",
  "title": "System",
  "type": "drupal-module",
  "description": "Handles general site configuration for administrators.",
  "extra": {
    "drupal": {
      "package": "Core",
      "version": "VERSION",
      "required": true,
      "configure": "admin/config/system"
    }
  }
}
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Crell’s picture

Another bonus would be thinks like "recommends" and "suggests" in addition to "depends". What we do with that is an open question, but it's already there in the format waiting for us to leverage, and we didn't have to do any work for it. I like that.

msonnabaum’s picture

I really like this idea.

Composer's json file is saner than pear's package.xml and it's much closer to what we've already done in .info files. We could leverage this in drush to improve our package management and it could potentially allow us to shed a lot of code.

Also, I don't see JSON as being in conflict with CMI's XML. Choosing a standard format and packaging system is unrelated IMO.

joachim’s picture

Downside though: JSON is harder to read with the naked eye; also it doesn't allow comments in its original spec at least.

amateescu’s picture

I don't find this json harder to read than a .info file. True about comments though...

https://github.com/symfony/symfony/blob/master/src/Symfony/Component/Bro...

Another advantage is that IDE's add pretty colors to the mix, which we currently don't have for our .info files.

In other words, big +1 :)

joachim’s picture

In a project where I used JSON for something I effectively extended the standard by simply stripping out lines starting with '//' before passing it to drupal_parse_json(). We could consider doing that here.

> I don't find this json harder to read than a .info file.

The quote marks and the { } make it noisier, I find. It adds a little bit of extra programmy-scariness for non-developers.

nod_’s picture

One could ask why a non-developer would need to open a .info file.

I'm all for it, very nice feature.

joachim’s picture

> One could ask why a non-developer would need to open a .info file.

Themers need to set regions.
We should consider getting some themers to look at this proposed change.

joachim’s picture

Issue summary: View changes

Add some related issues

msonnabaum’s picture

Everyone, this is not a thread about file formats. It is a thread about packaging and the benefits of leveraging existing packaging systems.

neclimdul’s picture

Comments is an interesting observation but I personally think the schema is very self documenting. We can not under any circumstances put random bits that need to be stripped before processing. This would be yet another drupalism that would completely defeat the point of using a standard. Even so, I've updated minuses in the summary to note the lack of comments since its technically true.

I don't disagree that we should have lots of people review this including themers, we shouldn't push a format on the community without review. I think the usability is vastly overstated though.

Consider that almost everywhere else on the web these days "non-developers" are using js if not json itself. A themer that hasn't done at least a little jQuery or javascript is probably going to be struggling with Drupal anyways. Also, you can easily google and find information about both json and composer.json that's not drupal specific. The community of knowledge available to new users will be larger since its not just the Drupal community, its also the web.

The INI format on the other hand is mostly only used for ancient unix config files and weird bits of Windows normal people don't touch. But if we where really using the format we could maybe have syntax highlighting but we don't even use the actual ini format so normal parsers can't parse it. Its weird fragile Drupal magic.

So for non-developers(and developers) you're getting syntax highlighting and checking in your editor, increased documentation through the broader community of the web, and just a general empowerment to use the tools of their trade. I personally believe we commonly underestimate the common designer and themer(at least the one's I've met) and over complicate their experience by building "easier" drupalisms to support them. I think this issue would fix one of those cases.

A final important point, if I can be so bold as to lead this discussion a little, lets not start discussing the ease of other formats. CMI bikesheded that for months and you're welcome to read it. The real benefit is as @msonnabaum noted that this is a larger php community format with a package standards that will empower the more common user of the format, developers. If the format is acceptably usable for the community, the real discussion should be the merits of the package management.

joachim’s picture

> Comments is an interesting observation but I personally think the schema is very self documenting.

; Information added by drupal.org packaging script on 2011-04-28
neclimdul’s picture

> Comments is an interesting observation but I personally think the schema is very self documenting.
; Information added by drupal.org packaging script on 2011-04-28

https://github.com/composer/composer/blob/master/doc/composer-schema.jso...

neclimdul’s picture

Issue summary: View changes

Note JSON's lack of commenting.

neclimdul’s picture

Updated summary with some possible pain points brought up in #drupal-contribute discussion.

neclimdul’s picture

Issue summary: View changes

Document some possible pain points brought up in #drupal-contribute discussion

neclimdul’s picture

Issue summary: View changes

use the correct related module issue.

pounard’s picture

We could support both .info and composer.json at first, if we choose to go this way: it's doable I guess since in the end, the module ends up as a single line in system table. The only remaining problem would be to chose either one of the formats for the serialized info field, or drop it definitely since it's memory waste to load it for anything else than pure module/package/dependency management.

sun’s picture

Component: system.module » base system

Now.

If you'd replace that JSON format with YAML, then that would actually make sense.

The YAML spec natively allows for comments and also supports complex data types (arrays ["sequences"], and hashmaps (associative arrays) ["mappings"]).

There are plenty of YAML libraries for PHP, but also for other languages (as partially listed on its homepage).

There's a PECL extension, which means you could perform the parsing (and also writing) in C code.

A short introduction to YAML can be found here (coincidentally by the author of Symfony). It basically has all the pros and none of the cons.

Now, that would make a lot more sense.

(I don't really care for Composer / packagist.org -- not even the OP clarifies why that is relevant in any way in the first place.)

The only downside I can see is that all of the user-space libraries suffer from the PCB syndrome (including Symfony's, which scores and possibly wins among all with ~45 KB in total, which is inane).

To put things into perspective, that PCB syndrome is way worse than the NIH syndrome. I'm a pragmatic realist, and so contrary to others, and despite all of the PSR-0/framework-ification/etc efforts, I do neither foresee nor expect Drupal to load less code anytime soon (rather more and more). Apparently, the sheer amount of code we're loading is one of the topmost performance problems we have.

Thus, even if the proposal was YAML, this is a very tough decision to make.

In total, however, count me -1 on JSON (and composer.json). If we replace the .info format with something else, then my new benchmark is YAML.

Crell’s picture

The OP has a lengthy list of the pros of moving to composer.

If you mean Packgist in particular, I don't anticipate Drupal code living there as a matter of course. However, it does mean we could run the same code base as Packagist, or slight variation thereof, rather than a custom (Drupal-proprietary) metadata file, server-side system, and XML-based format.

As a side benefit, it means pulling in additional 3rd party libraries as a contrib module dependency are using the same system we are.

JSON vs. YAML vs. XML is not the key issue. They key issue is less Drupal-proprietary code that we have to maintain ourselves, and that no existing developers outside Drupal comprehend.

msonnabaum’s picture

I dont get why file formats are being discussed.

We're talking about leveraging composer, a package and dependency manager. We can argue that composer is not a good fit, but it's insane to discount everything composer buys us because the file format is json.

Composer overlaps with .info, project module, and even drush make. It's not just a different file format, it's a huge improvement in how we manage and think about dependencies. THis would make it trivial for a contrib project to declare a dependency on an external library and provide autoload information for it.

Composer is clearly influenced by node's npm and ruby's Budler, which I find to be a pleasure to use whenever I work in node or ruby. Both those systems make coming back to Drupal/php feel like we're years behind in how we share and leverage code outside the project.

sun’s picture

Sorry, this apparently wasn't properly communicated from an earlier IRC discussion into this issue:

The file format is an important issue, because there are things like especially theme .info files, which are authored and edited by people, who may not be technically versed.

It is in our best interest to have and use a format that is "forgiving" about minor inaccuracies. JSON apparently is the exact opposite of that; it's ridiculously rigid, and comes with a plethora of special cases, which not even most technically versed developers understand.

In addition, it is very common to document special or otherwise noteworthy conditions in an extension's definition through comments, especially in contrib.

Composer appears to only support JSON, so the format implicitly becomes a central issue.

msonnabaum’s picture

I understand your point, I'm just saying we should discuss this without worrying about the file format until we have a better idea of how composer could be used within the project. Once we do, we can weigh all the potential benefits against the file format issue.

joachim’s picture

> They key issue is less Drupal-proprietary code that we have to maintain ourselves, and that no existing developers outside Drupal comprehend.

I'm sure that's a goal we're all behind.
But in the range of ungrokable concepts and unmaintainable code, info files and their parsing is surely pretty low down. Let's keep a sense of perspective here.

And at least from the user point of view, YAML looks much better. Let's not forget the primary audience of code is humans.

Seldaek’s picture

Heya. Being one of the devs of Composer and Packagist I'll try to address some of the concerns.

First of all if you are not too familiar with Composer please take some time and read my two blog posts, we are still missing docs but that hopefully enlightens you a bit.

As noted in #17, the main benefit I see to something like Drupal using Composer, beyond the fact you'll have less code to maintain, is that your modules (or any contrib stuff) can start depending on third party library transparently, without having to bundle them or anything. That means "reconciling" the Drupal and PHP communities to me, which sounds great.

Now the way I see it, a project like Drupal would typically embed/wrap Composer as a library rather than use it on the command line, because your users are used to interact with that stuff in a web interface, and I assume many also don't have CLI access anyway. What this means is you are much more flexible and you can extend/override stuff from Composer. Should the file format really be considered a major issue, you can add another loader that reads Yaml, .info files or even from the DB, and let Composer work with that. That said, I would really advise against it because I think having one file format for packages is just much more straightforward (everyone knows it and easily spots a composer.json) than if there are many formats around. That is the main reason we only support JSON. It makes integrations quite easy, for example knpbundles.com listing Symfony2 Bundles started recognizing composer.json and linking back to Packagist automatically. If they had to check for arbitrary files it would make it more complicated.

I hope this helps move the discussion forward.

P.S. @neclimdul: Composer is not -as is written in the proposal above- a Symfony application. It uses Symony's Console component to provide CLI interaction, but that's about it.

Crell’s picture

Just so people don't take it as a given that YAML > JSON: I actually detest YAML, and don't consider it more readable at all. I also rather like XML, as long as it's a well designed schema. Let's not pretend that there's a universal consensus that JSON is bad. There isn't.

pounard’s picture

Agree, and Seldaek is right about one particular detail: using composer.json would allow us to handle third party dependencies; Even thought I'm still unsure it would reconcile us with the PHP community: Drupal is too far away from it since too long, it would need more than just that.

neclimdul’s picture

Component: base system » system.module

@seldaek Oh my bad on the Symfony's app thing. That's what I'd heard and I should have looked into it more. Thanks for all the other bits and I agree about your points about the format.

I think there are at least 2 other strikes against YAML though other then not being the standard.
1) YAML is really really slow[1]
2) We can't really count on the PECL module being installed and we already require JSON support[2]

heyrocker also noted in his CMI review that its fragile and easy to break which isn't really a selling point either.

neclimdul’s picture

So we're really hung up on this format thing but I think we're missing the point. First off, these files are specifically for driving code. We're building module pages, class autloaders and project hosting solutions. This is specifically /for/ developers. Themers are just going to copy and paste and change a couple of lines 99% of the time. Heck, I have always done that with .info files and I helped come up with the original standard! Whether its Drupal info, YAML, XML or JSON, that is going to be the case.

So, can we discussion the merits of sharing a package standard with the rest of the world?

I've got this dream that made me post this issue. In it, we don't need a "module" as we see it now for PHP libraries using composer.json because we just require composer.json and it registers the namespace with our autoloader. In my dream I can point Drush at a composer.json file on the web in any repository and it can install it for my site. Or it can even download bundled parts of a module hosted externally for licensing reasons.

I'd also like to not have to design, implement and support all those features in the OP when someone already did plus more. This is not NIH or PCB this is empowering Drupal to do more with less resources and make use of an even larger community. I also have a dream where I can walk into my local PHP users group and talk package management without them looking like I'm an idiot. That might be a long shot but this is my dream. Can we discuss that dream already?

Crell’s picture

My general feeling is that we need to focus on our core competency: Building a first-class content management framework that kicks ass. Server-managed package management is not part of that core competency. We struggle to get people to work on infrastructure tasks, in part because a lot of it is very specific Drupal-proprietary code. That seems a perfectly good thing for us to outsource and focus our ever-limited engineering resources on our core compentency, making the Drupal framlication rock.

Standard or emerging-standard tools that let us do that (and thus maintain less code ourselves) are therefore a good thing by default, in my view. Obviously the devil is in the details of a particular implementation, but in general I support "do less, focus more".

sun’s picture

Well. All of that being said, I'm sure there are almost zero technical barriers for making drupal.org additionally produce a composer.json file in tarballs, or however else you imagine composer.json files to work within the infrastructure.

What I mean is that there's nothing preventing you from introducing composer.json support on various levels; be it pre-declared/manually-authored or automatically generated.

If that happens to help some new-born packaging approach for PHP, then I don't see why we shouldn't be able to expose corresponding files/data at the right locations.

Aside from that, however, composer.json is merely a definition schema and format. It's not like Drupal wouldn't have some of the concepts already. And it's also not like there wouldn't be already existing efforts to enhance the concepts. And lastly, it's also not like the concepts used by Composer are entirely new blasphemy; most of our existing efforts are oriented on the *nix packaging patterns already, and so is Composer.

What I mean is that the main logic for the definition still needs to be implemented in the application layer. Changing the definition on its own does not change anything at that layer. So while having a definition schema is a good idea (but nothing new), the actual challenges for Drupal are in the application layer's implementation.

And to continue on that, it's not like we wouldn't have had all of the ideas for the application layer in the past already. Fact is that most issues are stalled, blocked, and depending on "hard-core" people working on them. Assuming that a definition file/format change would automatically resolve all of those issues is a bit naive. ;) Even with the envisioned WSCCI/Framework initiative efforts and changes, those systematic challenges will remain the same.

Closely related to that, there are various claims in here that this change would "ease" the handling of external library dependencies. While I could see that to be the case and nicely work out for custom PHP frameworks à la Symfony & Co, I refuse and totally reject that argument for a modular system like Drupal. If it was that easy in Drupal, the Libraries API project wouldn't exist at all, and on top of that, Libraries API wouldn't still struggle hard to find answers for seemingly unsolvable problems. (Reality shows, only few people understand what 4-Dimensional actually means, and even those that do don't have any answers.)

joachim’s picture

I must confess I did make the mistake of thinking this was just about the format... :/

But I've read both the blog posts linked in #21 and read them again I still don't feel terribly enlightened as to how this actually would work.

As in, suppose module A declares module B as a dependency and module C as an 'enhancement' and module D as a 'conflict'. How does Composer make that happen? Surely we still need stuff in system.module to enforce the dependency on the modules page and flag up the problems etc etc. I'm not understanding how much of the work Compose would do for us, and indeed *how* it would do that work. For isntance, it sounds to me like we'd need to have a package repository it can talk to on d.org.

Also, if the format is swappable, here's a radical suggestion: why don't we keep our existing format? Then the net change to DX is zero. We're all about how hiding the internals means we can change them at will without breaking APIs: this would be changing the internals without breaking the API to humans. Also, the code for parsing info files is about a dozen lines.

(And finally: what on earth does PCB stand for? Yes, I googled it.)

neclimdul’s picture

So, great question on how composer would make that happen. I'll look into it.

The code for parsing info files is in fact not a dozen lines. We've got this goofy brittle regex thing going on which is(apart from the constant magic) the equivalent of just json_decode.

re PCB: I couldn't figure out why we where talking about printed circuit boards either but apparently its sun's acronym for "Poor Code Bloat."

RobLoach’s picture

FileSize
108.16 KB

I've put together the beginnings of a Drush wrapper for Composer over at #1419058: Switch scope of the Drupal Composer namespace. Might help people understand Composer a bit more. Screenshot attached.

Also opened up a Drupal fork over at GitHub with composer.json used to download jQuery UI for some experimental funtertainment. Of course, we'd have to define our own Composer repositories and such, but the idea is there.

pounard’s picture

Hehe, it seems actually really nice! It would make life of people that deploy Drupal into a closed network environment really hard thought, but it sounds appealing still!

RobLoach’s picture

It would make life of people that deploy Drupal into a closed network environment really hard thought, but it sounds appealing still!

You can host your own private repositories with Composer. You can even remove the Packagist one if you really want to.

Damien Tournoud’s picture

Some clarifications are probably in order, because this thread seems to imply that our .info files can be directly replaced by composer.json. They cannot, as they both have slightly different (and partially overlapping) purposes:

  • composer.json is a packaging / archive metadata. Its purpose is to tell you how to get a particular package (and in addition which other packages you might need, etc.). It covers the same ground as packages metadata in OS distributions (.deb, .rpm, etc.), and to CommonJS Packages specification. The equivalent in Drupal is information stored on Drupal.org in the Project and Project releases nodes that are exposed to the world in the update status feed.
  • Our .info file is a component-level metadata. Its purpose is to tell your how to use a particular component, and especially what you can find in it (introspection). It covers roughly the same ground as the config/environment.rb, CommonJS Modules, etc.

(Note that Symfony doesn't have any introspection capability. PSR-0 either.)

We tried at one point to *also* use .info files as a packaging metadata, and it became the untangled mess it is right now: modules can (but are not forced to) declare belong to a "project" and a "project status url"; one module of a project is going to be elected as the "main" module, in a way that is undefined, etc. (just look at the code of the update module or of Drush PM to get convinced).

So, composer.json as a clean packaging metadata, yes. composer.json as a complete replacement for our info files, I don't think so.

sun’s picture

#33 expresses exactly what I had in mind since this issue's inception, thankfully better articulated than I could ever have.

pounard’s picture

I agree that .info files don't need to be replaced if it's only in order to keep iso-functionallity, but if it happens that we need more features, it might worth it to at least study potential replacements such as composer.json. There is good reasons why we may want it, one I can see right now is that we could reproduce the download and make features of Drush without Drush.

cosmicdreams’s picture

Great stuff gang,

How could this help the use case of finding out when my modules need to be updated or installing new modules straight from Drupal's admin ui. Would composer.json be able to handle the problemset of discovering new modules to install?

neclimdul’s picture

So, I guess not surprisingly I disagree with Damien. I think the core of it is I see the system table as tell us how we use components. In random places in the Drupal interface we act like a package manager and read from those files and populate the system table building a working environment but the files themselves only tell us information about the component. Any where else in the world this would be called what it is, package management. I think this is demonstrated even more clearly by the fact that the vast majority of the fields in .info files are package related(name, description, version, grouping, dependencies, php compatibility, etc). info files map directly to composer.json.

Info files where always setup in this manner and the things that are less package management related are actually late arrivals to the format. The fact that things are spread out in the update feed and that the project url thing is goofy and underused has nothing to do with them not being a package format and everything to do with not being able to commit large changes to core/growing organically and us not being very good at it(dww, you're awesome and this is not a crack at you). This alone should be a reason for us to consider using composer.json actually.

Looking at the few components of .info files that don't map directly they are mostly our Drupalism asset registration systems and maybe theme regions. I'm not sure how these are introspective but I do think they can fit comfortably in composer.json if we think about our metadata files as dynamically building the application, not actually running it.

  1. Class files. Our current autoloading system is Weird. PSR-0 is better for the ease of registration, is consistent with PHP land, is already being adopted into d8 and is built into composer.json so that's the easy one.
  2. CSS and JSS files. So, this is a clever system and useful especially when your thinking in the building the application sense. I think adding an 'asset' field in the 'extra' section of composer.json seems like a perfect place for this sort of information and the module page(or drush or whatever) can "install" the files into the theme systems "put these assets on pages" system. We could probably even work out a standard with the composer people for conditional information and cool stuff.
  3. Theme regions. OK I'm not actually sure on this one but we could probably put it in the extra field too. I like PE/Panels's layout system better and I'd like to see blocks in D8 work more like that anyways but that's to be seen.

I've got some ideas on how we can address some other disconnects between drupal and composer to come.

pounard’s picture

neclimdul++

Crell’s picture

neclimdul: This would actually be a good Core Conversation proposal, methinks. :-) http://denver2012.drupal.org/core-conversations

Crell’s picture

Issue summary: View changes

note module per directory policy issue

neclimdul’s picture

So the novel solution to address some of the minuses didn't pan out so the list hasn't changed.

Would composer.json be able to handle the problemset of discovering new modules to install?

So, discovering new modules on the file system this would be mostly the same minus the module per directory requirement. Discovering new modules through the UI this would also work mostly the same but we would have the additional keys like keywords, suggests, and recommend to empower the module redesign people.

However discovering packages available for download(and are not in the suggests and recommended category) that would be out of the immediate scope of converting to the file because it would require non-composer code inside of Drupal for querying that system. The two options I would see though are:
1) We use d.o's project functionality mostly as it stands now without any real composery/packagist stuff and just have composer do its thing out of git like RobLoach is playing with now.
2) We could look at how Packagist is setup and move to something similar. This enters us into a community of package hosting instead of a sort of walled garden. This would also likely lower the bar for your own module hosting solution not on d.o for better or worse(a CC theme site?)

RobLoach’s picture

Not sure how Drupal could use Composer for module/theme deployment, but Drupal Core itself could use Composer for its third party vendors: #1424924: Use Composer for updating Symfony components (without removing Symfony code from repo). This makes upgrading and management of third party dependencies much easier than managing them all ourselves.

RobLoach’s picture

Status: Active » Postponed

Not sure Composer makes sense for the module/theme space. It might make sense for third party libraries though: #1424924: Use Composer for updating Symfony components (without removing Symfony code from repo). I'm going to set this to postponed for now, maybe even won't fix. Let's keep the discussion open at #1424924: Use Composer for updating Symfony components (without removing Symfony code from repo) though.

patcon’s picture

Issue tags: +Composer

tagging

gagarine’s picture

I agree with #33 .info and composer.json files have different targets.

We should use composer to manage dependency without trying to use the tools for what it was not made for.

One composer.json file (optional, themes do not have always dependencies) and a .info (or a better format if we find one) can certainly leave together.

RobLoach’s picture

Still don't see composer.json as a replacement of .info files. But, we now have composer/installers, which is helpful. LESS, Symfony and Zend Framework are a few examples of Drupal modules that support installation via Composer.

#1612910: [policy, no patch] Switch to Semantic Versioning for Drupal contrib extensions (modules, themes, etc) would allow versioning to work properly with Composer.

a.ross’s picture

This issue may be interesting in the context of the discussion here as well: #1545684: Separate version data from the .info file

gagarine’s picture

Title: Consider composer.json as a replacement for .info files » Consider composer.json to manage dependencies instead of .info files

composer.json is made to manage dependencies and package info but not stuff like theme regions. Let's make it clear in the title.

This is what it looks like

{
    "name": "monolog/monolog",
    "type": "library",
    "description": "Logging for PHP 5.3",
    "keywords": ["log","logging"],
    "homepage": "http://github.com/Seldaek/monolog",
    "license": "MIT",
    "authors": [
        {
            "name": "Jordi Boggiano",
            "email": "j.boggiano@seld.be",
            "homepage": "http://seld.be",
            "role": "Developer"
        }
    ],
    "require": {
        "php": ">=5.3.0"
    },
    "autoload": {
        "psr-0": {
            "Monolog": "src"
        }
    }
}
RobLoach’s picture

There's an extra option, which can be used to hold any extraneous information, like theme regions if desired.

gagarine’s picture

This can perhaps be a good things but I have to say I can't think about all the implications of using this extra option.

sun’s picture

Status: Postponed » Active
cweagans’s picture

Related: #1793074: Convert .info files to YAML

EDIT: Whoops, I guess I should refresh before posting a comment.

cweagans’s picture

Issue summary: View changes

Better information about Composer

RobLoach’s picture

RobLoach’s picture

Issue summary: View changes

Moved from #52

RobLoach’s picture

Issue summary: View changes

Updated issue summary.

RobLoach’s picture

Issue summary: View changes

Updated issue summary.

RobLoach’s picture

Issue summary: View changes

fg

sun’s picture

RE: 'title' property:
https://github.com/composer/composer/issues/1140

In a corresponding PR for it, @stof mentions that the Composer schema would already allow for the property, but it simply wouldn't be used (or rather ignored) everywhere.

RobLoach’s picture

Thanks, sun. Updated it accordingly. Following that PR. Will take a look once I get the chance.

dcrocks’s picture

There seems to be a lot going on in drupal 8. Is this going towards a json file for package management and a yaml file for module/theme metadata? Is this 'either/or' or 'and'

patcon’s picture

Looks awesome Rob. Thanks man.

@dcrocks Composer natively expects json, but if we really want, we can write a wrapper around composer and use it as a library to read the same stuff out of a yaml file. I kinda prefer that approach, but any step toward composer is awesome.

There's also this, but it's pretty clunky:
https://github.com/igorw/composer-yaml

podarok’s picture

#52 should be moved into summary

podarok’s picture

Issue summary: View changes

Updated issue summary.

donquixote’s picture

++ Damien in #33.

A module (e.g. devel_node_access) is not a package.
A Drupal "project" (e.g. what you download at drupal.org/project/devel) can be considered a package. Only with a different download mechanism.
composer.json would make sense as a file to put at the root of a *project*.
But we still want something to put at the root of a module folder, e.g. of views_ui. This is currently views_ui.info, and could become views.yml.

So, currently we have:

Drupal package / project: devel, admin_menu, etc.

Drupal module: devel, devel_node_access, etc.
Usually shipped with a Drupal package / project.
The special thing about a Drupal module is it can be enabled or disabled. And it can declare a schema to Drupal. Etc.
Modules can have dependencies to other modules.

Composer package: monolog, guzzle, etc.
You can not disable or enable a composer package. You can just download it, and it is there.

-----------

Module dependencies are not package dependencies.
If devel_node_access depends on devel, this has nothing to do with package dependencies.
it does not mean we want the package be downloaded, but one of the modules within the package be enabled. (and actually we don't care which package that is, so long as it contains a module with that name)

Btw, it does make sense for a module to declare a dependency on a composer package.
It also could make sense for a Drupal "project" to declare a dependency on a composer package. But it is probably better to do that on module level, because then we don't need unnecessary code as long as the module is disabled.

-----------

This issue can still make sense:

1) Look for modules within downloaded composer packages.
E.g. we provide a mechanism to download packages with composer in a dedicated folder, e.g. sites/all/composer/vendor (D7), or just composer/vendor (D8). Then we allow those packages to contain Drupal modules. These need to be declared in the composer.json, so we don't have to do ridiculous file scans.

2) Build on composer as a package manager, and make drupal.org the main repository for that.
That is, every Drupal "project" (e.g. admin_menu) will have a composer.json - but NOT replacing the module info file.

3) Allow modules to depend on composer packages.
This does not need to be in a composer.json, instead it can be in the module.info or module.yml file. Because a composer.json is to describe a package (= sth you can download), whereas the module.info or module.yml describes a module (sth you can enable/disable within Drupal)

Xano’s picture

Bump. I predict a lot of contributed modules for Drupal 8 will make use of composer to include external code, seeing as core already does this.

cweagans’s picture

This issue has been open for around 13 months without a patch. I think that it's safe to say that this is not happening for Drupal 8.

However, I know that this would be useful for some people, so the alternative that I propose is that we allow contrib modules/themes/profiles to define a composer.json file, and for the ones that choose not to, we'll autogenerate a composer.json file as part of the packaging process. This will allow people to manage their modules/themes/etc with Composer, and I don't think there's anything that needs to happen in core in order for that to happen. This differs from the existing proposal in that we wouldn't really care about it in core - it's just a nice thing to have for people to build their sites. In fact, there's no reason that this can't happen right now for Drupal 6 and 7 packages.

What I'm saying is that we can essentially built out the packaging system such that we can use composer.json as a replacement for Drush Make files. This requires no core code changes and is relatively easy to do.

cweagans’s picture

Version: 8.x-dev » 9.x-dev
sun’s picture

Component: system.module » base system
Issue tags: +Extension system
klonos’s picture

@cweagans, #60: Hey Cameron, did you file an issue for that proposal or is there one already filed?

RobLoach’s picture

@cweagans: http://drupal.org/project/composer_manager ... There are a bunch of modules that use a composer.json file already.

cweagans’s picture

@klonos, I have not opened an issue for that.

@Rob, Interesting. I hadn't come across that yet. Looks pretty cool. Correct me if I'm wrong, but wouldn't all modules need to provide a composer.json file to use composer as a replacement for drush make?

patcon’s picture

*Redacted*

RobLoach’s picture

@cweagans Drush Make is rather different than Composer and I don't really see one replacing the other. Although their use cases are similar, there are things that one does that the other should not do.

That said, there are some things we could port into Composer Installers' DrupalInstaller. I just put up a pull request to locate the Drupal root location. This is similar to how you can run drush dl stringoverrides from anywhere and it'll find out where to install String Overrides.

I've updated the Symfony module to use composer/installers. That means you can stick the following in a composer.json anywhere in your Drupal 7 folder, and it'll install the Symfony module to sites/all/modules/symfony:

{
  "require": {
    "drupal/symfony": "7.2.0-alpha5"
  },
  "repositories": [
    {
      "type": "vcs",
      "url": "http://git.drupal.org/project/symfony.git"
    }
  ]
}

Either way, Composer Installers is a much different discussion than using a composer.json in place of .info files in Drupal Core :-) . Get in the Drush Composer issue queue and we'll talk there!

lotyrin’s picture

Re #67:

> Drush Make is rather different than Composer and I don't really see one replacing the other. Although their use cases are similar, there are things that one does that the other should not do.

Could you elaborate?

patcon’s picture

Like Rob said, probably not the place :) This thread pings like a 60 ppl every comment

Can you recommend another thread Rob?

cweagans’s picture

RobLoach’s picture

Status: Active » Postponed
RobLoach’s picture

Issue summary: View changes

gvx

xjm’s picture

Component: base system » extension system
Issue summary: View changes
Issue tags: -Extension system
Mile23’s picture

catch’s picture

Title: Consider composer.json to manage dependencies instead of .info files » composer.json for modules
Version: 9.x-dev » 8.1.x-dev

Some kind of composer.json support can be added/recommended without removing support for .info.yml, so shifting back to a minor release.

timmillwood’s picture

Status: Postponed » Needs work

Even a marginal step toward this would be awesome to see in 8.1.x. If not for a while I think we will see a need for all contrib modules to have a composer.json, therefore I'd like to propose the MVP for this issue is that in 8.1.x is to require a composer.json in a module, much like we require an info.yml

cweagans’s picture

Personally, I'd love to see composer.json files *completely* replace the info.yml files. Obviously, we can't drop support for .info.yml files in 8.x, but we can start discouraging it as much as possible, and make it so that modules can omit it entirely.

I don't think we can require it in 8.x, but we can certainly use it instead of the .info.yml file if it's present.

catch’s picture

Status: Needs work » Postponed

Still postponed until 8.1.x opens up, and yes we can't require a .json since that would be an API change that would break all modules, but it might be possible to something similar to what cweagans said.

Although I'll add that we chose YAML for various things as being easier to humanly read/write than JSON, comment support etc., and there's no mention of that here.

Also we have lots of information in info.yml that has nothing to do with dependencies and packaging - especially for themes.

cweagans’s picture

Although I'll add that we chose YAML for various things as being easier to humanly read/write than JSON, comment support etc., and there's no mention of that here.

I did the conversion patch for switching info files to yaml, but I intentionally didn't mention it here. It's a Drupalism as far as I'm concerned. Everyone and their dog in the PHP world is writing composer.json files and not having a difficult time with it, especially since composer is reasonably good about validating composer.json files and telling you where things are wrong.

That said, I think I recall somebody saying that the composer file format is pluggable somehow, so you could have composer.yml if it's really that big of a concern.

Also we have lots of information in info.yml that has nothing to do with dependencies and packaging - especially for themes.

This information can go into the "extra" section. If we need to require specific things, we can even add the justinrainbow/json-schema package and validate "extra" against a schema (similar to an xsd for xml) to require certain data points.

catch’s picture

Do we really want theme regions in composer.json, even in an 'extra' section? Just seems like a mis-use of the file at that point.

We have services.yml and etc. so I'd more expect it to move to something like regions.yml

A lot of stuff has been dumped into .info over the years, and we should not just push it around into a different format if it doesn't make sense. Main point is this should not necessarily be a 1-1 conversion.

cweagans’s picture

That's fair. regions.yml sounds like a good path forward there, and we could do a similar thing where we look for regions.yml first, and if it's not there, fall back to the .info.yml file region information.

I was thinking that things like the "package" key in a module.info.yml would be a good thing to put in "extra".

In any case, doing this without a BC break is going to be a bit annoying, but it'll pave the way for stripping out the BC features in 9.x.

catch’s picture

In any case, doing this without a BC break is going to be a bit annoying, but it'll pave the way for stripping out the BC features in 9.x.

Yes I moved literally dozens of issues from 9.x to 8.1.x/postponed where the BC layer would be ugly, but feasible.

As much as possible we should be aiming for bc breaks in 9.x to be 'removing backwards compatibility layers', not 'add entirely new thing and entirely remove old thing + convert everything in core'. See #2608062: [META] Requirements for tagging Drupal 9.0.0-alpha1 for a bit more thought on that.

If we commit to making changes in 9.x and removing the BC layers, then even ugly and potentially confusing BC layers become a lot more desirable - because at that point it's as much a forward-compatibility layer with 9.x as a backwards compatibility layer with the previous 8.x minor release. Changes we wouldn't have added to 8.0.x even 6 months ago might make sense again when we get to 8.3.x.

Also regions.yml (and potentially the scripts/libraries stuff in .info.yml) is something we could do independently, before we even introduce composer.json support anywhere.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Mile23’s picture

Status: Postponed » Active
Issue tags: +Needs issue summary update

I think this question of having an official behavior for contrib modules with composer dependencies is a big one and we need to answer it.

Setting back to active.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Saphyel’s picture

Issue summary: View changes
gpap’s picture

Issue summary: View changes

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

markhalliwell’s picture

Title: composer.json for modules » Replace .info.yml with composer.json for extensions
Related issues: +#2958021: Proposal: Composer Support in Core initiative

Was looking for this issue, this is a more apt title for what this issue is proposing.

Mile23’s picture

We'd use composer.json files instead of info.yml files if we were abandoning the current module system for one that uses Composer to install modules.

Extensions can currently have composer.json files which declare dependencies, and these extensions must be installed using Composer.

Here are the docs on how some of this works: https://www.drupal.org/docs/develop/using-composer

markhalliwell’s picture

I know how core and composer both work... I'm not a n00b.

That wasn't the point. I was searching for this issue and "composer.json for modules" does not describe what the majority of this issue has been about, which is in fact about abandoning .info.yml files in favor of using composer.json since the majority of this info is duplicated there.

I really don't have an opinion one way or the other, just felt like the title was too simplistic for a topic that has large consequences.

donquixote’s picture

Was looking for this issue, this is a more apt title for what this issue is proposing.

composer.json for modules » Replace .info.yml with composer.json for extensions

The old issue title was open-ended, which I think was good.
Adding a composer.json does not mean that it will or should replace the *.info.yml file.
See #33 and #58, and possibly other comments.

The "one package, multiple modules" (which is the current reality) is not an accident, but actually makes sense:
Multiple modules (e.g. "submodules") should live in the same package, if they benefit from a shared git repository, a shared release schedule and version history, a shared module page.

A package is what you download, a module is what you enable.
Hence, composer.json and *.info.yml can happily coexist and contain distinct pieces of information for different consumers.
Composer will never look at the *.info.yml, Drupal shall never look at the composer.json.

I see no benefit in mixing this together.

markhalliwell’s picture

Project: Drupal core » Drupal core ideas
Version: 8.6.x-dev »
Component: extension system » Idea

The old issue title was open-ended

Exactly. It didn't mention .info.yml at all. Do you know how many "composer" issues there are...

Adding a composer.json does not mean that it will or should replace the *.info.yml file.

Except that in this proposal/IS, that is exactly what it means.

---

Listen, I'm not saying this is a great idea or exactly how it should be implemented (in stages or as is).

However, I needed to reference this issue to someone and couldn't find it. After a while, I finally found it.

This is a proposal to do exactly what the title now says (according to the IS and my memory from long ago of reading the issue).

If a different idea/proposal/implementation needs to be created (a hybrid if you will), then a new issue should be created.

I'm going to actually move it to the appropriate IQ now that I think about it.

Mile23’s picture

I know how core and composer both work... I'm not a n00b.

This issue is From Before The Beginning Of Time and some parts are relevant and some aren't. I thought I'd contribute a little towards that end for whoever ended up seeing it in their dashboard all the sudden. I didn't mean to be sanctimonious. Sorry I came off that way.

But hey! This is a core idea now... Excellent. That means we can talk about this: #2494073-305: Prevent modules which have unmet Composer dependencies from being installed

This is a 300+ comment thread where we successfully implemented adding a build_dependencies key to .info.yml. This allows us to say that a given extension requires Composer in order to be installed. Since it's an array, we can also specify other build tools. For instance a module could require that you say yarn install or whatever.

This allows us to establish a relationship between the .info.yml file and composer.json. It means that if any module needs a composer-based dependency, we can prevent it from being installed or updated. It also means that the d.o interface for projects can display a 'needs Composer' badge so people know what they're getting into.

I suggest we adopt that key for the betterment of all, keep the .info.yml file, and then redesign everything when we finally figure out how to turn extensions into packages that live in vendor/.

donquixote’s picture

Except that in this proposal/IS, that is exactly what it means.

You are right.
So, I express my -1 for this specific proposal,

If a different idea/proposal/implementation needs to be created (a hybrid if you will), then a new issue should be created.

In practice, there is already a work flow that uses Composer to build Drupal sites and to download contrib modules.
https://www.drupal.org/docs/develop/using-composer/using-composer-to-man...
https://www.drupal.org/project/project_composer
(Most of you know this already, possibly better than me, I just want to put together a short summary)

Modules that do not already have a composer.json are still provided through packages.drupal.org.
I think this mechanism actually works quite well.
Module packages can optionally provide a composer.json, if they have something meaningful to say, but they can also skip it, if it would be redundant.

I suggest we adopt that key for the betterment of all, keep the .info.yml file, and then redesign everything when we finally figure out how to turn extensions into packages that live in vendor/.

Sounds reasonable.

turn extensions into packages that live in vendor/.

I am ok to put extensions into vendor/ one day.
I still think that we should distinguish "something you download" vs "something you enable/install".
Composer itself already blurs this distinction with its autoload and include files, but I still think we can make this distinction.
If we ever drop this distinction, it will indeed be a real full-on redesign, which goes beyond the question of *.info.yml vs composer.json.

osopolar’s picture

@donquixote you said:

Composer will never look at the *.info.yml, Drupal shall never look at the composer.json.

I was surprised when I installed the Bootstrap Paragraphs module with composer (composer require drupal/bootstrap_paragraphs) that also dependencies from bootstrap_paragraphs.info.yml where downloaded, as there is no composer.json file. I didn't find how it works, but composer seems to look at *.info.yml.

Mile23’s picture

@osopolar: That happened because we have the Drupal-specific Composer façade.

Your root composer.json file probably has something like this:

{ 
    "repositories": { 
        "drupal": {
            "type": "composer",
            "url": "https://packages.drupal.org/8" 
        }
    }
}

That tells Composer to use packages.drupal.org to resolve metadata for packages. The facade looks at the info files of projects maintained on drupal.org and then keeps track of their dependencies, and presents them to Composer as a composer.json file through packages.drupal.org.

Some docs to help: https://www.drupal.org/docs/develop/using-composer/using-composer-to-man...

jibran’s picture

Mixologic’s picture

Okay, so a lot of this issue is out of date.

over at #3005229-5: Provide optional support for using composer.json for dependency metadata @donquixote said :

Module info files and module package composer.json files contain different information for different consumers,

Module info files are what is used to *build* packages.drupal.org, and there is a lot of code on d.o. that has to account for both info.yml files and composer.json files. Essentially they are both metadata about modules, with a significant amount of overlap.

drupal.org's project_composer has been responsible for drawing the link between the project (the git repo and its dist releases) and the modules themselves (be they primary, or sub modules). This is the key disconnect between composer and drupal -> namely that multiple dependencies can be embedded in one dist, or within one git repository.

Additionally, there has always been a problematic packaging step where drupal.org modifies the contents of the repository by adding what version a module is into its info.yml file. This breaks sometimes due to modules wanting to use particular commit hashes of a dev release, or when using a git clone of a module repository. Hence we have to use things like 'git_deploy or composer_deploy' to fake those version numbers after the fact.

If we start allowing composer.json's to replace info.yml files, and deprecate those info files for d9, then in d9 we can rely on composer's installed.json file to let us know what versions of the code we have. But we could only rely on module version numbers if the modules were installed with composer. If we continue to support tarball installs of modules, we'll have to have some way of determining the version of those tarballs, and determining if they satisfy the requirements, which that actually seems like its going to take a pretty significant effort to make happen, even before d9 is launched.

So given all of that, I think it'd be nice to start adding composer.json support as an alternative to info.yml files, and only drop info.yml support once we reach the point where the only way to extend the site is using composer (either via an upgraded update manager, or via the command line).

donquixote’s picture

I was going to go on a long rant, but in the end I noticed that perhaps I don't disagree as much as I thought I would :)

I think it is important to conceptually distinguish "module" vs "package".
I also think we should say "A package contains one or more Drupal modules", not "a package IS a Drupal module.".

If we want to express this information as part of the composer.json file, via $json['Drupal']['modules'][$module_name] = ['path' => $module_subdirectory, 'name' => $submodule_label, 'description' => $submodule_descripton, 'dependencies' => [...]], maybe I would find that unpractical or awkward, but I would not be fundamentally opposed from a conceptual point of view.

--------------

About the problems you describe:

Perhaps we should stop thinking of "module version"?
If we are honest, a module version was always tied to the module package, so it was always some kind of a lie.

- A package has a version.
- A package can depend on other packages, with specific version requirement.
- A package can contain modules.
- A module does not have a version.
- A module can have a database schema version, but this cannot be used in dependency declarations.
- A module can depend on other modules, without version requirement.
- A module can depend on packages, with version requirement.
- A module can depend on another module from a specific package with specific version.

Example:
- Package "weather" contains the main module "weather" and submodule "weather map".
- The "weather" package depends on some basic other packages.
- The main "weather" module depends on some modules in these basic packages.
- The "weather map" submodule depends on "acme/maps" package version ~2.5 (which is totally independent of drupal).
- The "weather map" submodule also depends on "sparkly visual effects" module, which can be found in "sparkly visual tools" package. It requires that "sparkly visual tools" is version ~3.1. (*)

(*) Not sure what to do with Drupal version compatibility number. This is a separate question.

To enable the "weather map" submodule, one would need to manually require these 3 packages in the main composer.json:
- "weather" Drupal module package.
- "acme/maps" composer package, version ~2.5.
- "sparkly visual tools" module package, version ~3.1.

These last two packages are only required by the "weather map" submodule, so Composer does not download them by itself. Bummer..
This is indeed a bit unpleasant.

Alternatively, the "weather" Drupal module package could require all the optional packages, so that they get downloaded even if they are not needed.

So why keep this separation?
- The installed/uninstalled status of a module is defined in the database, not by Composer.
- A module package can be downloaded without enabling/installing the module(s).
- A module can be uninstalled, without removing it from the filesystem. In fact, it should only be removed AFTER it was uninstalled. (**)
- It can be useful to have more than one module in a package, to share the git repo, version history, issue queue, module page, README, etc.

(**) Here it gets interesting again. With the model above, Composer does not know which modules are installed/enabled, and which are not.
You can remove a module package from composer.json that is still enabled in the database, and then run composer update to remove it.
There could be two ways to prevent this:
- Let Composer look into the database.
- Produce a file somewhere telling Composer that some module packages are "locked".

Both of these "solutions" are fragile and flawed, and invert the food chain of "Drupal on top of Composer".

The correct solution would be for Drupal to refuse to boot, if required Composer packages are missing. This can be implemented with an initial hash lookup, and a more expensive computation if there is a hash mismatch.

catch’s picture

A lot of this is outdated, but I think we should still look at #78-81 still. i.e. moving theme region support out to a regions.info.yml and similar changes to that, so that there's less in 'extra' in composer.json

That could be split into spin-off issues and worked on in parallel alongside supporting the absolutely necessary stuff between .info.yml and composer.json

AaronMcHale’s picture

This is the first time I've looked at this issue, and after reading lots of it and skimming the rest my brain is a bit fried, but here are my very initial thoughts:

Firstly, this issue is long, was created about 6 years ago, in that time I'd say Drupal has become a lot more Composer friendly and Composer focused, lots of Drupal sites use Composer and there's now a initiative to add better Composer support to Core, and a lot of the early discussion on this issue may no longer be relevant or an accurate representation of current opinion. So given that maybe it would make sense if we closed this issue and opened a new one?

Secondly, a good step initial step forward might be to unify the module.info.yml and theme.info.yml files into a single extension.info.yml file schema. I support the proposal of moving regions to a different file, but if we do go for this unified approach we'd need to find places for the other keys in theme.info.yml. Maybe instead of spinning off a *.regions.yml file we spin off a different file (*.theme.yml ?) which has all of the theme specific keys which currently exist in *.info.yml.

Thirdly, as a developer it slightly annoys me that I have basically duplicate information sitting in the info.yml and composer.json (especially for dependency management), and for custom modules/themes have to make sure those two files are in sync. We can argue back and forward about what the semantics of packages vs projects vs modules are but at the end of the day every module/theme and every sub-module/sub-theme can have a composer.json file, define their own dependencies, and all live happily together in the file system regardless of semantically what we call them. My point here is that these semantics shouldn't prevent us from moving key pieces of information from info.yml to composer.json, because technically speaking there's nothing that would horribly break if we implemented that properly.

jcisio’s picture

Issue summary: View changes
Mile23’s picture

+1 on #102 (and the discussion of, for instance, regions).

composer.json is already super-duper overloaded as it is. It's supposed to be a list of dependencies, and some metadata like where's the github for it.

The basic use case here is that if you make an extension, you'll need the info.yml file to specify what Drupal-ecosystem dependencies are required at install/run time. The only reason you'd absolutely need to have a composer.json file in your extension is if it's contrib, or otherwise reachable by the Drupal Composer facade, AND it has dependencies that don't come from D.O.

Otherwise, dependencies on contrib and Composer-based packages should be in your site root's composer.json, not your extension.

And if we can get #2982680: Add composer-ready project templates to Drupal core in, then that'll be super easy instead of kinda weird.

AaronMcHale’s picture

#105

The only reason you'd absolutely need to have a composer.json file in your extension is if it's contrib, or otherwise reachable by the Drupal Composer facade, AND it has dependencies that don't come from D.O.

Even if an extension has D.o Dependencies they should still be listed in the extensions composer.json, that way when you install the module or theme with composer, composer will automatically install those dependent modules or themes and ensure they stay at compatible versions.

Mile23’s picture

Re #106

that way when you install the module or theme with composer, composer will automatically install those dependent modules or themes and ensure they stay at compatible versions.

Correct. However, not all extensions *can* be installed with Composer.

If I make a module for a specific project and it lives in the codebase (say, under modules/custom/) then no amount of re-typing composer update will add its Composer-based dependencies.

AaronMcHale’s picture

#107:

If I make a module for a specific project and it lives in the codebase (say, under modules/custom/) then no amount of re-typing composer update will add its Composer-based dependencies.

Technically speaking, you could add module/custom as a location under extra > installer-paths, say with the type "type:drupal-module-custom", then specify that type in your module along with your dependencies, then add the Git repo as a repository. Doing that does work and does allow you to manage any custom module dependencies through Composer. Technically speaking it aloo does give you .git folders so you can changes, commit and push, but it's more useful for keeping everything up to date on your testing/staging/live environments.

Of course that solution isn't entirely perfect, there are potential performance impacts when running Composer Updates if you have many Git repos as Composer repos, so it's not necessarily a perfect solution, but none the less is does plug that gap.

As I'm writing this one thought also comes to mind, which would make something like this be more advantageous for local dev, it would be interesting to see if there is a way to make Composer scan for composer.json files in modules/custom and themes/custom, simply to allow those to be used for dependency resolution when running composer update.

gapple’s picture

Regarding custom modules and composer, if the module is unique to that project/site, then the site's composer.json is the correct place to specify any dependencies, and the module itself shouldn't have a composer.json file. Submodules also shouldn't have their own composer.json file, and the root composer.json should include all dependencies of all submodules (this can result in unused third-party dependencies if a site doesn't use the submodule, but there is a way to prevent the third party dependency from being installed with configuration in the site's composer.json).

If the module is / should be reusable, then there are other options.

For one past project, I was creating a library that was potentially going to be published later, so I wanted to keep it independent. The "path" repository type helped here, pointing to each library in a root /src/ directory separate from the rest of the project code. When composer installs dependencies, it just creates a symlink in the necessary location to the source dir. When the library was later published, I could change the repository to point to a remote git repository, or just remove it if it's available through packagist. (Unfortunately PHPStorm is a little confused by the symlink this setup, and complains about duplicate code).

"repositories": [
  {
    "type": "path",
    "url": "src/project-name"
  }
],

If the module is private, but kept in a remote git repository, then there is the "vcs" repository type in composer. Composer will resolve any version constraints against the git tags if they are in semantic versioning format, just as if the module came from packagist or drupal's package repository.

"repositories": [
  {
    "type": "vcs",
    "url": "https://github.com/organization/project"
  }
],
larowlan’s picture

larowlan’s picture

FYI #3005229: Provide optional support for using composer.json for dependency metadata is green with a BC layer, so this is actually feasible in 8.8

donquixote’s picture

dependencies[] = taxonomy
dependencies[] = exampleapi (>7.x-1.5)

Dependencies are moved over to "require".

"require": {
  "drupal/taxonomy": "7.*",
  "drupal/exampleapi": ">1.5"
}

We need to keep in mind: Any contrib module required in this way can no longer be disabled in the UI (without disabling the requiring module), because it is now considered as a dependency.

So if the goal is to have Composer download a module, but not require it being enabled, then you need to add the "require" in the root composer.json of the website project, not in any module composer.json.

mmjvb’s picture

See nothing wrong with treating drupal dependencies as composer requirements, regardless the composer.json (project or module).
Consider it wrong to treat composer requirements of the module as drupal dependencies.

Would like to see properly normalized data, but afraid that the info is not provided to the same audience.

So, -1 to replacing .info.yml with composer.json.

pfrenssen’s picture

So if the goal is to have Composer download a module, but not require it being enabled, then you need to add the "require" in the root composer.json of the website project, not in any module composer.json.

I think the main use case for downloading modules that are not intended to be enabled is to have them available for development purposes, then it can be added to "require-dev". This will not formally declare it as a dependency.

fgm’s picture

I suspect this could cause a problem when downloading modules not from packages.drupal.org but from packagist or other packagist-type registries, which won't have the specific module-within-a-project logic and for which the composer.json used is the one at the topmost directory, would it not ?

shaal’s picture

I love this idea, but the issue wasn't updated for the last 2 years.
Does anyone know if it's possible?

neclimdul’s picture

So... maybe? There are some problems to completely replacing still. Maybe more accurately, a lot of work and hurdles still.

The most commonly discussed problem is submodules. composer.json is a package/project level file where there are possibly multiple modules inside a project. The current .info format largely supports those needs.

Assuming we figure out the submodule problem, we're going to have to generate an entire sub-schema to shove into extra for the module/theme handlers. Open up a theme info file and you'll see that can covers quite a bit.

ATM I think its just been "good enough" and the effort to figure out the remaining hard problems has just overwhelmed any benefits for most people.

jwilson3’s picture

we're going to have to generate an entire sub-schema to shove into extra for the module/theme handlers. Open up a theme info file and you'll see that can covers quite a bit.

Good point, but a lot of the additional items that appear in the theme info.yml file structure feel more appropriate in other places, definitely not in a composer.json. For example:

  • libraries - It has always bothered me that you have to define global libraries in the info.yml file, things need to be defined and described in one single place. Imagine a library definition in themename.libraries.yml file containing a prop load: globally.
  • ckeditor_stylesheets - similar to ^, this should be specified inside the library definition: load: ckeditor
  • libraries-extend - similar to ^, this should be specified inside the library definition: extends: drupal/somelibrary
  • libraries-override - similar to ^, this should be specified inside the library definition: overrides: drupal/somelibrary
  • features, hidden, logo, favicon, screenshot, regions, regions_hidden - move to default theme config in config/install/themename.settings.yml
  • Any other custom theme-specific entries should move to default theme config in config/install/themename.settings.yml
dww’s picture

As pointed a few times above, the most active work on this effort would be #3005229: Provide optional support for using composer.json for dependency metadata. That’s the first thing.

+1 to a lot of #118. There’s also talk of a dedicated regions.yml file for defining theme regions. We definitely don’t have to put everything currently in .info.yml files into composer.json. Let’s use composer.json for what it’s already good at, and put everything else in other places.

catch’s picture

Status: Active » Postponed

#1398772-79: Replace .info.yml with composer.json for extensions mentioned regions.yml back in 2015. I think we should officially postpone this on moving definitely-not-composer-y things out of .info.yml, and then revisit once that's done. #1398772-118: Replace .info.yml with composer.json for extensions would be worth opening a core meta issue + sub-issues for (don't think splitting those out requires it's own ideas issue).

Pasqualle’s picture

Created #3250765: [meta] Clean up .info.yml files to move #118 and this issue forward.

voleger’s picture

BTW I tried to create a converter for the *info.yml files.
Maybe it would be helpful for the separation of the data into separate files on the fly to composer.json and additional files (regions.yml, etc.).

Apply the patch for the drupal repo, and perform following command:
php core/scripts/converter/info_yml_to_composer_json.php --core-only

voleger’s picture

Also, the converter can be reused for contrib projects, so it will help to make the faster transition and help the community adoption of the new structure of Drupal extensions.

andypost’s picture

Status: Postponed » Active

It's been postponed in #77 in waiting of 8.1 but there's 2 days till PHP 8.1.0 and D10 soon be opened /cc @catch

catch’s picture

Sorry I still think this should be postponed on #3250765: [meta] Clean up .info.yml files. Moving stuff out of .info.yml is worth doing in its own right, once that's done, we can evaluate what's left and moving it to composer.json - but it'll still have to deal with issues like multiple modules in one project which composer doesn't have a concept for.

voleger’s picture

We can try to use the "replace" key in composer.json to replace the required submodule by the module or with composer installer API.
Example:
- "project1" requires "drupal/module1:submodule1"
- composer resolves dependency and requires "drupal/module1"
Here is option 1:
- composer resolving replacements and see that "drupal/module1" replaces "drupal/module1:submodule1"
Here is option 2:
- composer read the configuration of the composer-installer provided by "drupal/module1", and, on the appropriate hook, the composer locates the "drupal/module1:submodule1" package in the file system regarding the definition provided "drupal/module1" package.

So for submodules, we can generate some pseudo-package names as the module will replace them.

But I'm not sure whether it can be doable without some additional scripts.

catch’s picture

Status: Active » Postponed

Moving back to postponed.

AaronMcHale’s picture

I like the idea of being able to do drupal/module1:submodule1.

Is this something Composer could support out of the box, or would we need a special Composer plugin or additional functionality to make it work?

I wonder if it's worth opening a separate issue to explore that idea further, because regardless of what we do here, that could still be useful in its own right.

Eric_A’s picture

Please note that the Drupal composer shim already supports submodules. At least last time I checked. I'll write down some details. And then let's all leave this issue alone for now and link to here from (newly created) issues. :-)

IIRC, last time I checked the docs, whenever you require a submodule, the shim would provide the complete Drupal project package.
To distinguish between different Drupal projects, a (sub)module can be prefixed with the project name followed by a separator, i.e. "drupal/project_name[SEPARATOR]module_name". I forgot what the separator is, but I think it's all documented.

EDIT: Note that the above is about composer install, not about Drupal install. Enabling Drupal (sub)module dependencies is what the info file does. Which is completely different from ensuring submodule code from a certain Drupal project is lying around.

sun’s picture

Just drop the feature of "sub-modules" already. One module is one package is one project. Clean up the mess.

Don't invent a new Drupalism while adopting standards.

Sub-modules only exist to group their issues in the drupal.org issue tracker. That's a search/filter question at the application level, which should not affect packages at the code level.

jcisio’s picture

Re #130 in that case, I think we should also figure out how to sync the releases between submodules, too. Like how Drupal would do with its 50 "submodules".

Chi’s picture

I think, if the projects follow semantic versioning and properly define their dependencies they don't have to sync their releases.

donquixote’s picture

#130
Submodules exist to sync versions and to have a shared issue tracker.
Similar to what composer packages do with subtree split, where issues are done on the "parent" github repository.
Any CI testing also needs to happen from the "parent" package, if this is where issues and pull requests happen.

This said, I still don't get why we need to blur the lines between two distinct concepts.
A composer package can be downloaded / installed or removed, a module can be enabled or uninstalled.
E.g. typically after a module is "uninstalled" in config, it will still be kept in code for one release cycle, because we need the uninstall hook.

And the "one module per package" is already violated in core and in custom modules (modules shipped with the root package). I don't see what we would gain by enforcing this 1:1 rule for contrib, but not for core and custom modules.

shaal’s picture

TIL: Drupal modules that are missing composer.json cannot be installed as a symlink via composer.
(Installing symlinked projects is the main mechanism that allows easy contributions with DrupalPod)

So even when a module doesn't have any dependencies, it should have composer.json included in the module, otherwise you get unexpected results when using Composer.

Wim Leers’s picture