Problem/Motivation

To use 3rd party code right now, we have to include it directly in our repo. That makes updates to it annoying to do as well.

Proposed resolution

Composer is an increasingly popular tool for project dependency management that auto-downloads stuff for you. We're already looking to use it for getting code into Drupal in the first place, and it lets us not keep duplicate copies of thousands of lines of code in our repo

Remaining tasks

(reviews needed, tests to be written or run, documentation to be written, etc.)

User interface changes

(new or changed features/functionality in the user interface, modules added or removed, changes to URL paths, changes to user interface text)

API changes

(API changes/additions that would affect module, install profile, and theme developers, including examples of before/after code if appropriate)

Original report by [username]

Now that Drupal 8 is bringing in some of the Symfony components, we need a way to manage them. Composer is a PHP package management system which is heavily inspired by npm, and with Packagist, it provides great support for Symfony.

What this means for Drupal.org is having the the packager process composer.json file during the package build process.

drush dl composer
drush composer install
Files: 
CommentFileSizeAuthor
#42 git-instructions-os.png103.85 KBwebchick
#42 issues-user-os.png104.38 KBwebchick

Comments

  1. The issue summary needs a clear Problem and Goal statement.
  2. Based on my wild guess of what is being proposed (and why), -1 on requiring any additional tools to make a Drupal core git clone/checkout work.

Problem: To use 3rd party code right now, we have to include it directly in our repo. That makes updates to it annoying to do as well.

Solution: Composer is an increasingly popular tool for project dependency management that auto-downloads stuff for you. We're already looking to use it for getting code into Drupal in the first place, and it lets us not keep duplicate copies of thousands of lines of code in our repo.

Thanks Crell, didn't really have the time to write a proper issue summary. I just copied and pasted that in for now, probably have some time tomorrow to write up more docs.

So whenever I do a git pull you require me to do an additional

php composer --blah blub bleh

on the command line?

That's not going to fly at all for me.

git is supposed to control, ensure, and maintain the state of files. Not a separate tool that completely bypasses the version control mechanism.

I don't even know whether my git client would support that (TortoiseGit). Can't find an option for defining custom post-checkout/post-pull hooks. (which of course, require to have a working PHP CLI in the first place)

In turn, there'd be no guarantee that everyone works on the same revision of files.

This would make contributing to Drupal considerably harder.

You specify a version of the dependency in the composer.json file.

We're talking about 3rd party libraries; the Git-native option is git submodules, which few people like. :-)

If your git client doesn't support a post-pull hook, then it's Doing It Wrong(tm). (I'm pretty sure all git implementations do that.) That said, it's not something you should need to do with every pull since I don't expect us to be changing code in those libraries all that often.

Status:Active» Postponed

Note that "building Core" is pretty ill-defined. There are at least 4 ways this happens, probably more:

  1. The d.o packaging script (what you're talking about here).
  2. Direct git clone
  3. drush make
  4. drush pm-download

Obviously, we could handle #1 here if there was agreement this is a good thing to do. #3 and #4 are do-able, as well, although that requires a separate issue and patch (at least both now live in drush core, so that slightly simplifies things).

#2 is the main thing that will be broken if we do this. That seems like a pretty big problem. I believe the testbot is just doing direct checkouts, so we'd need to teach it about this change, too. That also requires a separate issue if we go this route, since that code lives in PIFT/PIFR (I can never keep those straight).

In an ideal world, I agree it's better to just pull in 3rd party code at "build" time instead of stuffing it in our repo. However, this would be a pretty drastic change to basically cripple native Git clones without an additional step, and I don't think that's just a d.o infrastructure discussion to have.

The experience of the (cvs|git)_deploy modules is that effectively no one ever found out about this additional step if you just get our code directly from the VCS. A handful of people In The Know(tm) use them, but we still regularly get bug reports/support requests wondering why the Update manager is "broken" when you clone from Git. No one reads README files anymore, nor release notes, so I don't know what we can do to help teach people they need this step. Ideally, if you visited a D8 core site that hadn't taken this step we'd print a nice helpful page about it instead of WSOD. But, that's additional complexity and a performance hit in the lowest possible levels of bootstrapping the system, so I doubt that's going to fly. Maybe that's at least something the installer could know about. And maybe we can reuse the authorize.php/update manager plumbing directly in the installer to run Composer for you if you haven't already. This is totally off-topic here in the d.o infrastructure queue, but that's what I mean -- the non-d.o-tarball case is the one where this is going to create the most trouble, so I feel like we really need a discussion about that, first. Once that's agreed and decided, the actual fix to case #1 above is pretty trivial.

There is also the increased complication of development when it's less clear that everyone's actually running the same code, as sun points out. While this isn't a show-stopper, it is a concern. Perhaps there's something we can do to mitigate this, like make it really easy to snapshot your /vendor directory (or wherever all this code is going to end up) to see if you're actually up to date or not, etc. Maybe Composer itself solves this, I don't know.

This is also a double-edged sword when it comes to making sure those 3rd party dependencies are kept up-to-date. In some ways, managing them outside the Drupal core Git repo makes it easier to update these things when necessary. However, that also implies there's going to be a long tail of people who will never bother. That's already a problem with Drupal itself, and I imagine it's also going to get worse if there are other cryptic steps (from the POV of end-users that just want a website and aren't PHP developers deeply in the know) they have to take. If we start going this route, we should open an issue for Update manager (if there isn't one already) about teaching it to notice if your vendor tree is missing updates and to warn the user about it (and ideally, run this for them).

And, I can almost guarantee there are going to be cases where 3rd parties rev their code in ways that Drupal isn't prepared to handle and the flip-side will be equally problematic -- people upgrading vendor before Drupal is ready and breaking their sites that way. Fun!

Of course, as we start relying more and more heavily on 3rd-party code for chunks of core functionality (YAY!) all of these problems are only going to get worse at the same time that the need for *not* putting all that code in our repos gets greater. So, I'm really torn. Mostly I think we should do this. But I don't think we should sugar-coat what a PITA/WTF this is going to entail. That said, this has been a problem in software development for decades (I won't even begin to start telling stories of the nightmares this stuff used to cause us when I worked on Condor), and the tools to manage it have only gotten better and better. So it's not like we're the first people to run into this and have to solve everything completely on our own. I haven't actually used Composer myself yet, but it sounds like a Good Thing(tm), and if it's reliable and simple, it's probably the right way forward.

That said, I'm going to flat-out refuse to touch the d.o packaging script until some of these other issues are hashed out in more appropriate places, and the summary here (or somewhere) links to all the various issues I've mentioned that need to be submitted (at least a couple of core issues, drush, PIFT/PIFR, etc).

Thanks,
-Derek

p.s. Or we can turn this into the meta issue for this whole initiative, mark it active, and open a separate issue specifically for the trivial change to the packaging script (which would then be postponed). I didn't mean to obstruct progress, just that if this issue is about the packaging script itself, it's blocked on other things happening first. Thanks.

the need for *not* putting all that code in our repos gets greater.

Frankly, I don't get this argument at all. This entire proposal came out of the blue and tries to fix a "problem" that I haven't heard of a single time from anyone else before.

Apparently, #2 states only "updating" as a problem, nothing else. Updating the libraries we're using within core is an issue, yes. But mostly a policy issue. And for the scope of updating, we already have a proposal in #1424924: Use Composer for updating Symfony components (without removing Symfony code from repo).

However. Once that Symfony code is in core, and once we're actively relying on it throughout core, your chances for being able to update it without adjusting tons of code throughout core at the same time are close to zero.

And that's where the version control system comes in and plays a key role in the game. If we update external libraries shipped with core, then we also need to update the consuming code in almost all cases. Updating only one of both will - as far as PHP library code is concerned - lead to fatal errors.

In the end, I don't get why we're trying to rely on an external tool to do the job that the version control system tool is supposed to do. Use the right tool for the job. And apparently, git was designed from the ground up to handle larger repository sizes in a smart way.

Title:Process Composer components when building Drupal coreRemove symfony components from the core repo and let Composer manage the dependencies instead
Project:Drupal.org infrastructure» Drupal core
Version:» 8.x-dev
Component:Packaging» wscci
Category:task» feature
Status:Postponed» Active

Per dww's suggestion in #7 and seeing as this thread has so far been about whether this move is a good idea, changing its details to reflect that. Not sure whether to mark it as Policy, seeing as it's about making the decision and we can't provide a patch until #1424924: Use Composer for updating Symfony components (without removing Symfony code from repo) itself is in.

Will open sub-issues and mark them postponed for now.

Note: I'm actually pretty new to the core issue queue (apart from small stuff) so let me know if I am Doing It Wrong (tm) - I won't take it personally ;-)

Oh, and here's an interesting commit on the silex project: https://github.com/fabpot/Silex/commit/4ce45f1eecc3e7cfb69c70a7ba5abdd5d...

Sun, I don't get what you mean when you say

If we update external libraries shipped with core, then we also need to update the consuming code in almost all cases.

Surely for that to be true, the external library in question would have to have been extremely poorly architected, with no conception of abstraction or encapsulation or any of the lovely OO design principles that constitute the reasons we are embracing Symfony. Or maybe I am simply misunderstanding things here :-/

Re: @sun #8

Once that Symfony code is in core, and once we're actively relying on it throughout core, your chances for being able to update it without adjusting tons of code throughout core at the same time are close to zero.

Not sure how Symfony rolls, but if they're following some sort of semantic versioning policy (which package managers tend to get people into the swing of), then it's entirely possible that patch- and minor-level updates could be happening fairly painlessly. Only major-level should break backwards compatibility so long as we're using the public methods we're supposed to be, if I understand correctly -- and it's entirely possible I don't :)

In the end, I don't get why we're trying to rely on an external tool to do the job that the version control system tool is supposed to do. Use the right tool for the job. And apparently, git was designed from the ground up to handle larger repository sizes in a smart way.

My angle would be that version control is NOT intended to store large amounts of code, but meant to store all the information that is sensibly required to rebuild the current codebase at any future point in exactly the same way as it's built today. With custom code, that involves storing everything. With dependencies, that might be a simple as repo+project+version. Yeah, we could say that the external sources might go down, but so could pear.php.net or any of the other things we and other projects depend on for builds. We need to have a certain degree of trust in outside sources, and cover our asses with caching where it makes sense.

Anyhow, originally came here to post this link, which seems relevant:
http://stackoverflow.com/questions/2050450/git-hooks-management

Drush might be a good first start: #1316322: Add PSR-0 autoloader to drush

Anyhow, originally came here to post this link, which seems relevant:
http://stackoverflow.com/questions/2050450/git-hooks-management

Somewhat related, was working on some documentation for Composer which would allow you to automatically run a Composer update whenever a git checkout was performed:
https://github.com/composer/composer/pull/467/files

Not sure how Symfony rolls, but if they're following some sort of semantic versioning policy

I just asked #symfony, and they said they're using semantic versioning. Composer supports version wildcards to force a specific version branch. The following would retrieve the latest in the 2.1, but would not move to 2.2:
"symfony/finder": "2.1.*"

In turn, there'd be no guarantee that everyone works on the same revision of files.

Per #1424924-79: Use Composer for updating Symfony components (without removing Symfony code from repo) and the comments following it, we can and I think should commit a composer.lock file to our repo to address this. Updating a vendor component in turn would require a normal core process of a d.o. issue with a patch file (containing an updated composer.lock) that is tested by bot (confirming no Drupal functional tests break as a result of the vendor update) and committed by a core maintainer.

So whenever I do a git pull you require me to do an additional "php composer --blah blub bleh" on the command line? I don't even know whether my git client would support that (TortoiseGit).

This is a valid concern that I raised in #1424924-85: Use Composer for updating Symfony components (without removing Symfony code from repo), but katbailey pointed me to this issue as the better place to discuss it. I think we need to balance this productivity concern with the productivity concern of people working on core issues that require a vendor component update plus the drupal code they're working on. We've seen on some issues already that posting a patch that includes both results in large patch files that intimidate reviewers. In some cases we've dealt with this by splitting the vendor update in a separate issue, but then there's delays in getting that committed, and meanwhile people are less motivated to work on the drupal issue that's blocked on it. By having a patch file include only a composer.lock (and possibly composer.json) update along with the Drupal code, would that increase number/quality/timeliness of reviews? I hope we can either find a solution that works for people who don't like command lines or a solution to the "big patch on d.o. issue is scary" problem. In the end, we want as many contributors feeling productive as possible.

Also, katbailey makes a good point in #1424924-64: Use Composer for updating Symfony components (without removing Symfony code from repo) about helping site developers and distribution builders manage their builds.

As a data point, right now I have a branch in the WSCCI repo to fix error handling tests (and change the way the database handles error reporting) that I am not merging yet because it's blocked by #1542710: Update to latest Symfony 2.1 code. I don't want to just merge it to the kernel branch now because that's going to result in a lot of fugly merge history when both huge Symfony updates end up in the same merge commit (something a number of developers have objected to with a git-based workflow, rightly or wrongly). Instead I want to wait until I can resynch the kernel branch, then rebase the new feature branch onto it.

If I could just include a tweaked composer file as part of that, I wouldn't even need to bother updating core when I needed a newer snapshot of Symfony. I could just merge and go and let Git sort out a few 2 line commits later on.

Issue tags:+composer

tagging

I feel like this is a bit of a distraction. Let's stay focused on the minimal viable products (e.g. web services in core), even if that means there are some annoyances for the time being. It feels like this issue could easily distract some of our best people for a couple of weeks. I'd prefer not to do that.

For anyone interested, there's a summary of the composer-related issues here:
#1424924-91: Use Composer for updating Symfony components (without removing Symfony code from repo)

Also, here are all issues tagged "composer".

That other issue is generally for the step #0 that could be executed very painlessly: adding composer.json and composer.lock to repo
#1424924: Use Composer for updating Symfony components (without removing Symfony code from repo)

I respect @dries' rationale for postponing this larger issue, and anything I post is for later consideration. @sun and @katbailey made some relevant comments in the other issue that belong here:

@sun said

Introducing Composer for updating these libraries in core more easily is one thing. I can get behind that idea.

But removing these libraries entirely and completely relying on Composer sounds like a disaster.

Regardless of whether you're using a UI or the command line, this is not guaranteed to work:

git checkout 8.x-my-issue-dev-branch

Every checkout will have to be followed by a php composer whatever command whenever you are switching branches, because you cannot be sure anymore whether the files you have are complete, correct, and compatible.

While you CLI coders might be OK with that and will write a git alias or shell script, I'm seriously not. I'm using a git GUI for development, and I want to get the proper files when switching branches.

Pretty please don't make contributing to Drupal even harder than it is already.

@katbailey said

If no git GUI had existed for Windows back when we were switching to git, would that have been a good enough reason for us to stick with CVS? (I wasn't involved in the discussion so I have absolutely no idea whether such considerations factored into it, but I feel strongly that the answer to this question should be a resounding "NO!").

Changes like this don't affect the end user in the slightest and they affect developers only insofar as they force them to embrace the tools and practices that the wider PHP community has adopted in order to more effectively manage projects. That, to my mind, can only be a good thing.

I'd imagine that the vast majority of pulls from upstream when working on D8 will not include a change to composer.json (or composer.lock), and so no extra step will be required. Does your GUI client show you which files have changed when it merges changes from upstream? Is it possible to add git hooks in the GUI? If not, then I'd say it has some catching up to do, but that shouldn't hold up a change to Drupal that otherwise makes a lot of sense.

don't make contributing to Drupal even harder than it is already

I honestly see this change as having the opposite effect!

Dries: I'm unclear which "best people" would be tied up on this. Right now, it's a pain in the neck to "chase master" on Symfony. The more we want to push improvements upstream to Symfony (and we want to, and have a few times now), the more we'll want to stay up to date.

Unblocking this means we can get better velocity on Symfony-related stuff. Plus, the main tasks that need to be done to make it happen are testbot and infra tasks; those people are always busy, but mostly not on core. Plus, the tasks on the surface seem really easy: Just add an extra command that gets run before the packager and testbot do their final step. I don't see how that would "distract our best people for weeks". (Not to say that the infra people are not best people, mind; they do yeoman's work. I mean that it wouldn't be pulling people off of new-feature development for D8 to do this and make new-feature development for D8 easier.)

Using Composer to update the libraries in a simple way makes sense.

However, removing the libraries entirely from Drupal core results in more negative consequences than it actually helps. The benefits of doing so have single, one-time implications only. The benefit only affects the initial addition. Contrary to that, the negative consequences are permanent and affect everyone who is trying to hack on, develop for, and contribute to Drupal core in any way.

I understand that this has caused some troubles for D8 core development, but that should be considered as a side-effect of a large transition process. Post-release, i.e. once all these new libraries have been added, it is quite predictable and reasonable to expect that our main issue will revolve around updating the libraries. (which still means update != add)

I also understand that "others" are using Composer in this way, but at the same time, those others have an entirely different understanding of the "product" they are developing and delivering, which is not comparable to the Drupal core product at all. The vast majority of them consider themselves as developer libraries or frameworks that can be and are actually supposed to be manually mixed, adjusted, and changed in order to produce an actual end-product that works. Contrary to that, Drupal core is a solidly defined set of code, subsystems, and libraries, which not only needs to work by itself, but also requires that solid definition of what's contained in order to allow contributed projects as well as its entire ecosystem to work against a particular major version. The moment you loosen or remove that fixed definition, you are opening the door for a huge range of conflicts and problems that haven't been there before.

What we're talking about is the management of external libraries. However, not just some additional libraries, but instead we're talking about replacing the libraries that are driving the very core of the system. Sorry for being blunt, but my impression is that most people have only thought about the direct benefit for themselves, without thinking about the system-wide, organizational, and technical implications. I'm not sure whether I am in a privileged position, but having spent a lot of time with Libraries API, jQuery, jQuery Update, and jQuery UI modules in contrib, I know that this change proposal has plenty of consequences, which haven't been considered at all in here yet.

(Before anyone is going to ask: No, I won't provide a more concrete list of consequences. The question itself would mean that one wants to change something without understanding the consequences. If I had solutions for most of the consequences, then stuff like Libraries API wouldn't be as crappy is it now. But that's contrib material. And that's where experimental stuff belongs, for which not all consequences are known or resolved yet. The consequences of this change proposal here have an even larger impact than that.)

Next to that there are practical issues. First of all, as the engineers who are developing, building, and delivering a final product, we should not only know the version of some external library we're pulling in. We are also the final line of defense that is responsible for ensuring that the code we're shipping for our core product is sane, contains what we want and expect, and is neither bloated nor too limited. We should know the code we're adding and shipping with core. Just adding a pointer to a package and version that consists of 500k of possibly bloated code and perhaps even nonsense, and basing an entire set of architectural changes on that, heavily increases the chance that almost no one actually studied the code that is going to be packaged before it is added. In fact, we just did something like that via #1465584: Commit external library dependencies to Drupal core, and even though I created that issue myself, I never really reviewed the code being added. I strongly object to that approach. I agree with the "Not Invented Here"/"Proudly Found Elsewhere" efforts, but the moment we start to add and use functional code without even knowing and fully understanding what we're adding exactly, we're Doing It Wrong™, it couldn't get more wrong. That is, because it is us who are being held responsible for the final product we're delivering. If we do not fully understand the code that we're shipping, or which particular changes exist between a previous and a new version, then we will run into serious maintenance and security issues.

Second, the fact that we pulled in the latest version of external components and committed them allows people to easily hop between core and core sandboxes using alternative versions. A simple git checkout results in exactly the code you need. Presuming that you've fetched your remotes before leaving, everyone is able to work offline, without requiring a working Internet connection to (re-)download the packages you're possibly missing for another branch. As a direct consequence, we would destroy git's built-in/native capability of working offline. There's also no difference or conflict in what exactly "latest version" means, because the files are properly versioned.

Third, the troubles of novice core contributors even begin with getting a git client up and running already. In almost all cases, novice contributors are using a UI-based git client. It is absolutely clear that we need to on-ramp many more contributors in order to meet the demand of the market. While the additional usage of Composer might be clear and trivial to more advanced PHP coders, it is going to be a huge additional barrier for many novices to get involved, especially those who never really worked on the command line.

That said, the practical issues could be resolved in some ways. E.g., 3) by developing a git command/extension/fork that has built-in Composer support, 2) an offline cache for Composer (but which in fact would boil down to git mirrors of the pulled in packages, à la Drush), and 1) by requiring proofs of extensive reviews of code to be pulled in, before it is specified and committed as new Composer package.

However, the other issues I've mentioned earlier would still remain.

sun: While you raise some valid points, the one you spend the most time on is simply untrue.

You imply that by using Composer to pull in 3rd party code on-the-fly, we'll be pulling in untested code we've not looked at and vetted and decided was a good thing to do. That is completely false.

A composer file lets specify an exact version of a package you want to download. The composer.lock file, which gets generated and *should* be checked into the repository, specifies the version of a package you want to run down to the precise git commit sha1 hash to use. If we check into the repo a composer.lock file saying to use, for instance, Twig commit 123abc456, then every single person who checks out Drupal and runs composer update will get that precise, exact same version of the code. Yes, I am assuming that we are vetting the code we're talking about before checking in new versions of composer.lock, but that is no more or less safe an assumption than assuming that we're vetting periodic updates to the Symfony components now. I would presume we'd apply the same level of care to such updates that we do now.

The difference is that such patches are then 2 KB in size, not 2000 KB in size. It also means we do not need to first add an unused library to core just to have a vaguely readable patch for our own code; The "put Twig in core and use it" issue can have a single extra line in it to pull in Twig version whatever; Then that version (that precise, specific version on which we have done whatever auditing we feel like) will be downloaded by anyone applying the patch and running composer update. Similarly, in the routing work I'm doing now I won't need to manually add a commit that pulls in another 300 KB of Symfony updates because I need a newer snapshot than what's in core, because Niklas added some new functionality to Symfony's Routing component that I need.

Libraries API, jQuery, jQuery UI, etc. have nothing to do with it. The packager can run composer install/update as well, so people downloading tarballs will never know the difference. The entire "unvetted code" line of thought is FUD. Please stop with the FUD.

Your other points about branch switching while offline are valid; I'm not sure of the solution there, and agree we should discuss that. But let's stick to discussing real issues, not imaginary ones that have already been pointed out to be false multiple times.

I'm with Crell. Composer will make it simpler to bring in and roll with outside deps, but it won't unhinge code from common sense. That's still up to us to vet the single-line `composer.lock` patches that will represent large changes.

But you're right @sun about working offline. I'm lucky enough to live in a place where I don't need to worry about that, but it's likely a real issue for those living in areas with less reliable connectivity. Composer caching is in the works, and maybe that's a valid feature to wait on:
https://github.com/composer/composer/issues/54
https://github.com/composer/composer/pull/62

As for GUI users, I'm unsure whether there's a fix for this which would be ideal for the newest of core contributors, but there is this: Adding a default post-checkout hook on their system to run a script (from d.o docs presumeably) on each checkout.
post-checkout hook: http://stackoverflow.com/a/1011745/504018
Git hook templates: http://stackoverflow.com/a/1977655/504018

From now on, it's just:

$ git clone --branch=8.x http://git.drupal.org/project/drupal.git
$ mv drupal/.git/hooks/post-checkout.sample drupal/.git/hooks/post-checkout

We talk a lot of about new core contrib, but I really wish we had more data on who these people are. My thought was always that with the curve of core, it would be rather advanced users who began contributing so deeply in the code.

@Crell / #22 - just a small correction, you must run composer install, not update. update will ignore the lock file and update dependencies to the latest possible versions. Other than that you're correct.

As for the points mentioned by @sun - the offline thing is valid but hopefully will be mitigated in the future.

Regarding the contributors, I would just say.. why not give it a shot and see what feedback you get from contributors? It really is just a small-ish decision that can be reverted at low cost. Just commit the vendors again if it turns out it creates too much friction. I understand the uncertainty, but in terms of commitment, it's hardly as big as the other leaps that have been taken recently (Sf2 components and whatnot).

@Seldaek

update will ignore the lock file and update dependencies to the latest possible versions. Other than that you're correct.

Just to clarify, that's "the latest possible" given the version constraints in composer.json, right?

Seldaek: Oops, you're right. I got those two confused. What's the mitigation you're talking about for offline use?

Otherwise, what #24 said. This is being blown way out of proportion.

@patcon: yes.

@crell: local caching of packages mostly, but proxying via private satis repos might help too if you have some spare disk space, and if we build a CDN of mirrors it might help people in remote areas.

I see no problems with that, in Java world we are using Apache Maven which is composed of a package repository manager.

When you fetch a new project, you get the last version of the pom.xml (here the composer.json) which indicates to maven which depencencies are needed for a given scope (e.g. at build time, run-time, or testing). This has no conflict with a configuration management such as git, they are complementary tools, just like git submodules for instance.

That would imply also that the qa.drupal.org testbot install the required dependecies before runing a test-suit. Composer would become a drupal dependency for the build and testing phases.

Sylvain: How does that address the "switched branches on an airplane" problem, where you need a different version of dependencies? That's the only conceptual blocker here.

I am sorry I didn't get the idea, what do you mean by "switched branches on an airplane" ?

I am not english so there is some concept that I may not catch directly or needs some further explainations.

It is the fact of switching from branches to branches ? Well, for maven this is not a big deal, because maven keeps a local repository under a .m2 directory. This local repository is composed of all artifacts that you previously downloaded, meaning a switch of dependency is extra fast, plus is managed by maven itself. When you switch or change the pom.xml, maven automatically "Update Dependencies" for your project.

I am not sure for Composer however, if a local repository is not supported and you have to fetch a new version each time you switch a branch, that may cause heavy problems. Maybe the tool is not mature yet.

See #21 and #22.

Composer is certainly going to have a local cache, their is already some functional code https://github.com/composer/composer/pull/915. This is where the offline issue should be fixed not in drupal git.

Yea, definitely seems like correcting this upstream is the correct fix and would be beneficial to other projects.

So...

I really tried to re-evaluate this proposal with an open mind in the past weeks. And I warmed up to it. To some extent.

However, my main problem still is the practical barriers that this change would cause.

I'm not sure how your reality looks like, but mine looks like this:

- Pull latest origin:8.x
- Hack.
- Switch to wscci:bundles-sun, hack.
- Switch to cmi:config-next, hack.
- Switch to cmi:config-thingie, hack.
- Switch to platform:user-picture, hack.

The important part is in "switch to". EDIT: To clarify, this means to switch to these entirely different branches multiple times per day.

All of these branches have a completely different merge-base and thus a completely different set of vendor files in core (and core was fixed to be compatible with recent vendor updates). Most of the branches have no need to merge in latest origin:8.x/HEAD, since they are the driving force for focused topics/components, so origin:8.x/HEAD only changes when one of the branches was committed/merged into core.

My platform's git UI (TortoiseGit on Windows, which has evolved into an excellent git client in the meantime [compared to other platforms as well as the CLI]) allows me to simply checkout and work on the branch I want to hack on.

Having to bootstrap the command line into the right location in order to switch/update the core dependencies accordingly — even when presuming there's a caching mechanism in Composer that's able to handle git/$VCS commit hashes, which inherently becomes a problem, since Drupal core does not specify exact commit hashes to include, but only min/max dev versions at this point, so Composer would have no idea what to actually switch to between those git branches... which inherently brings me back to my point in #8... — well, yeah, even if that major design flaw was resolved (e.g., by specifying explicit commit hashes in composer.json — which alas, would inherently break the original idea of making vendor updates "trivial" in core), this change would still hinder me in contributing.

That said, I can totally understand the point of this when working off a code base that has dependencies that are not supposed to change. I.e., essentially translating into "the released Drupal core." — though doing that sounds nonsensical if the dependencies are in the repo during development already.

In the end, the negative consequences of this change proposal heavily outweigh the positive ones (btw, what exactly are those, actually? "popular" does not count).

FWIW, in case Composer has a --do-not-touch-existing-vendors option, then I'd totally +1 a change proposal to make the testbots execute Composer with that flag in order to pull in entirely new dependencies being proposed for a particular Drupal core patch.

Drupal core does not specify exact commit hashes to include, but only min/max dev versions at this point, so Composer would have no idea what to actually switch to between those git branches

That's not true. Composer.lock specifies the exact version, and if we allowed chasing non-release commits, would specify the exact commit. So, if/when #32 is resolved, then all that will be left is integration with git, such that a git switch (checkout to another branch/commit) of Drupal could automatically trigger Composer to update the vendor code to match what composer.lock specifies. Seems like once that's possible, all objections to this issue are resolved, but I don't know what it will take to get these things resolved.

Drupal could automatically trigger Composer to update the vendor code to match what composer.lock specifies. Seems like once that's possible, all objections to this issue are resolved, but I don't know what it will take to get these things resolved.

A while ago, I put together a pull request to add documentation about running a composer install after any git checkout: Use Git Hooks with Composer. That PR could probably use a touch up. Whether git hooks work with TortoiseGit is beyond me. TortoiseComposer, anyone?

Component:wscci» base system
Issue tags:+WSCCI

I still want to do this, but refiling...

Update to #32 gagarine:

Composer is certainly going to have a local cache, their is already some functional code https://github.com/composer/composer/pull/915. This is where the offline issue should be fixed not in drupal git.

First pass at cache support merged in:
https://github.com/composer/composer/pull/1282

Yep local cache works now :).

Good news, I was also wondering if you guys mentioned git submodules but Crell did it already in #5 ;).

Oh, and for anyone interested in contributing, @Seldaek (composer maintainer) seems interested in the git clone --mirror/git clone --reference approach that drush takes to caching git repos:
https://github.com/composer/composer/issues/1323

StatusFileSize
new104.38 KB
new103.85 KB

Some folks were asking about this at DrupalCon Sydney, so I decided to pull some numbers to see what exactly the impact of "The Windows Problem" is, so we can throw some data into this discussion of heated opinions. :)

I went into drupal.org's Google Analytics account to segment the URL "project/drupal/git-instructions" URL (since xjm and I couldn't think of a better URL that would help pinpoint "yes this person is most probably a core developer/contributor") with operating system. It turns out, despite what DrupalCon photos might indicate, there's about 60%(!) of our contributor base using Windows:

git-instructions-os.png

In case you think to yourself something snarky like "Well, duh. Windows users are stupid, so of course they use this page more," :P I also checked /project/issues/user, which would catch contributors across projects, and that the numbers still shows a pretty large Windows majority:

 52% Windows, 35% Mac, 15% Linux

Granted, this doesn't let us know if someone's using Windows but nevertheless has Cygwin, VMWare, etc. installed. But I don't think we can exactly poo-poo this concern (nor the one about switching branches and forgetting to update composer).

kim.pepper was working at the sprint to try and determine some more data around the problem space, as well as if there are workarounds. His concern, and I agree with this, is that people from outside Drupal will see we're using Composer, and want to use it in the same way they can use Composer in any other project. However, currently, the way we use Composer for just an update is a "Drupalism," which is the very opposite thing we want to do when we're bringing in external libraries. :P

Is the 'Windows problem' that users can't be expected to use the command line when needed? Composer runs on Windows according to the docs, and I don't think we should lump all Windows developers into a category of not being willing to use the command line if need be.

I decided to pursue some evidence in terms of the sizing of the repository. For anybody using drush make, every time they build a project that has a dev release of core, the repository is cloned from drupal.org (using --working-copy—otheriwse, drush does have a ref-cache after the first clone). I did the following to see the size of the actual download:

jhedstrom@hyperion:/tmp (@sp)☠ git clone dm:drupal
Cloning into 'drupal'...
remote: Counting objects: 200544, done.
remote: Compressing objects: 100% (48043/48043), done.
remote: Total 200544 (delta 144690), reused 192009 (delta 138125)
Receiving objects: 100% (200544/200544), 55.33 MiB | 592 KiB/s, done.
Resolving deltas: 100% (144690/144690), done.
jhedstrom@hyperion:/tmp (@sp)☠ cd drupal/
jhedstrom@hyperion:/tmp/drupal (8.x) (@sp)☠ git bundle create drupal-with-vendor.bundle --all
Counting objects: 200544, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (41478/41478), done.
Writing objects: 100% (200544/200544), 55.30 MiB, done.
Total 200544 (delta 144690), reused 200544 (delta 144690)
jhedstrom@hyperion:/tmp/drupal (8.x) (@sp)☠ du -sh drupal-with-vendor.bundle
56M     drupal-with-vendor.bundle

The current clone size is 56M. I then ran a series of commands to completely remove core/vendor from history (which I'd recommend we do if this goes in) on both the 8.x and 9.x branch.

jhedstrom@hyperion:/tmp/drupal (8.x) (@sp)☠ git bundle create drupal-without-vendor.bundle --all
Counting objects: 209319, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (38787/38787), done.
Writing objects: 100% (209319/209319), 35.37 MiB, done.
Total 209319 (delta 155618), reused 209319 (delta 155618)
jhedstrom@hyperion:/tmp/drupal (9.x) (@sp)☠ du -sh drupal-without-vendor.bundle
36M     drupal-without-vendor.bundle

Without, the clone size is 36M.

For everybody who isn't currently working on 8.x, this is 20M extra download everytime they clone 6.x or 7.x.

Once 8.x is the current stable version, the size of the repo is not as big an issue (in fact, with a new git clone and a composer install, far more code is downloaded, since the history of each vendor project comes down as a git repository).

It has been stated above that using Composer may make contributing to Drupal more difficult. I disagree, but regardless, what the current system does without a doubt is to discourage contribution back to the non-Drupal projects such as Symfony and Twig, since with those checked into the repo, contributors have no easy way to contirbute a pull request upstream to those projects.

It has been stated above that using Composer may make contributing to Drupal more difficult. I disagree, but regardless, what the current system does without a doubt is to discourage contribution back to the non-Drupal projects such as Symfony and Twig, since with those checked into the repo, contributors have no easy way to contirbute a pull request upstream to those projects.

This is a major point.

I feel like this discussion is getting a bit out of scope. Let's dial it back a bit...

  1. Composer works on Windows
  2. If you have Drush working on Windows, the Drush Composer wrapper works as well
  3. With the addition of Composer, Drupal core has the ability to bring in external libraries without needing to copy/paste. If someone wanted to contribute to Drupal, they could put together PHP package (like msonnabaum's KeyValueStore), and have Drupal's composer.json bring it in. It's not about having people contribute to Drupal, it's about having Drupal work well with others.
  4. The good thing about forcing updates upstream is that it forces updates upstream. We should be contributing fixes to the projects we use, and not forking them locally. Let's teach best practices, not bad habits.
  5. This issue is about removing core/vendor from the git repository and having Composer handle that for us. Benefit is that updates are managed for us (we don't need to submit update patches against core/vendor). Con is that it would need some d.o infrastructure changes. Very doubtful this would happen in Drupal 8, maybe for Drupal 9.

I have written a simple patch in #1920666: Allow additional external libraries with composer that provides:

  • An example.composer.json for each Drupal install, which references Drupal cores composer.json for dependencies. This can be then used to add new libraries used in the project
  • Removes core/vendor
  • Allows composer to download vendor libs into the project root /vendor dir
  • Changes Drupal's classloading code to look in /vendor

I agree that the biggest hurdle is infrastructure changes: running composer install when generating the drupal download, and testbot running composer install when running tests. The code changes are very trivial.

Kim

what the current system does without a doubt is to discourage contribution back to the non-Drupal projects such as Symfony and Twig, since with those checked into the repo, contributors have no easy way to contirbute a pull request upstream to those projects

I either disagree with or don't understand this claim. We are cloning projects in /core/vendor/, not forking them. I'm not aware of any point in the D8 cycle in which we accepted a change to an upstream project as a Drupal-specific fork. If you want some code in Symfony or Twig to be different, the only way to do that is to submit a pull request to the upstream project. The annoying step is then after that pull request is accepted by the upstream project, we also need a Drupal patch that brings it into Drupal's code base. However, at this point, we're setting composer.json to target tags / commit ids anyway, which means even if we didn't commit the vendor code into our codebase, we would still need a Drupal patch that updates composer.json to the new tag / commit id. For these same reasons, I similarly don't get what #46.3 and .4 are saying.

Is the 'Windows problem' that users can't be expected to use the command line when needed?

I think #34 sums it up nicely. When you switch to a different branch/tag/commit of Drupal's code base, it should automatically update vendor code to match what's in the composer.lock file of the Drupal code you're switching to. Having to require more than 50% of Drupal contributors to manually run an additional command every time they run git checkout seems worse for total core development productivity than having /core/vendor/* in our git repo.

However, #1920666: Allow additional external libraries with composer lists good reasons why pulling out /core/vendor would help contrib and custom site developer productivity, which is certainly worth considering and might tip the scales.

I either disagree with or don't understand this claim. We are cloning projects in /core/vendor/, not forking them.

I didn't mean to imply we were forking the vendor projects, nor did I mean that contributing to those would be impossible. I wanted to point out that using composer as it is used elsewhere makes contributing upstream to vendor project much easier since one has all the necessary git repos in their working project.

I wanted to point out that using composer as it is used elsewhere makes contributing upstream to vendor project much easier since one has all the necessary git repos in their working project.

We're not currently using the git repository method for pulling in dependencies.

Kim

We're not currently using the git repository method for pulling in dependencies.

@ykim.pepper
whenever someone wants to fix an upstream bug:

rm -rf vendor/twig
composer install --prefer-source
# Just twig cloned

@patcon Oh nice!

One more argument to remove the component from the repo https://github.com/theodo/TheodoDrupal8Bundle/blob/master/Resources/doc/...

If you want to integrate drupal8 back in symfony (and yes you want to do that if you want to add a CMS to an existing symfony project) you will have to hack the autoloader...

EDIT: At least to resolve the issue on TheodoDrupal8Bundle we need a switch (in settings) to disable the autoloader. Their is an issue for that already?

Issue summary:View changes

Updated issue summary.