I released a new version of the DrupalAPI distro last night, and even though the makefile was nearly identical, the new version is 170mb bigger than the older version. Crazy right.

The reason being, the new package includes the .git directory for the libraries, which being Drupal 6 and Drupal 7 are actually quite large.

Given that the older version was also using the same repos but wasn't packaging the .git directory I have to assume something has changed on the packaging systems end.

Is there anything to be done? Because a 200mb distro that should really only be about 5-10mb is completely unacceptable.

Comments

jhedstrom’s picture

Crazy! I'd guess this may be a bug in how drush make handles libraries fetched via git, but haven't verified by testing locally.

deciphered’s picture

I'm unable to reproduce the issue locally with Drush 5.8 and the ensuring the latest version of the Drupaorg Drush module.

It's almost as if 'working-copy=1' is being enforced during the packaging.

deciphered’s picture

I take that back, I am able to reproduce the issue locally, seeing if I can determine the cause.

bojanz’s picture

I can confirm this issue, we ran across it in Commerce Kickstart, see #1873148: Libraries should be downloaded from zip files, not cloned from git.
All of our libraries had .git directories packaged, leading to issues when users tried to commit the packaged distro to a git repository of their own.

jhedstrom’s picture

deciphered’s picture

So it looks like from the sounds of things even though this has been resolved in Drush it hasn't yet been rolled out on D.o?

bojanz’s picture

Status: Active » Reviewed & tested by the community
Issue tags: +needs drupal.org deployment

DamZ tells me this is the way to do it :)

(Or do I need to do that to the actual Drush issue?)

mgifford’s picture

How does this actually get implemented? I'm looking at #1427752: Support drush make includes[] in drupal-org.make files and just not sure who is responsible for doing it? What happens after it gets RTBC?

dww’s picture

Yeah, sorry. The problem is that d.o doesn't have a regular standard process for deploying changes. It happens ad hoc based on who has time to responsibly follow the deployment through completion (including being available to deal with anything that might go wrong) and what particular itches are being scratched at the moment. For example, most of the folks with the power to deploy changes like these (including myself) are busy working on the d.o D7 port. :/ Ultimately, we desperately need #1929526: Drupal.org Software Working Group Charter (DRAFT). Meanwhile, it'd be good to still be able to roll out bug fixes like this. I'd vastly prefer not to be a single point of failure, it happens a lot, and I'm trying to put my energy into making that not the case each time it crops up, instead of just doing all the work myself. But maybe there's no good solution here in the short term, and I'll just have to eat it and do this sometime soon this week. Stay tuned.

Thanks/sorry,
-Derek

drumm’s picture

Title: .git directory being pacakaged? » Upgrade drush used for distro packaging with drush make
Project: Drupal.org drush » Drupal.org infrastructure
Component: Code » Packaging
Status: Reviewed & tested by the community » Postponed

At the moment, there hasn't been a drush release lately. Ideally, we want to upgrade to 5.9. (We can upgrade to -dev if a release isn't happening in a timely manner and we need the fixes.)

mgifford’s picture

Thanks! There's lots of process stuff that has just happened because people make it happen in the community. Only so much time, and always so much to do.

deciphered’s picture

Is there any chance anything will happen here? Latest release of DrupalAPI is 230mb when it should be 5-10mb.... If there's anything I can do I am willing.

drumm’s picture

Issue summary: View changes
Priority: Normal » Major
Status: Postponed » Active

5.9 is now out. We have 5.10.0 and 6.2.0 to choose from, https://github.com/drush-ops/drush/releases. I think we should upgrade to 5.10.0, incremental upgrades are less stressful.

Setting to major because we want to be in a good place to fix #2213315: Packaging loop more permanently.

The next steps are:

  • Make a copy of /var/www/drupal.org/tools on devwww.
  • Make upgrades to that codebase, keeping good notes in this issue.
  • Test packaging a few distros on a dev site configured to use that tools directory. A release's packaged files on the dev site can be removed in the filesystem and re-added to trigger packaging. Check that all code paths of the drupalorg_drush work well.
drumm’s picture

Issue tags: -needs drupal.org deployment

Not ready for deployment. Needs testing first.

Mixologic’s picture

Assigned: Unassigned » drumm

Seems like drumm is on this one.

drumm’s picture

Assigned: drumm » basic

Will need some help from Rudy first, building on #2310409: Upgrade to Drush 6 on buildvm.

We will want drush6 available on every box that has the drupalpackaging Puppet module. Likely as part of that module, keeping this Drush for packaging independently deployed from other Drush instances. Being able to have both in parallel will be a big help for testing.

basic’s picture

Assigned: basic » drumm

drush (symlink to drush5) and drush6 are now installed concurrently on all hosts with drupalpackaging.

drumm’s picture

Assigned: drumm » Unassigned

This can now proceed. On util and git7staging:

bash-3.2$ drush6 --version
 Drush Version   :  6.3.0
[drumm@git7staging ~]$ drush6 --version
 Drush Version   :  6.3.0

Next we need to test that these work reasonably well:

  • verify-makefile, as called by drupalorg's project_verify_package/project_verify_package.pages.inc
  • make and related code, as called by drupalorg's drupalorg_project/plugins/release_packager/DrupalorgProjectPackageReleaseDistro.class.php

I'm not actively working on this right now, so I won't assign it to myself yet. When deployment is scheduled, it will be announced with https://www.drupal.org/news/change-notifications-drupalorg.

drumm’s picture

I'm not incredibly familiar with Drush make upgrades. Is this change important enough to email all distribution maintainers about?

We do require that maintainers are subscribed to the maintainer news email list, so it would be okay to email important things.

drumm’s picture

Assigned: Unassigned » drumm
Status: Active » Fixed

This was done quite awhile ago

btch1:~$ php /var/www/drupal.org/tools/drush.phar --version
 Drush Version   :  8.1.12

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.