I'm wondering if we use drush_make to create a drupal site, and then we update the make file, say change a module to a new version, can we then update/re-make changes by rerunning 'drush make'?

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

smokris’s picture

Status: Active » Needs review
FileSize
7.05 KB

Currently this is not possible --- drush_make only builds fresh sites; it can't modify existing sites.

I've attached a first attempt at implementing this. The attached patch adds a new command, "drush remake [makefile] [basepath]", which does the following:

  • Builds the site from scratch in a temporary directory, just like "drush make".
  • Copies sites/default/files/ and sites/default/settings.php from the existing site to the temp build.
  • Moves the existing site to a backup folder (basepath/../backups/date-sitename/).
  • Moves the temp build into the existing site's folder.

(And, of course, the eventual workflow would involve switching the site into maintenance mode prior to doing "drush remake", and running database updates afterward.)

mradcliffe’s picture

An initial look: I think it might be better to parse the make file, and invoke the same functionality as the "upc" command in drush (or the update command, but I like to run db updates separately).

That's the approach I was going to take.

smokris’s picture

Thanks for your feedback, mradcliffe. Could you elaborate on what's better about doing it that way?

(I understand that my method is more resource-intensive, since all the packages are downloaded again, but this can be easily mitigated with a proxy server if bandwidth is a problem.. And I see some advantage in producing a fully fresh rebuild when updating, to fight bitrot and those kitten-murdering core hackers.)

I considered parsing the makefile and performing incremental changes to the existing site, rather than rebuilding it, but ran into a few snags:

  • If a package has been removed from a makefile (or if the selected version of a package is changed), there's no credible way to detect this (unless drush make starts keeping its own versioned repository of makefiles for each site).
  • Even if we can figure out whether a package needs to be removed (permanently, or for update), there's no credible way to find exactly which files that package references (unless we download a fresh copy of the package, which approaches the same bandwidth issue as above).
  • We can't use "upc" to update core (whereas the above patch allows for this).
dmitrig01’s picture

drush updatecode.

smokris’s picture

Oh, cool. I hadn't realized #434944: Allow minor upgrades of drupal core was already committed.

smokris’s picture

So... @dmitrig01 and other maintainers:

1. Are you open to including a "remake" feature in drush_make?

2. If so, what are your thoughts regarding how to implement it?

  • A) lazily rebuild site from scratch as in #1
  • B) derive changes + "drush updatecode" as in #2
  • or C) a brilliant solution we haven't thought of yet
dmitrig01’s picture

Status: Needs review » Postponed

oh I've thought of 2c... its coming. So yes, I'm open to this, but the current solution isn't robust enough. Let's postpone this.

smokris’s picture

OK. Could you elaborate on what would make a robust solution?

drewish’s picture

subscribing. this would make drush_make much more useful for some projects i'm working on.

dmitrig01’s picture

basically, we need to keep track of what's already been downloaded. but this is going to have to wait until after the 2.0 release. at least.

rjmackay’s picture

subscribing

Steven Jones’s picture

Subscribe

bibo’s picture

Subscribe

TheDoctor’s picture

Subscribed.

franz’s picture

sub

djroshi’s picture

Subscribed

aidanlis’s picture

sub

guddomeNt’s picture

FileSize
6.78 KB

Hi.

I really think this feature should be prioritised up, as drush make is otherwise only suitable for initial install - Which is a big shame.

I couldn't use the supplied patch, as it will override any checked out working copies. Instead, I've created a new implementation of the same thing, where the make is built in-place - repositories updated, rather than re-exported, if checked out as working copies. Instead of a new "remake" command, I simply added a --remake option to the existing make command. Combine it with --working-copy for to keep working copies in sync.

There's clearly still some overlap between drush make and drush upc, but until this gets sorted out, I think a remake-option is an acceptable workaround.

Patch attached. Here's a brief summary of the changes within:

  • Created drush_make_tmp_build_path and updated all hardcoded references to __build__ to use this.
  • Patched drush_drush_make_make to set global drush_make_tmp_build_path if --remake option is set.
  • Patched DrushMakeProject#findDownloadLocation and DrushMakeProject#applyPatches so they will pass with --remake option
  • Patched drush_make_download_git to update working copy, if --remake is set and it's a working copy.
  • Patched drush_make_download_svn to update working copy, if --remake is set and it's a working copy.
  • Patched drush_make_download_cvs to update working copy, if --remake is set and it's a working copy.
  • Extended DrushMakeProject#applyPatches to read and parse PATCHES.txt and skip previously applied patches

I also fixed a few places where drush_make_error was called with a single argument.

phayes’s picture

subscribe

dman’s picture

Status: Postponed » Needs work

I attempted a re-roll of this from #18, as it sounds great
I tried to get that applied to 6--2 dev. Failed to apply cleanly, with a few broken hunks.
Tried to patch by hand but the patch didn't have context info, so it's a bit hard. Also, the tabs/spacing is off, I can't get that in with confidence that the right thing is happening.
Can we get a re-roll? or at least a patch with cvs diff -up?

achton’s picture

Subscribing.

guddomeNt’s picture

FileSize
21.22 KB

Here's a new patch without any whitespace mumbo-jumbo, rolled against latest cvs.

dman’s picture

Status: Needs work » Needs review
FileSize
21.23 KB

Hey, thanks for that!!
2 syntax error typos, (fix attached)
but after that it seems to do the job pretty well!
I note it works more as a version update than a 'revert' (which I sorta expected) But it is still a great tool for the workflow I think we can use.

I'll play with it a bit harder...

guddomeNt’s picture

Yeah sorry - I didn't test it after patching. Good to hear you got it working.

I note it works more as a version update than a 'revert' (which I sorta expected) But it is still a great tool for the workflow I think we can use.

What do you mean by that?

dman’s picture

I discovered the logic was nice and careful to not re-download stuff that was already downloaded.
I was trying to test it by making some local modifications and seeing if they would be reverted ('rebuild' means that to me - rebuild from scratch).
What I saw was an 'update'. Which is good and all.

I need to work a bit more with it and see how it cooperates with our version control checkouts.

guddomeNt’s picture

I need to work a bit more with it and see how it cooperates with our version control checkouts.

I use git, so I haven't really tested it with svn and cvs, but I did make changes and I believe they should work. With a git repo, it will re-use the checkout and simply update you to the revision listed in your makefile (or HEAD). It'll stash any pending changes first, so you won't lose anything.

What I saw was an 'update'. Which is good and all.

Yes, perhaps "update" would be a better name. But on the other hand, I don't really think there's any reason to distinguish between the initial make and subsequent updates. One command to do either would be simpler. Currently I have a shell script that wraps drush make, to hide the difference.

franz’s picture

I agree with troelskn, 'drush make' itself on an existing installation previously made with drush make should be all you need to trigger an 'update'.

inductor’s picture

Subscribing. This would be a killer feature for drush!

dmitrig01’s picture

so basicaly, this is like drush updatecode, with rcs-specific code too?

I'd like to see ab it more than this. I think that the code should be modified so you can make 2 files on top of each other, and when you maek the second file, it will overrwrite the versions in the first.

inductor’s picture

so basicaly, this is like drush updatecode, with rcs-specific code too?

Exacty. We`re developing a bunch of modules in our svn repository, and I`d like to be able to update them as simply as with plain 'drush updatecode'.

dman’s picture

Yeah, I've been developing our workflow. ..
Using svn exports from our own repositories, pointed at by the makefiles. That is so far being very rewarding.

I was planning to try to use remake to 'freshen' those distributions inline. That's not how it's working right now for me.
I'll be going with periodic full re-downloads of everything it seems. Plus some work with svn checkouts in the middle.
I'll keep pushing the corners and see what I really need.

guddomeNt’s picture

I'd like to see ab it more than this. I think that the code should be modified so you can make 2 files on top of each other, and when you maek the second file, it will overrwrite the versions in the first.

Unless I misunderstand you, that is what this patch will do.

dmitrig01’s picture

hm, i may have misread -- i haven't actually tried it out, sorry for the misunderstanding. I will look into it soon. for now, i'm quite busy w/ schoolwork

inductor’s picture

Any progress so far?

dmitrig01’s picture

Ok. I looked a little bit more about the patch. What I don't really like about it is the fact that it's modifying the download library. The download library is there to download, and do (nearly) nothing more -- and it's quite good at what it does. If possible, it should use the download library to download the newer versions of projects, but I don't think it should actually be modifying the download library,

smls’s picture

subscribe

Wim Leers’s picture

Subscribing.

willieseabrook’s picture

subscribing

drnikki’s picture

subscribe

crantok’s picture

subscribe

perandre’s picture

subscribing!

alozie’s picture

subscribe

areynolds’s picture

subscribe

mxmilkiib’s picture

nt - sub

johngriffin’s picture

subscribe

ablevine1’s picture

subscribe. This seems like a very key feature, especially if you have staging and production environments whose module versions you would like to keep in sync

izmeez’s picture

subscribing

Leeteq’s picture

Title: How to update modules using drush make? » Update modules with Drush make

Subscribing.

Zach Harkey’s picture

Subscribing (You guys are my heros)

Anonymous’s picture

subscribe

Grayside’s picture

Status: Needs review » Needs work

Going by #820954-35: Update core and modules with Drush make, a solution which adds logic to the download library is not acceptable.

mlncn’s picture

I'm subscribing/bumping, but really i'm stalking Grayside.

So it's better for the patch to do exactly what it's doing, but duplicate some code from the download library, rather than modify it? Am i understanding that correctly?

greg.1.anderson’s picture

I am ambivalent about drush remake. On the one hand, I might use it in drush patch #1078108: Drush issue queue commands; on the other hand, I also wonder if it is not better to just do the remake workflow in two-steps:

drush make mysite.make tmp
drush rsync tmp @mysite

The reason I am interested in this is not to turn drush make into drush deploy, but rather, I am interested in simplifying what I commit to my repository for my Drupal sites. It just seems wrong to mirror the releases already archived on d.o. when I could do better and make an easier-to-review history by generating and committing a drush make file, and leave the actual module contents out of my repository.

For modules that are patched, dog looks like it is really the way to go. I am trying to figure out how to make a nice workflow involving the hacked module to automatically detect modified modules, and then generate a makefile that does the right thing vis-a-vis the patches. If the patches come from some issue queue (which you could detect if the module was installed via drush make or patched via drush patch, then just putting a "patch" directive in the makefile would be simplest (and would preserve the source of the modification). For modifications that do not come from a patch, then we could get on the dog sled and make a local repo to store the changes to, and set up our makefile to pull from there. I'm experimenting with a modified variant of drush generate-makefile to do this sort of thing, but have not come to resolution on exactly how it should work or where it should live; ergo, I go slightly OT on this existing thread. Sorry about that. To tie it back in, my point is to explain the workflow where I find drush remake to be interesting & bring up alternate and related projects in that same workflow.

marji’s picture

subscribe

Robin Millette’s picture

subbing

Cyberwolf’s picture

Subscribing.

Bußmeyer’s picture

Subscribe

likewhoa’s picture

Subscribed

helmo’s picture

Status: Needs work » Closed (won't fix)

[ Powered by #1115636: Issue Macros and Templates - Drush Make]

Drush make is being merged into drush core (discussed in issue:#1310130: Put drush make in drush core)
This means that the issue queue is also moving. The Drush project has a component called 'Make' for this purpose.

We would like to take this opportunity to leave behind old/obsolete issues, allowing us to focus on a stable make command in core. E.g. one of the major tasks ahead is making more use of the Drush core code for handling downloads and decompression.

If you feel that this issue is still relevant, feel free to re-open and move it to the Drush queue.

More information will be posted on the drush_make and drush project pages.

smokris’s picture

Title: Update modules with Drush make » Update core and modules with Drush make
Project: Drush Make » Drush
Version: 6.x-2.x-dev »
Component: Code » Make
Priority: Normal » Major
Status: Closed (won't fix) » Needs work

This issue is still relevant, and many users are interested in seeing it happen.

Still waiting on a plan from @dmitrig01.

michaek’s picture

I agree that this issue is still relevant, and it seems like a pretty important feature of drush make. It seems to follow philosophically from "bundle install" in the (Bundler-using) Ruby world. http://gembundler.com/bundle_install.html

moshe weitzman’s picture

FWIW, I am not too interested in this. We are working on integrating the download classes of make so they reuse the caching features of pm-download and pm-updatecode. That means that re-make is going to be a very fast operation. From there, you can diff or rsync or do whatever operations you want. You an even use shell aliases to give short command names to those combo operations. I'm not yet seeing a clear use case for this, despite this being the 64th comment. I'm not a big Make user so I might be missing how folks plan to use this.

greg.1.anderson’s picture

I agree with #63, and propose that we set this to "Won't fix".

michaek’s picture

I can see that it might not be an important use case, but it seems to me like there's value in being able to re-run your makefile in order to add new configuration. I know it's possible to always run make with a new directory, keeping your existing code by rsync-ing it in, but that doesn't feel right to me.

Looking to Bundler as a pattern seems worthwhile - there's a single place where a developer configures dependencies and their versions, and running `bundle install` get the new goodies added to your Gemfile. I think there's some confusion in this issue on rolling in updates automatically (which the above patches seem to be about) and being about to re-run `drush make` to reflect changes in the makefile (which is what the initial issue seems to be about).

I may be missing something, but it also seems like allowing make to re-run is as simple as disabling the check for an existing directory in 319-322 of make.drush.inc, or running `drush make` on '.'. I probably don't know enough of the reason Drush is making that check - is there a reason not to run make on an existing directory?

I, also, am not a big user of Drush make, but I'm trying to work out how to use it sensibly. Thanks!

michaek’s picture

If `drush re-make` is meant to do what I'm describing, then whatever I've said here is pretty worthless. Searching for "drush re-make" pretty much just brings me to this issue, though, so I don't have much perspective on it other than the above!

I'm pretty excited about the caching. Definitely looking forward to it.

smokris’s picture

@michaek (comment #65) :: running make after disabling the check-for-existing-directory doesn't handle the following situations:

  1. removing a module/library/theme from the makefile (the already-installed copy will remain on the filesystem)
  2. updating a module/library/theme if files inside that module/library/theme have been deleted or renamed (the files with the old names will remain --- this issue bugged us a few years ago when date.module was reorganized)
greg.1.anderson’s picture

#67.1 is kind of hard to solve through drush re-make. In its normal mode, drush make will merge in with existing projects on the site; this is kind of necessary if recursive make is to be used.

#67.1 is easy to solve per #63; re-make the whole site from scratch in a clean directory, then you can rsync it with --delete if that's what you want to have happen. (Beware of your 'files' directory.)

We could also implement drush re-make as a shell alias that did those two operations. Trying to have a re-make command that "fixed up" an existing directory structure seems like a really bad idea to me. If the operation failed in the middle, your site would be trashed. Hopefully folks will only re-make scratch sites, but you never know.

mradcliffe’s picture

#67.1 is dirty as hell. You shouldn't have to rebuild your entire site if you're updating a module. That doesn't fit a rapid change model if you broke something.

Here are some valid use cases re #63 and #64:

  • User creates drush make file
  • User runs drush make to build site
  • User changes three module versions in drush make file
  • User runs drush remake to go update said module versions. You might *not* want to go to the latest version, but to a specific one. All in one go.

A better, more ideal use case:

  • User creates drush make file and pushes to version control
  • Site is initially built with drush make
  • User changes 2 module versions or adds a project to drush make file and pushes
  • Automated process issues a drush remake command, which essentially does a drush pm-update, drush dl, git checkout, wget, etc... on projects that don't match version or don't exist.
marianov’s picture

sub

helmo’s picture

[ Powered by #1115636: Issue Macros and Templates - _default]

@marianov: Please do not subscribe anymore. Just click the Follow button next to the issue summary.

michaek’s picture

I think the use cases in @mradcliffe's comment (#69) are well-described. Out of curiosity, are folks here familiar with using Bundler in Ruby development? I really think it's a pattern that's worth looking to for inspiration.

Regarding comments about what "re-make" does and doesn't do - is there someplace we can see either discussion about the scope/functionality or the current source of re-make? As I mentioned in #66, searching for it just leads me back to this thread.

mradcliffe’s picture

@michaek: Maybe a new topic on the Drush working group on groups.drupal.org. Now that this is in the Drush issue queue.

joelcollinsdc’s picture

cross posting this here, i think there is a use case for a feature like this; or else i'm missing something. faster build times are awesome, dont get me wrong...but it doesn't seem to help a common use case.

http://groups.drupal.org/node/207678

Here is my current workflow:
1) download an install profile (i'm trying out openpublic) from git.
2) modify the distro.make (its called something else in OP) so you can build from your local repo.
3) drush make to a folder called build with the --working-copy flag
4) install drupal, do work, make changes to code as well as openpublic.make (for module changes and such)
Now, here comes the crappy part...
5) rebuild with drush make to take advantages of the new modules you added to the openpublic.make file
6) copy in your sites folder to the newly build folder
7) continue working

Has to be a better way, right? This is even more frustrating because you have to keep a repo inside the build as well as globally.

For reference, this is my folder structure

master-profile-repo
builds/
       openpublic-build1/
               (drupal core)
               sites/...
               profiles/openpublic (another copy of the repo)
       openpublic-build2/
               (repeat)
jsagotsky’s picture

+1 for #69's ideal use case. That describes exactly what we're trying to do. I'm going to give the make-then-rsync alias a try, but would prefer something a little less hackish.

Grayside’s picture

I would use the first use case in #69 as part of a deployment script. post-receive hooks are awesome though :) "remake" might be an alias, but not the actual command name.

The alternative to this feature request is probably to run an expensive, complete drush make process on a server somewhere and use that as a deploy source. Avoiding impact on D.O. might mean looking into creating a local drush dl/git clone cache.

sir_squall’s picture

FileSize
22.04 KB

Hi,

I have tried the #23 patch with the last version of drush_make, and i have some conflict during the patching.

Hi have fixed this conflict and redo the patch, is exactly the same witouht the conflict.

greg.1.anderson’s picture

For the record, it is still my opinion that this issue should be closed 'will not fix'. Drush re-make implies #67; if you don't remove items from the site that are no longer in the makefile, then you are not really doing a re-make, which could lead to confusion an use error. If you do remove items, you risk bigger problems.

I think that the comments in #69 are on the right track vis-a-vis what sort of use cases are desirable; however, I think it might be more fruitful to think about modifying pm-updatecode to take a makefile. pm-updatecode could then use the specific version number supplied for each project in the makefile, or default to its usual behavior for projects that do not have a specified version number, or that do not appear in the makefile at all.

DamienMcKenna’s picture

An additional use case that pm-updatecode does not handle: automatically applying patches. I haven't read through the code to know for certain, but I have a feeling that it might be easier to handle as an argument for "drush make" rather than running it as a separate command, bicbw.

DamienMcKenna’s picture

Oh. BTW you can currently do this with Drush 5:

cd ~/Sites/mycoolsite
drush make mymakescript.make .

That updates everything in-place, kinda the way that everyone wants it to work.

DamienMcKenna’s picture

One thing to watch for when using the 'make' command indicated in #80 is that it will, by its very nature, remove many files, e.g. custom modules, features, themes, etc that are not loaded via lines in the make script but which only exist in the repository. Do you think it would be difficult to get it to not remove these files?

dman’s picture

That super-dangerous quirk has now been fixed in dev (though still exists in todays stable)
#1539076: drush make overrides modules folder with downloaded modules

DamienMcKenna’s picture

@dman: Thanks for the update, yay contributors!

DamienMcKenna’s picture

Status: Needs work » Fixed

Per #82 this can be marked as done.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.