moshe suggested in #1342652: Allow shell aliases to be defined in more than one drushrc.php file, or in site aliases. that we support replacements such as %root in shell aliases. I was contemplating something similar myself. One I thought would be useful would be to replace "@self" with the target alias, if any.

For example, drush @mysite site-push might be implemented by a site alias 'site-push' => '!drush rsync @self @peer && drush @self updatedb'. I haven't tested to insure that the compound command syntax is correct there, but the idea is that both instances of '@self' would be replaced with '@mysite'. Presumably, the alias @peer would be defined in @mysite, or alternately, if the alias was defined in @mysite, it might refer to the target site directly. I think that replacements in shell aliases would give us a lot of power to easily build interesting workflows not easily replicated by drush scripts or drush commands. For example, 'site-push' might use git for some sites, and rsync for others.

There are probably good use cases for using %root and %file replacement as well; however, in some cases it might not be desirable. For example, consider 'site-pull' => '!drush sql-sync @peer @self && drush rsync @peer:%files @self:%files to bring the database and 'files' directory down from the peer (live or stage) site to @self (dev). In this case, rsync will already do replacements on @peer:%files and @self:%files, which might be different paths. If you wanted site-pull to bring down files from git, though, then it would be nice if drush did the replacement prior to executing the shell commands.

To allow for both kinds of use cases, perhaps we need to make the replacement be a little more cumbersome, such as {%files} or even {{%files}}, to avoid the complexity of escaping and the risk of undesired replacements.

Comments

greg.1.anderson’s picture

Issue tags: +Release blocker

I'd really like to see this in drush-5 if possible. The code isn't hard if we can decide how it should work.

I think that replacing @self is dubious; there might be some cases where a literal '@self' is still desirable. Perhaps we should define a replacement {{%target}} or {{%target-site}} that evaluates to the alias provided before the command (or '@none' if none given).

moshe weitzman’s picture

As for a replacement syntax, I think {{%target}} is fine.

I'm a little ambivalent about this though. Complex workflows like site-pull may want a Help page with options and so forth. At that point, a command is better. I'm curious to hear more about 'not easily replicated by drush scripts or drush commands'

greg.1.anderson’s picture

The part that is not easily replicated is selecting your workflow on a per-site basis. You might push some sites with git, and just rsync others (scratch sites). I have some sites where I'd like to experiment with using Postgres journal files (postgres calls them something else) with point-in-time recovery instead of sql-dump / sql-sync (should be way faster for large databases). You might have some operations that you always want to perform when you pull stage down to dev (for example, turn on devel). If site-pull is a command with options, then it gets rather complicated as it slowly grows to meet all workflows. My thought is that you make commands like sql-sync and prepare-devsite (turn on devel, etc.) with help and options, and then call the sequence of commands appropriate for your site with a shell alias that is customized on a per-site basis in an alias file (maybe with a default alias in drushrc.php for your default behavior). We can have a few examples in the drush documentation on some common ones. Simple workflows can be expressed with just a few commands; as things get more complicated, then the alias can be changed to call through to a custom command if desired.

To use commands like sql-sync, we need to replace {{%target}}. To use git and other shell commands, we need to replace {{%root}} and {{%files}}. Not sure what other replacements are necessary, but including everything in path-aliases would be a good start.

moshe weitzman’s picture

With all due respect, I think that only drush maintainers and their next of kin will honestly use workflows that have the same name but vary under the covers depending on a site-specific strategy. I mean, even a consulting firm like Palantir (et al.) will (ideally?) manage all their client deployments with a single strategy.

Even so, this probably doesn't cost us that mucvh code or complexity so lets see the patch and evaluate.

greg.1.anderson’s picture

Status: Active » Needs review
StatusFileSize
new4.73 KB

:D My progeny will thank you, then.

Perhaps no one will use multiple workflows for the same command name, but different users might pick the workflow that suits them best for commands such as site-pull and site-push. If it is ever useful to chain together multiple commands in a shell alias, then it is useful to support replacements inside shell aliases.

Here is a patch. I tested it with the following shell aliases:

$options['shell-aliases'] = array(
  'test' => '!echo {{@target}} {{%boo}} {{%root}}',
  'site-pull' => '!drush sql-sync {{#peer}} {{@target}} && drush rsync {{#peer}}:%files {{@target}}:%files',
  'site-push' => '!drush rsync {{@target}} {{#peer}} && drush {{#peer}} updatedb',
);

I used these shell aliases with the following site alias:

$aliases['dev'] = array (
  'root' => '/path/to/drupal',
  'uri' => 'mysite.org',
  '#peer' => '@mysite.live',
  'path-aliases' => array (
    '%boo' => '/usr/local/boo',
  ),
);

Why '#peer'? Because site alias elements whose keys begin with '#' never propagate in backend invoke. Why {{%boo}}? Becuase path aliases by convention usually start with %. Why {{@target}}? To differentiate from {{%target}}. That gives us a fair number of possible leading characters in shell alias replacements, but I think they all make sense based on the source of where they came from. {{@target}} is the special one; it could be {{#target}} or {{%target}} if desired; that would give one fewer valid leading characters in the available replacement patterns.

Something of note in drush_core_execute:

-    $args[$x] = drush_escapeshellarg($args[$x]);
+    // escape all args except for command separators.
+    if (!in_array($args[$x], array('&&', '||', ';'))) {
+      $args[$x] = drush_escapeshellarg($args[$x]);
+    }

I omit escaping command separators, &&, || and ;, so that compound commands will actually work. I'm not sure if there might be other special shell sequences that would also need similar treatment.

There are a couple of minor bugfixes in here as well. I will continue with tests, but not sure if I'll have them ready to post tonight. Will put them up soon, though.

greg.1.anderson’s picture

StatusFileSize
new7.61 KB

Here's a patch with tests that pass. There's one failing test -- site upgrade. I'll look into whether that is related to this issue or not next.

greg.1.anderson’s picture

Status: Needs review » Needs work

Looks like the failing test is caused by this patch. Will investigate that next.

moshe weitzman’s picture

Code looks good.

I don't know how common or comfortable it is to reference a site alias from a site alias (your example is '#peer' => '@mysite.live'. We might have to explain in docs when thats useful.

Needs docs, and then I think this is RTBC (pending all tests pass).

greg.1.anderson’s picture

Status: Needs work » Reviewed & tested by the community
StatusFileSize
new0 bytes

Doing a pull and rebase seems to have cleared up the failing test. No code changes in this patch other than that; will work on docs and commit.

greg.1.anderson’s picture

StatusFileSize
new8.22 KB

Whoops, here's a non-empty version of the above patch file.

greg.1.anderson’s picture

Status: Reviewed & tested by the community » Fixed

Committed bbe039f, 71a9ca2 and 63b2a4e to master; includes tests and docs.

msonnabaum’s picture

Sorry for commenting late here, but although I *really* like the basic idea here, I'm worried about the complexity.

The peer stuff is most confusing to me. I would almost rather we left it out until it was argued for by users, but it doesn't appear to add much complexity to the code, so I'm torn.

moshe weitzman’s picture

I also am a bit troubled by that. I think the word peer is unusual. I would prefer to use @dev, @stage, @prod as examples.

jdonson’s picture

The nomenclature-by-environment makes sense, naturally. *Some simple network diagrams would also clarify these client-server patterns.*

Are there any logical models of client-server drush architecture?

'Peer' creates some additional client-server architectural options that remain somewhat unclear.

Drupal instance metadata [local | remote | peer? ] should prolly be clarified a bit first?

greg.1.anderson’s picture

Title: Support replacements in shell aliases » Simplify documentation for shell alias replacements
Component: Base system (internal API) » Documentation
Category: feature » task
Status: Fixed » Active

I will work on simplifying the documentation for shell alias replacements.

* There are simpler use cases for replacement; I'll find something appropriate to use in the basic documentation. pull-files as an alias for cd %files followed by git pull might be a good candidate.

* I will avoid using the 'peer' nomenclature, and will try to make the examples use concrete terms (@dev, @stage and @prod) instead of the abstract #peer.

* I'll document the push and pull examples in a drush topic, and keep this complexity out of the base examples.

moshe weitzman’s picture

Assigned: Unassigned » greg.1.anderson
moshe weitzman’s picture

Status: Active » Fixed

I replaced the #peer stuff with #live and hopefully clarified a tiny bit. Calling this fixed for now. Anyone is welcome to do more.

Automatically closed -- issue fixed for 2 weeks with no activity.