Concurrently execute a single drush command on multiple sites

Built-in alias groups are a great idea. Perhaps 'local' should be all of the local sites, 'remote' should be all of the remote sites, and 'all' should be all local and remote sites.

Hot fresh steaming patch for the adventurous.

This defines groups via the 'site-list' item of a site alias (rather than 'group') so that site lists can also be specified on the command line via --site-list.

Modifications to drush sql sync and drush rsync have not been made yet, and I may have introduced a problem with remote site specifications -- but this works okay for local sites. More will be forthcoming in the ensuing weeks.

Update: Look but do not use; as suspected, this patch does not work quite right.

I have considered automatic detection of multi-sites, but I'm not sure how to best do it. Multiple roots are common; people have d5, d6 and/or d7 sites on the same machine, and I, like jorgbert described in Should I use Multisite or Stand Alone Installations?, put my dev sites at a separate document root. I considered adding an option that contains a list of Drupal root folders as one possibility. Searching siblings of the Drupal root for other valid Drupal installations. If we can find all of the document roots, then we could make a group alias for all of the local sites automatically.

You are right that config file options should not be passed along to the backend invoke call. Only options from the command line, and options from the active alias should be passed along. The alias options are important for the function you recommend in #658420: helper functions for drush_backend_invoke(), I think. After I fix this function, maybe these two routines can eventually become the same thing.

Here is a new patch that fixes the problems from #4. This should work well, but is still for the brave. At this point there's still some chance that you might execute commands on the wrong site, so only use it on scratch systems. Running with -s first is also advisable.

Regarding the issue mentioned in #5 and #6, see my comments there regarding code unification.

Thank you. My replies:

1. Yes, I misunderstood you in #6 above. I realized my mistake, and although I forgot to say anything about that, the current version of redispatch does pass along the config file options, just as your version does. If the command cannot be found locally, though, this step is skipped and you only get the command line args. Now, as you mention, there might be some instances where you don't want to pass the config items along to a remote machine. For example, if you had config settings set one way on the local machine, and another way on the remote machine, passing the config options in $data "promotes" them to the 'options' context, which causes them to take precedence over the config options on the remote machine. Overall, at the moment I think that the benefits to passing the config variables outweighs the potential downsides, but if there was some way to flag certain command line args to get them to be processed in a lower-priority context, that might be better still. I'll mull this later.
2. I'll remove the underscore. I also moved these functions to includes/drush.inc
3. Yes, that's a kinda-sorta bug. The site alias code does this too--pops just one element, even though the command might take up two elements. Even though the string goes in wrong, backend invoke still generates the correct command, so it's hokey, but it works. My recommendation for un-hokeying it is to not pop at all; see #658432: drush_backend_invoke() assumes single word command names #3. You could also get the $command object to find out exactly how many elements you needed to pop, but if you did that, you'd be stuck if you were trying to call a remote command that was installed on the remote machine, but not on the local machine. I think that just having a flavor of backend invoke that does not specify what the contents of $args is is better.

Design update - derived site lists

It occurred to me that using site lists in both the source and destination of drush sync and drush sql sync would be very useful if you could guarantee that the order of the sites in the source and destination lists were aligned. Then you could, for example, do something like this:

drush sync all-live all-dev
drush sql sync all-live all-dev
drush updatecode all-dev
drush updatedb all-dev

Then, after you test all of your upgrades, you could push them back to the live site. (The scripts I currently use to publish are a little more involved, as they clear caches and specify some skip tables, but you get the general idea.)

So, we need a way to easily specify groups of machines. One way to do it is to pull in all of the groups from the filesystem, by finding all of the multi-sites at a given drupal root:

$options['site-aliases']['all-live'] = array(
  'site-search-path' => '/srv/www/drupal-live-sites'
);

If you wanted, you could have multiple drupal roots inside the search path; if you do, drush would concatenate all of the multi-sites found at each document root.

Now, naively some might think that we could define another list for 'all-dev' using the same construct, and we'd be done. Unfortunately, doing that would be a bad idea, because you cannot always guarantee that the order of sites pulled from an unknown filesystem will always be stable. To improve this a little bit, I could sort the sites alphabetically after the group is constructed, but this still seems fragile to me. Therefore, drush will flag site lists created by listing all multi-sites, and will issue a warning (or fail with an error?) if two are used together.

Instead, the right thing to do is to define a derivative site list, which is a site list that is created from some other list. While the design is not finalized, one way to do this would be to define parameters that are passed to preg-replace:

$options['site-aliases']['all-dev'] = array(
  'parent-list' => 'all-live'
  'uri-pattern' => '/http:\/\/(.*)/';
  'uri-replacement' =>'http://dev.${1}';
);

Drush could then insure that the sites defined in the derived site lists were always in the same order as the sites in the parent list.

Facilities for transformations on the root using filesystem APIs would also be useful. Any comments or suggestions are appreciated.

(Edited: changed 'group' to 'site list' in above description)

Site specification equivalents for filtered site lists

It is convenient to have site specification equivalents for site lists and site searches. For example, the "all-live" alias in #10 can also be represented as /srv/www/drupal-live-sites#all. Either of these constructs will search the filesystem for 'settings.php' files and return a list of site specifications for the sites thus found.

One difference between the "all-live" alias and /srv/www/drupal-live-sites#all is that the site list alias may contain a list of drupal roots, whereas the site specification can search from but a single path. The specified path does not have to be a drupal root, however; if you search for /srv/www, then all drupal roots under that path will be searched for sites. Another shortcut I am planning is simply "#all". The #all specification will build a site list alias that searches all drupal roots defined by the 'site-search-list' global option. If there is no such option defined, then the current drupal root (defined by -r/--root, or found from cwd) is used.

Currently, the site specifications mydrupal.com and #mydrupal.com both refer to the site specified by the settings.php file found in the folder 'mydrupal.com' in the 'sites' folder. I am considering changing the definition of the second form to search for 'mydrupal.com' in all of the sites returned by #all.

n.b. There may be more than one site that matches (d6 requires copies of sites to have the same site folder name); for example:

$ drush sa /srv/www#all
/srv/www/drupal-dev-sites#mydrupal.com
/srv/www/drupal-live-sites#mydrupal.com

If #sitename is used in contexts where a single site is needed, then the first site from the list would be used, or the site that is stored at the drupal root specified by -r/--root or cwd, if a drupal root has been found.

Changing #sitename to do a search becomes interesting when filtering is added. If the specification #mydrupal.com/dev is provided, then the #mydrupal.com list is filtered such that it only contains items whose drupal root contains "dev". This also means that #all/dev is all of your dev sites, and #all/live is all of your live sites. This would allow:

$drush sql sync #all/live #all/dev

The problem with this is the same as #10 -- the order of this sites might be all higgley-piggley, and in this case, derived site lists are not a convenient solution. I am therefore considering sorting the site lists by some heuristic. In D6, we can pretty much count on the fact that the site folder name must be the same, so aligning the list by the site foldername will cover all but a small number of D6 users who fix up the database post-migration. In D7, though, there will be no such restriction, and there will likely be more people with dev sites with site folders with different names.

If the heuristic worked, though, it would be possible to do something similar to sync a series of remote sites locally. For example, if 'all-live' is a site list of aliases to remote live systems, you could pull them all to your local dev sites via drush sql sync all-live #all/dev (this is an alternative to using an all-dev site list that is manually maintained, or defined as a derived list).

Questions:

1. Is anyone put off by the idea of parsing the uri fragment to get the filter component, as described above?
2. Is filtering by the drupal root likely to be a good separator for different kinds of sites stored on the same system (e.g. dev and stage)?
3. Are there better or additional heuristics that could be used to order site lists (especially in the D7 era)?
4. What do you all think of a "site clone" command? It couldn't configure vhost conf, and editing settings.php might be hard, but if the user did these steps, the clone command could insure that the setup matched the heuristic...

Requiring manual maintenance of site lists / derived lists would avoid these problems, but would require more setup time on the part of the user.

Yes; allowing multiple sources and multiple destinations to sql sync and sync introduces all sorts of complexities. I think that for now I'll go back to disallowing that, and just try to get a system as described at the top of this page finished and committed. I can revisit #10 and #12 later if warranted.

Here is a patch that does everything described in #0, including drush sync \!drush all-dev:\!drush.

There is no "all" or "#all" or "remote" or "local" aliases yet, but /drupal/root#all does work. I didn't take it any further than that for now, pending better clarity on how to handle the issues that arise when we start walking down the path of #10 and #12, above.

The "site alias" command basically works, but needs a little more love to keep up with some of the changes made here.

When you run a multiple-target command, the current code gives you a warning and prompts you what to do, including an option to run once in simulated mode. Running in simulated mode is a great idea for sql sync and rsync, but be aware that many other drush commands ignore the -s flag, so be careful. There currently is no way to skip the confirmation message, but do not fear, I only coded it like that to protect users during these early development periods when code errors are likely. Once this feature is better tested, I will convert these prompts to simple "drush confirm" style warnings that will obey the -y flag.

Review is welcome, but this patch needs more work before being committed.

Here is an update that fixes some bugs in #15, and also adds the capability to drush rsync and drush sql sync from multiple targets to multiple targets.

A very simple form of derived site lists is supported:

$options['site-aliases']['all-live'] = array(
  'site-list' => array( 
    'user@isp.com/srv/www/drupal#greenknowe.org',  
    'user@isp2.com/srv/www/drupal#edit.dickensfair.com',  
    'user@isp3.com/srv/www/drupal#westkingdom.com',  
  ),
);

$options['site-aliases']['all-dev'] = array(
  'from-list' => 'all-live',
  'remote-host' => '',
  'root' => '../drupal-dev',
);

The regexp ideas of #10, and the filtering ideas of #12 are unimplemented and probably shelved. I'll probably add simple replacements to derived site lists, though, so that you can compose the root from the site. We'll see; it might be just as well to yank derived lists. It's not too bad maintaining the lists by hand.

Code still needs testing and work, but it's getting closer.

Happy New Decade.

There are in total four places where drush will redispatch to itself or some remote drush instance.

1. In drush_remote_command: If the --remote-host flag is specified directly, or set via an alias or a site specification with a remote host component, then a single redispatch is done to a remote drush instance (pre-existing).
2. In drush_do_multiple_command: Called by drush rsync and drush sql sync, this routine will redispatch to the local drush instance once for each source/target pair, which must be in alignment.
3. In the inner loop of drush rsync: If there is a single source and multiple targets, then the single source is rsync'ed to each target in turn.
4. In the inner loop of drush sql sync: If there is a single source and multiple targets, then the source will be dumped once and then pushed to each of the targets.

The routines in #1 and #2 look pretty similar, but they are actually different enough that they cannot be unified.

In the case of #3, this code can be simplified by slightly enhancing case #2. I'll make this improvement eventually, but #3 was written before #2, and I haven't gotten there. Once I clean this up, there will only be a single check for multiple targets in the rsync command.

In the case of #4 and #2, it is actually necessary that these cases be handled individually. Having the separate code in #4 allows sql sync to dump the database once; if it just called through to #2, then the database will be dumped every time through the loop instead of just once at the beginning. If the 'cache' and 'dump' settings are initialized correctly, this wouldn't be such an issue, but it seems like the extra code is worthwhile.

Regarding the all-dev example:

$options['site-aliases']['all-dev'] = array(
  'from-list' => 'all-live',
  'remote-host' => '',
  'root' => '../drupal-dev',
);

'from-list' causes this site list to start off by copying all of the items in the from-list, which in this case is 'all-live'. The 'remote-host' parameter replaces the 'remote-host' items in all of the elements of its list. Since 'remote-host' is empty, it removes the parameter. Similarly, the 'root' item adjusts the root element of every item in the list. If it is an absolute path, then a replacement is done. If it is a relative path, then it is appended to the root of each item in the list and then simplified.

Here is the output of drush sa for all-live and all-dev:

# drush sa all-live --full
$options['site-aliases']['greenknowe.org'] = array (
  'remote-user' => 'root',
  'remote-host' => 'reetstreet.com',
  'uri' => 'greenknowe.org',
  'root' => '/srv/www/drupal',
);
$options['site-aliases']['edit.dickensfair.com'] = array (
  'remote-user' => 'root',
  'remote-host' => 'reetstreet.com',
  'uri' => 'edit.dickensfair.com',
  'root' => '/srv/www/drupal',
);
$options['site-aliases']['westkingdom.tekazu.com'] = array (
  'remote-user' => 'root',
  'remote-host' => 'reetstreet.com',
  'uri' => 'westkingdom.tekazu.com',
  'root' => '/srv/www/drupal',
);
# drush sa all-dev --full
$options['site-aliases']['greenknowe.org'] = array (
  'uri' => 'greenknowe.org',
  'root' => '/srv/www/drupal-dev',
);
$options['site-aliases']['edit.dickensfair.com'] = array (
  'uri' => 'edit.dickensfair.com',
  'root' => '/srv/www/drupal-dev',
);
$options['site-aliases']['westkingdom.tekazu.com'] = array (
  'uri' => 'westkingdom.tekazu.com',
  'root' => '/srv/www/drupal-dev',
);

This needs some documentation, but as you can see, it wouldn't be too hard to set up your local dev sites to mirror live sites located on various different servers with this feature. Another alternative, though, might be to just provide some php functions that could be called in code to copy a list and then modify the items in it. Just drop some functions in your drushrc.php file. These routines would have to be specially coded to look for sites and site lists both in $options and in the default context, which might be a little bit inconvenient. (Maybe each routine could start by copying the appropriate options to the default context; that would be workable.)

Would there be any objection if I committed this in-progress implementation? It seems to work pretty well, warns people before they use it, and shouldn't affect anyone who doesn't try to use it.

I have some other issues I need to work on that involve global changes, and I don't want to have to deal with the source merge afterwards.

Committed an updated version of this code with a little more testing and a little more fixing. The attached patch is for reference only; this has already been applied to drush-HEAD.

Reviewers can try out this code to see how it works. The current version prompts before execution; selecting "simulated" on the first run is a good idea. Once this gets some more testing and feedback I'll replace the multiple-choice question with a yes/no that respects the --yes flag.

Setting this to 'active' since the patches above have been committed, and perhaps the only changes to come will relate to simplifying the confirmation messages later.

@anarcat: This feature benefits people who call drush from the command line. If calling drush from a script (or a system like Aegir) that already manages lists of sites, there isn't much difference between passing the list as a drush multi-site arg, or just calling drush N times. There is some optimization provided in the case of using sql-sync to copy from one source site to multiple destination sites, but that is a rare enough situation to be discounted.

Simplify warning message, taking out the 'experimental' nomenclature and replacing drush_choice with a simply drush_confirm.

Concurrency wouldn't be too hard to add. I think this is only necessary in drush_remote_command; I just need to look up the php APIs for threading and semaphores (references welcome; will be asking my friend Google later).

Pseudocode:

// (A) Create  a semaphore with N resources, where N == max # of threads
// (B) Create a semaphore with 1 resource (mutex)  to control screen output
foreach ($site_list as $site_spec) {
    // (C) get a resource from semaphore (A), blocking the main thread if all are used
    // (D) fork a new thread
    if (fork()) {
      $values = drush_do_site_command(_drush_sitealias_get_record($site_spec), $command, $args);
      // (E) Grab the mutex from B, blocking the current thread if it is not available
      drush_print($values['output']);
      // Release the mutex from (E)
      // Release the resource from (C)
   }
}

Then, the example from #32 can be done as follows:

drush all updatedb

(Renaming and keeping this issue active rather than starting a new one for the benefit of the current subscribers.)

I presumed that concurrent rsync would not be terribly useful, as you would quickly become bandwidth-limited, and might as well do them serially anyway. It could be supported if that were not the case for some.

I recognize that on some systems, multiple updatedb's would be a bad idea. I always upgrade on a different machine than the live server, but you probably wouldn't want more than one updatedb running at the same time if your dev sites were on the same machine as your live sites. I therefore think that the default should be unthreaded. With a little bit of wrapping, the above pseudocode could fall back to single-threaded behavior when the forking functions are not available.

Coding the forking semaphore to be load-based rather than number-of-threads based would be hard, because in the example above, you fork a bunch before you know how much CPU time your commands are going to use. I'll think about it and see if I can think of a simple way to adjust the thread count dynamically, but in any event, once a thread gets going, it's going to run to completion. It would be better, I think, to experiment with different values for number-of-threads and just pick the right threadmax for your particular situation.

Thanks for the pcntl_fork reference.

Yes, serial execution will be the default.

A few minutes with my friend google shows pcntl_fork as the only obvious method to natively fork in php. Are there other ways to fork from php?

Another option would be to not fork from php, but to instead use backend invoke to essentially use the shell to fork: sh -c "drush <command> <args> &". I'd have to do some research to see how that would interact with proc_open. You'd have to let backend_invoke return right away, and keep an array of open processes that you polled to get the output. I'm not even sure if that would work right; in theory it would. It would be more complicated than the pcntl_fork solution, and I have to say I'm not too excited about it.

I'm certainly willing to consider alternate proposals that would be more widely available and also as easy to implement, but barring any better ideas, I'll probably go the pcntl_fork route. That would benefit those who were fortunate enough to have access to the pcntl functions, and would be easy to implement. An alternate implementation that supported Windows could perhaps come along later.

(Note: removing the "drush-3.0" tag, as the patch that this tag initially applied to has been committed, and concurrency will definitely be post 3.0-stable.)

That is a lot more complicated than pctl_fork, but with all that useful sample code, we may have something. Thanks.

@carlos8f: The drush multisite feature has been committed to drush-HEAD. Your version looks nice and has some improvements over some of the existing handling for finding multisites. Patches against includes/sitealiase.inc would be welcome. In particular, see function _drush_sitealias_find_record_for_local_site() and function _drush_sitealias_check_sitelist_fields(). The former defines 'current' as an alias to the current bootstrapped site. In function _drush_sitealias_get_record(), /path/to/root#all means all of the multisites at a given root, so you can say:

drush /path/to/root#all update

That would call 'update' on all of your sites at the specified path.

Note, however, that the alias code is about to go through a big overhaul, so it might be a frustrating time to submit patches (or exciting, depending on how you look at it.) See #727058: Site aliases should not be cached in the options context and others.

@carlos8f: You've hit the nail on the head. _drush_sitealias_check_sitelist_fields() goes partway by defining a concept of a Drupal root search path, and it's possible to specify #all for one particular root, but there is no provision at the moment to put the two together, or to figure out (say, based on cwd or -r) which root 'all' should apply to. Since this is just a "half feature", it's not documented anywhere.

If you want to work on this, it would be best to put it in a separate issue.

@dbt: Don't worry; if you don't need or want concurrency, you won't need to use it. The default will be one at a time, and you'll need to change the configuration option to make it do more than one at the same time. It sounds like you're wondering about the "all" shorthand, which is currently /path/to/drupalroot#all. This is going to be removed. I may replace it with a relative alias, e.g. /path/to/drupalroot/@all.

As far as updating is concerned, though, I prefer to put all of my sites in separate drupal roots so that they are independent vis-a-vis module and core updates. If you are updating multisite installations, though, you'll want to do them more or less all at once, because as soon as you updatecode on one, you've done an updatecode on all, and all are going to need an updatedb. In this respect, drush /path#all update doesn't quite do the right thing; what you want is drush mysite.org updatecode && drush /path#all updatedb. But again, #all is vestigial.

1. I expect this will work.

2a. Why is it important to update core only once? Is it transmission time? If so, you could rsync one site to the others. Not saying you have to do this; a multisite install is perfectly valid. I just find single-drupalroot sites more convenient.

2b. If you think there are additional error checks or confirmations that updatedb should be doing, please create a new issue and specify exactly how you think it could be improved. It would also be helpful if you posted your script at the same time.

I thought that Drupal itself assumed that the anon uid had to be zero, and would have problems if it is not. In contrast, I don't think any drush commands will fail if the anon uid is not zero. Did you mean to say Drupal / drush instead of drush / drush? How are you getting non-zero anon uid anyway? I would imagine that would be a very rare circumstance.

What does it say when you substitute status for updatedb? Try adding -d and see if there is any more useful information provided.

I realized suddenly in the dead of night that this is a duplicate of #831272: '@sites' alias doesn't work unless drush can find at least one local site at the drupal root, a bug I had forgotten about.

n.b. I fixed and committed #831272 yesterday evening; your problem in #65 should be solved if you update to the 4-August or later drush-HEAD.