Study fabric and build similar on top of backend.inc

I took a cursory glance at Fabric; some of the primary differences between Fabric and drush seem to be:

1. Fabric allows you to call any function from the command line, whereas drush defines named commands that can be called.
2. Fabric allows you to execute on any host or any number of hosts, whereas drush site aliases allow you to execute on a single remote host.
3. Fabric contains utilities to execute blocks of code under certain contexts (e.g. disable messages, etc.), whereas drush site aliases allow you to set particular command-line options when that alias is used (with some capability to specify contexts where the command-line options are set).

What particular features of Fabric are you looking to add to drush? Do you want to be able to execute a particular command on multiple machines, or is one machine at a time sufficient? Do you want to be able to separate contexts from site aliases (e.g. drush mysite.com context installcore)? Are there other useful Fabric features I didn't notice that you'd like to see in drush?

The things I'm most interested in trying next include: installcore on local and remote machines (try what is already there), make publish and merge commands to handle forked sites (simple examples, like edit content on one site and re-merge with updated user and comment data on another site, and check for differences between sites), manage groups of modules and their settings to define features (c.f. Drupal features project) for both local and remote machines. It does seem like we have the building blocks ready, and we just need to do some integration work. Are there specific features of Fabric that will make these sorts of things easier to build?

Looks interesting for sure!

Installing drush on remote machines would also be pretty interesting, from a bootstrap point of view...

I was impressed with the ability to specify multiple hosts in a single command. I'd like to see that in drush.

Otherwise, I think the packaging of Fabric is impressive. drush_backend_invoke() does most of the same stuff but has none of the documentation/polish. Fabric is nearly a ready to use solution; backend_invoke() is a backend component. I'm not sure how we do it, but we need to highlight all of the apps that are being built and can be built with drush as a backbone. Perhaps we start with Aegir.

I created a new issue, #628996: Execute a single drush command on multiple sites, for the feature request to specify multiple hosts in a single command. Please add any comments on how that feature should work to that issue, and continue other Fabric-related discussion here.

FYI, installcore (soon to be renamed) does work on remote hosts. I tested that.

@Owen - Installing drush on a remote host is as easy as drush sync \!drush stage:\!drush. this rsyncs drush dir to a host named in your 'stage' site alias. it doesn't set the executable bit which is a small annoyance. We're getting spoiled by drush.

installcore on a remote host -- awesome. I thought that would probably just work.

Note that drush sync \!drush stage:\!drush might be held up by #634720: Buglets with sql sync and sitealias, which I will fix as soon as I can (but there will likely be some delay).

Perhaps there should be a way to define contexts inside of a drushrc.php file, and specify them easily from the command line, or perhaps define a mapping from directory names to contexts for automatic selection based on the cwd. In the later case, it would become possible to apply a context to an entire directory tree without introducing the need to walk from cwd all the way up to the document root to find it.

I think this would be a better solution to some of the issues discussed in #599698: Load a drushrc.php from cwd.