Adding context information for @sites output

Hmmm. Good point. I don't really know what the answer is. The primary user of --backend has been Aegir in the past and they don't give a hoot what prints out to interactive users.

This may lead nowhere, but one thought on my mind is ....

If you run this command with --debug, you will see the actual commands that drush issues to each site. They all end with --backend which means that drush will return all the output in a JSON array (enclosed by a string delimited for easier parsing. can be overridden). Thats an ideal return format for script consumption (especially if vget's 'output' were more structured. perhaps we would not a site_alias name in there too). For example, here is the command on my system. Note the real vget values are in first element called 'output'.

~/htd/h$ /opt/local/bin/php /Users/mw/contributions/modules/drush/drush.php  --php=/opt/local/bin/php  --uri='default' --root='/Users/mw/htd/h' vget 'site' --backend

DRUSH_BACKEND_OUTPUT_START>>>
{
    "output": "site_403: \"\"\nsite_404: \"\"\nsite_default_country: \"\"\nsite_frontpage: \"node\"\nsite_mail: \"admin@example.com\"\nsite_name: \"Site-Install\"\nsite_slogan: \"\"\n",
    "object": [],
    "error_status": 0,
    "log": [{
        "type": "bootstrap",
        "message": "Drush bootstrap phase : _drush_bootstrap_drush()",
        "timestamp": 1281060937.36,
        "memory": 3059952,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Loading drushrc \"\/Users\/mw\/.drushrc.php\" into \"user\" scope.",
        "timestamp": 1281060937.36,
        "memory": 3069872,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Loading drushrc \"\/Users\/mw\/.drush\/drushrc.php\" into \"home.drush\" scope.",
        "timestamp": 1281060937.36,
        "memory": 3075264,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Drush bootstrap phase : _drush_bootstrap_drupal_root()",
        "timestamp": 1281060937.41,
        "memory": 5408072,
        "error": null
    },
    {
        "type": "notice",
        "message": "Initialized Drupal 7.0-dev root directory at \/Users\/mw\/htd\/h",
        "timestamp": 1281060937.42,
        "memory": 7160608,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Drush bootstrap phase : _drush_bootstrap_drupal_site()",
        "timestamp": 1281060937.51,
        "memory": 7683304,
        "error": null
    },
    {
        "type": "notice",
        "message": "Initialized Drupal site default at sites\/default",
        "timestamp": 1281060937.52,
        "memory": 7855640,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Drush bootstrap phase : _drush_bootstrap_drupal_configuration()",
        "timestamp": 1281060937.58,
        "memory": 7959480,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Drush bootstrap phase : _drush_bootstrap_drupal_database()",
        "timestamp": 1281060937.63,
        "memory": 7970232,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Successfully connected to the Drupal database.",
        "timestamp": 1281060937.63,
        "memory": 7971424,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Drush bootstrap phase : _drush_bootstrap_drupal_full()",
        "timestamp": 1281060937.74,
        "memory": 8834928,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Drush bootstrap phase : _drush_bootstrap_drupal_login()",
        "timestamp": 1281060938.01,
        "memory": 27459744,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Found command: variable-get (commandfile=variable)",
        "timestamp": 1281060938.08,
        "memory": 28456808,
        "error": null
    },
    {
        "type": "bootstrap",
        "message": "Initializing drush commandfile: user",
        "timestamp": 1281060938.08,
        "memory": 28461768,
        "error": null
    },
    {
        "type": "notice",
        "message": "Command dispatch complete",
        "timestamp": 1281060938.09,
        "memory": 28462088,
        "error": null
    },
    {
        "type": "memory",
        "message": "Peak memory usage was 27.92 MB",
        "timestamp": 1281060938.09,
        "memory": 28460648,
        "error": null
    }],
    "error_log": [],
    "context": {
        "php-notices": "warning",
        "structure-tables": {
            "common": ["cache", "cache_filter", "cache_menu", "cache_page", "history", "sessions", "watchdog"]
        },
        "skip-tables": {
            "common": ["migration_data1", "migration_data2"]
        },
        "script": {
            "script-path": "\/Users\/mw\/.drush\/scripts"
        },
        "config-file": "\/Users\/mw\/.drushrc.php",
        "context-path": "\/Users\/mw\/.drushrc.php",
        "script-path": "\/Users\/mw\/.drush\/scripts",
        "php": "\/opt\/local\/bin\/php",
        "uri": "default",
        "root": "\/Users\/mw\/htd\/h",
        "backend": true
    }
}
<<<DRUSH_BACKEND_OUTPUT_END~

Looks like our old friend Fabric does just as sime suggests. See the first example there. We should do what they do, IMO. I'm interested in hearing from adrian and greg on this.

@sime: Rather than running with the --backend argument, you might instead try to write your code in php and make it a drush command. Then you could call drush_do_site_command to get the results that you wanted. For example, drush gets the database info from a remote machine like so:

$values = drush_do_site_command($alias_record, "sql-conf", array(), array('db-url' => TRUE));

Rather than using a site list for $alias_record, you could loop through your sites and call drush_do_site_command for each one. (I don't think that drush_do_site_command would necessarily do the right thing with a site list in any event.) Note however that only a couple of drush commands provide nice values usable with the above wrapper; if you want access to the whole data structure that moshe references above, you'll need to call drush_backend_invoke directly. See the implementation of drush_do_site_command for an example of how to do this. You'd then have to take the 'output' ASCII and parse it. You could also submit feature requests to have pm-info & c. behave like sql-conf and insert their result objects into the backend invoke results.

Regarding grepping for 'enabled', you could also pass --status=enabled to get only the enabled projects.

Finally, as to the original request, I think it would be reasonable to have drush prepend the site name to each line of the output as shown in #0. I might prefer to have the site name separated from the rest of the output by some special character or characters; for example:

emspace.com.au   >> views  6.x-2.10   enabled
other.com.au     >> views 6.x-2.10   disabled

Note that in some cases, you might have two sites with the same uri; for example other.com.au on the dev machine, and other.com.au on the live server. In that case, I might do something like this:

emspace.com.au                  >> views  6.x-2.10   enabled
localhost#other.com.au          >> views 6.x-2.10   disabled
live.isp.com.au#other.com.au    >> views 6.x-2.10   disabled

In other words, if there are two sites in the @sites list with the same uri, then the remote-host will be prepended.

I would expect that putting the machine name in front of the output would be the default behavior, but there would be some flag that could be used to remove it if it was not desired. (If we put this feature in 3.x, maybe it defaults to off.)

If the format above looks good, then I can roll a patch.

Hm, there is a problem. If I add on the name of the drupal site to the output via post-processing, as I intended, then the table formatting would be messed up for commands that used tables. Inserting the name of the site into the table for proper formatting would be complicated, as it would require extensive changes.

Maybe we could just adjust the column width downward by the length of the info we're going to add; then everything would come out okay.

I was wondering how you were gonna handle that :)

Right now, we just print out the $output for each do_command call. What if we force it into the logs such as drush_log($output, @localhost#other.com.au). I think that would look nice and tabular, at the expense of bloating the context object some (double entries for $output)

Hmm. My idea will probably look bad when the remote command returned a table.

Right now, in environment.h we have drush_set_context('DRUSH_COLUMNS', $columns);, with some logic to calculate $columns. I propose adding a new private global option, similar to --backend, that can be used to subtract N from $columns before setting the context variable. This also implies one more parameter to drush_backend_invoke.

This, while not squeeky-clean, is probably the cleanest solution available to us under the circumstances. I'd prefer it to adding extra log entries, anyway.

I'll put this on my list for when I get around to it...

For me - maintaining a multisite environment - drush's most useful ability is to run update.php on all sites with just executing one command:

drush @sites updb

If I do a verbose DB update on a single site like this

drush -v updb

it outputs both the site and the update results:

Initialized Drupal 6.19 root directory at /usr/share/drupal                                                                 [notice]
Initialized Drupal site www.icc.or.at at sites/icc.or.at                                                                    [notice]
No database updates required                                                                                               [success]
Calling call_user_func(drupal_flush_all_caches)
'all' cache was cleared                                                                                                    [success]
Finished performing updates.                                                                                                    [ok]
Command dispatch complete                                                                                                   [notice]

However, when trying to execute that command in the @sites scope like this:

drush -vy @sites updb

the output is only one empty line per site. So fixing the "empty line output" bug with @sites would provide a solution.

Until the @sites bug is fixed, another solution that is working right now on one line is:

for s in $(drush -n @sites st | sed -e '/#/!d' -e 's/^[^#]*#//'); do echo "$s:"; drush -l http://$s updb; echo; done

This will give an output like this:

icc.or.at:
No database updates required                                                                                               [success]
'all' cache was cleared                                                                                                    [success]
Finished performing updates.                                                                                                    [ok]

icc-austria.or.at:
No database updates required                                                                                               [success]
'all' cache was cleared                                                                                                    [success]
Finished performing updates.                                                                                                    [ok]

healthgrain.org:
No database updates required                                                                                               [success]
'all' cache was cleared                                                                                                    [success]
Finished performing updates.                                                                                                    [ok]

:-)

To list all sites in @sites, try drush sa /path/to/drush/@sites --pipe, and see if you like that better.

Oh yeah - this would shorten

drush -n @sites st | sed -e '/#/!d' -e 's/^[^#]*#//'

to the code

drush -p sa @sites | sed -e 's/^[^#]*#//'

giving the same output - like this:

icc.or.at
icc-austria.or.at
healthgrain.org

So I will now use

for s in $(drush -p sa @sites | sed -e 's/^[^#]*#//'); do echo "$s:"; drush -l http://$s updb; echo; done

to update the DB on all sites of a multisite environment.

While I have not tried it, you should also be able to shorten the above to:

for s in $(drush -p sa @sites); do echo "$s:"; drush $s updb; echo; done

Of course, if output with multiple-site commands worked better, just drush @sites updb would be fine. See also related issue #823388: pm-list returns empty array when used with --pipe and @sites alias.

Emitting piping dush's output to sed would let $s become something like "/usr/share/drupal#somesite.tld", while sed sanitizes the output to a clean domain-only string like "somesite.tld", as the directories under the sites directory would be named. So I still prefer my above solution - it's tested, gives a clean output and works well. Of course this is only a workaround until drush itself can handle it nicely.

Subscribing! Was just wishing for this today. Drush rocks!

Grrr.. Component changed on me, but "Code" isn't an option. Leaving as PM until someone smarter than me comes and sets it to what they want..

Yes, the downside of changing the components is that you can't touch an existing issue without updating the setting. In a way, this is a good thing, though, as it forces the reclassification of old issues. ;)

@greg - reducing DRUSH_COLUMNS seems like a reasonable solution to me.

possibly related: #920936: Add XML output mode for CI integration

Hey, sorry about the duplicate issue.

So guys, what's the status on this - any progress? Or are we daily @sites drushers wishing for site name display stuck with the "sed" workaround? It's not very smooth.

Without knowing the exact nature of the table formatting, would it not be possible to add a header with the site name? That should not break columns unless other scripts depend on exact line numbers.

I think greg's idea of reducing the COLUMNS width is our best option.

This patch adds context information to @sites and other multiple-site commands. The following options are introduced:

--autoconfirm-multi: Execute the command on all @sites without asking y/n
--label-multi: Add the name of the site on every line before the command output
--reserve-margin: Reduce COLUMNS by the specified amount

The last is used internally by drush to make sure that there is room in the formatted output to insert the site name; it does not need to be specified directly.

Suggestions on better names for the options welcome.

Slipped on the status.

Code looks good. IMO, the labels should be on by default and one uses the option to disable them.

I am ambivalent about --autoconfirm-multi. Part of me wants just inform the user about what will happen up front and then propogate a --yes to each site. Thus, only one prompt is ever shown. We probably made the decision for multiple prompts a while ago so it is OK not to deal with that here.

Awesome to see this moving forward -- thanks, Greg. I strongly agree with Moshe that labels should be on by default when using @sites.

The reason the multiple-site code prompts at the beginning is that @sites runs each command via backend inovke, so --yes is implicitly added for each subcommand. Given this situation, from the commandline, --yes is as good as --autoconfirm-multi. However, if you wanted to set --yes for @sites always, then $options['autoconfirm-multi'] = TRUE; gives you a way to do this without setting $options['yes'] = TRUE;

I'd like to at least change the name of --autoconfirm-multi, but I'm not sure what to call it.

I'll convert the labels switch to simply --no-labels, and make labels on by default. Limited time now, so I'll do this later. Any further input on option names, etc. still welcome.

does it make sense to allow folks to extend a built-in alias like @sites using a site_alias? i tried the following but 'yes' does not get set.

DEFINE CHILD ALIAS
-------------
$aliases['foo'] = array(
  'parent' => '@sites',
  'yes' => TRUE,
);

OUTPUT
----------
~/htd/d6/sites/default$ drush sa @foo --full
$aliases['.Users.mw.htd.d6#default'] = array (
  'uri' => 'default',
  'root' => '/Users/mw/htd/d6',
);

Even if we do nothing about the oddness above, i think --yes is enough and we don't need autoconfirm-multi. i think folks will be mostly OK with setting it in their site drupal root using $command_specific.

Okay, I will proceed without --autoconfirm-multi in this issue. We should consider #28 as a separate issue; children of alias lists are not explicitly supported at this time. I suppose it does make sense.

I'll also have to see if $command_specific is processed before or after the @sites dispatch.

Here's a patch adjusted per #29.

Just curious: can we expect this feature - once committed - in drush 4.0?

I notice that this patch defines $output_handle but that doesn't seem to be used. Perhaps it is a remnant from other patch.

Looks real good to me. RTBC after above is fixed.

Yes, $output_handle is residual.from the other patch. For a while, I considered that 'ok' should go to stdout, but later changed my mind on that, so $output_handle got left behind when I separated the forked branches.

Committed without $output_handle.

@roball: drush-4 stable will be tagged from drush-HEAD when ready -- which should be fairly soon.

Neither 'drush -y @sites updb' nor 'drush -py @sites updb' still works with the latest official drush (4.5) - there is no output.

What about the idea of autoconfirm-multi? It still does not seem that is possible to automatically confirm the alias list, but give -n to each site's question. This could be important to - for example - check which updates are available on all sites, automatically via a script. At the moment I still have to do this by

for s in $(drush -p sa @sites | sed -e 's/^[^#]*#//'); do echo "$s:"; drush -l http://$s -n upc | grep -i "Update available"; echo; done

Try this on Drush-5.x; many improvements were made in this area. It is unlikely that these changes will be backported to 4.x.

No feedback. Back to fixed.

@sites does not resolve in any site anymore using the 'st' command with drush 5.0:

[root@server ~]# drush @sites st
You are about to execute 'st' non-interactively (--yes forced) on all of the following targets:
Continue?  (y/n): y
Invalid argument supplied for foreach() backend.inc:604                                                                    [warning]

Do you have a Drupal site at ~?

No, but when I execute 'drush sa' from the same directory, it DOES list all the sites, like this:

[root@server ~]# drush sa
@none
@self
onesite.example.com
anothersite.example.com

where onesite.example.com and anothersite.example.com are the sites represented by the @sites alias. This is another bug reported at #1031170: Invalid argument supplied for foreach() backend.inc:709 on "drush @sites ups" since that output should be

@none
@self
@sites

instead.

Where are the onesite.example.com and anothersite.example.com sites located? It's not clear to me how Drush is finding these; it shouldn't see them unless your cwd is at the Drupal root that they're stored at.

Didn't mean to set status.

Where are the onesite.example.com and anothersite.example.com sites located?

The answer is shown by 'drush sa @sites':

[root@server ~]# drush sa @sites
$aliases['.usr.share.drupal6#onesite.example.com'] = array (
  'uri' => 'onesite.example.com',
  'root' => '/usr/share/drupal6',
);
$aliases['.usr.share.drupal6#anothersite.example.com'] = array (
  'uri' => 'anothersite.example.com',
  'root' => '/usr/share/drupal6',
);

Do you set $options['root'] = '/usr/share/drupal6'; in your drushrc.php?

Does drush @sites st work if you first cd to /usr/share/drupal6?

Do you set $options['root'] = '/usr/share/drupal6'; in your drushrc.php?

Yes, exactly, I have set

$options['r'] = '/usr/share/drupal6';

in my "drushrc.php" file.

Does drush @sites st work if you first cd to /usr/share/drupal6?

Yes, bingo!

[root@server ~]# cd /usr/share/drupal6
[root@server drupal6]# drush @sites st
You are about to execute 'st' non-interactively (--yes forced) on all of the following targets:
  /usr/share/drupal6#onesite.example.com
  /usr/share/drupal6#anothersite.example.com
Continue?  (y/n): 

Duplicate of #1031170: Invalid argument supplied for foreach() backend.inc:709 on "drush @sites ups"; re-setting original title and status of issue.

Sorry to re-open this issue, but the patch #8 from #1031170: Invalid argument supplied for foreach() backend.inc:709 on "drush @sites ups" actually did NOT fix the issue described here: 'drush @sites st' still does not work if you are not in the root directory:

[root@server ~]# drush @sites st
The drush command '@sites st' could not be found.  Run `drush cache-clear drush` to clear the commandfile cache if you       [error]
have installed new extensions.

[root@server ~]# cd /usr/share/drupal6
[root@server drupal6]# drush @sites st
You are about to execute 'st' non-interactively (--yes forced) on all of the following targets:
  /usr/share/drupal6#onesite.example.com
  /usr/share/drupal6#anothersite.example.com
Continue?  (y/n):

Could we please handle defects related to #1031170 in issue #1031170? The original purpose of this issue was to add labels (e.g. "onesite.example.com >>") to the output of multiple-site execution, which is still working. There is no need to have two issues open here.