Consider commands as pm-list, pm-update or user-list (not in drush still). Those commands print out information in a table format and are also allowed to print out raw data suitable for pipelining. The --pipe output.

Columns and the form of pipe output is hardcoded. Several issues have raised asking to add some columns or discussing on how the pipe output should looks like. The proposal here is to implement an output engine for tabular data flexible enough to let the user customize their outputs.

Here's a description of the proposal:

1/ Engines can be classes since PMTNG 1, the next natural step is to let them receive parameters to be configured. We also want to configure them from the command definitions:

  $items['user-list'] = array(
    'engines' => array('output_tabular'),
    'output_tabular' => array(
      'columns' => array('uid', 'name', 'mail' , 'status', 'roles'), // All of the available keys in data objects
      'column names' => array('User ID', 'User name', 'E-mail', 'User status', 'User roles',),
      'table columns' => array('uid', 'name', 'mail', 'status'), // Columns for table output
      'pipe columns' => array('name', 'mail'), // Allow for a composed output
      'pipe separator' => '-', // Separator for the composed output: name1-mail1, name2-mail2
      'pipe delimiter' => ',', // For a CSV output
    )
  );

Obviously pipe separator or pipe delimiter can be default values not needed to be defined in each command. Also, the pipe output could default to the first of the table columns.

2/ Commands implementing output_tabular engine will be cleaner, with no need to compose the table rows or the pipe output. They could just return output_tabular_print($data);.

$data may be a keyed array of arrays or objects that must define all of the columns from the command definition.

3/ With the above in place, the output engine options could be: table, pipe, json, yaml,... json or yaml may output all of the columns.

4/ Output is custimizable with command line options such as --table-columns="uid,name"

This, although laborious, doesn't seem hard to implement and may be suitable for drush 5.

CommentFileSizeAuthor
#9 drush-1364808.patch17.71 KBjonhattan

Comments

moshe weitzman’s picture

moshe weitzman commented

Sounds pretty good to me ... An alternative that occurs to me that is shipping with some example bash/awk code in example.bashrc that demonstrates parsing our CSV output. We could also add examples for common tricks like xargs. This wouldn't get you json/yaml etc. but would make dealing with our output easier.

jonhattan’s picture

jonhattan
Primary language Spanish
Location Plasencia
commented
Issue tags: +Release blocker
msonnabaum’s picture

msonnabaum commented

Glad to find this issue already started. I've run into a few commands lately that don't have --pipe (test-run) and it would be great to make this more consistent. I also really like the idea of supporting any of the --format options.

Is this something we need to tie to the concept of tables though? Perhaps it would be too abstract otherwise. Not sure yet.

moshe weitzman’s picture

moshe weitzman commented
Issue tags: -Release blocker

If this gets done real soon, it can happily go into Drush5. But I don't think we will hold release for it. Removing tag.

jonhattan’s picture

jonhattan
Primary language Spanish
Location Plasencia
commented
Title: Tabular output engine proposal » Command output format engines proposal

After a chat with Mark I think of this:

Consider two output engines: output_tabular and output_object. Both using --format= for the command line.

output_tabular is almost expressed in OP. Their engine types are table, csv, pipe...

output_object's engine types are: var_dump, json, yaml (#1366098: Automatically download Symfony YAML component). It could print one or several objects. It can be extended in contrib with new formats (#1396178: Add properties output as a --format)...

The big difference between both engines is that output_tabular must declare its objects' columns. For the rest, output_tabular is a subclass of output_object, so tabular commands can just print json et al.

Commands optionally declare its output type. Perhaps with 'output' => tabular | object instead of including the engine explicitly. That is, engine options become first level command options.

If a command declares its output type, the command callback is not responsible of printing out. It just needs to return $data and drush will send it to the engine.

clemens.tolboom’s picture

clemens.tolboom
Primary language Dutch
Location Groningen, 🇳🇱/🇪🇺
commented

For http://drupal.org/project/drush_entity we have --fields which are a more general concept then columns. With fields you can ask for a path in the array/object tree.

Apart from the column names which are hard do define on ie entity_info() results it would be nice for this issue to support path as drush_entity does.

    'output_tabular' => array(
      'columns' => array('uid', 'name', 'mail' , 'status', 'roles'), // All of the available keys in data objects
      'column names' => array('User ID', 'User name', 'E-mail', 'User status', 'User roles',),
    )
  );

could be defined as drush entity-type-read user node --fields=label,bundles/*/label

  'columns' => array(
    'label' => 'Label',
    'bundles/*/label' => 'Bundle label',
  ),

thus this defined the data path and Column name at once. In this example 'Bundle label' value occurs multiple for each node type (and user).

For more --fields examples see http://build2be.com/content/drush-entity-support-patch-needs-review

damiankloip’s picture

damiankloip commented

I think me and Clemens started a similar issue (http://drupal.org/node/1441026) in the drush queue, but this is probably a more substantial and comprehensive solution to the whole formats issue. I am happy to do some work on this, I really want to see more unified --format, --pipe options etc...

How do you see contrib modules adding/extending the available formats?

jonhattan’s picture

jonhattan
Primary language Spanish
Location Plasencia
commented

With this proposal contrib could extend formats by defining an engine.

jonhattan’s picture

jonhattan
Primary language Spanish
Location Plasencia
commented
Title: Command output format engines proposal » Proposal for an output formats engine
Status: Active » Needs review
StatusFileSize
new drush-1364808.patch17.71 KB

I wonder if the title is more comprehensible now.

Attached is a format-patch with three commits:

1. Extend engines subsystem to allow for parametrized engines.
2. Implements the output engine. I called it printout. A better name is welcome.
3. Changed pm-releases command to use printout engine.

I started from the perspective of a table to get something running. This patch still needs work and focus to define a broad coverage of use cases: better support other formats and be usable by other commands. Just set it to needs review to get some feedback.

Secondary anottation: I added a new file includes/engines.inc with some new declarations. The idea is to put in it all code related to engines from commands.inc and drush.inc. however it should be deferred to the end of this issue or to a follow-up for a better understanding.

Explanation of commits in the patch:

1. Parametrized engines: This allows to shape a engine for a particular use case. For example a tabular engine must be parametrized with name of valid fields.

1a) Parameters are defined in the engine_type declaration (i.e. hook_drush_engine_type_info()) as:

+    'parameters' => array(
+      'fields_all' => 'All fields available for command output.',
+      'fields_name' => 'Associative array to optionally give a human name to any field.',
+      'fields_default' => 'Default fields used by the command.',
     ),

1b) Parametrization is done in the command declaration. For example in pm-releases:

     'engines' => array(
       'release_info',
+      'printout' => array(
+        'default' => 'table',
+        'fields_all' => array('name', 'version', 'tag', 'version_major', 'version_extra', 'status', 'release_link', 'download_link', 'date', 'mdhash', 'filesize', 'release_date', 'release_status'),
+        'fields_name' => array('version' => 'Release', 'release_date' => 'Date', 'release_status' => 'Status'),
+        'fields_default' => array('version', 'release_date', 'release_status'),
+      ),
     ),

1c) Parametrized engines must implement drush_parametrized_engine interface.

1d) Parameters are passed to the engine constructor.

1e) Those engines need to alter command help in order to fill in parametrized values in the help text.

2. Implementation of printout engine.

Implemented table, pipe, json and yaml for the sake of completeness. json and yaml are just quick&dirty for demonstration. Not much more to say here.

3. Changes in pm-releases.

The command doesn't deal with table business. Basicly it obtains the releases and pass them to the engine. So its implementation is simplified.

I'm not satisfied with:

+    if ($printout->engine == 'table') {
+      $printout->caption = dt('------- RELEASES FOR \'!name\' PROJECT -------', array('!name' => strtoupper($name)));
     }

Perhaps it can be managed by introducing parametrization at engine level (parametrization explained above is at engine_type level, that is, for all implementations of a engine_type).

4. Screenshots

a) Help text is parametrized. See the fields option.

#drush help pm-releases

...
Options:
 --dev                                     Work with development releases solely.                                                                                                                     
 --fields=[version, release_date,          Fields to output. All available fields are: name, version, tag, version_major, version_extra, status, release_link, download_link, date, mdhash, filesize, 
 release_status]                           release_date, release_status.                                                                                                                              
 --source                                  The base URL which provides project release history in XML. Defaults to http://updates.drupal.org/release-history.                                         
 --format=yaml                             YAML markup.                                                                                                                                               
 --format=pipe                             Pipeline format for scripting.                                                                                                                             
   --delimiter=<,>                         Delimiter for items in the output. Example: item1,item2,item3                                                                                              
   --separator=<->                         Separator for a composed items output. Example foo1-bar,foo2-baz,foo3-qux                                                                                  
 --format=table                            Default printout engine. Tabular format.                                                                                                                   
 --format=json                             Javascript Object Notation.

If you'd like to see the options grouped by engine, goto #1421570: Command options grouping

b) Standard usageworks as before:

# drush pm-releases drupal
------- RELEASES FOR 'DRUPAL' PROJECT -------
 Release  Date         Status                 
 7.x-dev  2012-May-18  Development            
 7.14     2012-May-02  Supported, Recommended 
 7.13     2012-May-02  Security

... as before.

c) Pipe output by default prints the first of the default fields (version).

# drush pm-releases drupal --format=pipe
7.x-dev,7.14,7.13

d) Pipe output with three fields and custom separator and delimiter.

# drush pm-releases drupal --format=pipe --fields=tag,status,release_date --delimiter=";" --separator="/"
7.x/published/2012-May-18;7.14/published/2012-May-02;7.13/published/2012-May-02

e) Json output defaults to all fields:

# drush pm-releases drupal --format=json
Undefined index: version_extra json.inc:17                                                                                                                                                              [warning]
Undefined index: version_extra json.inc:17                                                                                                                                                              [warning]
[["Drupal 7.x-dev","7.x-dev","7.x","7","dev","published","http:\/\/drupal.org\/node\/156281","http:\/\/ftp.drupal.org\/files\/projects\/drupal-7.x-dev.tar.gz","1337299948","0b6bbd78948611da1bf9fbb49aefb399","3129368","2012-May-18",["Development"]],["drupal 7.14","7.14","7.14","7",null,"published","http:\/\/drupal.org\/node\/1558424","http:\/\/ftp.drupal.org\/files\/projects\/drupal-7.14.tar.gz","1335997556","af7abd95c03ecad4e1567ed94a438334","3128473","2012-May-02",["Supported","Recommended"]],["drupal 7.13","7.13","7.13","7",null,"published","http:\/\/drupal.org\/node\/1558412","http:\/\/ftp.drupal.org\/files\/projects\/drupal-7.13.tar.gz","1335997261","80587b66375c7fc539414a20a2c6f2de","3088448","2012-May-02",["Security"]]]

f) Json with limited fields:

# drush pm-releases drupal --format=json --fields=version,release_status
[["7.x-dev",["Development"]],["7.14",["Supported","Recommended"]],["7.13",["Security"]]]
moshe weitzman’s picture

moshe weitzman commented

THis looks terrific to me. I'd love input from drush users and drush maintainers.

moshe weitzman’s picture

moshe weitzman commented

I should say that I looked at the code and found only minor issues:

  1. Needs some tests
  2. parametrized should be parameterized
damiankloip’s picture

damiankloip commented

jonhattan, this patch looks pretty awesome. I think it's a much more solid and flexible solution going forward. We can probably ditch #1441026: Add a hook_drush_output_formats in that case too :)

moshe weitzman’s picture

moshe weitzman commented
Status: Needs review » Reviewed & tested by the community

@jonhattan - I'm ready to create a branch for 5.x and then commit this to master. Lets just address my comments in #11 .

greg.1.anderson’s picture

greg.1.anderson commented

Sorry for being late to the party. The code looks really good, and I like the direction. The one thing that occurs to me is that some code is going to want to output a 2d array, and other code is going to want to output a 1d array. Some data types, like xml and json, are well-suited to both types, but it is kind of hard to shoehorn a 1d array into a table, or a 2d array into --pipe output. Maybe it would be helpful if commands specified the kind of output they produced, and output engines specified what kind of data they could format. This would be helpful for formatting command help output (list relevant available output formats in command help). This augmentation is minor and could be a follow-on issue, though.

pm-releases is an example of a command that produces 2d data that works well as either json or tabular output, but what about something like pm-updatecode? That command produces tabular output, but when representing this data as structured output (xml, yaml or json), it would be most useful to output an associative array, where one column of the tabular output (e.g. the module machine name) is the key, and the other columns in the table are returned as a sub-array value of that key. Another example where this would be useful is drush status output as json or xml, which ideally would be represented as name:value pairs. It looks like the existing design could be extended to allow this option. Without this feature, the json output of drush status would be something like [["Drush version", "6.0-dev"],["PHP configuration", "/etc/php5/cli/php.ini"]] (or if done literally from the existing code, [["Drush version", ":", "6.0-dev"],["PHP configuration", ":", "/etc/php5/cli/php.ini"]]), which would not be as useful as an associative-array-based output. Actually, the output drush status might instead be simply ["6.0-dev","/etc/php5/cli/php.ini"], but that is also somewhat inconvenient, since the fields output by drush status is variable depending on the status of the site being examined.

I might be inclined to say that we should figure out how drush status should be implemented with this scheme before committing, but I suppose that I wouldn't mind if this was committed to a drush-6 branch w/out the associative arrays feature. I suspect we are going to need it before this feature is generally useful, though.

moshe weitzman’s picture

moshe weitzman commented
Status: Reviewed & tested by the community » Needs work

Lets get drush status working nicely before committing

moshe weitzman’s picture

moshe weitzman commented

fbcmd, a CLI for facebook, has a similar feature. See http://fbcmd.dtompkins.com/output.

Hope someone can get back to this. It is close,

greg.1.anderson’s picture

greg.1.anderson commented

This should be unified with #1866344: Make --pipe awesome with output format engines for more awesome. That issue pretty well takes care of #14. 'pipe' is a mode, not a format, though; see #1866344, which allows --pipe=json, etc. (as different commands may have different output formats for --pipe). One thing I realized, though, is that some formats are only good in regular (non-pipe) mode, and others are best in pipe mode. We could further unify --pipe and --format by making the later put Drush into pipe mode for certain formats, and make --pipe shorthand for --format=[pipe default format].

greg.1.anderson’s picture

greg.1.anderson commented
Status: Needs work » Closed (duplicate)

Continuing work at #1866344: Make --pipe awesome with output format engines.

greg.1.anderson’s picture

greg.1.anderson commented
Version: » 8.x-6.x-dev

Closed #1866344 as a duplicate of #1366098: Automatically download Symfony YAML component.