Experimental project

This is a sandbox project, which contains experimental code for developer use only.

Introduction

This Drush extension is meant for individual field data manipulation from commandline. It uses Drush commandline interface. Aim is to be able to perform all CRUD operations and more on all data in all core field types and contributed field types. Some fields have field-specific support other fields have general support.

New functionality is added by feature requests.
This project is currently located at:

Status of the project

This project has been decided to be duplicate of Drush Entity, but until the merge actually happens download this Drush extension.

Meanwhile new features are added here.

This was project application.

Here is merge proposal issue.

Code review by Clemens and further thoughts on integration.

Currently working on merge.

Audience

This Drush extension is mostly for developers, admins, but not limited to.

  • Admins
    • Quick data inspection
    • Quick data fixes
    • Helpful in data analyses
    • Helpful for finishing up migration
  • Developers
    • Integration with other systems
    • External data processing
    • Asynchronous data manipulation
    • Testing for module development
  • Regular user
    • Importing directoryful of files or images

Supported field types and operations

Update = Add, Set, Import.

Field Read Update Find Replace
Date Yes Yes
Datestamp Yes Yes
Datetime Yes Yes
Email Yes Yes
Entity Reference Yes Yes
File Yes Yes
Image Yes Yes
Link Yes Yes
List: Boolean Yes Yes
List: Float Yes Yes
List: Integer Yes Yes
List: Text Yes Yes
Number: Decimal Yes Yes
Number: Float Yes Yes
Number: Integer Yes Yes
Taxonomy Term Reference Yes Yes
Text Yes Yes
Text Long Yes Yes
Text with Summary Yes Yes
General Yes Import Yes
Node title Yes Yes Yes Yes

All field types:

  • Count, Delete, Read, Find

Requirements

  • Drush is installed
  • Field module (in core) is enabled
  • Entity module in installed and enabled
  • Relevant field module to be enabled. (Example: If you want to add/del/read a file field, then you need File module enabled.)
  • This Drush extension assumes there are some fields with that type to work with.
  • You have commandline (SSH) access to your Drupal installation.
  • Understanding about following concepts:
    • Entity type (Node, User, Taxonomy Term, Comment, Profile2, ...)
    • Bundle (Article, Basic Page, ...)
    • Entity ID (1, 100, 1234, ...)
    • Field name (field_pdf, field_thumb, ...)
    • Machine name (only_lowercase_letters_and_underscores)
    • Field type (File, Image, Email, Text, Integer, ...)
    • Drush
    • Unix shell

An Introduction to Entities: https://drupal.org/node/1261744
About Drush: http://www.drush.org/about
GNU Bash: http://www.gnu.org/software/bash/

Install

  1. Download the Drush extension
    cd ~/.drush
git clone --branch 7.x-1.x http://git.drupal.org/sandbox/ragnarkurm/2274189.git drush_fields
  2. Refresh Drush commands cache
    drush cc drush

Configuration

This Drush extension has nothing to configure.

Usage

General

Generally command line consists of 5 parts:

  1. drush
  2. command - For example: fields-update, fields-node-title-read, etc
  3. target arguments - This specific to each command and it specifies which data to access. Please check drush command --help. For example: node 1234 field_myfile 6
  4. required options - For example: --value=/tmp/a.txt
  5. other optional - For example: --mime=text/plain

Detailed help:

drush help
drush fields-count --help
drush fields-del --help
drush fields-read --help
drush fields-info --help
drush fields-update --help
drush fields-node-title-find --help
drush fields-node-title-read --help
drush fields-node-title-replace --help
drush fields-node-title-set --help

fields-find

To search field values there is a command for that fields-find. It searches within one field instance, but it can be across many entity types.

There are two sets of values available for filter and for output.

General

  • bundle
  • delta
  • entity_id
  • entity_type
  • language
  • revision_id

Field-specific

It varies from field to field what kind of values are available. For example, for date(iso), there are additional field-specific values: value, value2, timezone, offset, offse2.

Output format

It is almost identical what fields-read has. It is based on template and --template and --header options work similarly. Differences are: here are no structural output like JSON or XML etc and secondly no ~key|key|key data structure traversal.

Conditions

But how does actually find works. Maybe more accurate is to say filtering. It works by adding command-line arguments, each being a condition. Each condition consists of 3 parts: value, operator and operator argument. There are 3 value types supported: string, integer, float. Each type has separate set of operators available:

  • String: =, <>, like, not-like, ~, !~
  • Integer: =, <, >, <>, <=, >=, in, not-in, between, not-between
  • Float: <, >, <=, >=, between, not-between

There are 2 categories of operators: database level and php level. Database-level filtering is fast, PHP-level filtering requires fetching all data and then perform matching and this is slow. Most of operators work on database level and ~ and !~ work on PHP level.

Tilde ~ performs regular expression match and !~ performs negative regular expression match.

Examples

Suppose you want to find common misspelling 'wich' to replace it later manually:

$ drush fields-find field_text 'value ~ /wich/'
Type       Bundle         ID  Delta Value      Format    
node       page            1      4 Test wich  (NULL)

Lets make life easier:

drush fields-find field_text 'entity_type = node' 'value ~ /wich/' --template='http://example.com/node/{%d:id}/edit' --header=n
http://example.com/node/1/edit

fields-update

import

It is possible to import all field data (not individual elements in multi-value field) at once using following formats:

  • serial
  • json
  • xml

It is possible to read input from file (absolute path) or from stdin.

In order to perform import, use --op=input. Use --value to supply input source and format. Examples: --value=/tmp/location.serial or --value=-.json -y. The input data type is determined by extension, including stdin (-.json). For stdin must supply -y.

Note that during importing only rudimentary data checks are done. If the data has wrong structure for the field (for example export date field and import as file field) then results are unspecified.

Examples:

$ drush fields-update node 1 field_geo --op=import --value=/tmp/a.json
$ drush fields-update node 1 field_geo --op=import --value=/tmp/a.serial
$ cat /tmp/a.xml | drush fields-update node 1 field_geo --op=import --value=-.xml -y

Advanced example - clone/copy field from one entity to other and change case on the fly:

drush fields-read node 1 field_text --template=json \
| tr a-z A-Z \
| drush fields-update node 2 field_text --op=import --value=-.json -y

fields-info

While working with fields, coding PHP, there is often need to inspect field itself (not it's data). Most frequent inspections are:

General info example:

$ drush fields-info field_text
Array
(
    [translatable] => 0
    [entity_types] => Array
        (
        )

    [settings] => Array
        (
            [max_length] => 20
...

Instance info example:

$ drush fields-info field_text node page
Array
(
    [label] => text
    [widget] => Array
        (
            [weight] => -3
            [type] => text_textfield
            [module] => text
            [active] => 1
            [settings] => Array
...

fields-read

Command 'fields-read' lists data in one field in one entity. Data is printed out in tabular form. Each field type has its own default template how to print data. User can specify own format (good for automation). template consists of text and data-placeholders which will be filled with data. Placeholders consist of two parts. First printf()-style formatter (like %s or %-10d) and key. Key may be preset keyword which is constant over Drupal versions or data structure specific keys which lead to data. If user-specified template is empty, then all possible keys are listed specific to selected field. Header is attempted to print aligned to columns, but does not always work properly with custom template.

drush fields-read node 1 field_image 
 Delta    FID    UID  Width Height           Created      Size Filename
     0     65      0    400    225 2014-Jul-04 00:03     23543 images-background.jpg
     1     74      1 (NULL) (NULL) 2014-Jul-11 00:33       379 aaa.txt
     2     75      1   1920   1200 2014-Jul-11 00:33    380478 img.jpg

This is default output.

fields-read node 1 field_image --template='{%5d:delta} {%05d:width} {%-8.8s:filename} {%s:~metadata|height}'
Delta Width Filename Value
    0 00400 images-b 225
    1 00000 aaa.txt  (NULL)
    2 01920 img.jpg  1200

Here you can see custom template with custom formatting in tabular form. Not vary practical, but serves purpose of demonstrating power of formatting options. First, delta is 5 digits wide integer, secondly width is zero-padded 5 digits wide integer, thirdly filename is 8 characters wide and truncated at 8 chars aligned left, finally string is fetched from value data structure $value[delta][metadata][height].

drush fields-read node 1 field_image --template='{%d:width}:{%d:height}:{%s:filename}' --header=n
400:225:images-background.jpg
0:0:aaa.txt
1920:1200:img.jpg

This is more practical example. Data is listed csv or delimited format which is excellent for shell scripting and works well with cut, grep, sort, uniq and other standard text processing tools.

$ drush fields-read node 1 field_geo --template=print_r
Array
(
    [0] => Array
        (
            [geom] => POINT (22.33 11.22)
            [geo_type] => point
            [lat] => 11.220000000000
            [lon] => 22.330000000000
            [left] => 22.330000000000
            [top] => 11.220000000000
            [right] => 22.330000000000
            [bottom] => 11.220000000000
            [geohash] => s3zzu4
        )

)

It means, it is possible to use following template placeholders:

  • {%s:~geom}
  • {%s:~geo_type}
  • {%f:~lat}
  • {%f:~lon}
  • etc.

For example --template='{%f:~lat}:{%f:~lon}' gives following result:

Value:Value
11.220000:22.330000
12.120000:23.230000
16.160000:24.240000

Generic data handling

There exists few generic ways to export data from fields. Use --template=format to get desired format. Hopefully format IDs are self-explanatory. Available formats are:

  • print_r
  • serialize
  • var_export
  • var_dump
  • json
  • xml

Examples:

$ drush fields-read node 1 field_date --template=print_r
Array
(
    [0] => Array
        (
            [value] => 1999-12-31 22:00:00
            [value2] => 1999-12-31 22:00:00
            [timezone] => Europe/Tallinn
            [offset] => 7200
            [offset2] => 7200
            [timezone_db] => UTC
            [date_type] => datetime
        )

)
$ drush fields-read node 1 field_text --template=serialize
a:3:{i:0;s:10:"/tmp/a.php";i:1;s:8:"asdfasdf";i:2;s:4:"test";}
$ drush fields-read node 1 field_text --template=var_export
array (
  0 => '/tmp/a.php',
  1 => 'asdfasdf',
  2 => 'test',
)
$ drush fields-read node 1 field_text --template=var_dump
array(3) {
  [0]=>
  string(10) "/tmp/a.php"
  [1]=>
  string(8) "asdfasdf"
  [2]=>
  string(4) "test"
}
$ drush fields-read node 1 field_text --template=json
[
    "\/tmp\/a.php",
    "asdfasdf",
    "test"
]
$ drush fields-read node 1 field_text --template=xml
<?xml version="1.0"?>
<field_text>
  <value_0>/tmp/a.php</value_0>
  <value_1>asdfasdf</value_1>
  <value_2>test</value_2>
</field_text>

Note that the values are always printed as arrays, even if min (required) = 1 and max (cardinality, number of values) = 1.

Note that if field is empty no data structure is returned, but indicative text is issued.

Note that the formatted data is based on $wrapper->field->value().

Versions, milestones, todo

Current code is 7.x only.

Following features are planned:

  • Find operation implementation.
  • Replace operation implementation.
  • Read, Update, Find, Replace are supported for common entity type's (node, user, comment, taxonomy term, profile2) pseudo fields (title, published, username, email, etc).
  • Server-side counterpart (a module) for assisting file handling (file permissions).

Please submit your feature requests!

Misc, todo

  • Merge into Drush Entity
  • Tests
  • Resolve class autoloading with drush.
  • Drush extension updating
  • Change File handling, since files are entities and file fields are references to entities

Security and data access

Drush default Anonymous user and user 1 can do all operations for practical reasons.
All other users (-u or --user) are checked if they can access particular data.
This data access approach may change according to feedback.

Troubleshooting

  • This Drush extension contains a lot of checks and verbose error messages which should be helpful for solving problems.
  • Check: drush command --help
  • Check watchdog log
  • Search issues
  • Post an issue:
    https://drupal.org/project/issues/2274189

Related projects

Use cases

If you develop a system base on this Drush extension or find some other valueable use, it would be nice to share a short description of it. Also get a link to your site.

Simple cases

  • Creating invoice .pdf in shell and add to Drupal Commerce, Order entity.
  • Adding (importing) collection of files or images to a field.

Case: Export images or files from node

I had situation where one of my sites accommodated many galleries and it grabbed too much disk space (drush backups were too big). So I decided to move gallery images to Flickr. So how to export images from node at once? The problem is that I have all images mixed in the directory - how to separate them easily?

$ cd /www/myprojectname/html/sites/example.com/files/images
$ drush fields-read node 15 field_images --template='{%s:filename}' --header=n | tar -cf /tmp/15.tar -T -

I have configured field_images to save images into files/images directory. With help on Drush Fields I extracted all image file names and fed them into tar for archiving, -T - means reading filenames from stdin. Now the archive file can be downloaded and then uploaded to Flickr.

Case: Linux distribution ISO compilation system

A system has been developed where user enters parameters (Language, region, packages, etc) and after a while will be able to download customised Linux distribution ISO. Techinically user posts parameters to node edit form. Cron runs regularly, exports the data (Views Data Export) and compiles ISO images (ca 700mb). These images are transferred back to system by Drush Fields. Users will be notified when the task is done and distribuition is ready for download. It is very easy and handy to perform the compilation and insertion to Drupal by shell script.

Maintainers

Current maintainers/developers:

Project information