Supporting tokens for my module's field types

Since the Field API (formerly CCK) was merged into Drupal 7 core, but failed to provide token support in time for the 7.0 core release, Field tokens are now supported through Token module. For the most part this support is automatic for the base-level of tokens like [node:field_tags], [user:field_first_name], etc because we use the field_view_field() function with custom display settings to output base-level tokens.

Overriding default field formatter for base-level tokens.

By default the display settings passed to field_view_field() uses the default field formatter for the field specified in the field's respective hook_field_info().

For example, the taxonomy term reference field provided by the core Taxonomy module defines its field as the following:

function taxonomy_field_info() {
  return array(
    'taxonomy_term_reference' => array(
      'label' => t('Term reference'), 
      'description' => t('This field stores a reference to a taxonomy term.'), 
      'default_widget' => 'options_select', 
      'default_formatter' => 'taxonomy_term_reference_link', <-- used to render base-level tokens with field_view_field()
      'settings' => array(
        ...
      ),
    ),
  );
}

By default any token like [node:field_tags] would get output as a list of terms with links to their term pages. For most users this type of token is not useful. The default formatter for tokens can easily be overridden by adding a 'default_token_formatter' value with the preferred field formatter to use with token generation.

To make plain text terms the default output for term reference field tokens, we would change the following:

function taxonomy_field_info() {
  return array(
    'taxonomy_term_reference' => array(
      'label' => t('Term reference'), 
      'description' => t('This field stores a reference to a taxonomy term.'), 
      'default_widget' => 'options_select', 
      'default_formatter' => 'taxonomy_term_reference_link', 
      'default_token_formatter' => 'taxonomy_term_reference_plain', // <-- Added here
      'settings' => array(
        ...
      ),
    ),
  );
}

Default token formatters for core field types

Token.module defines the default token formatters for the following core field types:

Field                    Default formatter
------------------------------------------------------
taxonomy_term_reference  taxonomy_term_reference_plain
number_integer           number_unformatted
number_decimal           number_unformatted
number_float             number_unformatted
file                     file_url_plain
image                    file_url_plain

Rendered field output

After running field_view_field() for the token output, token_tokens() runs a #pre_render callback on the field output to remove the default <div> markup surrounding the fields, as well as join multiple values together with commas.