Meta-Data Plan and Structure

You coming to DCDC then, kwinters?

I am very interested in metadata handling with images. Here's some relevant information:

• EXIF
• GPS (which maybe should somehow work with location module)
• IPTC (spec)

Not only am I interested, I'm a (entry level) php person, so I'm more than willing to learn what's needed and help out. I just may need some direction from time to time to make sure whatever I'm working on integrates nicely with all the other multimedia solutions.

after a lengthy discussion w/ arthurf, alexua & drewish, we decided to forgo trying to store metadata ourselves entirely, since there are already better things in place, notably the rdf module.

instead, we'll simply give various form tab to mime type mapping, and allow implementations to plug into that, defining the metadata they want and storing it themselves. we'll provide a default, probably just 'title' for now, and leave it at that.

a new issue will be to provide an rdf-media bridge module.

there's now a placeholder form in place for metadata now, currently a rough fake tabset at $form['media_browser']['media_browser_metadata'] in media.module. this needs to pull in the real forms built by registered modules (and altered by helper modules, such as imagefield-media or rdf-media bridges.

Yes, I agree full-heartedly. We had a long discussion about this at the media sprint. I leaned on the side of including official meta-data support in the module, while arthurf & drewish argued on the other side. In the end, we agreed that creating yet another metadata solution probably wasn't the best way to go, and to let other modules do the heavy lifting for this.

With the current drive to put RDF in core, RDF seems the best place to look. Thus, I opened the #438388: RDF - Media Bridge Module issue.

However, although I favor the media module leading the way somehow, at the same time, I don't want to require the RDF module be enabled, if an administrator simply wants the current imagefield behavior of title/alt being the extent of stored metadata.

Thus, at the end of our discussion, we decided to implement a few bridge solutions. So if the filefield/imagefield media bridge module is installed, then title/alt will be provided (and stored in the current system as is). If the RDF module (and its bridge) is installed as well, then metadata storage & handling will be deferred to that module. Media will provide a hook for these modules to insert and modify their respective forms, and no more.

Updated Implementation Plan, from discussion with @aaron at Drupal Camp Colorado

media_metadata (table structure)
-----------------
mid (serial) | fid (int, relates to 'file' table) | key (string) | value (string)
1 | 21 | genre | documentary
2 | 21 | genre | action
3 | 21 | title | Awesome Video
4 | 21 | license | CC by SA

Two new hooks
----------------

/*
 * @return list of keys that this module cares about
 */
hook_media_metadata_keys ($mimetype) {
  return array();
  // return array('genre', 'license'); // if module cares about these two 
}

/*
 * @param $file is a file object
 * @return array: field(s) in accordance with Form API.
 */
hook_media_metadata_form ($file) {
  if ($file->mimetype == 'video') {
    $form['genre'] = array(
      '#type' => 'select',
      '#title' => t('Genre'),
      '#options' => my_module_genre_options(),
    );
    $form['duration'] = array(
      '#type' => 'item',
      '#value' => my_module_duration($file),
    );
    return $form;
  }
}

"@return array: field(s) in accordance with field api."

actually, that's FAPI (Form API).

Thanks Aaron, I've corrected that now. (#13)

I do not know if this is the right place, but could this be integrated, or be somehow related to getid3 module? to get metadata filled automatically in case it is audio, for example.

davebv: The short answer is yes. We were actually thinking about modules like getid3 when deciding whether or not to implement this rather than attempt building on RDF. We settled on this simple system because we didn't want to force the complexity of RDF on users. In contrast, this simple key/value system will generally be transparent to the user.

Edit: So this is a win/win situation. It's not initially complex, but leaves the door open to those that do want RDF and other modules with the additional complexity of configuration and mapping.

I've begun implementing this now, in media_metadata.module (available in the latest d6 dev from cvs).

needs some work -- for instance, i currently have it build a unique array, but now i remember from discussions that we need to respect multiple modules who want to implement 'title', for instance.

that means we'll need to add 'module' to the table, and invoke each module individually when building the array so we can associate it with the module, probably.

I made made a copy of aaron's work in HEAD (Drupal 7). What API functions shall we provide?

Here is my draft:

/**
 * Returns an array of file IDs that match the given key value pair.
 *
 * Given a key name or data (or both), the related file IDs will be returned.
 *
 * If there is data to be returned, the associative array will always contain
 * mid, fid, name, and data.
 *
 * @param $name
 *   An optional key name.
 * @param $data
 *   An optional data value.
 * @param $unhandled
 *   TRUE returns ALL key value pairs even if they are no longer managed. By default only
 *   key value pairs that have a handler will be returned. This is to protect against
 *   metadata added by another module that no longer exists.
 * @return
 *   Returns an array of file IDs ({file}.fid) that match the given key value pair.
 *   An empty array is returned if there are no results.
 */
function media_metadata_by_pair($name = NULL, $data = NULL, $unhandled = FALSE) {}

/**
 * Returns the key value pairs of the given file ID.
 *
 * Given a file ID, this function will return an associative array containing
 * it's metadata.
 *
 * If there is data to be returned, the associative array will always contain
 * mid, fid, name, and data.
 *
 * @param $fid
 *   A file ID.
 * @param $unhandled
 *   TRUE returns ALL key value pairs even if they are no longer managed. By default only
 *   key value pairs that have a handler will be returned. This is to protect against
 *   metadata added by another module that no longer exists.
 * @return
 *   Returns an array of file IDs ({file}.fid) that match the given key value pair.
 *   An empty array is returned if there are no results.
 */
function media_metadata_by_fid($fid, $unhandled = FALSE) {}

/**
 * Returns the key value pairs of the given URI.
 *
 * This is a convenience method. If you already have the file ID use
 * media_metadata_by_fid(), otherwise unecessary resources will be wasted.
 *
 * If there is data to be returned, the associative array will always contain
 * mid, fid, name, and data.
 *
 * @param $uri
 *   A stream such as public://foobar.txt
 * @param $unhandled
 *   TRUE returns ALL key value pairs even if they are no longer managed. By default only
 *   key value pairs that have a handler will be returned. This is to protect against
 *   metadata added by another module that no longer exists.
 * @return
 *   Returns an array of file IDs ({file}.fid) that match the given key value pair.
 *   An empty array is returned if there are no results.
 */
function media_metadata_by_uri($uri, $unhandled = FALSE) {}

/**
 * Add a new metadata key value pair to the file ID.
 *
 * @param $fid
 *   The file ID.
 * @param $name
 *   The key name.
 * @param $data
 *   The data value.
 * @return
 *   Returns the new mid of the key value pair on success, or FALSE on failure.
 */
function media_metadata_add($fid, $name, $data) {}

/**
 * Deletes a metadata key value pair.
 *
 * @param $mid
 *   The metadata ID.
 * @return
 *   Returns TRUE on success, or FALSE on failure.
 */
function media_metadata_delete($mid) {}

This should cover general use and anything more complex would probably need to be done through a custom query. Are there any other functions that we need?

Subscribe

I'm wondering why not store file url+meta data as a node, the way the image module does? That way you'd have the taxonomy features, it'd be searchable, and you wouldn't have problems with multiple out-of-sync meta data stores (@#12).

There's a module that already basically does this, without all the other good stuff that Media plans to have: http://drupal.org/project/filenode

it's a fine balance, discovering metadata that would always be stored w/ a file (such as GetID3's duration, artist, title, etc), and what would be stored with a specific instance (such as imagefield's title/alt). i definitely am leaning in the direction of the first, which is what this would implement. it doesn't force the issue, and modules such as filenode and imagefield are still free to create their own instantiated metadata. however, i definitely do not want to require everyone to create an entire node for each file. doing that defeats the purpose and intent of this module. and frankly, you still have the same issue of "multiple out-of-sync meta data stores (@#12)", because the whole intent of filenode is to allow files + metadata to be attached to other nodes, so you just sugarcoat the issue. it just adds all the baggage of another node to a file. forcing the hand here would tell folks, 'you can't really use upload or filefield w/o creating a new node', which is bad mojo.

you bring up a good point of searchability though. we'll definitely want to hook this into the search system.

i've implemented media_metadata_add() from #21. i wonder, however, if we should allow multiple values for the same key of a file? currently, this is the behavior.

i suppose that yes, we should: that allows for storing, say, multiple video genres without needing to serializing the array.

it means that we leave it up to implementing modules to determine if a title exists, for instance, so that the key can be overwritten. we'll need to add that to the api.

http://api.drupal.org/api/function/db_delete/7 returns 'A new DeleteQuery object for this connection.' I'm not sure how to turn that into the TRUE/FALSE required by media_metadata_delete($mid), so for now I'm going to not return anything.

with RDFa going into core #493030: RDF #1: core RDF module we should examine that before going forward.

I've created File Metadata module to store metadata attached to a file. Using the new D7 file hooks, we are able to load and save data easily.

Your module merely needs to add data to the file:

$file->metadata = array(MODULE_NAME => 
    array(IDENTFIER => array(
      KEY1 => VALUE1,
      KEY2 => VALUE2,
   )
); 

Here, MODULE_NAME is your module name, IDENTIFIER is a unique id (if your module implements separate kinds of metadata- eg: one set for video, one set for audio). KEY is the name of the data you are storing and VALUE is the value.

The module hasn't even been tested yet, but in the interests of moving things along......

fixed; using fields now.

I am testing the media module, and I would like to use the id3 metadata. I am a bit confused on how to get the metadata, I do not find how to get the "fields" for that.

Any help would be much appreciated.

@davebv : There is nothing implemented for this presently. My guess is it would be implemented in a couple ways - 1 as a bridge between fields - so that fields could be created for id3 related stuff and as the file is updated, the fields get copies of those values.

Alternately, it could be loaded as needed through field hooks.

Basically, it needs to be built.

this was closed pre-maturely, and needs to be dealt with.

Looks like this issue wasn't important enough to block the beta? Any progress on this over the past year?

Seems like #780848: Merge getID3 with FileField Meta in Drupal 7 should fit into the plan somewhere?

We've got fields on files, I'm closing this.

@arthurf -- Mind providing a link to some documentation so users arriving at this page from Google have an idea where to start? Thanks!

Documentation can be found on http://drupal.org/documentation/modules/media Like most documented projects this is mentioned on the project page.
Of special interest for this issue are probably:
Using file fields in content types Media 2.x Quick Start Guide
Using fields on file types Displaying media