Problem/Motivation

In #1447790: image_load makes admin/content/media unusably slow the {image_dimensions} has been added to cache image dimensions and avoid re-reading an image file when its dimensions are needed.

Instead of doing it only for image dimensions, the File entity module should provide a generic framework for its consumers to retrieve files meta-data. This way we could prevent having the same kind of performance issue with other kind of meta-data (EXIF, ID3, etc.).

Proposed resolution

On file insert, update and load the meta-data for a file should be read and stored in some kind of generic store and added as an array in a property of the $file object (something like $file->metadata).

Getting a file meta-data should of a file should be pluggable. Additional modules should be able to provide meta-data to File entity.

The meta-data storage should be flexible enough for any type of meta-data (string, number but also more complex types).

The main goal should be to provide an unified solution for consumers of file entities to efficiently retrieve meta-data when using file_load and file_load_multiple. To keep things simple, providing efficient querying over the meta-data should be the concern of separated and dedicated modules. For instance, a module could store selected ID3 audio and video meta-data as fields values while another could store the selected EXIF meta-data in a dedicated table exposed to views.

An alternative solution may be to lazy-load meta-data when requested once we have a FileEntity class (needs #1361226: Make the file entity a classed object which is blocked by #1401558: Remove the usage handling logic from file_delete()). This way we avoid the overhead of loading too much rarely used data on file loads.

The two approaches could also be combined with basic meta-data loaded on file loads, but with lazy-loaded extended meta-date lazy-loaded. For instance, image dimensions and base ID3 tags (title, artists, album, etc.) could be loaded on file load while things like video/audio play time, bitrate, channels number, codec, etc. could be lazy-loaded (in groups) when requested.

API changes

A new hook or plugin should be added to allow modules to provide their own meta-data.

Files: 
CommentFileSizeAuthor
#20 1496942-20-metadata-api.patch19.95 KBDevin Carlson
PASSED: [[SimpleTest]]: [MySQL] 874 pass(es).
[ View ]
#17 1496942-metadata-api-17.patch14.53 KBaaron
FAILED: [[SimpleTest]]: [MySQL] 768 pass(es), 24 fail(s), and 38 exception(s).
[ View ]
#16 1496942-metadata-api-16.patch14.6 KBaaron
FAILED: [[SimpleTest]]: [MySQL] 298 pass(es), 5 fail(s), and 0 exception(s).
[ View ]
#14 1496942-metadata-api-14.patch14.59 KBaaron
FAILED: [[SimpleTest]]: [MySQL] 298 pass(es), 5 fail(s), and 0 exception(s).
[ View ]
#10 1496942-metadata-api.patch14.52 KBDave Reid
FAILED: [[SimpleTest]]: [MySQL] 298 pass(es), 5 fail(s), and 0 exception(s).
[ View ]
#8 1496942-metadata-api.patch13.83 KBDave Reid
FAILED: [[SimpleTest]]: [MySQL] 808 pass(es), 20 fail(s), and 427 exception(s).
[ View ]
#7 1496942-metadata-api.patch10.94 KBDave Reid
FAILED: [[SimpleTest]]: [MySQL] 808 pass(es), 20 fail(s), and 427 exception(s).
[ View ]
#6 1496942-metadata-api.patch10.94 KBDave Reid
FAILED: [[SimpleTest]]: [MySQL] 808 pass(es), 20 fail(s), and 1,587 exception(s).
[ View ]

Comments

Issue summary:View changes

typo

Issue summary:View changes

Add lazy-loading idea.

Issue tags:+sprint, +Media Initiative

Assigned:Unassigned» Dave Reid

We are going to go ahead with a {file_metadata} table and convert the {image_dimensions} table to use the new format since it will be compatible with the new Entity Property API for D8. The only things that should be stored in this table are things that can be extracted from the raw file in the file system without any kind of Drupal context: things like image dimensions, audio track length, video size and length, etc.

this was discussed in very early meetings, as far back as 2008. It is a very good idea, that should allow for all kinds of cool uses, such as integration with GetID3, storing YouTube information, such as title and duration, and lots of other useful tidbits

Issue tags:+7.x-2.0 alpha blocker

I would really like to add this to the blocker list as something to push for in D8 before freeze and since it adds an API I want people to start being able to use it.

Status:Active» Needs work
StatusFileSize
new10.94 KB
FAILED: [[SimpleTest]]: [MySQL] 808 pass(es), 20 fail(s), and 1,587 exception(s).
[ View ]

Initial patch adding a metadata API.

StatusFileSize
new10.94 KB
FAILED: [[SimpleTest]]: [MySQL] 808 pass(es), 20 fail(s), and 427 exception(s).
[ View ]

Revised patch, still need to remove some references to image_dimensions.

Status:Needs work» Needs review
StatusFileSize
new13.83 KB
FAILED: [[SimpleTest]]: [MySQL] 808 pass(es), 20 fail(s), and 427 exception(s).
[ View ]

One more version.

Status:Needs review» Needs work

The last submitted patch, 1496942-metadata-api.patch, failed testing.

Status:Needs work» Needs review
StatusFileSize
new14.52 KB
FAILED: [[SimpleTest]]: [MySQL] 298 pass(es), 5 fail(s), and 0 exception(s).
[ View ]

Revised patch that fixes a major bug in the new function.

Status:Needs review» Needs work
Issue tags:-sprint, -Media Initiative, -7.x-2.0 alpha blocker

The last submitted patch, 1496942-metadata-api.patch, failed testing.

Status:Needs work» Needs review

#10: 1496942-metadata-api.patch queued for re-testing.

Status:Needs review» Needs work
Issue tags:+sprint, +Media Initiative, +7.x-2.0 alpha blocker

The last submitted patch, 1496942-metadata-api.patch, failed testing.

StatusFileSize
new14.59 KB
FAILED: [[SimpleTest]]: [MySQL] 298 pass(es), 5 fail(s), and 0 exception(s).
[ View ]

Status:Needs work» Needs review

StatusFileSize
new14.6 KB
FAILED: [[SimpleTest]]: [MySQL] 298 pass(es), 5 fail(s), and 0 exception(s).
[ View ]

StatusFileSize
new14.53 KB
FAILED: [[SimpleTest]]: [MySQL] 768 pass(es), 24 fail(s), and 38 exception(s).
[ View ]

Status:Needs review» Needs work

The last submitted patch, 1496942-metadata-api-17.patch, failed testing.

Category:feature» task

Status:Needs work» Needs review
StatusFileSize
new19.95 KB
PASSED: [[SimpleTest]]: [MySQL] 874 pass(es).
[ View ]

A patch to address a number of small issues with #10 (mainly using db_merge instead of db_insert and accommodating changes in tests).

Status:Needs review» Reviewed & tested by the community

bravo!

Status:Reviewed & tested by the community» Fixed

Committed #20 to File entity 7.x-2.x. Thanks everyone!

Please file separate follow-up issues for any enhancements you require, locations the API could be implemented or troubles with the upgrade path you come across, etc.

Automatically closed -- issue fixed for 2 weeks with no activity.

Issue summary:View changes

And related issues references.