Media Image Transform (formerly Cropping)

Jacob,

I'm glad you're excited about this - thanks for all of the work you've put in to the media module. I'm launching a new project on D7 as soon as possible with a heavy focus on ease-of-use and the media handling aspect of D6 just isn't cutting it. This is coming out of necessity, so I'm looking to get a base implementation up ASAP, then refine it as time goes on.

At first, this will only support the Multimedia asset field via a "Media image selector with cropping" widget, where users can configure cropping dimensions on a per-instance basis. Down the road, I want to try to integrate this into the browser to allow cropping on a per-insert basis into a WYSIWYG textarea.

From looking at the code, I was thinking of this as a development strategy (using the widget):

1. User clicks "Select media", then uploads or selects an image.
2. Module JS intercepts the selected image information using the mediaBrowserOnSelect() callback.
3. An interface is created where we can specify the crop dimensions using jCrop. The image is maximized to the maximum available area using jQuery's innerHeight/innerWidth functions and jCrop's box sizing methods for scaling down large images (see attached media-widget-crop.jpg).
4. Image thumbnail is replaced and dynamic preview is shown and Re-crop button is added to edit the crop (see attached media-widget-crop-finished.jpg).
5. Crop data (start x, start y, delta x, delta y) is stored with the field data (which stores it in the _data column of the field's data table).
6. When the image is called, we would need to alter the URL to specify the cropping parameters, such as appending the crop data to the end of the image url (image.jpg?crop=100,100,500,250) since it seems that the only data is passed to the image is the filename and style. I believe the Imagefield Crop module actually made a copy of the image with the cropped dimensions, then ran any Imagecache actions on the copy. As I don't think it would be a good idea to have a bunch of cropped copies of images littering the Media library, we'd need some way of communicating the crop information when loading the image from the URL.

I'm a bit limited with my knowledge of the inner workings of the new Image API and the Media module so I'm not sure if this is the "proper" way of doing this but any input or insight would be appreciated.

This is my start - I'm eager to hear your thoughts.

Actually let me revise - I don't think that appending the cropping params to the image would be a good idea since that can open a security door to overload a CPU by rapidly requesting different crops, so one of three things will probably have to be done:

1. Make a copy of the image in the library and use that file. Unfortunately this may clutter up the image library unless we can add some sort of flag to these images for them not to display in the browser.
2. Create a database table that holds cropping info (or use an existing one such as the data column in the media field's table perhaps?) and pass a row identifier with the image. However, I'm not sure if usage records are created when images are inserted through the WYSIWYG editor (seems like params are just included in a JSON object).
3. Pass a HMAC together with the crop size to verify an approved crop size is being requested from the server.

As I've said, I have limited knowledge with how Media and some of the newer D7 architecture works, so there may be a way that I'm not thinking of.

Jacob,

I can see the data blob being useful for the field instances. However, the one problem I see is that including an image via WYSIWYG creates a JSON-based placeholder that seems like it has enough information to render an image by itself:

[[{"type":"media","view_mode":"media_large","fid":"6","attributes":{"alt":"","class":"media-image","title":""}}]]

Since this is the case, it seems that no database record is created, and since it has enough information, an image with the correct style could be rendered on demand. Since no database record is created, we would either have to create a table that held all metadata information and update it whenever changes were made in the WYSIWYG browser, introducing syncing issues and AJAX callbacks for in-editor WYSIWYG deletions, etc. (which you noted) or pass this metadata along with the URL when generating the image (which seems like the easier approach). Other than these two options, I'm at a loss on how we would communicate metadata to the image in question.

If we were to add cropping metadata to this, say:

[[{"type":"media","view_mode":"media_large","fid":"6","attributes":{"alt":"","class":"media-image","title":""},"metadata":{"cropping":[100,100,500,250]}}]]

it would translate to:

/sites/default/files/styles/large/public/grass.jpg?metadata=<?php http_build_query($metadata); ?>

Now, we have the information we need for cropping, but then it would make it easy for an attacker to perform a DoS by rapidly requesting random (and non-cached) crops. One way to solve this is to sign the metadata with an HMAC (we could use $drupal_hash_salt). The only downsides I see to this is a long query string appended to the image URL, and an image that isn't cropped if the query string isn't appended.

If anyone has any thoughts regarding this, I'd be eager to hear them.

Jacob,

I have the images cropping from a URL and verifying with a signature successfully, however I can't figure out how to create and register a javascript plugin for Media. I looked at the wysiwyg-media.js file, but it seems like it's being invoked from the WYSIWYG module's plugin system, not Media's, since it's namespace is Drupal.wysiwyg.plugins.media.

Since there's no up-to-date documentation, can you or someone who's familiar with Media's JS plugin system write a sample basic plugin outline that includes some sort of mediaBrowserOnSelect callback that works with the popup created by the Media widget? It would really help get this module going and also could be a reference for other would-be Media integrators.

Thanks.

Jacob,

Work is continuing on this and as long as you would be able to review and commit the JS patch in #972830: Create a sample plugin module to show how to extend the media browser, I should have a mostly-working cropping module to submit for review by the end of the week.

Thanks for all your help.

-cocoloco

Jacob,

I'm really close, but it seems I've hit a snag. I'm trying to figure out how to save crop information in the field table, but it seems (correct me if I'm wrong) but it doesn't appear that there's the ability to save data in the field's data column yet. The only field element that looks like it's in the form is the "fid" element.

I tried to see if there was a hook that I could use to preprocess the variables before they were saved, and it looks like hook_field_presave would be able to do that job. However, this can only be set on a per-field basis (not a per-widget basis) and I'd hate to create a new field (and consequently, table) when the media field has all the columns it needs to store crop information just fine.

I thought one possible solution would be to also include the data column as a hidden field in the form, just like you did with the fid column (from media_element_process()):

$element['fid'] = array(
    '#type' => 'textfield',
    '#default_value' => $fid,
    '#attributes' => array('class' => array('fid')),
);

// Create a place where we can access and edit the data column
// Transform current data where it's storable in a hidden field
// We'll need $.deparam to unserialize the data in the hidden field
$element['#attached']['js'][] = 'misc/jquery.ba-bbq.js';
// This can be decoded in jQuery like so:
// var data = $.deparam($('.data').val());
// Then updated like so:
// data.crop = {x: 100, y: 100, w: 500, h: 250};
// $('.data').val($.param(data));
$element['data'] = array(
    '#type' => 'hidden',
    '#default_value' => http_build_query($current_data), // $current_data is unserialized
    '#attributes' => array('class' => array('data')),
);

Then you can easily get this with all of the other form data:

// I forget the variable names that get sent back after a form is submitted, however you should get the idea:
parse_str($form['data'], $unserialized_data);

I'm not sure where in the media.module it saves this data to the field table, but if you could let me know, I could make a patch for the media module if you think this is a good approach. On the other hand, if you have a better approach, or if something is already created, I'm eager to hear your thoughts.

Jacob,

I think we have everything we need to currently add cropping data to the current schema:

--
-- Table structure for table `field_media`(only relevant columns shown)
--
`field_fid` int(10) unsigned DEFAULT NULL,
`field_title` varchar(255) DEFAULT NULL,
`field_data` longtext,

The existing longtext data column is more than enough room to store a serialized array used for cropping as well as other data, however I guess in D7 they made a push to store any serialized data in BLOBs instead of text-based fields (#690746: Text column type doesn't reliably hold serialized variables). In that case, the data field should be a BLOB anyway to do what I believe it intends to do (hold extra data about a field instance).

Of course I could also make another field type called "Media image w/ cropping" that users can select instead of "Multimedia asset" however it doesn't make much sense to create a whole other database table just to store a bit of metadata, which may be useful to other contributors.

Jacob,

Follow-up from my previous comment:

Looking through the code, it seems that the existing data field was made exactly for this purpose:

function media_field_schema($field) {
  return array(
    'columns' => array(
      'fid' => array(
        'type' => 'int',
        'unsigned' => TRUE,
        'not null' => FALSE,
      ),
      'title' => array(
        'type' => 'varchar',
        'length' => 255,
        'not null' => FALSE,
      ),
      'data' => array(
        'type' => 'text',
        'not null' => FALSE,
        'size' => 'big',
        'serialize' => TRUE,
        //'description' => 'Used for storing additional information.  Can be harnessed by widgets',
      ),
    ),
    'indexes' => array(
      'fid' => array('fid'),
    ),
  );
}

If that's the case, we just need to convert this to a BLOB field (see linked issue in previous comment) and provide a way that I can line in to update it (such as what is shown in comment #12).

If there is a performance issue with LONGBLOBs (though there shouldn't be any more than with the current LONGTEXT) I would say it would be OK to just use a regular BLOB - I don't think additional data will exceed 64KB (not for crops anyway) and we certainly wouldn't need the 4GB available as a LONGBLOB. MEDIUMBLOB (16MB) would be a good compromise but I don't think we could use that since it is not defined in the core database handler's getFieldTypeMap().

Let me know your thoughts.

@effulgentsia: That did the trick! Thanks for pointing me in the right direction.

@effulgentsia: Hopefully you have time to share your knowledge regarding the Fields API for one more thing.

I have my cropping widget saving the crop data to the Media field's data column successfully. However, when the field is being displayed, it needs to alter the URL of the image if cropping information exists in the field's data column. Since this is module currently only implements a widget, do I have the power to do this? Or would I need to:

• Create a field type specifically for image cropping? I've tried not to go down this route, since Media's field table already has all the columns needed for storing cropping metadata, and I'd essentially be creating the same table structure for this new field. However, it seems that hook_field_prepare_view() is perfect for this, although it seems most hook_field_* (with the exception of hook_field_widget_*) calls can only be attached to field modules, and widget-only modules cannot invoke them.
• Write a formatter and attach it to the Media field somehow?
• Ask the Media module to provide a hook somewhere?

If you could point me in the right direction, I'd appreciate it.

I've been busy with deadlines but am still working on this, and got it to the point where it will show the cropped image in the default display, and I'm still working on have it render a style using the cropped image. Will provide more updates soon.

I wanted to give everybody an update on this since it's been a little while since I've made any noteworthy news to this, however now that Styles 2.x is pretty well ironed out, it helps since last time I was heavily working on this, a rewrite was still being worked on.

In that time, I've been doing some thinking about the approach I took before, and I think it can be improved. This is because there are two parts to the Media module - 1) attaching media images as a field and 2) attaching media images inline in the WYSIWYG editor. Both of these have drastically different cropping metadata handling, since the field can store it in the database, however inline would require storing cropping metadata in the embedded JSON code. It has been my intention to store cropping metadata on an instance-by-instance basis so that we don't have duplicates of images in the library, all with different crops, which IMHO would be a mess.

As stated before, with a couple of tweaks to the Media module, I was able to store (and display) crops for field-based Media images and tweaked it to store this metadata in the database.

Then I got thinking, since we have this never-used-before metadata option, why should this be limited to crop data? Since implementing the metadata was the hard part, why wouldn't we add the ability to also specify vertical/horizontal flipping, rotation, or even specify ImageCache Action presets to image instances via this metadata system, in addition of the crop data? I feel like it may be a good idea to make this an instance-by-instance image filter system, so we can have something like (this would be an inline example, formatted for easier reading):

[[ { "attributes" : { "class" : "media-image",
	    "height" : "209",
	    "width" : "262"
	  },
	  "process" : {
	  	"crop" : {
	  	  "x" : 0,
	  	  "y" : 10,
	  	  "w" : 209,
	  	  "h" : 262
	  	},
	  	"flip" : ["horizontal"],
	  	"actions" : ["my_imagecache_action_preset"]
	  },
	  "fid" : "3",
 	  "type" : "media",
	  "view_mode" : "media_large"
} ]]

This is a pretty intensive use of this, however this snippet would crop the image, flip it horizontally, and run a custom ImageCache Actions preset on it (the backend would cache the images so this instance would only be generated once).

The only difference between the inline implementation of this versus the field implementation is that the process directives are stored in JSON vs. the database. It would be great to have a nice GUI that people can use to specify all of these directives (like WordPress uses), however that would have to come after we got the actual implementation working.

I'd like to know your thoughts on this, or if anyone has any ideas.

Thanks,
cocoloco

Jacob,

Again, thank you for all of your continued hard work on the Media module. I do agree that this should be a separate module that provides extended functionality (as you said, media_transforms or media_images or something...) and should be easily extensible for other effects/filters. It seems that this will heavily rely on using the styles_alter hooks or formatters to apply the effects (something that effulgentsia was trying to tell me a long time ago - before Styles 7.x-2.x came out) but didn't sink in until a lot later). Edit: I didn't see your note about the module_invoke hook, however that can be an option as well.

Media is a complex module, and it took awhile for all the things I originally learned about it through going through the code to sink in. Additionally, it took a bit of working with D7 and Styles for everything to make sense.

When trying to get this to work before, I did do some hacking to use the field data column that you provided in Media, which I used to store the serialized metadata. However, at that time, I didn't understand the difference/relationship between field formatters and image styles.

I agree, code shouldn't be duplicated at all where possible. I would like the related part of the JSON structure (when used inline) to essentially match the corresponding array structure (when used as a field) so we can essentially pass either one to the same function that creates and caches the filtered image.

The only problem I see is that, when used inline, it will have to translate some part of the JSON string into an image URL, and that URL could be quite long. Additionally, the URL would have to be signed so that people can't bring down a server by rapidly requesting different parameters (I've already coded a solution to this problem).

It's not the most elegant solution, however another solution to be only allow authenticated users to request image formats via URL (since we need this for previewing in WYSIWYG) and then, on node save, parse the body and save the inline images/params into a database. For instance if your body is:

<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nullam felis nibh, porta et pulvinar ut, iaculis non velit. [[{"attributes":{"class":"media-image","height":"209","width":"262"},"process":{"crop":{"x":0,"y":10,"w":209,"h":262},"flip":["horizontal"],"actions":["my_imagecache_action_preset"]},"fid":"3","type":"media","view_mode":"media_large"}]]Curabitur gravida, tellus in scelerisque accumsan, sem felis rhoncus ipsum, vel dapibus dui enim in ipsum. Nam nec molestie quam. Suspendisse sed nulla tellus, ut volutpat metus.</p>

It would find and parse the media tags on node save (while still allowing the WYSIWYG to render it via a URL). It would go ahead and insert it into a database table (sample SQL call, ignore the data serialization - it could just as easily be serialize()'d instead of JSON):

INSERT INTO media_image_inline (nid, fid, field, delta, metadata) VALUES (1, 3, body, 0, '[{"process":{"crop":{"x":0,"y":10,"w":209,"h":262},"flip":["horizontal"],"actions":["my_imagecache_action_preset"]}]');

Which would return an identifier (for this example, say 255), which we can use to call the image. So instead of calling:

http://example.com/sites/default/files/styles/media_large/public/my_image.jpg?crop=0,10,209,262&flip=horizontal&actions=my_imagecache_action_preset&sig=033292b555545793d62faa6073bc6545

we can use:

http://example.com/sites/default/files/styles/media_large/public/my_image.jpg?mfid=255

Then, every time a node is saved, we can update this table. We wouldn't have to rely on tracking via JavaScript since the WYSIWYG could still request the params via GET (as in the long example above) since the user was logged in and they would have the proper permission to.

I thought of this while writing this response so I may not have thought it through, but on the surface it seems like it may be a possible solution (hopefully what I wrote makes sense).

-cocoloco

I'm going to be working as much as I can in my free time on making the above specs a reality as a separate add-on module to Media. As soon as I have enough code to release a full project that can be reviewed, I will post the project details here.

Thanks for your input and interest in this.

Just so people can follow along with the development of this project if they are interested, I've started an experimental project at http://drupal.org/sandbox/cocoloco/1090384. Once it is mature enough for others to test, I will submit it for review.

I'm looking to have an alpha in May/June - unfortunately this takes some changes to the Media module, which I am in the process of branching so I can make integration improvements.

Just an update - I'm making decent headway on this - it will need some patches to media and styles in the meantime but should have something ready for testing in the next couple weeks.

I've made a preview of the initial functionality for the Media Image Transform module - can you please take a look at the video and let me know your thoughts?

http://bit.ly/mitransform

I'd like to hear your feedback. Thanks.

I've been incredibly busy lately however I would like to help integrating this into 2.x (or at the least make sure it will be eas(ier) to integrate into 2.x). I also have diffs of changes that I had to make to 1.x to make what you see in the video work properly.

Feel free to message me if you need anything - I'll follow up shortly.

drzraf,

Please note that the last thing this has been tested on is media-7.x-1.0-beta4 that I had to make the attached changes to and this project still has a long way to go. However, it should work in Chrome with the code found here: http://drupal.org/sandbox/cocoloco/1090384. Since I'm short on time, I'll attach a previously-sent e-mail on the subject (please ignore references to the repo since you won't need it since it's just a copy of media-7.x-1.0-beta4 with the attached patch applied):

I'm currently looking to minimize the amount of changes required to the Media module - currently this requires a change to the createTag function of wysiwyg-media.js that parses the transform key in the JSON tag. Altering this function in Media for this purpose is unacceptable since it would have no use for other modules; consequently, this functionality should be in the Media Image Transform module. I've tried to put this functionality in MIT's attach() function, however since Media's attach function gets called first, it parses the tags and strips the transform data. The solution I am working on involves having the WYSIWYG editor generate an ID for each image and maintain it's own tag map of IDs to transform data, which it will attach to the JSON tags on detach(). I've created a new branch that will incorporate this internal tag map, so for the time being, please clone from the master branch to test. Once I remove this functionality from the wysiwyg-media.js file, essentially the only change left (at this point in time, at least) will be adding an alter function to allow other modules to modify the settings array (which I think other modules could benefit from).

Just a reminder - please try this with Chrome or Firefox - I think Safari has some problems and I didn't even try to test on IE (I wanted to keep my morale up). Additionally, it only works with TinyMCE - not CKEditor support as of yet.

Please let me know if there is any other way I can help or if you have any questions.