Quick Process Overview [#830864]

Could someone please post a quick overview of the File Framework process for uploading and converting a file?

For example (NOT real ... I'm just making stuff up!)

1) Installing File Framework REPLACES the existing file upload feature. Note that since the existing file upload feature will still be available (there is no way to remove it), anyone who wishes to use the File Framework file conversion process MUST be forced to use ONLY the File Framework file upload feature, which looks like this: [screenshot]. This may result in at least TWO file upload mechanisms, so be sure to distinguish between them.

2) Use ONLY the NEW File Framework file upload mechanism (as seen in the above screenshot) to select and upload a file type listed in the "Handlers" configuration section (or MIME Types or Converters or whatever section is appropriate ... I sure can't tell you!)

3) The "file_convert" module is then called to process the temporary file

4) The file_convert module tries to process the temporary file through the File Framework Converters by invoking the "find_converter" function, found in some_dang_file.module

5) If the conversion fails, an entry is made in the [x] database table indicating the failure, and the next conversion is attempted by repeatedly passing the file reference back through the find_converter function until it runs out of converters to try.

6) When the conversion process is complete, the original and generated files are stored in the Bitcache storage directory, and references to each successful converted file are stored in the [x] database table.

Or something like that, please. Something that describes the path through the functions that each attempt to convert a file takes.

Are there any plans to put together some basic documentation for file conversions with File Framework/Bitcache/RDF/etc.? (Not the "what to install" docs ... those exist. I am referring to a "how to do a file conversion like we say you can" document.)

I really, truly appreciate ANY help at all with this. Seriously. ANY help. It's been a long month of unsuccessfully trying to get this working on a brand new system set up just for this purpose.

Thank you very much for your attention.

Comments

Comment #1

miglius commented 2 July 2010 at 14:15

Thanks for posting this write up. You have covered it right. Here is some addition to make things clearer:

file_convert module itself does not define the conversions. It is responsible for the execution of the file conversions defined in the file format modules (file_document, file_image, etc.) which are located under the "format" sub-folder.

There are two types of conversions - from one MIME type to the other like convert the PDF file to the snapshot image which are configurable at the file_convert configuration pages and those conversions which does not change the MIME type, for instance creation a thumbnail of the uploaded image. The conversions which change the MIME type to avoid endless conversion loops can happen only two layers down. So for instance if the MS Word file is uploaded it will be converted to the PDF file (1st MIME conversion), then the PDF will be converted to the Flash video (2nd conversion). All possible conversion paths in the tree are triggered, so the PDF generated after the 1st conversion will be also converted to the image and the thumbnail created (the last conversion does not change the MIME).

The relations between converted files is stored in the RDF.

From the cron each uploaded file is executed in the sliding window manner (a number of files in one go) and missing or newly configured conversions are executed for the files being inspected. It means if you install a missing conversion utility, it will be triggered for all already uploaded files.

If file conversion failed for any reason, then this is marked in the RDF and this conversion won't be run for the same file on the next cron runs to save the cpu time and server load. However, there is a setting under the module settings allowing to clear all marks about failed conversions, then all failed conversions will be re-run again from the cron. This can be handy if one fixes the conversion string or installs missing utility.

If you would volunteer to compile a documentation I would be glad to contribute as much as I can.

Also, if you have any questions about the design or want to improve it, don't hesitate to open issue about it.

Comment #2

zwhalen commented 16 July 2010 at 20:31

I seem to be having a lot of trouble with this module, and since this issue is related to process, maybe it's better to post here than start a new issue.

Basically, I'm not getting any previews, any conversions, anything. It's probably because I haven't set up the module correctly, but since there's no documentation, I'm a bit at a loss.

Mainly, I'm interested in dealing with .doc files, and I want the ability to view them inline in the browser.

I followed the directions here, and got everything working except unrtf.

I enabled the File module and the Document format, and I set the conversion settings for .doc -> .pdf with all the correct file paths to jodconverter.

When I upload a doc, however, it just gives me the option to view, download, info, or properties. The "view" option doesn't show inline as advertised -- just downloads the doc.

I notice you refer here to pdf conversion with a thumbnail generation. That would be awesome, but I can find no relevant settings to make that happen. You also refer to the pdf converting it to SWF. Again -- awesome, but how do I make that happen?

This seems like a super powerful module, so I'd love to get it working! Any suggestions?

Thanks
Zach

EDIT: Hm, well, I seem to be making progress. Apparently, one needs to enable other format modules -- even for formats I'm not specifically using, like Image -- for the conversion options to show up. I'm still having a bit of trouble with openoffice and a never-ending cron job, but I think I'm on the right track. Suggestions still welcome!