Identify missing features in File_Entity

We have been creating a scald_mediasync module that syncs media entitys into scald atoms. This gives as lots of possibilities to use scald, but also the media modules features (feeds import, mass import)

I hope we can push the code into sandbox soon.

@wroxbox I did see your scald_file sandbox ;) http://drupal.org/project/scald_pdf extends it to use more File Entity features, and for a specific use case (PDF), but it's generalizable. Thus I'd love to see your scald_mediasync (or rather scald_filesync?) module. I've created an outline about migration to Scald http://drupal.org/node/1806816 (a real outline!;))

That said, I do think a scald_filesync is a stop gap solution. If every file should become an atom, Scald would have been built on top of File Entity. A classical question is: we don't want that any file uploaded via an arbitrary file field goes into the library (but the reverse is easy, i.e. if you want all files uploaded via a specific field become atoms). Besides, I'm not confortable that a YouTube video or a tweet is a file, in the sense that a file must have a size (a file in Drupal is not a Linux file ;-)) or a MIME type. An atom is more general (thus less specific) than a file. Certain types of atom could be built based on File Entity, not all. I do think these two entity types are different and it is difficult to merge.

I'm currently working on the scald_file / scald_filesync modules @wroxbox mentioned.

I ran into a problem of not being able to identify the (Drupal) files created in Scald compared to the ones created elsewhere. By not being able to tell where the file originates from, we're locked in an infinite loop of creating a file, which creates an atom, which creates a file, which creates an atom, etc. I would need Scald to add some kind of a flag to the created file / entity so we know where it's from.
I was thinking about using the file_usage as an identifier, but I can't figure out where we could access the file early enough and already have a scald id.

Any thoughts on how we could resolve this?

#3 When the $file->status is marked as permanent in file_field_presave(), it's too soon for the atom to have an sid. I think you are using hook_file_insert() to sync. To do it without hack in Scald core (add a flag etc.), I track the whole workflow:

- When a file is uploaded, it fires hook_file_insert() and hook_entity_insert(), with $file->status = 0.
- Then when the host entity is saved, in field_presave it marks that file as permanent, then saves that file (and triggers hook_file_update() and hook_entity_update() for that file), then triggers hook_entity_(insert|update) for the host entity.

Thus, to have an unidirectional sync file -> atom without infinite loop, I'd:
- Use hook_entity_(insert|update) and check that is not a file entity. Check if there are permanent files that don't have an atom then create an atom.
- Use hook_file_insert(), then add the atom creation job into a queue. In this case there is a lagged for independently uploaded files, like files created through the Media module (I don't check if there is a hook to help that).

@afox should you create a sandbox for Scald File module? Then we could move this issue into that, so that we can see code and submit patches. Or leave this issue as a meta issue, and create a new issue for infinite loop problem.

Funny thing... we spoke with @wroxbox just 15 minutes ago and decided to POC Solr indexing for Scald atoms, thus removing our need to sync with Files.

We did come to the conclusion that in our current project Scald currently does everything File entity does, and then some... But it doesn't yet have the deep support from other related modules (like searches). I think Scald atoms for everything Files are used for, is more scalable in terms of content management and editing.

However, the question is, is there anything Scald can't do, which files can? Should we focus more on getting the same support for scald than to try to sync scald with files?

I do think Scald supports well Search API (there was an issue for that). About the File Sync module, I think need is simply to have some files (like: all files uploaded in the field_book_cover) available as atoms. Also, we can reuse some modules that support files (and don't have to develope/clone a new provider) - it's the case with many video providers.

Other than that, I don't see anything in File that we need in Scald.

@afox, @wroxbox I think the File Sync module could perfectly fit #1951276: Migrate from Media 2.x to Scald (I guess it needs an retroactive sync for the migration). Could it be published as an independant project?

There is Scald File provider http://drupal.org/node/1874994. Scald sees media entity differently and tries to build upon file entity brings nothing to Scald.