Mapper for Media

I second that!

+1 Subscribing

+1 Subscribing

Subscribe.

There is a patch to get media:youtube fields to be filled through feeds #1109490: media(_youtube) integration in Drupal 7 - this issue is under Feeds: YouTube parser https://drupal.org/project/feeds_youtube . I have noted on that issue that such work should be directed over to this issue queue.

a duplicate or related issue within feeds module #1041804: Feeds and Media Module: A Drupal 7 Media Utopia?

Let's see what we want.

• Source:

  If the source is youtube://v/rF1X12PE6PY or public://x/y the import will be pretty easy. Maybe this would be something to start with.

  But what if we are dealing with http://example.com/bla. Is this an image or a video? Shall we request it and see what the mimetype is? Shall we save it on the server?

  What if it is http://www.youtube.com/watch?v=rF1X12PE6PY? A youtube video, obviously. Should that be hardcoded in feeds or should there be a plugin mechanism? Media itself uses modules for Youtube, Flickr and so on ...

• Target:

  The target could be a media field on a node. At least that is what I would use it for.

  But there are more entities in Drupal 7. Most of which behave generic (I hope). What Stephen Piscura suggested in #1041804: Feeds and Media Module: A Drupal 7 Media Utopia? would also be useful. Here, however, if we manage the first (media field on a node) we are already creating media assets.

-- Kind regards, Niklas

Throughout this response i refer to Media Fields (Multimedia Asset fields) as Media Entities, since Media Fields within an Entity (Content/Taxonomy/User/etc.) can have fields of their own -> hence Media Entities refer to the actual Multimedia Asset field & its attachedfields and not the entity of which the media field/ entity is attached to.  Media Asset should have probably been used in some places that I used Media Entities (as Media Asset refers to the actual unique content/media file & the data in its corresponding attached fields as structured by the Media Entity and created by populating/filling a Multimedia Asset field within any Entity with such a field attached.
Media Asset = the actual multimedia stored as a unqiue Media Entity (a.k.a. actual media content),
Media Entity =  an entity composed of a Multimedia Asset field with "sub-fields" attached to it (a.k.a. Sub-Entity: turns Multimedia Asset field into an entity by allowing extra fields to be attached)(structure)
Multimedia Asset Field (Media Field) = a field which attachesMedia Assets as a Media Entity into any Entity.  (structure)
Entity = Anything that can have fields attached to it  (structure)
Sub-Entity = Any entity with an entity within it (acting as a fieldable field) (structure)

• Source:

  
  

  - I agree, starting with known media providers should be 1st. 

        "But what if we are dealing with http://example.com/bla. Is this an image or a video? Shall we request it and see what the mimetype is? Shall we  save it on the server?"

  - URLs (sources) which are questionable (ex. unknown if it is an image or video) would historically be dealt with by a independent Media provider module.  BUT if you can develop a system which alters to support new media providers on the fly -> then this would be ideal. 
    - Extracting the mimetype would be the better of the 2 options you presented -> because there is no way to download EVERY external media file type to the server for further analyzing.   For ex. you can only download YouTube videos by caching them in your browser (hence a module which would automatically download youtube videos to your server upon creating a media entity (or by tiggering an action) is not possible - at least this is what i have been told by developers.  if this is possible -> please contact mel and i will surely fund it!). 
    - If you want to use the download method, the FileField Sources module http://drupal.org/project/filefield_sources could provide the basis for developing a "Media Sources" module - since FileField Sources allows you to populate a CCK file/image field by transfering files from a remote server (cURL required)-> the Media Sources module could do the same.  Altough - as outlined under "Target," Media Entities may soon be able to become File Entities, so Filefield Sources may integrate with Media-based File Entities at that point.  Although a solution for determining file type(mime type) from a "random url pattern" would still be needed.

       "What if it is http://www.youtube.com/watch?v=rF1X12PE6PY? A youtube video, obviously. Should that be hardcoded in feeds or should there be a plugin mechanism? Media itself uses modules for Youtube, Flickr and so on ..."

  - The full URL is all that needs to be mapped into the Media Entity - as the Media provider modules (Youtube, Flickr, etc.) handle the extraction of media data from the source.  Therefore it does not need to be hardcoded as the determination of which media type (video/ image/ etc.) it is -> is determined by the Media provider modules.  The actual display of the Media Entity is controlled by the Styles module http://drupal.org/project/styles
  - If you want to develop a separate "Media Sources" module to provide a general plugin mechanism for uknown/non-supported media providers (as outlined above) -> then that would be great.

• 
  

  Target:

  
  

  1) Media Entity - Media Field on an entity/ content type (a.k.a. Multimedia Asset field (since media module compacts all media types into a single field type -> and in the config. you determine 1a) types of media (video, image, etc.) 1b) types of sources (public, private, youtube (http://drupal.org/project/media_youtube), flicker (http://drupal.org/project/media_flickr) , etc.)
  2) File Entity - According to #1086958: Switch from Media Entity to File Entity + #1087372: Extract file/media entity api into separate module, media module may soon be able to store the data as a file entity instead of a media entity.   This should be kept in mind when developing the targetting system

        "But there are more entities in Drupal 7. Most of which behave generic (I hope). What Stephen Piscura suggested in #1041804: Feeds and Media Module: A Drupal 7 Media Utopia? would also be useful. Here, however, if we manage the first (media field on a node) we are already creating media assets."

  - What Stephen Piscura seems to be suggesting is to have the meta data attached to the Media Field (Entity) itself rather than just the Entity to which the Media Field (Entity) is attached to. 
     - INSTEAD OF MAPPING THE FEED TO THE MEDIA ASSET LIBRARY (AS HE OUTLINES), IT SHOULD JUST BE MAPPED TO A MUILTIMEDIA ASSET FIELD COMPOSED AS A MEDIA ENTITY WITHIN ANOTHER ENTITY.  Upon intitially creating entities with Multimedia Asset field(s), the URL of the Media Asset Field would be compared to the existing Media Assets library.  If it already exists, then that pre-existing Media Asset would be embedded into the new entity/node.  If the media entity type is configured to accept updates then actions would take place accordingly(see BELOW for full details)
      - For ex. in Drupal 6 using Embedded Media Field & Feeds/ FeedAPI to embed youtube videos -> would allow you to only have access to the related (meta)data provided by 1) the meta-data the youtube API provides through their embedded player via a specific URL(video) -> 1a)video url 1b)duration 1c)video status 1d)etc. (ratings, # comments & so on) & 2) the "add-on" data which can be extracted from the rss feed upon mapping it to a node -> 2a) video title 2b) author 2c) post date 2d) video description 2e)tags/ categories 2f)keywords 2g) etc.   So with this system it was impossible to attach all the meta-data to the actual embedded media field (as that was limited by the youtube api) & the embedded media field could not become a fieldable entity (in D6) -> and therefore it couldn't have the additional add-on data extracted from the rss feed attached to the actual (embedded) media field.  Thereby you needed always refer back to a specific node to provide ALL the meta-data surrounding a specific video stored in an embedded media field.    This setup is no longer ideal as Media Assets (a.k.a. individual populated Multimedia Asset fields) are stored and controlled by a central "library".  They can be re-used in various places throughout the site.  Through the new media field/entity system, by applying the methodology used in D6 to attach add-on data to an embedded media field -> upon re-insertion of the media asset through the media library, the media asset would become detached from its initial node & would therefore loose the add-on data which was on the initial node. 
     - Ensuring that this add-on meta-data (a.k.a. related data from sources beyond the (youtube) api.  example: RSS Feeds) stays closely attached to the Media Entity (Field) itself is the ideal solution. 
        - APPLICATIONS!  Sync specfic (meta-)data fields with remote server via API (determined by Media Provider modules) or Feed-based updates (potentially in real-time thanks to PubSubHubbub)=>  Control which (meta-)data/fields get automatically updated based upon changes to a remote server via API (ex. youtube) OR via rss.  So instead of replacing the media asset as a whole upon update (based upon feeds current duplicate items actions) you could update specific customizable parts them (specfic fields or meta-data) while keeping other (non-synced) data unchanged.  Perhaps specific meta-data fields could be triggered to update at certain times or upon certain actions occuring.  For ex.  you could sync the Ratings, # Comments, or # Views from the YouTube onto the Media Asset in real-time while allowing other fields to remain unchanged.  If you had 1 rating (or # views) field for youtube & then one for on-site, you could create popularity views comparing or combining the latest statistics from your website vs. youtube stats.

• ABILITY TO MAP TO MEDIA ENTITY (MULTIMEDIA ASSET FIELD) USING FEEDS (patch started already!)
• Extracting Media Mime Type/ File Type data from a "random pattern url" on a remote server (random pattern url = no discernable media/file type in url) -> Automatically determining Media Entity type (video, image, etc.)
• Downloading media (determined by media entity) to local server from a remote server

POSSIBLE SOLUTIONS BY AUSTIN (TimelessDomain)

• NEW Media Sources module - Same as Filefield Sources http://drupal.org/project/filefield_sources but for Media Entities http://drupal.org/project/media
•   - Mime/ File/ Media Type Extraction
•   - Sources for populate Media Entity/Field: 1)Transfer files from a remote server (cURL required) ) - the following features are already provided in-part by a pre-existing Media module - Media Browser Plus http://drupal.org/project/media_browser_plus, but i included them since I figuire Media Sources could be the counterpart to Filefield Sources. 2) Re-use files by an auto-complete textfield 3)Select existing files through the IMCE file browser  (some of these are already provided by other Media modules
• - To get Feeds to work properly with Media module, as per Stephen Piscura's recommendations we would -> Ideally be storing all meta-data & add-on data in fields attached to the Media Asset as a Media Entity, while still being able to pull all that data into another Entity which has the Media Asset embedded through a Multimedia Asset field in the form of a Sub-Entity Media Entity. Media Assets would be checked for uniqueness
• 1) Need the abiltity to map to an Entity with a Multimedia Asset field (currently possible w/ patch)
• 2) Need the ability to map to Sub-Entities while mapping Entities within Feeds (in this case ability to map to the fields of a Media Entity which is acting as a Sub-Entity to some other Entity)
• 3) Need to automatically create Media Assets from creating Entities which have Multimedia Asset fields OR Sub-Entity Media Entities
    3A) Need ability to check for duplicate Media Assets (based upon URL OR filename of media)
    3B) Need ability to update OR replace Media Assets automatically based upon default settings (for the true potential of this see ABOVE for my explanation of how to sync/ update specfic meta-data fields with a remote server via API or Feeds)
    3C)Integration with diff. module to shows changes to updated Media Assets?
  THANKS Niklas for the vigor.

Thank you TimelessDomain for clearing this up.
Especially for your suggestion to add a media_sources module. In fact that suggestion is so good that this module is called Media Internet Sources (media_internet) and comes with the media module.
If it was called media_internet_sources I might have seen earlier that it provides the plugin mechanism we need.

I have rewritten my patch for feeds_youtube to use that module and be more generic. It adds the file media.inc.
* Patch attached *
* Download linked *
(At least on my machine) I can now import media from every source that plugs in to media_internet.

_

Two issues remain:

• media_internet is great, but it was meanted to interact with the user. If the user tries to add a file that already exists, an error message will be shown and the user can resolve the issue.

  We however want to do that programatically. Best practise would be:

  // Find the right provider (Youtube, Flickr, anything)
  $provider = media_internet_get_provider($source_value);
  
  // Let the provider validate the $source_value.
  // Sadly, this will fail if we have duplicate files
  $provider->validate();
  
  // Now get the file
  $file = $provider->getFileObject();
  

  But because $provider->validate() will fail for duplicate files we have to:

  // Find the right provider
  $provider = media_internet_get_provider($source_value);
  
  if (/* Find out, if the file is a duplicate */) {
    // Skip validation :(
    // Make the file ready for update
  } else {
    // Do validation
    $provider->validate();
  }
  
  // Now get the file
  $file = $provider->getFileObject();
  

  (I hope I manged to explain what I mean, using this pseudo code.)

  Nonetheless the import currently works fine, because currently the providers don't rely on validate() being called before getFileObject(). And currently they don't do anything other than checking for dupes in validate().

  Still, this should be fixed.

• I don't know how to handle errors that occur while importing. So I just used drupal_set_message. Doing so the user will never see errors that occur during an automated cron run. Better use watchdog here?

_

Now. This only maps a source to a media field on a node (or another entity). Meta data is only added by the media_internet provider. With the patch it is not possible to import meta data from a feed source and attach it to the media asset field - nor did I add a processor to directly import media entities.

_

@ Maintainers of the feeds module: Please comment on the drupal_set_message thing. Maybe you could also create a feature branch in GIT.

@ Maintainers of media_internet respectively media: What would be the best way to solve the $provider->validate() issue? They probably don't hear this, so I should go ask them^^

_

That said: Downloading youtube videos to the server is possible. In youtube-dl rg3 did a good job at writing python code to do it. That could be ported to PHP and cURL. (Maybe there are working PHP solutions already.)
A similar thing was the "download it to the server and detect mime-type thing" I suggested.
These two could be done by writing a plugin for media_internet (if there aren't any yet). As TimelessDomain proposed this is not an issue for feeds anymore, then :)

_

Phew.

-- Kind regards, Niklas

They probably don't hear this, so I should go ask them^^

#1121808: Change file_uri_to_object to re-use files by default and remove duplicate validation checks in handlers

Any chances of getting this comitted? Do you need test coverage for that? Or would this be better placed in a feeds_media module? And, of course: How to handle errors, while importing?

subscribe

After discussing with some people I found that a seperate Media Feeds integration module would be the better way. Media is a fast moving target, Feeds is relatively stable yet.

Here is the sandbox: http://drupal.org/sandbox/niklas/1150302
Here is the application to make it a full project: #1150410: Media Feeds

I am happy to announce that Media Feeds is now a full module and has it's first release. Thanks to all who helped reviewing and testing.

My friend, you sir are a golden god. Thanks.

+1