Mapper for FileField / ImageField

I'd love to see this :) Mind to post a patch?

Hi Simon,

can the patch in #311404: image and audio file mapper and yours be combined to solve issue #224235: Mapper for CCK Image Field also?

Yep,

the EM-fields just generate the code to include the media from the original server. They do not download the files.

This is starting to look good. Two things:

* Please don't use drupal_set_message() in the mapper.
* I'd love to see a real patch.

#4 - imagefield builds on filefield functionality, right? Where is the overlap here? Once we have a file field mapper we should be only a step away from an image field mapper, right?

This is a dev version issue.

Tried code in post no. 3 but does not seem to work with latest version of the mapper...Any update on this code? Anyone has it working yet?

Thanks,
Patchak

Hi everybody,

this is an amazing work, thanks lyricnz for sharing your code. As Im working with the lastest feedapi_mapper, I had to made some changes in order to make it work.

You just have to put this file within the mappers directory of feedapi_mapper module, that's all.

I've changed the drupal_set_message for watchdog functions.

Hope it is helpful.

Thanks!

#12 worked wonderfully for me.

I had to make a customization to it to get the download to work, but that is because I was working with an extremely odd feed.
+1

Please create patches properly - if you don't have write access to the repository use "cvsdo add" - this makes it much easier to review a patch quickly - thank you.

copy() is using stream wrappers - http://us2.php.net/copy - Drupal file API does not yet support them #227232: Support PHP stream wrappers.

I think it's fine to use copy() but we should test whether it was successful and add a watchdog warning if it wasn't.

Filefield support is image field support. Renaming this to make it findable for people who are looking for imagefield support.

Hey! I'm currently working on and testing a version of this for 6.x-2.x, I'll post the patch shortly.

David

Attached is patch for 6.x-2.x . 6.x-1.x should be very similar. Tested primarily on feeds from flickr with enclosures--works for both file and image cck fields. Enclosures and plaintext urls should be supported, works best with SimplePie parser.

Correct version then. The difference between 1.x and 2.x is the function signature of the hook and the call to feedapi_mapper_get_item_node_type() that 1.x does not require.

So far, I have seen five image fields mapper patches proposed since last year. They are

1) In this topic #19
2) In this topic #15
3) #535970: Mapper for imagefield
4) #311404: image and audio file mapper
5) #224235: Mapper for CCK Image Field

These are the ones that I have seen so far. Maybe there are more. It is confusing. I think this is a highly anticipated feature, so people keep writing similar patches doing the same things since last year. Maybe we should compare these patches and choose one of them and concentrate on it.

For example, a feature that I expect is the ability to import all images in a Flickr set to a multivalued imagefield in a single node. I want to store all my images (Flickr set) in a single node. Currently, all images are created in separate nodes. I checked the patches, but it looks like the patch at #15 in this topic and the patch at #535970: Mapper for imagefield do not support it. Right? I could not check #19, because it does not have 1.x version.

#21 - I agree, we should merge these issues. I've already done that with 3), and now also with 4). I've posted a notice on 5).

Right now, this patch is the front runner. If we add tests we can commit this. Version 1.x and 2.x are only mildly different in regards to the API and the UI for testing, so we should be able to commit this patch and tests to both branches.

Needs to query file destination via filefield_widget_file_path(). Use feed item node's user for that.

Fixed as per #23. Now working on a simpletest, will post that patch when finished.

#24: Does not set $account parameter. Needs to take uid from $node, load user and pass it to filefield_widget_file_path().

Please see http://drupal.org/node/224235#comment-1802668 for a mapper which supports image files only:
- support images that are returned by scripts (like getimage.php?foo=)
- can save images in sub-directories
- support token & transliterate in pathname and transliterate in filename
- support feedapi_inherit module

Updated patch, with test and uid tokens.

#26

- support images that are returned by scripts (like getimage.php?foo=)

Doesn't Drupal automatically handle this by not overriding files and storing them as file.png, file_1.png, file_2.png?

- can save images in sub-directories
- support token & transliterate in pathname and transliterate in filename

This should be supported by using filefield_widget_file_path().

- support feedapi_inherit module

I am not sure what this means in this context. Could you clarify?

cleaned up comments and style

#28

Doesn't Drupal automatically handle this by not overriding files and storing them as file.png, file_1.png, file_2.png?
For a url like getimage.php?foo=, my guess is that the images would be saved as getimage.php, getimage_1.php, getimage_2.php, which would have incorrect extensions. Not sure how to extend this to support all file types though. Maybe by integrating with http://drupal.org/project/mimedetect ?

filefield_widget_file_path()
Currently filefield_widget_file_path() only replaces tokens and transliterate pathname but not filename. It also only returns the path, but doesn't create it.

support feedapi_inherit module
This was supposed to mean that it'll use feed item node's user to replace tokens instead of currently logged in user. However, after a recheck, it seems this has nothing to do with feedapi_inherit module. haha

subscribing

Tests are passed, code is okay.
I used this as the URL:
http://gdata.youtube.com/feeds/base/videos?q=iphone&client=ytapi-youtube...
I created one CCK field. I mapped the ->link item to the field. I got non-existing "watch?v=IpQ9RESJnWM_0" files.
I'm going to find out what's this.
I assume if it could not download the file, that file entry should not be created.

I could not kill this bug, i tried to urlencode the filename what to save, then base64_encode, each had some issues. with urlencode, the problem was the same, 404, with base64_encode, i get file saving errors:

warning: copy(/tmp/d2F0Y2g/dj1fMUFRYUZjbl9ySQ==) [function.copy]: failed to open stream: No such file or directory in /home/aaron/developmentseed/feedapi_mapper_v2/mappers/feedapi_mapper_filefield.inc on line 64.

I only modified line 63 of feedapi_mapper_filefield.inc.

"watch?v=IpQ9RESJnWM_0" looks like you're mapping the HREF around the image, but not the image itself.

It does not matter, it's not an image, the mapper should save the actual html content of the youtube page. and it works. It saves the HTML content, all is fine, but the link is broken when i visit the node and click on the file.

Also needs work because of recent changes:
ctools needs to be added to test setUp()
mapper signature has been changed:

function node_feedapi_mapper($op, $feed_node, $active_processor, $node = NULL, $feed_element = array(), $field_name = '', $sub_field = '') {

Attached patches for both branches.

The special filenames are still an issue that i can reproduce.
Drupal handles a function to cope with these: file_munge_filename() - that's the way to go. Expect a next round of patches soon.

#37 is fine. My browser was too clever and removed ugly things after watch and i tried to find out where the heck Drupal filters out those nasty things. The special filenames are still an issue that i can reproduce, but it affects the core as well. An urlencode is missing somewhere, damn :)

Related core issue:
#549162: Uploading tricky filenames leads to 404

I tried Aron's patch for 1.x. It works perfectly for filefields and imagefields. I also tried multiple values from a custom feed that contains something like <file>http://localhost/d6/files/druplicon.jpg</file>. It also worked as an enclosure. Only problem was with filenames containing whitespace and/or special characters, but Aron has already mentioned this.

Tried Aaron's patch (v. 1.x) and got the following result when trying to refresh a feed that has url.wmv mapped to a flashvideo CCK field

warning: copy() [function.copy]: URL file-access is disabled in the server configuration in (...)/sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_filefield.inc on line 59

warning: copy(url.wmv) [function.copy]: failed to open stream: no suitable wrapper could be found in (...)/sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_filefield.inc on line 59

Is the patched feedapi mapper not handling things correctly or did I do something incorrectly ?

cURL !

#41: allow_url_fopen is Off on your server: http://us2.php.net/manual/en/filesystem.configuration.php#ini.allow-url-...

Yeah, some restrictive servers may make trouble.
I coded fallback for drupal_http_request. I'd not replace it because php url wrapper system is super flexible (exhaustive list of prococols, for example)

well mannered squirrel: can you test these on your system?

I've got

warning: copy(files/images/Kaeri.jpg) [function.copy]: failed to open stream: No such file or directory in /.../sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_filefield.inc on line 59.

with the patch in #37. using 1.x

I'll try #45

What kind of feed do you use? Are you sure that your feed contains valid, absolute URLs?

i'm constructing my feed with contemplate and views using:

<?php
$xml_elements = array(
  array(
    'key' => 'node_title',
    'value' => check_plain($node->title),
  ),
  array(
    'key' => 'file',
    'value' => $node->field_image[0]['filepath'],
  ),
);
?>

So the result would be something like

<node_title>Some title here</node_title>
<file>http://mydomain.com/files/images/cutedog.jpg</file>

So my guess is I should have something like

<file>http://mydomain.com/files/images/cutedog.jpg</file>

am I right?

Yes that did the trick

array(
    'key' => 'file',
    'value' => 'http://mydomain.com/'.$node->field_image[0]['filepath'],
  ),

#49: I'm glad, it's good to hear that it's unrelated to the patches :)

The patch is working great (1.x, didn't try 2.x). Thanks.

alex_b : I do not have access to the php.ini file

Aron : I used your new patch (1x) and it works perfectly. My feed only had one new item but I guess it would work fine with multiple items too.

I'll report back if I encounter any issues.

Many Thanks

I think there might be a few issues with the patch in #45:
- the temporary file used might not be cleaned up properly.
- the URL needs to be decoded with html_entity_decode(). Test feed: http://rss.news.yahoo.com/rss/us
- the output directories need to be created because field_file_save_file() doesn't create them.

^ ^

Should be in FeedAPI mapper 2.0 beta 1

ball.in.th: great insights! Thanks, i improved the patches according to your observations.

The patch in #55 works great!! ^ ^

However, during my test, the current user (1) was used to replace tokens instead of the feed node author. This is strange; maybe $node->uid has not been properly set yet? Should the author of $node->feedapi_node->feed_nids be used instead?

Also, perhaps the basename should be transliterated (transliteration_get) and munged (file_munge_filename)?

The images used to work, but now I'm getting this error when running cron via drush.

warning: unlink(sites/default/files/vygotsky2_7.jpg): Permission     [warning]
denied in includes/file.inc on line 435.

And now the images are gone. Using 1.x

Bumping this down again in an effort to keep our critical task list for FEMP 2.0 beta 1 as light as possible.

Does not mean that I'd love to get this in ASAP.

"However, during my test, the current user (1) was used to replace tokens instead of the feed node author. This is strange; maybe $node->uid has not been properly set yet? Should the author of $node->feedapi_node->feed_nids be used instead?"

I assume this is NW then.

I solve the problem I reported in #57 by deleting the items and doing a refresh, this might not be the best solution. Didn't try the patch in #55. Don't know if that one would work.

Subscribing...

subscribe

What version should I apply this patch (1.x) to - dev or stable?

Looks like I cannot to both of them...

*edit*, it's ok, working fine with 1.0 stable.

Szy.

subscribing

I've just tried patch #55 for 2.x-dev (not alpha3, dev doesn't appear as published, you'll have to click on view all releases) and it's working. I wanted to try Feeds but it seems CCK support it's not ready yet and honestly I don't know if it will ever be ready since Drupal 7.x it's just around the corner.

subscribing, will test the patch later

Anybody tried the patch with 1.3 version? Or 2-alpha4?

Szy.

Attempted to patch 1.3, got the error message

warning: Missing argument 3 for filefield_feedapi_mapper() in /public_html/mysite/sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_filefield.inc on line 7.

subscribing

I needed to add type='image' to get this patch to work for imagefields as well as filefields. Updated patch attached

#55 (along with #69) were perfect for me (using 6.x-1.x-dev). Thank you everyone for pretty much saving my life with these patches/add-ons!!

These work for me (thanks!) but any ideas how to get ImageCache to process any images which are imported?

What is happening right now is that, after a bit of hacking to #69, files are being imported from enclosure elements in the feed, and ending up in the directory on the server associated with the imagefield. However, these images should be getting processed by imagecache, and thumbnails etc ending up in files/imagecache/whatever -- this is not happening at the moment. So although I've got imagecache set up in theming, and everything works if I upload a file to the imagefield manually, it is not working following an import.

Makes me think that imagecache hooks are not being invoked somewhere. Any ideas?

Hmm... Ok, I hear what you are saying lyricnz :) That all makes sense, except that I'm not seeing the images when viewing the nodes after import, and there's nothing in files/imagecache/whatever :(

Gah! I've got it sorted -- it was actually a problem with special characters in the original filenames then getting mangled in the feed, and re-mangled again when being imported. Everything works now, thank you for helping me fix it :) :)

Subscribing

The php copy function just wasn't working for me even with "allow_url_fopen" enabled so I created an alternative patch which uses curl instead

Subscribing :)

After implementing the patch #69.How I show the image using views?

Thanks in advance.

Success

Please tell how feedburner images can be handled.