Mapper for FileField / ImageField
lyricnz - October 10, 2008 - 12:27
| Project: | Feed Element Mapper |
| Version: | 6.x-2.x-dev |
| Component: | Code |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Aron Novak |
| Status: | needs work |
Description
FileField seems to be the way we're heading for generic file attachments to CCK nodes. Would it be possible to add support for this to the Feed Element Mapper? This maybe a more generic version of something already done with the emvideo or imagefield mappers.
#1
I've got this working here, but I'm not sure that downloading enclosures on the fly is a good thing to do for a Feed Element Mapper. Thoughts?
My actual code is basically the same as feedapi_mapper_video_cck, except that it uses curl/copy to pull the file, and field_file_save_file() to save the file / add to {files}. Interested?
#2
I'd love to see this :) Mind to post a patch?
#3
Here's what I'm using. I should probably check $url before I put it in drupal_set_message()
<?php
/**
* Implementation of hook_feedapi_mapper for pulling a file from an enclosure into a node.
*
* @param string $op
* @param Drupal node $node
* @param string $field_name
* @param string, number or array of string or number $feed_element
* @param string or number as id $sub_field
*/
function mymodule_feedapi_mapper($op, $node, $field_name, $feed_element = array(), $sub_field = '') {
$field = content_fields($field_name);
if ($field['type'] != 'filefield') {
// if not an filefield field just return
return;
}
switch ($op) {
case 'describe':
// Describe what we are doing in this mapper. This shows up as help text on the mapping page.
return t('Maps a link to a file in a FileField CCK field. Use the original_url element for mapping to this field.');
case 'list':
// just for sub_fields
return TRUE;
case 'map':
// Here is where the actual mapping happens.
$items = $node->$field_name;
if (is_string($feed_element) && valid_url($feed_element)) {
// straight link usually from options->original_url or options->guid
$filepath = file_destination(file_directory_temp() . '/' . basename($feed_element), FILE_EXISTS_RENAME);
$ok = copy($feed_element, $filepath); // TODO: use curl?
if ($ok) {
$info = field_file_save_file($filepath, array(), file_directory_path());
if ($info) {
$items[] = $info;
}
else {
drupal_set_message("Could not save file $filepath");
}
}
else {
drupal_set_message("Could not fetch $url");
}
}
$node->$field_name = $items;
return $node;
}
}
?>
Putting it into a file should be cut'n'paste. Enjoy!
NOTE: I don't think it's a problem with the mapper, but since this takes a long time, if you select "refresh" from the UI, the server will timeout while it's still running. The exact behaviour here probably depends on your server/php setup.
#4
Hi Simon,
can the patch in #311404: image and audio file mapper and yours be combined to solve issue #224235: Mapper for CCK Image Field also?
#5
It certainly seems like several of these mappers share enough in common (parse a URL, download a file, maybe rename it, maybe put it somewhere special, and call a field-specific handler). We should probably share code between them, perhaps with a common library functions? or a handler-maker with a $function parameters for the unique bits?
However, it appears that the "Embedded Media Video CCK" mapper just inserts the original URL, rather than downloading the content?
#6
Yep,
the EM-fields just generate the code to include the media from the original server. They do not download the files.
#7
This is starting to look good. Two things:
* Please don't use drupal_set_message() in the mapper.
* I'd love to see a real patch.
#8
#4 - imagefield builds on filefield functionality, right? Where is the overlap here? Once we have a file field mapper we should be only a step away from an image field mapper, right?
This is a dev version issue.
#9
If we shouldn't use drupal_set_message() in the mapper, how do we report errors in fetching the files? Happy to provide patch, pending answer to above. (self-contained mapper, without considering reuse/commonality to other mappers, eg: imagefield).
#10
Tried code in post no. 3 but does not seem to work with latest version of the mapper...Any update on this code? Anyone has it working yet?
Thanks,
Patchak
#11
I just tried the code above, and it works fine:
- create a new module, and paste the code above into it - rename the method. I happened to use a module called "junk".
- create a new content-type for the feed-item, add a filefield to it
- create a new instance of "feed", and select the content type above as the target type.
- Go to the Map tab of that Feed, and select the options->original_url field, and map that to the file-field you created in step 1. You can use the "Feed item example" to check that it's a URL like you expect. See screenshot. Click update.
- then click Refresh
I used a feed a http://feeds.feedburner.com/motogpod (a podcast), and this created nodes like the second screenshot.
It would be really useful if this was in Feed API Mapper :/
#12
Hi everybody,
this is an amazing work, thanks lyricnz for sharing your code. As Im working with the lastest feedapi_mapper, I had to made some changes in order to make it work.
You just have to put this file within the mappers directory of feedapi_mapper module, that's all.
I've changed the drupal_set_message for watchdog functions.
Hope it is helpful.
Thanks!
#13
#12 worked wonderfully for me.
I had to make a customization to it to get the download to work, but that is because I was working with an extremely odd feed.
+1
#14
I used the code from #12 as-is again last night, and it worked perfectly. My only reservation is that we should probably use curl or some drupal API rather than "copy".
#15
Please create patches properly - if you don't have write access to the repository use "cvsdo add" - this makes it much easier to review a patch quickly - thank you.
copy() is using stream wrappers - http://us2.php.net/copy - Drupal file API does not yet support them #227232: Support PHP stream wrappers.
I think it's fine to use copy() but we should test whether it was successful and add a watchdog warning if it wasn't.
#16
#17
Filefield support is image field support. Renaming this to make it findable for people who are looking for imagefield support.
#18
Hey! I'm currently working on and testing a version of this for 6.x-2.x, I'll post the patch shortly.
David
#19
Attached is patch for 6.x-2.x . 6.x-1.x should be very similar. Tested primarily on feeds from flickr with enclosures--works for both file and image cck fields. Enclosures and plaintext urls should be supported, works best with SimplePie parser.
#20
Correct version then. The difference between 1.x and 2.x is the function signature of the hook and the call to feedapi_mapper_get_item_node_type() that 1.x does not require.
#21
So far, I have seen five image fields mapper patches proposed since last year. They are
1) In this topic #19
2) In this topic #15
3) #535970: Mapper for imagefield
4) #311404: image and audio file mapper
5) #224235: Mapper for CCK Image Field
These are the ones that I have seen so far. Maybe there are more. It is confusing. I think this is a highly anticipated feature, so people keep writing similar patches doing the same things since last year. Maybe we should compare these patches and choose one of them and concentrate on it.
For example, a feature that I expect is the ability to import all images in a Flickr set to a multivalued imagefield in a single node. I want to store all my images (Flickr set) in a single node. Currently, all images are created in separate nodes. I checked the patches, but it looks like the patch at #15 in this topic and the patch at #535970: Mapper for imagefield do not support it. Right? I could not check #19, because it does not have 1.x version.
#22
#21 - I agree, we should merge these issues. I've already done that with 3), and now also with 4). I've posted a notice on 5).
Right now, this patch is the front runner. If we add tests we can commit this. Version 1.x and 2.x are only mildly different in regards to the API and the UI for testing, so we should be able to commit this patch and tests to both branches.
#23
Needs to query file destination via filefield_widget_file_path(). Use feed item node's user for that.
#24
Fixed as per #23. Now working on a simpletest, will post that patch when finished.
#25
#24: Does not set $account parameter. Needs to take uid from $node, load user and pass it to filefield_widget_file_path().
#26
Please see http://drupal.org/node/224235#comment-1802668 for a mapper which supports image files only:
- support images that are returned by scripts (like getimage.php?foo=)
- can save images in sub-directories
- support token & transliterate in pathname and transliterate in filename
- support feedapi_inherit module
#27
Updated patch, with test and uid tokens.
#28
#26
- support images that are returned by scripts (like getimage.php?foo=)
Doesn't Drupal automatically handle this by not overriding files and storing them as file.png, file_1.png, file_2.png?
- can save images in sub-directories
- support token & transliterate in pathname and transliterate in filename
This should be supported by using filefield_widget_file_path().
- support feedapi_inherit module
I am not sure what this means in this context. Could you clarify?
#29
cleaned up comments and style
#30
#28
Doesn't Drupal automatically handle this by not overriding files and storing them as file.png, file_1.png, file_2.png?
For a url like getimage.php?foo=, my guess is that the images would be saved as getimage.php, getimage_1.php, getimage_2.php, which would have incorrect extensions. Not sure how to extend this to support all file types though. Maybe by integrating with http://drupal.org/project/mimedetect ?
filefield_widget_file_path()
Currently filefield_widget_file_path() only replaces tokens and transliterate pathname but not filename. It also only returns the path, but doesn't create it.
support feedapi_inherit module
This was supposed to mean that it'll use feed item node's user to replace tokens instead of currently logged in user. However, after a recheck, it seems this has nothing to do with feedapi_inherit module. haha
#31
subscribing
#32
Tests are passed, code is okay.
I used this as the URL:
http://gdata.youtube.com/feeds/base/videos?q=iphone&client=ytapi-youtube...
I created one CCK field. I mapped the ->link item to the field. I got non-existing "watch?v=IpQ9RESJnWM_0" files.
I'm going to find out what's this.
I assume if it could not download the file, that file entry should not be created.
#33
I could not kill this bug, i tried to urlencode the filename what to save, then base64_encode, each had some issues. with urlencode, the problem was the same, 404, with base64_encode, i get file saving errors:
warning: copy(/tmp/d2F0Y2g/dj1fMUFRYUZjbl9ySQ==) [function.copy]: failed to open stream: No such file or directory in /home/aaron/developmentseed/feedapi_mapper_v2/mappers/feedapi_mapper_filefield.inc on line 64.I only modified line 63 of feedapi_mapper_filefield.inc.
#34
"watch?v=IpQ9RESJnWM_0" looks like you're mapping the HREF around the image, but not the image itself.
#35
It does not matter, it's not an image, the mapper should save the actual html content of the youtube page. and it works. It saves the HTML content, all is fine, but the link is broken when i visit the node and click on the file.
#36
Also needs work because of recent changes:
ctools needs to be added to test setUp()
mapper signature has been changed:
function node_feedapi_mapper($op, $feed_node, $active_processor, $node = NULL, $feed_element = array(), $field_name = '', $sub_field = '') {#37
Attached patches for both branches.
The special filenames are still an issue that i can reproduce.
Drupal handles a function to cope with these: file_munge_filename() - that's the way to go. Expect a next round of patches soon.
#38
#37 is fine. My browser was too clever and removed ugly things after watch and i tried to find out where the heck Drupal filters out those nasty things. The special filenames are still an issue that i can reproduce, but it affects the core as well. An urlencode is missing somewhere, damn :)
#39
Related core issue:
#549162: Uploading tricky filenames leads to 404
#40
I tried Aron's patch for 1.x. It works perfectly for filefields and imagefields. I also tried multiple values from a custom feed that contains something like
<file>http://localhost/d6/files/druplicon.jpg</file>. It also worked as an enclosure. Only problem was with filenames containing whitespace and/or special characters, but Aron has already mentioned this.#41
Tried Aaron's patch (v. 1.x) and got the following result when trying to refresh a feed that has url.wmv mapped to a flashvideo CCK field
warning: copy() [function.copy]: URL file-access is disabled in the server configuration in (...)/sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_filefield.inc on line 59
warning: copy(url.wmv) [function.copy]: failed to open stream: no suitable wrapper could be found in (...)/sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_filefield.inc on line 59
Is the patched feedapi mapper not handling things correctly or did I do something incorrectly ?
#42
cURL !
#44
#41: allow_url_fopen is Off on your server: http://us2.php.net/manual/en/filesystem.configuration.php#ini.allow-url-...
#45
Yeah, some restrictive servers may make trouble.
I coded fallback for drupal_http_request. I'd not replace it because php url wrapper system is super flexible (exhaustive list of prococols, for example)
well mannered squirrel: can you test these on your system?
#46
I've got
warning: copy(files/images/Kaeri.jpg) [function.copy]: failed to open stream: No such file or directory in /.../sites/all/modules/feedapi_mapper/mappers/feedapi_mapper_filefield.inc on line 59.with the patch in #37. using 1.x
I'll try #45
#47
What kind of feed do you use? Are you sure that your feed contains valid, absolute URLs?
#48
i'm constructing my feed with contemplate and views using:
<?php$xml_elements = array(
array(
'key' => 'node_title',
'value' => check_plain($node->title),
),
array(
'key' => 'file',
'value' => $node->field_image[0]['filepath'],
),
);
?>
So the result would be something like
<node_title>Some title here</node_title><file>http://mydomain.com/files/images/cutedog.jpg</file>
So my guess is I should have something like
<file>http://mydomain.com/files/images/cutedog.jpg</file>am I right?
#49
Yes that did the trick
array('key' => 'file',
'value' => 'http://mydomain.com/'.$node->field_image[0]['filepath'],
),
#50
#49: I'm glad, it's good to hear that it's unrelated to the patches :)
#51
The patch is working great (1.x, didn't try 2.x). Thanks.
#52
alex_b : I do not have access to the php.ini file
Aron : I used your new patch (1x) and it works perfectly. My feed only had one new item but I guess it would work fine with multiple items too.
I'll report back if I encounter any issues.
Many Thanks
#53
I think there might be a few issues with the patch in #45:
- the temporary file used might not be cleaned up properly.
- the URL needs to be decoded with html_entity_decode(). Test feed: http://rss.news.yahoo.com/rss/us
- the output directories need to be created because field_file_save_file() doesn't create them.
^ ^
#54
Should be in FeedAPI mapper 2.0 beta 1
#55
ball.in.th: great insights! Thanks, i improved the patches according to your observations.
#56
The patch in #55 works great!! ^ ^
However, during my test, the current user (1) was used to replace tokens instead of the feed node author. This is strange; maybe $node->uid has not been properly set yet? Should the author of $node->feedapi_node->feed_nids be used instead?
Also, perhaps the basename should be transliterated (transliteration_get) and munged (file_munge_filename)?
#57
The images used to work, but now I'm getting this error when running cron via drush.
warning: unlink(sites/default/files/vygotsky2_7.jpg): Permission [warning]denied in includes/file.inc on line 435.
And now the images are gone. Using 1.x
#58
Bumping this down again in an effort to keep our critical task list for FEMP 2.0 beta 1 as light as possible.
Does not mean that I'd love to get this in ASAP.
"However, during my test, the current user (1) was used to replace tokens instead of the feed node author. This is strange; maybe $node->uid has not been properly set yet? Should the author of $node->feedapi_node->feed_nids be used instead?"
I assume this is NW then.
#59
I solve the problem I reported in #57 by deleting the items and doing a refresh, this might not be the best solution. Didn't try the patch in #55. Don't know if that one would work.
#60
Subscribing...
#61
subscribe
#62
What version should I apply this patch (1.x) to - dev or stable?
Looks like I cannot to both of them...
*edit*, it's ok, working fine with 1.0 stable.
Szy.
#63
subscribing
#64
I've just tried patch #55 for 2.x-dev (not alpha3, dev doesn't appear as published, you'll have to click on view all releases) and it's working. I wanted to try Feeds but it seems CCK support it's not ready yet and honestly I don't know if it will ever be ready since Drupal 7.x it's just around the corner.
#65
subscribing, will test the patch later
#66
Anybody tried the patch with 1.3 version? Or 2-alpha4?
Szy.
#67
Attempted to patch 1.3, got the error message
#68
subscribing
#69
I needed to add type='image' to get this patch to work for imagefields as well as filefields. Updated patch attached