FileField Sources: Remote files / Reference existing files

I'm deferring this for now. I hope to get this feature done around june.

i'd just like to put in another vote for this when it can happen. we use the images as icons which we reuse. currently, this means many copies of the same images stored on the filesystem.

thanks for your consideration.

This would be a really cool addition. Choose existing or add new would be great. +1 for this, I would help test and maintain

I'd really love to see this, too. This is what separates imagefield from being slightly usefull to being great. Seriously. I'd really love to see an image browser like IMCE for the imagefield that allows me to browse existing images and upload new ones.

Imagefield is great, thanks for all the work. This feature will really take it to another level.

subscribe

+1 for the features
big +1 for using imce to choose the file.

I might just do this one myself. I've already re implemented IMCE's interface: http://drupal.org/node/139387

I had a look at it this afternoon. It would be considerably more complicated that I initially thought to do this. I suspect it's probably better to implement a new CCK field widget called IMCEfield or something. The main reason being that imagefield ties in closely to the files table and therefore has properties like fid, where as IMCE tries to keep things really simple by ignoring drupals files table and related code. Therefore a property like 'fid' doesn't make much sense in the context of IMCE.

Another good reason not to mix these up is that IMCE has it's own upload function. If one were to implement IMCE on imagefield, then which upload form should a user use?

Subscribing

Subscribing

This feature is very useful.

You should consider the img_assist module as a base for the new CCK node type.

I wouldn't use img_assist for this as it rewuires the image being an image node -- which is IMCE's advantage for many cases.

This would be absolutely great...

@dopry What are your thoughts on this. Is using IMCE or similar what you had in mind? Any thoughts on the img_assist vs IMCE, or whether uploaded images need to become image nodes an d or whether an editor should be choosing an image node, or a image of the filesystem?

Found this today that sounds like it could provide a better solution: http://drupal.org/project/asset

Hmmmm, yeah, I can see what you mean. Once there's a CCK asset field type, it should definately be able to do what I've been trying to do with the image-field.

I watched dries talk at the last drupal con; http://video.google.com/videoplay?docid=7827250900890284668

He has recognized the need for asset management in core to allow for better handling of files and assets in contrib modules. My understanding from his talk was that such an asset manager to handle assets as nodes with relationships to other nodes, assets as files in multiple nodes, and possibly assets already on the filesystem (ala IMCE?). However it could take some time before we see this solution in drupal core (drupal 6 perhaps?).

Depending on the urgency of this feature for two projects (i.e. how long it takes before a large number of duplicate images becomes a problem for the hosting account) I may implement an IMCEimage CCK field or similar for drupal 5, perhaps utilizing assets module either in addition to or instead of IMCE...

I'm also looking for this functionality, but in the mean time I made a work-around using:
- image module
- node reference CCK
- custom template file

0/ Upload some images to play with

1/ Create a content type and add a node reference field (named myimages), limit the field to images (or use a view if you need more control) and check multiple values option

2/ Create a new node using this content type, and select some images

3/ Create a node-mycontenttype.tpl.php file (based on node.tpl.php) and add the following code

     if ($node->field_myimages.count) {
       print '<ul class="cck-field-myimages-list">';
       foreach ($node->field_myimages as $img) { 
         $imgnode = node_load(array('nid' => $img['nid']));
         print '<li>' . l(image_display($imgnode, 'thumbnail'),'files/' . $imgnode->images['_original'], array('class' => 'thickbox'), null, null, FALSE, TRUE) . '</li>';
       }
       print '</ul>';
     }
   

4/ add some css to display them inline

ul.cck-field-myimages-list
{
	padding: 0px;
	margin: 0px;
	list-style-type: none;
}

ul.cck-field-myimages-list li
{
	display: inline;
	padding-right: 5px;
}

maybe as a work-around the user could be prompted, when uploading an image with an existing name, if he wants to save it with a unique name, overwrite the existing image, or use the existing image instead. (this fuctionality could/should be optional, next to the current existing functionality)

i created a bi-lingual site, and need to upload all images twice. just upgraded my host as i ran out of disk space...

Imagefield should definitely support such a feature, but not directly. Instead, it should support the bare minimum: a hook to let other modules return if a particular image (that is about to be deleted) may be referenced in other contents. Each image is identified by a unique file id. So another module providing the re-usage of images uploaded via imagefield in other nodes would return FALSE.

I'm currently investigating, if image.module is needed at all in future. Basically, Image module provides nothing more than a node with one image attached. In contrast, that's basically a custom content-type with just one imagefield.
So I've (dirty) hacked Img_assist a bit - just to see, if it could be an Imagefield_assist module, too. Guess what? It basically worked out.

For those interested, I'm attaching a patch. Don't use this on any site and DO NOT EVEN THINK OF REPLYING WITH A BUG REPORT or criticism, please. It was just a quick'n'dirty test. And I like the idea even more. :)

IMCE could probably be used to a similar end. IMCE is like image_assist, but integrates with imageson the filesystem not image.module.

IMCE has gained a lot of traction and support, and is possibly in higher usage than img_assist.

Glad to hear that you support my suggestion to let Imagefield provide the bare minimum and have other contrib modules to do the actual job. (btw: Are you aware of Img_Assist and IMCE stats?)

I didn't know those pages existed. Are they from one of the recent GSoC projects?

Those pages only provide information about issues though -- which doesn't say a lot about the projects. Img_assist is a much older project so I'd expect it to have many more issues. But that doesn't make it stabler, better, or more popular.

Actually I don't think either project is better -- they're just different. I think for most sites IMCE's approach of not having images as nodes makes more sense. But there are definitely use cases for Img_assist's approach too.

IMCE definately has it's issues. Here's my pet issue! :) http://drupal.org/node/139387

Unfortunately the maintainer seems to awol. Or maybe he's just ignoring me. :(

beta of IMCE + CCK image field.

you can download it from http://www.hyrme.com/d5/files/imceimage-beta.zip

Cool!! I'd like to check this out, when I have time. Are you aware of Walkah's images-in-core campaign? Does this relate to that? If so how?

Please consider this if you have time; http://drupal.org/node/139387

Subscribing

Unrelated work - I am not modifying IMCE at all. The imceimage module needs imce to work - imce in any form is okay as long as it is "backwards" compatible with the existing 5.x version of IMCE.

subscribing

Cool thing, panis!
Are you creating an own module?

Subscribing

FWIW, I've put my work-around here.
http://drupal.org/node/194293

panis -

Cool beta.

I needed one fix (on Postgres). The column for field_..._imceimage_path TYPE was character(1) following install. I needed an ALTER to character(200) query.

subscribing

+1
Subscribing

Perhaps dopry could give us some hints on how to implement this correctly so we could try to implement it ourselves. The main problem might be keeping it in sync with the files table, because an image is tied to one node now, and we wouldn't want the image to be deleted when the original node is deleted. Perhaps we could do something like a reference counter, or copy the entry in the files table. I need to take a look at CCK widgets, but I didn't find a lot of documentation when i searched last time. Or perhaps it is just wrong to implement it here in this module, then I would be delighted if someone points me in the right direction.
Greetings

The issue is centered around short comings in Drupal core and the lack of file life cycle hooks. If you want to see this feature in D7 I suggest reviewing... http://drupal.org/node/142995 and helping get it into core.

For anyone who wants a start on this, as I needed this functionality, here's a starting point:
in the files.inc file change this:

function file_delete($path) {
  if (is_file($path)) {
    return unlink($path);
  }
}

to this:

function file_delete($path) {
  if (is_file($path) && !file_in_use($path)) {
    return unlink($path);
  }
}

function file_in_use($filepath) {
	return (0 != db_result(db_query("SELECT COUNT(*) FROM {files} WHERE filepath = '%s'", $filepath)));
}

Yep, it's a core change but it's also the most logical choice for this purpose - unless there is some API you could use but I think that would probably be overkill anyway - (I will bow out right now from any possible ensuing debate on this statement). This will make sure that you don't delete files still in use. This assumes that all modules "play nice" and use this call as they should - so far all I have used do play nice in this respect.

Next the exact code change will be dependent on your particular setup, however the basic idea is just to create a duplicate (the primary key will of course be different) of the entry in the files table for any give file you want to reuse. If there is a matching cck table entry like there is for images, duplicate that as well. Then, all other operations with that node should function as per normal because you are not reusing table entries (and therefore maintaining the expected data structures of the modules) - only the file itself is being reused.

Well actually if you want to see this in D7... You should review http://www.drupal.org/node/142995 which implements soft delete behavior in core using hook_file for reference counting. It's the patch that has been holding this up. It didn't make it in for D6 so I never got to work on this feature.

I'll try out the core patch this week. I'm still interested in a work-around for D5 though, is the additional widget the way-to-go? I imagine having a view on all imagefields in a popup or embedded where to choose a file from.

Sorry, the site forces me to change the version, 5.x-2.x-dev isn't listed in the dropdown.

Panis' workaround with imceimage actually does all I need it to do. So I think that could be developed into a stand alone module.

Just to be clear, from what Dopry is saying this feature may not even make it in D7! That would be a concern

dopry, can you perhaps help me with my questions from #35 and #39?

scroogie what I gave in #37 works like a charm for me. I have a DB with 1,000's of products and some of those products have 100's of variations. Each variation uses the same image in a CCK field as it's parent product and it works perfectly for me. I've been using a heavily modified sub-products module to automatically create an delete the product variations.

I can imagine it works, but I would prefer to do it right and possibly commit it back. I need an UI to choose images from, e.g. a View of images uploaded per imagefield. I think I could do it in a hackish way, e.g. with an additional seperate optional field which accepts URLs, but I think it belongs to imagefield.

mm subscribing cause i want this to..

Check out this new module:
http://drupal.org/project/imagepath

Currently provides a very nice method for adding an img tag for existing image nodes (any node with an imagefield in it) in you html. But the maintainer already has an IMCE integration working and is planning to release that with the module.
Certainly looks promising...

Hi, not to balk at testing, but has anyone tried panis' beta in #24? I ask because it's more than 6 months old & wondering what results (if any) have been before attempting to mess with that? Or maybe there's been an update? Panis, have you had any further luck with this?

mainly, I just want an image library that users can choose from (not edit)... alternately they could upload their own image (not to add to the library, just like normal - though I could do this with another field).

Anyone have any leads?

Thanks!
:) Scott

6.x module here

http://drupal.org/project/imceimage

Thanks panis!
I wonder – have you run into any bugs with the 5.x version? We're running Drupal 5.7 on the site I need this functionality for... And I'd be happy to try it, if you think it's functional.

I realize it won't be quite as sweet as the 6.x version, but as long as it works I think that should suit this particular purpose... (I can try the 6.x version on a different site!).

Thanks for any recommendations about this...

subscribe

subscribe

+1 even if there's no UI for selecting, I'll copy and paste a path, type in a fid, whatever :)

But what I'd really like is selecting from Image nodes by taxonomy - don't need a GUI for image preview/browsing in that case, just a dropdown select of existing terms and then a list with a checkbox/select all option.

I've got to believe something like this is in the works for Image to Imagefield migration purposes, but I guess if it were easy the issue wouldn't have sat around for 20 months! And oh yeah, I need it for D5. . .

Any suggestions in the meantime for a workaround? Even if it means duplicated files in the site file system (yech) - at this point disk space is NBD.

Bulk importing into a field (for a site that's already using Image module - Imagefield Import's apparently incompatible)?

TIA

subscribe

I'd add another +1 but that'd be cheating :)

I've been checking out Asset, and like the way it allows for this (re-use), so I thought I'd point it out as a currently shipping model for some future feature improvements.

The "canonical" reference to a file is stored outside the node relationship, so the CCK field just points to the reference in the Asset table, not the file itself.

The other big advantage over IF is that it lets you upload the files into an organized filesystem hierarchy, both in batch mode into one target folder, or one at a time into different ones, creating new folders as you go if needed. (Token support would be even better).

This makes it very easy to browse through your custom-organized folders to look for the images you want to insert into a given CCK field, and of course you can do this in multiple nodes without re-uploading. (Also allowing Image node-referenced files to be browsed by taxonomy would be even better)

Asset isn't image-handling specific though, more multi-media generalist, so (haven't done much detailed testing yet) doesn't offer other features that IF does, particularly in Views integration.

And of course the huge gaping hole is the ability to link to image files already in the filesystem and linked to Image nodes. Any integration along those lines should include IMO the ability to browse Image nodes by taxonomy, ideally by bringing Image Assist into the mix.

I almost got this to work. Except it turned "<" & ">" into html entities. So what I got was something like this:

<ul class="cck-field-myimages-list">
	<li>
		<a href="/drupal/sites/default/files/images/image_01.jpg">&lt;img src="http://localhost/drupal/sites/default/files/images/image_01.thumbnail.jpg" alt="Image1" title="Image1" class="image image-thumbnail " width="100" height="100" /&gt;</a>
	</li>

Could it be because I use drupal6? Btw, how could you use thickbox with it? Are image.module and thickbox compatible with each other?

This is a reply for comment by attiks.

The definition for the l function changed a bit, you have to specify that the text is full html, have a look at the api (http://api.drupal.org/api/function/l), you have to add 'html' => true to the options array

That solved it. Thank you!

apparently someone got an example of this done with the "Linodef" module: http://drupal.org/node/335191#comment-1117891

some progress with Inline API: http://drupal.org/node/80170#comment-1124808

and someone else looking: http://groups.drupal.org/node/17030

Subscribing; would make my day.

+1
Subscribing

+1 ;)

#343841: UI to import existing files from the server

+1

I had a look at this again today. I used hook_elements to add a process function to $elements['imagefield_widget']['#process'] that adds an additional fieldset to the upload form. In that collapsible I embedded a view of existing images. I used jquery to set the value of field_news_image[$n][fid] when the user clicks on an image. I ran into several problems:

Widget:
If I extend filefield_widget it's not shown on imagefields. On the other hand if I extend imagefield_widget it's not shown on "normal" filefields. Do we need to add it to all widgets?

Reroute View:
I tried to use hook_form_alter to set $form['#action'] to the current form page if $form_id equals 'views_exposed_form'. Is that the right way to do it? It didn't work initially. This would be needed for exposed filters for say the filename or taxonomy, which is a must I think.

If views not present:
what should we do if views is not present? Display a select box or autocomplete field for the fid or file name?

Thumbnails:
How can we show previews of the files? I think the admin thumbnail isn't permanent, and we can't rely on imagecache, right?

The very, very latest development version of FileField has a new functionality that should help with this task. I've spent a little time on it and come up with a new module, currently named "FileField Sources", which let's you use (or reuse) files from any location.

Right now I've written two implementations: Referencing existing FileField images, and downloading remote images (or files). This should be able to easily support IMCE or some other file browser, provided that file browser can read from the {files} database table instead of directly from the disk.

Unfortunately there's no configuration for these options yet, since we can't add options to the FileField configuration form. I've submitted a patch to CCK to allow this: #417122: Allow drupal_alter() on Field and Widget Settings.

scroogie, I like your approach of using a view to list images. I'd been thinking about something like this also. If we had a place to put configuration options, then we could let the user create view (or provide a default view), and select from a drop-down list which view should be used to view thumbnails, similar to how Node Reference lets you use a view for the referenceable nodes. The user's view could use ImageCache if available, or we could use theme_imagefield_admin_thumbnail() which automatically creates the thumbnails used for ImageCache's preview.

Now, just because this is brand new I feel obliged: Do not use this on a live site, it very may spit out massive amounts of errors, leave temporary data all over the place, and delete files it shouldn't. You need to install the latest FileField and ImageField development versions, or use beta4 or higher when they come out.

Edit: fixed src in img tag.

It's fantastic to see that your working on this. You even added pluggable sources already. Awesome. I'll try it out on a sandbox immediately.

Okay, it works quite well.
Somehow, even if I try to reference an existing file, filefield_source_remote_value() is invoked asynchronously on /filefield/ahah/$content_type/$field_name/$n
This is because in filefield_widget_value() the new $element['#filefield_value_callback'] is called for all functions that are registered in any case (being both reference_value as well as remote_value no matter which form you submit).
I'm also not sure about curl. If we check for allow_url_fopen we could also use the usual copy(), although we wouldn't get a progress bar then afaik.
About the views browser: It could live either as a collapsible fieldset just as I tried it before or as a browse button which opens a popup besides the select button in the reference tab. I think even without letting the user choose the view it would be very helpful (as a default view). I think if we use {files} as base table we cannot use the imagecache formatter, but we could code around that or provide a views plugin. How could we add that to only the image widget? Or do you want it for the general filefield?

Somehow, even if I try to reference an existing file, filefield_source_remote_value() is invoked

Yeah I was having a bit of trouble with the "hint" text which is added by JavaScript. Often times it's still in the field when you try to submit. It'd be best if we could somehow detect which button was clicked and only act on the related field.

I think if we use {files} as base table we cannot use the imagecache formatter, but we could code around that or provide a views plugin.

We should be able to do this (I think) by making a relationship to the Content: Field (field_name) field. Then the content based fields become available (and the ImageCache formatters). We might need to add a little extra to the FileField Views implementation to do this. It just makes more sense to start with the files table than it does with node, since we don't actually need the node table at all (and a file could be attached to multiple nodes anyway, causing duplicate results).

How could we add that to only the image widget? Or do you want it for the general filefield?

In the #process function we can always check that $field['widget']['widget_type'] == 'imagefield', though potentially we might this to work on FileFields also, not sure yet.

Changing the title so that I can consolidate issues here. This is really a purely FileField enhancement but I'm leaving in the ImageField queue since more eyes look here for these features.

Marked #85464: Use image from arbitrary filepath or URL and #257220: Specifying a file already uploaded as postponed while this develops.

How could we add that to only the image widget? Or do you want it for the general filefield?

In the #process function we can always check that $field['widget']['widget_type'] == 'imagefield', though potentially we might this to work on FileFields also, not sure yet.

Yes please open to any filefield type

Alarm bells. This issue turns into a direction where it duplicates existing efforts:

- http://drupal.org/project/media - a full blown media handling and file processing and management module, based on D7 file API, with potential backports to D6.

- http://drupal.org/project/img_assist - specifically #175812: Imagefield support, #208136: Imagecache/Image* support, and #352554: Use img_assist to browse for node ids (e.g. nodereference & image_attach) - removing the dependency on Image module and opening the doors for ImageField/Cache support (equally using Views as image browser) is a long wanted feature. Deferring the image browser to another module is what my comment in #20 already outlined.

- Several other file/image browsers in contrib.

I am also not sure whether this implementation properly accounts for file/image updates and deletions of re-used files/images (What about the optional alt/title fields?).

Note: I don't want to hinder progress, I just want to point out that this will lead to yet another custom functionality on Drupal sites. It might be worth to consider the low-level/basic functionality for ImageField/FileField, but make the high-level (UI) functionality pluggable or to even defer it completely to other modules.

Heh, it's not that bad sun.

Re: Media: Yes I'm aware of Media and what it does. It's abilities should eventually outstrip FileField since it can do things like reference files on S3 or Akaimai without downloading them. FileField is made to store files locally and that's what it will continue to do.

Re: Img_assist: That's great. Take a look at the source code for FileField Sources, you'll see it's primary purpose is making accessible APIs to set a FID value in the field, regardless of where it comes from. In the end, all FileField ever does (after saving the file), is associate a FID to the field, alt and title text is just a bonus ontop of the primary data: the FID.

Re: Any other browsers: Same story. If you have multiple ways to attach a file, they would all conflict if there wasn't an API with a unified way of switching between them. The two implementations included (Remote files and reference existing files) are just two implementations of what could be potentially a dozen or more (the existing implementation are also completely removable from the module and could be made into separate projects). If a user installs two methods (such as IMCE and a remote reference) they want to be able to use both of them on the same field, not choose between one or the other (which is the current situation).

I am also not sure whether this implementation properly accounts for file/image updates and deletions of re-used files/images

Handling file references across nodes is in no way different from handling files across revisions. FileField has full support for reference counting. Remember that dopry's work in FileField was the basis for hook_file() in D7.

What about the optional alt/title fields?

Each reference still gets its own row in the CCK tables, so each referenced image or file will have a separate alt/title field. The only thing actually being referenced is the FID from the files table.

You killed all my concerns, Nate. Awesomeness as usual. Thanks! :)

filefield_sources is great. Thank you!

Remote URL should have option for allowed/banned domains.

Just a quick note if you're thinking of a views-based file browser, inspired from nodereference :
nodereference plugs on the 'default' display of the specified view, which, I realized afterwards, is not really inline with how Views works (default displays are not really supposed to be executed).
Ideally, I think there should be a 'noderef selection' display plugin, and noderef field settings lets you pick one of those on one of the views that have such a display. Not easily changeable for noderef without breaking existing setups (+ I'm lacking time), but if filefield goes this way, it would probably be best to start on a cleaner base :-)

My version of noderef (CCK 6.x-2.2) is giving all the views displays except for the default display, so I guess they changed it. I have to admit it's cleaner.

Awesome

If you guys want to ban/allow certain domains just add some lines in sources/remote.inc after $url = $item['filefield_remote']['url']; (somewhere around line 110)

   if (parse_url($url,PHP_URL_HOST) != 'domain.tld') {
         form_error($element, t('Remote images are limited to domain.tld'));
      return;
    }

Remote URL should have option for allowed/banned domains.

There are many configuration options that could be added to both these new features. Some additional ones for references:

• Only allow users to reference their own files
• Reference files from within a view
• Reference files only within the same field, or all file fields

But alas, currently the problem is we don't have anywhere to put such configuration, unless we were to make an admin/settings/filefield-sources page (or something similar). Anyone capable or interested please contribute to #417122: Allow drupal_alter() on Field and Widget Settings, so we can put these settings where they belong on the field configuration page.

Yeah I was having a bit of trouble with the "hint" text which is added by JavaScript. Often times it's still in the field when you try to submit. It'd be best if we could somehow detect which button was clicked and only act on the related field.

Are you sure this is related? I think it's the filefield_value_callback that is called on all plugins instead of only on the used one. This might be especially bothersome for people that don't have the curl extension (or any other requirement of a source plugin).

subs

This module is now an official project (though no official releases yet). See http://drupal.org/project/filefield_sources. Now with IMCE compatibility. :D

Please open new issues in the FileField Sources issue queue. http://drupal.org/project/issues/filefield_sources

Great job! Thanks, guys!

@quicksketch: =)

the missing link - this is great!!!!!!

thanks a lot