Add ability to use file uploaded via FTP

In my opinion this should be the responsibility of IMCE, and not of Filefield sources. Filefield sources is meant to be the glue code to prepopulate a FID for the filefield code, so that filefield itself doesn't need any special code. The modules that allow uploading files should use the Drupal files table. Thats the idea of it, right? IMHO, files lying on the webserver which have no entry in the database are not 'content', but just files lying on the server. If you have CVS or SVN files in your directories, do you want them to show up in Drupal? I certainly don't.
I think IMCE is a bad candidate for filefield sources anyway, because of its weird thumbnail behaviour (showing up as seperate files). I'd favour just showing file-entries with views, allowing such things as search / filter by mime etc.

Wouldn't it be possible to choose any file on the server an when selected, add it in the database, just like if we were using the usual upload method?

Scroogie has it right, but I want to point out the serious danger if FileField allowed any file to be used. Because a file is not stored in the database, using it in a file field would make FileField add the file to the files table automatically. However, then if that node were deleted the original file would be deleted also. This is because FileField keeps track of when a file is in use by nodes, and when the last node is deleted the file is deleted also.

However, there are ways around this that could be implemented, I'm just not sure what to do about it yet. We could somehow set up a module that simply scans a directory repeatedly, and adds those files to the files table. Then implement hook_file_references() in that module and always tell FileField that there is an active reference, so the file should not be deleted even if there aren't nodes currently using it.

And then you get the problem that IMCE can delete the file without updating neither the references nor the entries in the database. I'd suggest not to start these kind of things.

Yeah, IMCE's feature set can cause all kinds of problems, but at least when used with FileField, we're able to disable the ability to upload, delete, thumbnail, etc. However as soon as use IMCE attached to a textarea, then you can do all those silly things again. However we should be able to make a patch to IMCE to simply check hook_file_references() before deleting files that it doesn't own.

subscribing

@#2
I don't see any conflicts. You either manage your assets through drupal (using filefield) or you don't. So, if you want to reference a file not in the database in this filelfield, it should be added to the database (possibly copied to the corresponding directory) and that's it. If you then reference the file not with drupal methods (filefield) but with html tags, it is your responsibility to keep track of the validity of your reference.

Just my two euro-cents.

it should be added to the database (possibly copied to the corresponding directory) and that's it.

It WILL be added to the database, that's actually the problem. After adding to the database, that reference will be the ONLY reference of that file, so if the file were to be referenced, it's added to the database. Then the user could click "Remove" to remove that node-file reference. Now FileField will automatically delete that file because as far as FileField knows, that file is no longer in use.

Now FileField will automatically delete that file because as far as FileField knows, that file is no longer in use.

And that's correct and desired behaviour. As I wrote: either you use filefield or not. It is not possible to be a little bit pregnant. And I also wrote: "probably copied to the desired directory". Normally the assets referenced in filefield reside in special directories. If the asset is copied to such a directory, the behaviour described above does not happen.

But, on linux systems, one can also create a softlink to the original source thus avoiding duplicating the file (with other caveats).

So, what's the problem?

I don't think duplicating the file is an expected or desirable behavior, especially if the file were referenced from 2 or more different FileFields that had different upload directories, you'd be duplicating the file several times into each directory. I think the files most likely to be manually uploaded would be large files on servers that do not support the upload progress bar, so then you'd end up with duplicates of very large files. Or if the user was well-organized, they'd probably FTP the file directly into the folder that contains the other files uploaded by the FileField, so the source and destination would be the same. In such a situation, FileField would end up needlessly copying a file from example.mov to example_0.mov within the same directory.

So you decide to be a little bit pregnant. Ask a woman, if that is possible.

As I wrote: you either let filefield do it's housekeeping or you don't use filefield for that purpose. You could directly use html tags in that case.

I'm simply disagreeing with the approach your describing. If you'd like to have FileField behave in this way, you can write your own FileField Source (it's a hookable system) that behaves in that way. I'd personally prefer to make IMCE behave in a way that only shows files that are already in the Files table. To solve the problem generally, I'd like to have a module (either as part of FileField Sources or a new one) that simply scans certain directories and says "I own these files". This would make it so that even if no nodes were referencing those files, the module would still say "I'm using that file" and prevent it from being deleted.

That's your decision and you can do it as described. But keep in mind that you then will never delete unused data. You can then simply write an alternative module for a cck field containing an url to any resource. You don't need any housekeeping as filefield does. You even don't need any database table to keep track of usage counts for resources as you would never delete an resource.

If you'd like to have FileField behave in this way, you can write your own FileField Source

Filefield behaves in this way (concerning houskeeping), so I don't have to write my own module. Besides, there are many non-programmers that enter issues and take part in this issue queue. So "you can write your own module" is quite impolite. I am not a php programmer, too, but I am an analyst (and a Drupal-user). So I have learned to count pros and cons and consequences of different approaches.

Do as you wish, you don't have to use the code I write and I don't have to take the suggestions you give. I'll take the approach I think is best, if you think otherwise hire a developer and file a patch.

I still agree with quicksketch here. In my opinion files that are just lying around the filesystem shouldn't even be visible. What you are describing (for uploading files per FTP) should be a seperate module or source adapter, not the "reference existing file". The latter is imho logically coupled to files that are handled by drupal.

Being able to reference existing files (in the DB) would be a huge time saver. Also, the ability to reference files as per comment #2 (scanning directory for new files) would help with manual importing of existing files and pre-organized directory structures (via FTP) - which would be a further time saver. There are sure to be many people needing the alternate upload methods other than the one that comes with Filefield. Being able to save them download times (which could be accomplished with FTP) is valuable. But at the same time the concept behind Filefield being a CCK field has some very important considerations with respect to Drupal's content management.

I second the call to be able to import files from an FTP server - with large video files, uploading one after the other is a massive hassle. I'm also getting the error around 'file doesn't exist in database'.

I followed the tutorial on integrating Filefield with JQuery and it's running like magic, except for this last piece. Some of the details in this thread pass me by, but as a principle it would be excellent to have the option to both upload or import/browse server-side files.

ta

I guys,
when I select a file from IMCE file browser I have the following error:

Referencing to the file used in the Documento field is not allowed.

What's the problem?

Can you link me the tutorial on integrating Filefield with JQuery?

Thanks

Nicolo'

here you go:
http://www.drupaltherapy.com/node/50

Any thoughts from the module maintainers whether we're likely to get FTP upload/import functionality? Just the last piece in the puzzle for ultra-flexible video integration

@wilgrace

Referencing to the file used in the Documento field is not allowed.

Did you solve this problem? I tried width JQ Media without results...

Thanks

Nicolo'

@nicoloconte: This is exactly the problem described in this issue. It happens when you try to use a file that was not uploaded using filefield, but using some other method (FTP, IMCE upload, etc.)

I agree with quicksketch's approach here in general, but I would like to show the user POV. When a user (=cotent manager, not web developer. One that does not know where filefield ends and IMCE begins) is browsing her files. She is looking for the 'upload' button in her file browser. If it is not there, it looks like a bug (and a serious usability issue), and if its there, she can't use it since she will get a validation error.

I understand the problem of file referencing, but I think that duplicating a file while adding it into the database might be a good solution for all. Most users won't care about the additional disk space, and you might even add a note that mentioning this fact in the UI. Think of it as an "external" file, like "reference from URL", but the word "external" means "external to filefield" in this context.

In addition to that, maybe you can even compare sha1 to identify files that exist in the database before duplicating them, but that is just an improvement on top of that, not a must.

My 2 cents.

Uploading a file through IMCE is a whole different matter. IMCE actually *does* put files into the files table, we could (in theory) make it possible to use files uploaded through IMCE just by giving the file a "temporary" flag until the node was saved. Or another alternative, if IMCE implemented hook_file_references() and kept a list of files uploaded through it, FileField Sources wouldn't need to do anything at all, since IMCE would "claim" the file as being in-use.

The only real problem is referencing files that were not uploaded through IMCE at all (such as FTPed files). Since these don't have a reference in the files table at all and no module is going to claim ownership on it. Of course we'd also have the same problem with all existing files uploaded through IMCE, since it hasn't ever kept a record of which files it uploaded in the past. They're just files "floating" in your Drupal files table.

I didn't know IMCE keeps files in the files table, it's good to know.

What I suggest is that if the user is trying to reference a file that has no references, filefield_sources should copy it, and start referencing the copy, exactly as if it was uploaded through filefield, or referenced through external source.

yhager: Yeah that's basically what druppi has suggested above. I posted a couple of unfortunate scenarios that could happen with that configuration in #9 (large files, two files in the same directory).

I think a large part of my resistance to the approach is that usually you don't want complete file-system based file browser. What users will want is a system that makes files easy to find, which almost immediately rules out a straight file-system based browser. Using IMCE normally, you end up with directories like "css" (css aggregator), "js" (js aggregregator), "imagecache", "resize" (Image resize filter). Not to mention the thousands of files upload.module will just plop right in the root files directory.

IMCE has huge amounts of potential for a better browsing experience. It supports (right now) any alternative browsing mechanism you can throw at it. So at the root level you could have something like:

Your files
Recent files
Files by type

Right now FileField Sources is *attempting* (but not very successfully) to not show files that are not in the database by starting with the file system listing, then filtering it down by checking the database.

Anyway, I'm going off on my own solution again. What I'm trying to say is that I don't think a database-driven solution (like FileField) is directly compatible with a file-system based solution (like IMCE out of the box). However I think file browsers are great and could have huge potential. FileField is intended for managed files and I think the best way to leverage IMCE would be to make it follow a managed approach also.

Hello,
I think that the file_import module very useful here
that you may import the file to dummy node, then used imported files with the imce brower normally.
thnx

I need a client to be able to upload files but FTP large files. With D5 I hacked the filefield module to do this.

Attached is a very, very ruff source file to ‘Attach’ files from the file system. EG> FTP to a folder, attach these uploaded files to a file field.

This was based on my very bad hack of D5, imce.inc and remote.inc.

Hope this helps other people.

Removed a debug message.

Has anyone tested these hacks? I'm not totally sure where to implement them so can't test.

I tried using the file_import module but am getting a 'File does not exist error' on files that are imported, so that route seems to be closed.

For my use (and it seems a few others), allowing FTP uploaded files to be imported through Filefield Sources would be ideal, but I understand the controversy above.

@wilgrace
To use my attachment source file. You just have to copy the file to filefield_sources/sources/ and make sure the filename is attach.inc.
Please not that the folder you would need to FTP files to is files/file_attach, see line 12-14.

@NaX
Thanks for this - the functionality is appearing on the filefield source area ("Select available files to attach...."), but the dropdown selector doesn't seem to be recognising the files in files/file_attach (fulll path is sites//files/file_attach ).

Does it need to be referenced elsewhere? I'm not completely clear how your include will import the file, whether it's dependent on other Filefield settings that need to be changed on the Content Type settings

Thanks for you help

The FILEFIELD_SOURCE_ATTACH_PATH path is relative to the site root. So you need to change it to the path you want EG. sites/default/files/file_attach. It would be nice to create a per file field settings on the widget form but I could not get the settings to work. I don’t think filefield_source supports this yet.

I could have made the path relative to file_directory_path() but I needed the path to be outside the normal files directory. You could technically set a path out side your sites public_html by setting the full path EG. /path/to/my/ftp/folder but this is untested.

Maybe it would be better to make this relative to file_directory_path().

I do appreciate your help on this mate

Still no joy though - I've tried a number of paths withing the attach.inc file, including:
define('FILEFIELD_SOURCE_ATTACH_PATH', 'sites/default/files/file_attach');

My site is in a subdirectory on the server, but that shouldnt make any difference. Whatever the case, I also tried including the subdirectory name

I'm trying with an mp4 and gif file - does the file format make any difference? Is it just scanning the directory?

I'm using the second of the text files above

Try this version out. Small change to the file_scan_directory() mask. It now does not use the allowed extensions settings any more and just list all FTP’ed files.

From what I can tell version 2 would have work just fine if only one allowed extension was set, but breaks when you set more than one.

Closer, much closer!

It's scanning the directory and naming the files, but seems to be appending the full path to the name of the file - I'm getting this error:
* The selected file could not be copied, because no file by that name exists. Please check that you supplied the correct filename.
* File attach copy error
ta

I think this one should work as expected.

OUTSTANDING!

Mate, you're a legend - this works a treat now. For anyone looking to FTP large files to a server, and then pick them up through Filefield, NaX has just saved the day.

Couple of things on this:
- You need to change the path to the FTP files on line 12 - it's relative to the base path
- I seem to only be getting .MOV files working - mp4 and avi won't play, but this may be other issues with my configuration
- See instructions above on how to activate it

Once again, thanks so much

@wilgrace
I am happy it worked for you.

Very nice NaX! At first read it looks like this addresses the major concern I had regarding the moving/deletion of arbitrary files. It looks like a user could potentially reopen the whole problem by not specifying a FILEFIELD_SOURCE_ATTACH_PATH directory. But I think this is definitely enough protection since it would take an effort to undo this precaution.

... upon second read it looks like you don't even delete the original file that was referenced, but copy it first to the temp directory and then to the final location. This might be *overly* protective, but I'd be happy to include it using this approach since it's better to be over protective than under.

Only a couple of things:
- Clean up the commented out code that is unnecessary now.
- The default file path should probably use file_directory_path() . '/file_attach' instead of hard-coding 'files/file_attach';
- Don't use hash sign comments and wrap the text after 80 characters.

Thank you for attach.inc file. However I found small issue there, because I can select any file from my computer (outside Drupal directory) using eg. C:/Common/files, but then drupal copies test.txt file from C:/Common/files/folder/test.txt to C:/Common/files/test.txt How can I fix it?

Thank you for help.

So, will this work for a private file system currently, or does it only work for public files? It looks to me like it depends on a path within the web root...

thanks

it doesn't work for provate file system at the moment.

subscribe!

Excellent patch, thank you. It works great on my D6.13 setup.

However, I notice that when I select "File Attach" and then change my mind and switch back to "Reference Existing" or "Remote URL", the file selector does not change back to the appropriate field types for those options. The selector that the File Attach option inserted stays there.

Any thoughts? Thanks!

@aasarava
I don’t seem to have the same problem. If I select a file and click transfer and the click remove, I can then select from any of the file source options again. I have not tried it on a multi file field scenario yet. How is your file field setup, and what web browser are you using?

Here is an updated version based the comments form quicksketch
This version is not as well tested (not in production) as the pervious version.
The only code change was the change to FILEFIELD_SOURCE_ATTACH_PATH default value.

@Nax (re #45)
I uninstalled the entire Filefield Sources module and re-installed, and now I don't have the problem anymore. I wonder if there was some .js or .css ordering conflict or a cache issue that was causing it. Sorry for the false alarm! (PS: yes, I'm using a multi file field scenario and it now all seems to work fine.)

Ack, this does not work for me. Put the attach.inc from #36 into the sources folder. I am on a local Xampp installation on windows, trying to use IMCE. Ufku's patch for IMCE appears to have made it to the relase I have. Still I get the message: "The selected file could not be used because the file does not exist in the database."

Does this thing only scan a certain folder for files? Or have I misread something in this thread that needs to be done to activate (reference) attach.inc?

@eigentor
The message you are getting is related to IMCE, the attach.inc file does not have anything to do with IMCE, it creates a new source called File Attach.

Each of the files in the sources directory creates a link above the file upload form fields. The attach.inc file adds the File Attach link. When you click on the File Attach link you should see a dropdown and a Transfer button. The files in the dropdown are built from a specific directory referenced by the FILEFIELD_SOURCE_ATTACH_PATH constant at the top of the attach.inc file. If you use the latest (#46) it will default to your files directory creating a sub directory called file_attach. Using FTP you can upload files into that folder and then they should appear in the dropdown.

To simplify things until you get it all working correctly I suggest you disable the sources you don’t want to use by ether deleting/moving them from the sources directory or renaming them EG. imce.inc.disabled.

Subscribing. Thanks

Hi,
Is it possible to remove the original files from file_attach folder once the transfer completed?

I FTPed a big file to file_attach folder, transfer the file via Filefield_sources File attach link, the file shows up in the node as expected, but the original file remains in file_attach folder. So currently I have two similar files after transfer, one in file_attach (which I supposed to be moved to the new location) and another inside the new location folder.

Just in case I missed anything, I also run the cron manually, flush cache tables and refresh browser cache, but the file remain in the original file_attach.

Is this supposed to be so or did I miss anything?

Thanks

@gausarts
Because your source location is meant to be managed by FTP we don’t remove the files after they have been attached.
If you want to remove a file you need to use FTP/SFTP/SSH to remove it. It is the same with IMCE (I think), because the files were uploaded and managed using IMCE they are not deleted when attached to a node.

Doing it this way has already saved me some bandwidth. I attached the wrong file to a node and then had to correct my mistake, because the files were still in the file attach dir it was no big deal.

I did think maybe we could place a checkbox under the dropdown, something along the lines of “Remove source file after transfer”, but the removing of the source file should really only be done when the node is saved.

That will be great.

Only after saving the node, then a removal button appears. Very thoughtful. This will be a great additional feature. I would be very grateful if you could incorporate the feature into the attach.inc as well. Any plan?

Thank you very much.

@gausarts
I don’t have time at the moment. Maybe closer to the end of the year, but I have a lot of deadlines at the moment.
But I can also live with using FTP to manage upload and removal. We will have to see if this is a feature I will prioritize, unless somebody else want to pickup the ball.

Does it work for you, as it was intended?

@quicksketch
Any thoughts on deleting of files?

Yes, it works fine so far. I think I can live with FTP as well in the meantime :) Thanks.

I could start a new issue, but this component is not official yet :)

file_attach does not transliterate filenames. We got transliteration activated on our site. If you upload a video directly through the file field, file names are transliterated. Only when transferred with the widget, this is apparently not triggered.

Am no big knower of the innerds of transliteration, so cannot tell why this happens or how it is triggered. I thought every file that is entered into the files table is automatically triggered by transliterate, and renamed. But apparently this does not happen.

What we also found out is that files with Umlauts and any special chars are not shown in the select box at all.

patch at #46 works, rocks my world. thanks muchness.

Once figured out, it works like a breeze. Once I made the folder for file_attach a different one than the one the filefield stores its files in, it is even neat and orderly.

We have a patch now working for the transliteration, but it is kind of hacky and only works for german Umlauts:

/**
 * Rename files and replace umlauts and spaces.
 */
function _filefield_source_attach_rename($files) {
  $replacements = array('ä' => 'ae', 'ö' => 'oe', 'ü' => 'ue', 
                        'Ä' => 'Ae', 'Ö' => 'Oe', 'Ü' => 'Ue', 
                        'ß' => 'ss', ' ' => '_', '?' => '_', 
                        '!' => '_', '$' => '_', '__' => '_',
                        );
  $clean_files = array();
  foreach ($files as $path => $file) {
    $new_name = strtr(utf8_encode($path), $replacements);
    if (rename($path, $new_name)) {
      $file->filename = strtr(utf8_encode($file->filename), $replacements);
      $file->basename = strtr(utf8_encode($file->basename), $replacements);
      $file->name = strtr(utf8_encode($file->name), $replacements);
      $clean_files[$new_name] = $file;
    }
    else {
      $clean_files[$path] = $file;
    }
  }
  return $clean_files;
}

you need to change function _filefield_source_attach_get to call the rename function (look in the middle)

function _filefield_source_attach_get($path, $allowed_extensions = '.*', $existing = array()) {
  $file_exists = array();
  $file_attach_options = array('');  
  $file_attach = file_scan_directory($path, '.*', array('.', '..', 'CVS'), 0, TRUE, $key = 'filename', 0, 0);

  if (count($file_attach)) {
    foreach ($existing as $file_exist) {
      if (!in_array(basename($file_exist['filepath']), $file_exists)) {
        $file_exists[] = basename($file_exist['filepath']);
      }
    }
    // Rename source files before showing them.
    $file_attach = _filefield_source_attach_rename($file_attach);
    foreach ($file_attach as $filename => $fileinfo) {
      $is_attached = '';
      if (in_array($filename, $file_exists)) {
        $is_attached = '(A)';
      }
      $filepath = str_replace(FILEFIELD_SOURCE_ATTACH_PATH, '', $filename);
      $filepath = trim($filepath, '/');
      $file_attach_options[$filepath] = $is_attached . ' ' . $filepath;
    }
  }
  return $file_attach_options;
}

The trick is the files are renamed when passed into transferring.
The original file is not changed.

Thx a lot for the work!

Sorry to report the bug here, since it has no official release yet.

I fileattached a file into a node. It was okay whenever I bypass the preview button. But when I went back and edited some text, and DID preview before hitting save button, then the error came:

    * The selected file D:\.....\files\file_attach could not be copied.
    * File attach copy error

It stopped all the editing process. Any hint? Thanks

Found the same problem. Method to find it was slightly different. I used the file attach method to add the file and saved it. I went back and changed some of the other fields and now it won't save. with the same error in #59.

sub..

I found a possible solution, behind the line 124 add the following code:

	$file = explode('/',$file_path);
	$number_of_words = count($file);
	$name = $file[$number_of_words-1];
	$ext =  explode('.', $name);
	$mime = '';
	
	switch ($ext[1]){
		case 'jpg':
			$mime = 'image/jpeg';
			break;
		case 'jpeg':
			$mime = 'image/jpeg';
			break;
		case 'pdf':
			$mime = 'application/pdf';
			break;
		case 'doc':
			$mime = 'application/msword';
			break;
		case 'gif':
			$mime = 'image/gif';
			break;
	}
	$date = time();

	$path = str_replace('%20', ' ', $file_path);
	
	$size = filesize($path);
	//$size = 2100;

    if (!$fid = db_result(db_query("SELECT fid FROM {files} WHERE filepath = '%s'", $path))) {
		db_query("INSERT INTO {files} (uid, filename, filepath, filemime, filesize, status, timestamp) VALUES (1,'%s','%s','%s','%d',1,'%d')", $name, $path, $mime, $size, $date);
	}

I am trying to use patch at #46 (with and without code in #62). I am getting errors with this patch when trying to attach large files.

When I hit transfer, the file, it
IS copied from my "upload" folder into the "destination" folder.
IS added to my "files" mysql database table
IS NOT added to the "content_field_multimedia_filefield"
and the little round icon to the right of the "transfer" button keeps spinning, indefinitely. It is getting hung up somewhere and just stops.

This filefield is getting handled by getID3 and SWFtools. The process works with fine with small files (e.g. <10mb). However, it does not work with large files (e.g. 55mb or 65mb).

From phpinfo(), my upload_max_filesize = 64M

Any ideas? This is exactly what I am looking for -- a way to upload my large (>50mb) video files via FTP, attach them to my cck+filefield node.

Thanks!

I tried code from #62, but I can't seem to make it work. The error persists when hitting save after previewing.

I am no English native, so I don't quite get what "behind" the line means. But I tried placing the code either before or after line #124 that says: $temp = file_directory_temp();, but the problem is still there.

Can you clarify this please? Thanks

Thank you guys so much for the work;
This seemed to work fine for me (#45) - although have not verified it as yet with files larger than 10mb

* The selected file D:\.....\files\file_attach could not be copied.
* File attach copy error

Same problem :(. Seems to accure only with file larger than maxsize allowed by php.ini.

Big thx to NaX, i am so close to getting the thing i need :).

@BurrowsDontMove

I have now used file_attach on a few sites and I don’t seem to have the same problem. I don’t know if it is a hosting setup issue or a code issue.
I am running on Linux it looks like you are running Windows.

When I get more time I will maybe try to look into testing with some really large files, but I don’t have the time at the moment.
Do you know what PHP setting is causing the limit IE memory_limit, post_max_size, upload_max_filesize.

I'm in linux too. And no, i don't kow the settings :(.

Today I downloaded the patch in #46, I manually created the file_attach directory, well I'm not sure this was necessary, anyway I have a problem.
I select a file from the dropbox, I press the transfer button and all is ok, I see the file, but when I press Save button I receive an error:
selected file /var/www/vhosts/spazioausili.net/httpdocs/files/file_attach can't be copied.
File attach copy error
The strange thing is that there is not the name of the file, only the path in which the file is.
And another thing, now the file is both in the files directory and in the file_attach directory, so the copy process is ok but something other doesn't allow to save the node.
Any ideas?

You guys are doing great work! +subscribing.

@NaX
Great work. I think attach.inc is the functionality end users are expecting, it is for me. Any development on the Preview/Edit issue yet?

@quicksketch
Do you see any solution to NaX's attach.inc function not working with Preview or Edit?

I finally found some time to do an update of file attach. I have updated and attached a new version of attach.inc.

I have added support for the transliteration module and it is based on eigentor hack in #58. I don’t really like the idea of rename the files in the file_attach directory and would rather do it on save, another away would be to change the $dest of field_file_save_file(), I don’t really understand why the filefield_widget_file_path() call is not working as expected. So I added a message that notifies the user that their FTP’ed files were renamed. Thanks for working this out eigentor. Maybe latter we can dig a little deeper into filefield_widget_file_path and be able to do it on Save.

I have added a commented out debug form_error() function call on line 166. For people getting "File attach copy error" errors you can uncomment this line and it will output a lot of extra info that we can use to debug this further.

Before un-commenting this function call please first confirm that your temp location is writeable by visiting the status page or the file system settings page.

I have tested this version including previews and it all seems to work as expected for me. The only thing I did not test was using very large files. All the files I used for testing were under 1MB.

I also did not test with the getID3 module.

Once others have tested this latest version I would like to call others to help with some setup documentation. Some people seemed to have trouble with understanding the basic setup as it is a little more manual than most modules. I would like to add an attach_README that later when/if it gets included can be copied to FileField Sources README. At the moment the only docs are the comments at the top of the inc file and I don’t think this is adequate.

subscribing

NaX,

These are great additions to this module...great work thank you! Below find my feedback for #72 attach.inc file.

Errors/Bugs after applying #72:

When uploading/attaching files on a node that has several filefields, I get the following:

---1st filefield: I attached a file using "Attach Files," this file usually transfers ok (I get a success message and I can see the file on the list)

---2nd filefield: I uploaded a file via regular upload (the "Browse" option).

However, I can't upload the file or the 2nd filefield...I am getting the following:

1) "Successful file attach message," even though I uploaded the file
2) The form disappears.
3) The file does not upload and I get these errors:

The selected file  /files/path/to/upload/folder could not be copied 
File attach copy error

4) Get same errors when I try to save the product.
Suggestions/feature requests:

Make the FILEFIELD_SOURCE_ATTACH_PATH part of the config options. And if possible, add token support so one can set paths using tokens (e.g. user directories)...is this something that you plan to do?

Thanks,

KH

Great work, NaX. You've just salvaged a primary feature of a planned site. Thanx.

I've done some testing on two separate installations, one on shared hosting, the other on a VDS. In both environments I still get the "File attach copy error" and am unable to save large .mp3 files AFTER previewing. I can save changes if I don't preview, which for me is no real big deal, and means I will be able to use file attach.

Oddly, I am able to preview then save .jpg files on the shared hosting, but not on the VDS. I uploaded via ftp the same .jpg and .mp3 file for testing into the respective file_attach folder for each. File permissions are the same. Only real difference I can see is the Server API for VDS is Apache 2.0 Handler, while it's CGI/FastCGI for shared hosting. I know this has an impact on the owner/group definitions, which probably means that even though my file permissions appear to be the same (numerically), they aren't.

Hope this helps.

Hello Nax,

I am currently in the middle of configuring your attach.inc to allow each user to ftp their files and attach them later.

I'd like to know how best to configure it so that each user has its own folder like files/[uid]/file_attach so each user can only attach his/her own files.

What is the best way of setting/achieving this on the attach.inc.

Thanks,

KH

@3dloco

I recommend trying to get it working on one directory first, and then trying to go to a multi user solution.

You do know that every user will have to have there own FTP account.
Or are you going to have one FTP account but a per user folder that forms the dropdown, so only the site admin will have FTP access.

I have never tried to do a multi user solution but if I had to...

Look at:

define('FILEFIELD_SOURCE_ATTACH_PATH', file_directory_path() . '/file_attach');

Something like:

define('FILEFIELD_SOURCE_ATTACH_PATH', file_directory_path() . '/file_attach/' . $user->uid);

OR:

global $user;
define('FILEFIELD_SOURCE_ATTACH_PATH', file_directory_path() . '/file_attach/' . $user->uid);

First of all, thanks all for the excellent work you're sharing here

I've been happily using the filefield sources module and attach.inc hack. But since i updated to the last module version of 2010-Feb-22 the module doesn't seem to acknowledge the attach.inc anymore. I didn't change it, it's still there, it just doesn't show up in the form. The other .inc's do. And they appear and disappear whenever I add of remove them. Just the attach.inc does not.

So, anyone else got this problem? And has a solution? And for those who haven't please don't update to the last version because I haven't been able to find the previous working version....

For those encountering the same problems, I found the older versions of the module in de CVS repository:
http://drupalcode.org/viewvc/drupal/contributions/modules/filefield_sour...

All is well once again.

Hello NaX,

Thanks for all your help I really appreciate it. With the code you provided above (#78) I was able to set the FTP for multi users, and its working great! The one that worked for me is the last code the one with global $user.

You do know that every user will have to have there own FTP account.
Or are you going to have one FTP account but a per user folder that forms the dropdown, so only the site admin will have FTP access.

Thanks for pointing this out. The way I currently have it setup is that each user has its own FTP account.

What I'd like to know is how best to further limit the dropdown to certain files. In my case I am thinking of just adding the field name to the directory, sort of like files/[uid]/file_attach/[%field_name].

This works for me because I can create folders for each filefield. For example, If I have filefield with the name “field_images” for attaching images only. For this field, I want to limit the dropdown to files inside the files/[uid]/file_attach/field_images. In this way, the dropdown will list only images and the users can only select images.

Is this possible and what do I need to do in order to get the attach.inc to use the field name.

Many Thanks,

KH

The file attach.inc worked quite well for me - thanks for the work.

I recently updated filefield_sources to the latest dev release (I think the date was 2-27-2010) and the functionality added by attach.inc went away.

When clicking the link to get the drop-down list of files that have been ftp'd, well, the link was no longer there. The attach.inc file was present, and it had been working well before upgrading to the latest filefield_sources module.

Any suggestions on getting this working again?

Thanks!

This file should continue working as before, it just that now you need to explicitly turn it on. Since it wasn't an option in the official release, the update did not automatically turn on this option like it does all the other ones. Just visit the field configuration and turn it back on.

NaX: this all looks pretty good, I think it makes sense that this is a separate "source" file so we can document it more extensively than the others. However, now that we have #436174: Configuration Options, it should be possible to make the server directory a configurable option, rather than needing to change it in the source code.

Hi, I hope it's ok to post this here, as I think it is related.

I'm using drupal 6.15, filefield 6.x-3.2, filefield_sources 6.x-1.x-dev, and the attach.inc file submitted by NaX in post #72 above. I'm having this error too:

* The selected file /home/claudo/site.com/sites/default/files/file_attach could not be copied.
* File attach copy error

I then upgraded to filefield_sources 6.x-1.0, and added attach.inc as per above, but still same error. I get this error whenever I try to attach a file, whether by using "Attach File" or "Upload" option. Even after I disabled the Attach File option on the field setting, I still get the same error.

The strange thing is that it was all working fine a week ago. I could upload files, attach files, etc, no problem. The only changes in the past few days that I can think of is that I upgraded ckeditor (from dev version to 6.x.1.0), installed ckfinder and disabled imce (as I don't want to use the Browse File option for attaching files). Would any of these break filefield_sources?

Some server settings:
memory_limit - 90M
post_max_size - 20M

The file that I'm trying to upload/attach is 16M.

Does anyone have any idea of what could be wrong? Any hints or pointers would be appreciated. Thanks.

Just to add some more info on my post #84 above, and also since it's happening on the non-dev version as well.

I'm also using FileField Paths 6.x-1.4

In the field settings of the file upload, I have under FileField Path settings, File path: sermons/[term-raw]. When I modify this, say to testdir, it is reflected in the database, as in the filefield_paths table is modified accordingly. However, the debug output from File Attach seems to indicate it is still using the old file paths. This leads me to believe the problem is not necessarily with FileField Paths, but only that the value is being ignored.

Here's an excerpt from File Attach debug output:

[widget] => Array
(
    [file_extensions] => txt mp3
    [file_path] => sermons
    [progress_indicator] => bar
    [max_filesize_per_file] => 
    [max_filesize_per_node] => 
    [filefield_sources] => Array
        (
            [remote] => remote
            [attach] => attach
            [imce] => 0
            [reference] => 0
        )

    [label] => Sermon download
    [weight] => 2
    [description] => 
    [type] => filefield_widget
    [module] => filefield
)

...

[uid] => 1
[filename] => 20090830.mp3
[filepath] => sites/default/files/sermons/20090830.mp3
[filemime] => audio/mpeg
[source] => field_file_save_file
[destination] => sites/default/files/sermons/20090830.mp3
[filesize] => 9054208[status] => 0
[timestamp] => 1268137518
[origname] => 
[filefield_upload] => Upload
[filefield_remove] => Remove
[upload] => 

See how filepath is still pointing to "sermons" directory even after I changed that value to "testdir". Also, are [filepath] and [destination] above supposed to be the same?

Any thoughts? If you think this could be better posted somewhere else please let me know. Thanks.

@pinkish
It looks like some sort of module conflict. I am next, when I get some time, going to work on addressing quicksketch points on #83. If I can, I will see about looking into your bug. Thanks for the debug out, it should help.

I don’t think I will have the time to sort this out this week, but hopefully soon, in the mean time you can test by disabling new modules or related modules. EG: Try disabling FileField Paths and see if it starts to work. I don’t think ckeditor could affect it, but you can try test with it disabled. If possible I would recommend testing on a backup/dev site and not on a live site.

Looking into a similar solution to enable me to upload video files that are 2GB and larger so I'm subscribing here. A big thanx to all that have worked on solving this.

Cheers!

+1 subscribing.

subscribing

I had success with "attach6.inc".

http://drupal.org/node/438940#comment-2460938

Attached the largest MP3 mix on the site first which is over 200MB. It took under 30 seconds to complete - no errors. This is altogether something that has me pretty excited to move ahead with this website project as it was a bit of an ordeal finding a solution.

Here is my original question:
http://drupal.org/node/742512

In case it's helpful I'm using Drupal 6.16 with some more server/setup trivia below.

PHP 5.2.12
PHP memory limit 250M
Web server Apache (Shared hosting)
getID3() 1.7.9-20090308

- getid3-6.x-1.3
- filefield-6.x-3.2
- cck-6.x-2.6
- filefield_sources-6.x-1.0
- swftools-6.x-2

Subscriving, Been looking for this functionality for a wile now. I will try this and hope for the best. Thanks for all the work,.

Hi, Figures things would not go smoothly for me. anyway I put the attachment.inc file in to the sources folder, went and turned it on, created the file-attachment filder in the site/default/files/ folder and put sme files in the folder, but no luck, nothing shows up in the combo box. would there be a way to change this from a drop down to check boxes or radio buttons as an option?

Did I miss something. I beleive I found the attachment source on 72, sorry if I am wrong, been at this all day searching for resolutions.

@Jim
Please see the documentation at the top of the .inc file. For a more advanced setup please see comment #78. In future I hope to make this configurable. When it comes to your request for check boxes or radio buttons, I am not planning on anything like this, but I don’t think checkboxes are at all possible but I think you could alter it to use radio buttons.

I have finally found some time to work on file attach again. I think we are finally almost ready for the big time. This update is a big one and I would really appreciate some testers give it ago as I can only do some limited testing on my side.

Please do not test this on a production system as a lot has changed and it should be considered beta quality until we get feedback from testers.

Informal Change log:

- Integrate with per field widget settings provided by FileField Sources module.
- Make file field path use token module when available. Code copied from filefield module.
- File attach path is forced to be in Drupal files directory. We can look at expanding on this at a later stage, but I feel this is the best option for the moment and the simplest to setup for most user cases.
- Revamped transliteration support to alter files in temp location before being attached to node. I was not happy with altering files in the users FTP file store, I also found a dangerous security issue with incorrectly formed file attach paths and this change should prevents all these issues. (Not full tested).
- Improved feedback to user when no files are found or file attach is not correctly setup.

One idea I briefly looked at but decided against was adding a remove source file checkbox to the transfer form that would delete the file from the FTP file store when the user clicks the transfer button, but I decided against this as I feel the file attach file stores should be 100% managed by FTP/SFTP. If there is major need for this we can look at this at a later stage, possibly making it a field setting.

subscribing

Subscribing

Hello NaX,

This is awesome! Thank you so very much for making these GREAT additions, especially the support for tokens and per widget settings. With these new changes to attach.inc now is possible for me and many others to have ftp for multi-users and have specific folders for each filefield.

I've done testing with my most recent config, which now includes:
FileField Sources 6.x-1.x-dev (2010-Mar-13)
File Aliases 6.x-1.1
FileField 6.x-3.2
FileField Paths 6.x-1.4
getID3() 6.x-1.3
CCK-6.x-3.x-dev (cck2 dev with multigroups)

Here is what I am getting:

--Attaching files from ftp for Filefields that are inside multigroups works great! I can attach multiple files and save node ok.

--Attaching files for Filefields that are inside a standard group only work for the first attach. When I click to attach another (add another button) I get a folder permission error even though my temp folder is writeable (777), as follows:

* The selected file /.../files/attach_folder could not be copied.
* File attach copy error. Please confirm you temp directory is writeable. See status report for setup errors.

--What is strange is that I am getting the error above regardless of whether I use upload, attach, browser or reference existing (as long as attach is enabled).

--And this error only occurs when I have already uploaded/attached/referenced the first file and then click on the add another button.

--This error does go away once I disable attach or if I move the filefield to a multigroup.

Appreciate your help,

KH

3dloco, I was finally able to reproduce this long standing bug and think it should now be resolved.

Up to now I have not tested it on an unlimited multi field and it turn out it was the "add another" button that was submitting the form and causing the process code to fire. I incorrectly assumed that the "add another" function would deliberately try to avoid additional processing.

The fix ended up being a very small change.

Please give this version a go. If you can, it would be nice if you could please test this version in all the same scenarios you did version 7. I am currently only running CCK stable on all my sites and not dev, so a multigroups test would great.

Hello NaX,

THANKS! I've tested the attach.inc_8 above with same config as in #97 and everything is working now! I can confirm that attaching files from ftp works for both standard groups and multigroups, when attaching single or multiple files.

I really appreciate all the work you've put into this and I am forever grateful to you for sharing it with us :-D

Thanks again for all your help,

Regards,

KH

Subscribing.

Subscribing

Subbing

Can we please reconsider the decision to force the FTP upload path into drupal's files directory for the following reasons :

* FTP upload directory will be publicly accessible if the drupal installations is using public filesystem
* FTP upload directory should (ideally) be on separate partition with "noexec" flag, and quotas

In my case the FTP upload directory is on a separate server and is exported through NFS to the www server.

Can we at least make it an option ?

Thank you for your hard work :) this feature is really worth its line numbers in gold.

@iamer
What you are asking for is next on my feature list, but it's not a priority at the moment and I probably will not be working on it anytime soon.

My priority at the moment is getting it stable and feature complete enough for quicksketch to consider it for inclusion. There is an argument for it being a critical feature that is needed before inclusion.

If you can’t find a workaround IE symlinks using the latest version or can’t wait, I suggest you give #72 attach.inc_6.txt a try. There is an irritating bug in that version that only affects multi file fields, that was only fixed in attach.inc_8.txt, but that version should allow you to configure your file attach path to be what every you want. To do this, have a look at the comments at the top of the file. The one major down side to this version is that it only support a single file attach path. From version 7 on it is per a CCK file field.

If you need all the new features and you know a little PHP it’s also a very easy hack in version 8. Have a look at the very last function.

From

  return file_directory_path() . '/' . $path;
}

TO something like this:

  return '/' . $path;
}

Yes I am already doing that , thanks for taking the time to answer.

I think the "attach" feature is essential for any site that wants to use FTP upload in a logical and secure manner.

Same issue as #106. I cannot set the directory and save. If I leave it as the default file_attach, it is not recognized. Is the latest .inc (#8) dependent on a specific version of CCK?

Thanks for all your hard work!

EDIT: I currently have...

CCK 6.x - 2.6
FileField Sources 6.x - 1.0
FileField 6.x - 3.2

not working either with the same configuration in #107

@#105, #106 and #107
I have only test version 8 with the latest dev version of FileField Sources (6.x-1.x-dev), and I think it requires the latest dev version.
Please test it with the latest dev version and let me know if the problem you describe goes away. I am unable to replicate your issue using the dev version.

great, it is working with FileField Sources (6.x-1.x-dev)

Wow...this is looking very promising. Can I get advice on what looks like a simple problem?

- running attach_8 with the modified closing line from #104
- latest .dev of filefield_sources
- in my cck file attachment field, I set the file sources path to: uploads/attached which is a writable directory sitting in the files path (../files/uploads/attached)

But when editing an existing node I get an error:

"* Specified path must be under file sytem path."

And clicking on the File Attach widget returns:

"File attach could not create or locate a valid directory. Please check your settings for this field."

Any clues? Thanks in advance...

@boabjohn, don't use the modified line in #104 unless you need to attach files elsewhere outside your files directory.
hope this helps

OMG: kisses and hugs to all of youse heros! Where do I send the beer!

xox

And (because I never know when to stop) where is the thread that is discussing the provsion of a sources/webfm.inc connector for WebFM?

There's a lovely table full of fid and path values sitting there, nicely managed by WebFM. But I don't see a native linkage between these files and fielfield_sources....

Is that a feature request, clueless, or already done?

Cheers!

where is the thread that is discussing the provsion of a sources/webfm.inc connector for WebFM?

WebFM is unfortunately in no way compatible with CCK or FileField. See #436396: support for webfm popup.

Ding: right. Thanks for the pointer. I believe I will have to dump WebFM as it is making file management a nightmare due to this lack of sync with the files table.

Thanks again for all the help and contrib.

Just in case anyone else is loving this thread but needs the capacity to insert a stack of ftp file references into the core files table, check out the excellent effort by John Locke:

http://www.freelock.com/blog/john-locke/2010-02/using-file-field-importe...

It works as promised. million thanks to all of you specially @NaX :)

about being unable to save the filefield settings. this is caused because of line 90 in filefield_sources.module:
$settings += filefield_sources_invoke_all('settings', $params);

duplicate keys will be overwritten so this should be turned into an array_merge i think

@Johnny: This appears to be fixed in 6.x-1.x-dev.

The attach.inc file from #98 seems to work very well for me along with the 6.x-dev version of filefield_sources. Thanks!

EDIT: In the actual "File attach" file dropdown, might I suggest asort() on the returned array first? It would make the perusal of the returned list of files much easier... (tested this and it works fine for me)

  // At line 169:
  asort($file_attach_options); 
  return $file_attach_options;

I think this will be the last version that adds features for now, until file attach gets included in FileField Sources.

New features that need testing:

- sorting files in dropdown by: torgosPizza #119
- absolute path support requested by: iamer #103

I need people to test the absolute path feature.

Hello, thanks for your wonderful module. I have tried all i can but just keep getting this message!
"The selected file could not be copied, because no file by that name exists. Please check that you supplied the correct filename.
File attach copy error. Please confirm you temp directory is writeable. See status report for setup errors."

Maybe i need more info on how to set the directory.

Thanks

@ibraaheem
Can you give us more info on what you are trying to setup? Are you using the latest dev version of FileField Source, because File Attach will only work with the latest dev version? Also are you using the new absolute path feature?

Hello Nax,

Thank you for the additions of these features! I have tested attach.inc_9 with the same setup that was working before (tested in #99) except I used the latest development version of filefield sources as follows:

-FileField Sources 6.x-1.x-dev (2010-Mar-30)
-File Aliases 6.x-1.1
-FileField 6.x-3.2
-FileField Paths 6.x-1.4
-getID3() 6.x-1.3
-CCK-6.x-3.x-dev (cck2 dev with multigroups)

1) Sorting files in dropdown by: torgosPizza #119: The asort is working great as I am able to select/goto a file by typing the first letter of the filename).

2) Absolute path support requested by: iamer #103: Could not really test this feature because the file attach no longer works with FileField Sources 6.x-1.x-dev (2010-Mar-30) and attach.inc_9 whether the path to the file_attach folder is relative or absolute.

I am getting the same message as ibraaheem #121 above even though I've set the folder permissions to 777.

The selected file could not be copied, because no file by that name exists. Please check that you supplied the correct filename.
File attach copy error. Please confirm you temp directory is writeable.

Thanks,

KH

@3dloco
I fix it by changing line #207 from

$attach_path = _filefield_source_attach_calc_path($field['widget']['filefield_source_attach_path'], $GLOBALS['user']);    

to

$attach_path = _filefield_source_attach_calc_path($field['widget'], $GLOBALS['user']);    

May be it's wrong

Thanks a lot KestL, your modification worked out and I can confirm that attaching files works when using an absolute path (using attach.inc_9 plus the change mentioned above in #124).

Note: I tested this on a node with multiple filefields (file and images), some filefields had absolute paths and others had relative paths. Also some of the filefields where inside standard groups and others inside multigroups...and all of them worked ok.

Thanks KestL, here is an updated version including your bug fix.

@3dloco - Thanks for all the testing.

Tested attach.inc_10.txt, works fine for me. Thanks @NaX, I second that RTBC!

Thanks. Sub'ing

I've given the last version in #126 a spin with the latest filefield release and filefield_sources dev. Everything seems to be working fine.

Thanks for all the wonderful work. RTBC!

Can anyone tell me how I apply the patch in this forum. I need to be able to use files already FTP'ed to the site, as the subject states here. Do I just copy the text from the attached file, and paste it into or on top of some other text in the .module file for this module?

Thanks for any info

I tried the patch from #126, but unless I did something wrong (entirely possible), then it's not working correctly.

What I did was copy/paste the text from #126 to the end of the filefield_sources.module file.

Now, I see a link that shows up saying "reference existing." And it has a text box where I can enter the values, but no matter what I put in the field, it just doesn't do anything... basically, I put the name of a file in the field (and/or path to file) and then click the Select button, but nothing happens, it just erases it, and looks like I never entered anything, so when I try to save the node, I get an error message telling me no MP3 file is attached (b/c I set it as required for the content type).

Can someone give me a clue as to what I may be doing wrong?

I can list a file which I've previously uploaded with the IMCE module for my custom "audio" content type, but if I just FTP'ed the file to the site, then I can't seem to access it. It seems like it should be pretty simple to force the files through the same process of adding to a table if they are already there.

Or what about a simple method that we could run and add any already existing files to the table in one fell swoop, then all of this won't matter for individual file selection? But then I guess I'm getting out of bounds of this request...

Any thoughts to my ramblings so far?

Right now, I have the audio module installed on my live site. It works fine, but I was wanting to use the SWF Tools module and CCK fields instead to give me more control over the player appearance, field filtering in views, and that sort of thing. Which let me to this module from suggestions in this thread:
http://drupal.org/node/796520

Thanks for any suggestions.

One more thought, question. If I can understand the table fields, and what they are, then I could simply manually add the file information from the files into the database table? I tried manually adding one, but I didn't know the exact file size, and didn't know what the timestamp was referring to, so I left those at 0. It added a file reference, but it ended up not being usable, gave an error when trying to save the node:

Referencing to the file used in the mp3 field is not allowed.

Thanks for any suggestions... (if I get get it working as per my previous post then I won't need to do this part.)

Well, if nothing else, this is giving me more reason to need to better understand the php code so that perhaps I could dive in and give more help rather than saying, "help!" ;0)

Well, here's another thought - something I tried, that worked out perfectly me in my scnerio.. I've only tried with one file so far, but I've no reason to believe it won't work with other files. And now I've just got to figure out how to get the getID3 info into the Content type, but that's a difference issue, and probably not an issue, but just something for me to learn/figure out.

Here's what I have done:

1. Created a test site on my main site's web server, as a subfolder for testing SWF Tools Module usage instead of Audio Module (which as I mentioned IS a great module, but I want more control over a few things.)
2. After already testing with all these various modules - SWF Tools, FileField, CCK, filefield sources, etc.. I created a folder in the site's Drupal files root, called audioimport... got this idea from the Audio Module, and how it's audio import works..
3. In my test node, I clicked the Remote Path option, and entered the path to the file within the audioimport folder.
4. Clicked the Select button
5. Viola! After saving the Node, it works. Frankly, so far, it seems to work better than when I just flat importated the file... maybe it has to do with the patch from #126, I've no idea at this point.

So, now I'm off to doing more testing, but I think I may very well use this method to get some things going that I've been wanting to do all along!! Yippeeee... things I thought I'd never get fixed so that I could try them out! Woohoo!!!!

subscribing, attach_10 works brilliant ... thx

@kjv1611, to use it place the attach.inc file from #126 inside the sources folder like this /../filefield_sources/sources/attach.inc, no need to modify .module file.

Thanks, I'll definitely give that a shot! That sounds much more reasonable.

By the way, is that just the case with this module, or is that how things work (in general) with php? In other words, if I've got a site running php - such as Drupal - and I want to add something, if I simply add a file that has php in it, will the server automatically see it, and use it (if possible)?

Thanks for any info - regardless, I'm definitely going to give this patch a try.

@kjv1611, it is not a feature of PHP itself to automatically discover PHP files and parse them. This is a common feature of MVC frameworks like CakePHP or Codeignitor and of CMSs like Drupal that are written in PHP, but that functionality is not intrinsic to the programming language.

Thanks for the info. That makes a lot of sense (not being sarcastic). It makes perfect sense that it's the particular CMS or tool rather than the language handling that, now that I think of it. Thanks for clearing it up for me.

By the way, is that file supposed to be named more like attach.inc or attach.inc_10.txt or does it make any difference? I mean, which name should I give it when putting it on the web server?

@kjv1611, the file should be named attach.inc.

wow long thread lol. I came late to the party. Is this ever going to make it into the dev? I'm going to try out one of these patches tmrw when i get to the office.

patch from #126 works for me. Thanks a lot :)
Everything works as expected. User selects a file from the drop down that is located inside the FTP folder. Upon clicking the Transfer button that file is copied to where the file field saves the files. This is great b/c now the user can reference it without having to re transfer it.

The only problem I've come across is if the user decides to overwrite a file in the ftp with a newer version that file is not recopied to the new location. If you try to re attach the file it gets copied over but with a _0 after it b/c it doesn't want to overwrite it.

Would it be possible to somehow keep the files in sync. Maybe a script that checks the files on cron? A small bounty could be paid if we could work something out. Just email me

Thanks a lot :)

@gooddesignusa
I think what you are referring to has to do with how Filefield works or how the Drupal file system works and does not have to do with FileField Sources. Have you tried removing the file and then saving the node and seeing if the file gets deleted and then re-editing the node and attaching the newer version.

I would run a test with a small file using just the normal upload method and seeing if it still does what you are referring to. I dont think the file gets deleted until there are no nodes referring to the file.

@gooddesignusa
After a short search I found following modules and I think you will find them interesting. I don't know if they work with FileField Sources, and if you test them some feedback on their compatibility with FileField Sources and attach.inc would be appreciated.

http://drupal.org/project/upload_replace
http://drupal.org/project/attachment_links

First let me start by saying I'm not a programmer and much of whats in this thread is a little over my head but I'm learning and trying my best. I've followed the thread through and would like to qualify the correct procedure for sorting the issue.

For me, I'm creating a site and have a number of podcasts and video to put on the site. Some exceed the hosting companies upload limit of 8 megs.

Having created my content type I have a filefield source for the videos. It uses filefield_sources in order for me to choose how to put my files up / reference the file that is already ftp'd to the files/video folder I created. It works great for those files less than 8 meg. Those that exceed that I get the error message : "The selected file could not be used because the file does not exist in the database."

To overcome this I used the fix 'attach.inc' at #126 and placed in sites/all/modules/filefield_sources/sources/attach.inc

After doing that I once again tried to browse to the file using IMCE and had the same error as before.

I also tried using 'Reference existing' and get this - An HTTP error 0 occurred.
http://domain.co.uk/filefield/reference/video/field_video_upload

I would very much appreciate guidance on this. A simple 123 of how to get this to work or a viable work round / best practice. I find it strange that this is an issue as I understand that most hosting companes limit the uploading to 8meg and ftp'ing seems to be a logical way around this. I see from the thread there are security issues / file issues that needed to considered but surely this sort of functionality is needed to keep Drupal in it premier position.

Finally thanks to all those who have written on here, I have learned much from your explanations as to how Drupal works.

@richard1966
To use the latest version of attach.inc you need to use the latest dev version of FileField Source. With the latest dev version of FileField Source you are able to enable and disable sources per a field and you do this in the file field CCK settings page. Once you have uploaded the latest version of attach.inc (currently version 10) to the sources directory you will be able to enable file attach in the field settings the same as the others. To use file attach you need to upload files to the file attach folder. Then when uploading the file you will see a link called attach. When you click that link you should be given a dropdown allowing you to select files in your file attach folder. The file attach folder is separate to IMCE folders and should be seen as a separate external FTP folder that could exist in the Drupal files directory or in a completely different location using the absolute path feature and it is best to not mix file attach folders with other folders. IE: don’t use the IMCE folders as your file attach folder. It should be fine to have one file attach folder that is shared across multiple file fields.

I hope that all makes sense.

NaX

Thank you for this. I can't tell you how loud I cheered when I saw the video load after following your method.

It never ceases to amaze me how generous people are with the time and knowledge. Many, many thanks!

Richard

Well all was going swimmingly, the fix was working thanks to Nax, I added loads of content to the site and then after a number of hours for no apparent reason using the file attach method for adding files that I added to the site via FTP I started getting error messages...
"The selected file /tmp/file_name.png could not be copied."
The selected file /tmp/jfile_name.mp3 could not be copied.

... when I clicked the 'transfer' button.

I've cleared the caches to no avail. Could anyone offer insight?

@NaX
Sorry if i wasn't clear on explaining. I did check out http://drupal.org/project/upload_replace the other day but didn't do enough tests. Today I reinstalled it and ran the following.

First I made sure the module indeed worked as expected. Using the normal file field and upload a file with the same name triggers the rename of the old file instead of the new one. Thus keeping the newest file with the same original name and the old one gets append a _0 to the file name.

1. Created a new page with a file field called files. I called this Page Import Files From FTP.
2. Edit that page and transfer a file from ftp using the new patch. Save page. ( example file.pdf )
3. Goto a different page that also has a file field called files. Edit that page and Reference the newly added file. (Since it was already transferred on the previous page it now shows up in the reference no need to reattach)

-- Everything is working as expected so far.

4. Open an ftp client , connect & upload a new revised copy of the file you uploaded before with the same name. Overwrite it. (Mine was files.pdf)
5. Goto the first page you set up and edit it. Mine was called "Import Files From FTP"
6. Fine the file field that holds the file you previous "transferred". Remove it and "re-attach" the new file with the same name which would be the updated version. Save page.

7. The new revised version of the file is copied to the new location BUT upload_replace doesn't seem to get triggered. The new file is does not take the place of the old file it is renamed. Thus all the other pages that reference that attached file does not get updated with the new revised version.

If upload replace did work it would still require the user to remove / re attach the new file after uploading via ftp.
What I was purposing is somehow we can keep these files in sync so that the user does not have to reattach after ftping a newer version of a file.

I was looking @ the 2 folders and comparing the files "mtime". The original file in the ftp directory has the correct mtime. The new file that was copied after the transfer has a mtime of when it was created. If we could use php touch to set the mtime as the time as the file in the ftp directory we could theoretically be able to tell if the file in the ftp directory is the same as the one that was copied. When a user uploads a new revised version to the ftp directory it would have a different mtime than the one that was previously attached.

Then maybe somehow make a function that checks to see if the files are in sync and if not copy the new file over to overwrite the old one. Maybe run this script on cron run ?

Another unrelated thing I was curious about was if we could somehow color code the files in the dropdown under the attach section to show which files are already "transferred". For example if all files that are already "transferred" were blue the user would know to use the reference area instead of re attaching it. This would hopefully help keep the files that are copied to the minimum. Or instead of color coding maybe have an option in the admin section that would hide files that were already "transferred".

Sorry for the long post. I tried my best to explain my situation as I'm not a expert PHP programmer.

edit: forgot to mention I can provide a bounty. Just contact me to talk further since these requests are somewhat specific for the site I'm working on. If others might need this functionality I'm fine with paying to get it added.

Please ignore #147 - I'd reached my limit on the webserver which is why it generated errors. The NaX recomended method works well!

@gooddesignusa
I had a very quick look at it and from what I can tell we using field_file_save_file() to transfer the file, but some how field_file_save() is not being called (See filefield/field_file.inc). The reason field_file_save() is important is because it calls hook_file_update() and upload_replace uses hook_file_update().

When I have some free time I will have another look at it. Please note that I can’t promises looking at it anytime soon, my work schedule is very long and tightly packed at the moment.

Just did some more tests regarding upload_replace and filefield sources. Regardless if you use this patch the referenced file will still point to the older file. I will start a new issue but i'm not sure if this is even a bug or expected behavior.

@gooddesignusa

I have an idea, but please note this is completely untested and I think we going about this the wrong way.

The idea is to make attach.inc work with the upload_replace module.

In attach.inc change this - line 235

if ($file = field_file_save_file($filepath, array(), filefield_widget_file_path($field))) {
  $item = array_merge($item, $file);
}

To this (taken from field_file_save() in field_file.inc)

if ($file = field_file_save_file($filepath, array(), filefield_widget_file_path($field))) {

  // Let modules update their additional file properties too.
  foreach (module_implements('file_update') as $module) {
    $function =  $module .'_file_update';
    $function($file);
  }
  _field_file_cache($file); // update the cache, in case the file has changed

  $item = array_merge($item, $file);
}

I can try this but I dont think this will solve my issue. I think i misunderstood how file field sources actually works. After I did those tests from comment #151 it seems like it wouldn't matter if upload_replace worked or not. Since the reference file wont be deleted if other nodes are referencing it. Thus the other nodes that have a ref file wont be updated with the new one.

Ideally I was trying to make one node where the user could store files that are shown on other nodes. So in case a file needed to be updated they could just change it in one place. File field sources does not do that. The user would have to go back to all the nodes where it was referenced and re reference them. So it kinda defeats the purpose of using this module for me. It does have some benefits though just not what I originally thought.

would be great subscribing

Wow, this really does work fine.

Suggestions to folks who are arriving at this thread late - grab the latest attach file, rename to attach.inc, and put in the 'sources' directory of filefield_sources.

And then you'll want to clear the cache, I expect.

I'm about to put out a 1.1 version and I was hoping to include this functionality, but it's not quite ready for prime-time and needs a bit more work. Since FileField Sources requires an update to work with the new version of FileField (3.4+), I'm going to shelve this feature and include it in the next release.

So far I've cleaned up the code a bit:
- Formatted according to coding standards.
- Fixed strings to match patterns throughout Drupal (Sentence casing labels, proper use of i.e. instead of EG, clarity fixes, etc.)
- This patch incorporates transliteration support but is not compatible with the issue in #784946: Add support for transliteration to remote URLs. If providing a transliteration mechanism it should be applied to both sources.
- Theming improvements.
- I'm not sure about the "Absolute" checkbox and the way that it works, I'll need to give it some thought.

Anyway here's the patch of the status so far, it still needs some work before being included.

well - I'm a little uncertain from your post, quicksketch, what you think needs to be done to make this ready for the next release?

The code still needs to be cleaned up and more strings revised for clarity. I'm not sure if we can consolidate code further, such as _filefield_source_attach_directory() seems like it's redundant from other sources. The patch also needs testing again since I've changed the way the code works quite a bit already.

I can confirm the patch from #156 works.
Already using it on a live site ( have to because of over 1000 videos).

What I would expect (maybe as optional checkbox) is that the files are moved instead of copied.
At the moment I still have the already used files inside the attach selectbox, which is a little
confusing by over 1000 files. That's the reason I would prefer a checkbox in the filefield settings,
letting me move the files instead of copy them after they are attached to a node.

Confirmation here too that #156 is just the feature I was looking for. I didn't test the "Absolute" check box, though, and I agree with MrKatz that I'd prefer that the files be moved (or just left where they are) instead of being copied. However, the feature as it is now is still immensely useful. Thanks, quicksketch.

@MrKatz, @Garrett Albright
When it comes to your feature request, it was briefly discussed earlier.
See last paragraph here http://drupal.org/node/438940#comment-2755528
I don’t know what quicksketch thoughts are about this, my focus up to now has been getting it to a point that quicksketch would consider it for inclusion. Based on #156 he is still working through it.

subscribing

Hello,
the problem is not with FileField Sources but with imce and I make a solution for this problem but I wait from ufku the owner for imce if its ok or not.

I'm working on this patch again, seems there are some large issues still at hand here including:

- Lots of cruft code left over from copying from the remote.inc file.
- Inefficiencies with copying files multiple times when attaching to a node.
- Security problems because *ALL* validation has been removed, allowing attachment of files that shouldn't be allowed. This could lead to unexpected attachment and execution of PHP code.

What I would expect (maybe as optional checkbox) is that the files are moved instead of copied.
At the moment I still have the already used files inside the attach selectbox, which is a little
confusing by over 1000 files. That's the reason I would prefer a checkbox in the filefield settings,
letting me move the files instead of copy them after they are attached to a node.

I also agree with this idea, though it'll be difficult to avoid copying the file at least once, since field_file_save_file() will always use file_copy(). Drupal's file_move() internally just does a file_copy() also, so unless we use straight-up PHP functions and duplicate a lot of code, we'll probably have to deal with copying the file and then deleting the original. Considering this may be necessary anyway for maximum compatibility (say you're moving files between disks or onto a network mount), I think we'll be okay with that approach. This should be an option as part of the configuration of the attach source.

I've committed the attached patch which finally adds this functionality to FileField Sources. Changes from the previous patch:

- Appearance cleanup in several areas, better markup and matching CSS.
- Added option for "move" or "copy" attached files.
- Slightly more efficient handling of files with only one file copying instead of two.
- Added transliteration of filenames to the remote.inc file to match the functionality in the attach.inc file.
- Improved language and consistency of help text.
- Fixed validation so that all normal validation routines are run, excluding file size checks.

Please try out the development version and we can get out a new release of the module later this week.

Patch for previous comment.

We ran into this problem after ftp'ing a few gigabytes worth of video files to a site and realizing that filefield_sources wouldn't recognize them without it being in the Files table so we developed a simple module to help us add those existing files to the DB that are on the server. Feel free to use it. You can download it here: http://features.centogram.com/node/8

I need to be able to use FTP to upload files used by Drupal via filefield sources / IMCE. In imce.inc I commented the code out that filters out files if they are not in the database. I try to use one of these files and I get this error obviously: "The selected file could not be used because the file does not exist in the database."

So there needs to be some sort of way to add these files to the database on-the-fly as they are selected for use in a node.

Any help or insight would be great! Thanks.

So I found a solution. In imce.inc in the filefield_sources module, instead of this:

  foreach ($directory['files'] as $filename => $file) { 
    if (!isset($db_files[$filename])) { 
			unset($directory['files'][$filename]);
			$directory['dirsize'] -= $file['size']; 
    } 
  }
  return $directory;
}

put this:

  foreach ($directory['files'] as $filename => $file) { 
    if (!isset($db_files[$filename])) { 
			//unset($directory['files'][$filename]);
			//$directory['dirsize'] -= $file['size']; 
			$mime_type = file_get_mimetype($filename);
			$sql = "INSERT INTO files(uid, filename, filepath, filemime, filesize, status, timestamp)
			VALUES('1', '" . $filename . "', '" . $sql_dir_name. "/" . $filename . "', '" . $mime_type . "', '" . $file["size"] . "', '1', '" . $file["date"] . "')";
			//echo $sql . "<br><br>";
			db_query($sql);
    } 
  }
  return $directory;
}

So this basically creates a database entry for new files that are not in the database. So now when I click insert it inserts the image, although, whenever I go to save my node I get this error: "Referencing to the file used in the field is not allowed."

Does anyone know what is wrong here?

Thanks.

Ok I found a solution for that as well so it does not throw the error. In the filefield module, I just commented out this block of code and it works like a charm.

if ($references['filefield'] == 0) {
          form_error($element, t('Referencing to the file used in the %field field is not allowed.', array('%field' => $element['#title'])));
}

I can't find that piece of code anywhere. You say it's in the filefield.module file?

@tacoparty, I found it in filefield > filefield_widget.inc. Comment out

        if (field_file_references($file) == 0) {
          form_error($element, t('Referencing to the file used in the %field field is not allowed.', array('%field' => $element['#title'])));
        }

Is there any consideration of these ideas for D7?

I'm approaching this as a photographer with multiple files to upload and display on each per post. Individual uploading of a batch of files is just too time-consuming to be useful, so FTP has been the way to do it.

Filefield_sources is great on D6 for allowing me multiple ways to display images with just a single source file - something that most CMS's for the artsy types don't do. I just wish I could upgrade to D7!

@artproduct
As far as I know this feature was committed to filefield source and should be available in the 7.x version. The source option you should be looking for is called "Attach". I personally have not tried 7.x yet, but as far as I know quicksketch has ported all of the 6.x features to the 7.x version. If you are still using one of the patches in this issue I suggest you first look at migrating to the latest version of 6.x as some things might have changed from the patches here and then look to upgrade. You might need to reconfigure each of your fields you using filefield source.

Thanks for pointing that out, NaX. I'd been focused on using IMCE to see the images. The File Attach certainly gets the job done though the purely textual list is a bit unwieldy in my case. And its default state copying files makes things problematic.

I still wish IMCE could see all the FTP'd files but I'll work with this for now. D6 had that in my instance, but I don't see it in D7. I'd been using the tweaks in this thread (like #176) along with ones at http://drupal.org/node/877452#comment-4499144.

In my perfect world there'd be some module or script for adding directories to the file database so they can be used here with a preserved directory structure. I see everything through IMCE just fine in the user/File Browser, but they don't appear for the file field. The comments at 176 suggests it's neither filefield_sources or IMCE that's at work here, but the File module, now in core.

I have a D7 install up and running smoothly, I just don't have an efficient way of authoring content on it without IMCE and filefield_sources working together on everything.

@artproduct
I suggest you open a new feature request issue describing your problem and how you would like to see it solved as this issue is now closed.

Any body that is interested in FTP import into IMCE, not related to filefield source please see this issue.
http://drupal.org/node/1260538

@artproduct
I assume you would interested in this.

This issue is closed so any IMCE based discussions should now move to that issue or a new issue.