Image previews fail to display with private files directory outside of web root

I have exactly the same frustrating issue.

I can create a node and upload an image to the private area.

I can see the image in the filepath, but if I try to view the image through the node it cannot load it.

Thanks
Stephen

same problem here

This worked for me:

http://drupal.org/node/308201#comment-1040035

I'm seeing this problem still in the latest 6.x-3.x code.

Uploading causes the form to simply refresh with no file upload attached. Changing to public files causes everything to work smashingly.

Version: 6.x-3.0-alpha2
Drupal: 6.6

Same problem. If I change to a public files system the thumb show on the form page, but if I use a private files system, I get no thumb.

The image seems to display correctly using imagecache though even if I use a private file system.

Some more info from tonight's debugging session:

When private files are enabled, then all file download go through the system/files's page callback, the file_download() function. This function only initiates the transfer if one of the modules that implements hook_file_download() returns some headers to explain how to display the file.

As merc mentions, there's a bug in filefield_file_download() in that it's only doing a check to explicitly see if the field is a 'filefield' type. In the case of imagefield, it's actually type 'image', so the logic says "Ok, nothing to do here" and returns NULL, which results in file_download() returning a 404.

I don't believe merc's fix is correct though, as that'll make it work for images and not for files. We could put in some || logic (if it's a type filefield OR it's a type image) but then that gets sloppy; I imagine modules like audio, embedded media field, etc. would also like to build off of filefields in the same way imagefield does.

So we need to basically say on that line is "If this field uses Filefield as its base, then do stuff." rather than base it off the type.

Ok, since #1 is a Filefield problem, posted a patch over here: #291383: hook_file_download() causes 404 for filefield-derived fields on private download

Going to dig into this "mysterious lack of image previews" problem this week.

More specific title. The images seem to show up fine with private files, it's the combination of private files + a files directory outside of web root that seems to cause the problem.

Some information from tonight's debugging session:

These images are being displayed via the function theme_imagefield_admin_thumbnail(), which looks like this:

function theme_imagefield_admin_thumbnail($item = null) {
  if (is_null($item) || empty($item['filepath'])) {
    return '<!-- link to default admin thumb -->';
  }
  $thumb_path = imagefield_file_admin_thumb_path($item);
  return '<img src="'. file_create_url($thumb_path) .'" />';
}

The problem is that $thumb_path gets returned as '/Users/webchick/private/path_to_file/image.jpg'. file_create_url(), however, only wants 'path_to_file/image.jpg' in order to do its thing, which is remap it to 'system/files/path_to_file/image.jpg'.

I then turned to imagefield_file_admin_thumb_path() to see if there was a way there to make it smarter about returning only partial paths for private files. However, the $file object found there is missing the critical "path_to_file" piece of information. $file->filepath contains the full path to the file (which is what you'd expect, actually).

I might be able to infer the path_to_file part (which comes from Imagefield's widget settings) and then dynamically build the URL from there, but at least this gets us in the ballpark of why this isn't working properly.

Hehe. Allow me to change my mind completely. ;) This must be why it takes 5+ hours to debug this problem. ;)

file_create_url() should indeed deal with the situation where an absolute filepath is passed in. So that was a dead end.

New theory! I noticed that even going to http://localhost/drupal/system/files/path_to_file/image.jpg directly resulted in a 404. We know from debugging the previous issue that this means that no hook_file_download() took responsibility for this file, and therefore file_download() defaults to 404.

In filefield_file_download(), there is this snippet:

      $result = db_query("SELECT ". implode(', ', $columns) ."
                          FROM {". $table ."}
                          WHERE ". $fid_column ." = %d", $file->fid);

      while ($content = db_fetch_array($result)) {
        $content['field'] = $field;
        $cck_files[$field['field_name']][$content['vid']] = $content;
      }
...
  // If no filefield item is involved with this file, we don't care about it.
  if (empty($cck_files)) {
    return;
  }

At preview time, there is a record for this file in the files table, but NOT in the content_type_foo table. This explains the 404 on initial insert -- we're returning a NULL value and passing the buck to some other module to take care of. However, it doesn't explain why it continues to not work on edit after the image is already uploaded. So, still investigating...

Okay. FOUND IT.

function filefield_file_download($file) {
  $file = file_create_path($file); ## $file ends up being '/Users/webchick/private/path_to_file/image.jpg.thumb.jpg'

  $result = db_query("SELECT * FROM {files} WHERE filepath = '%s'", $file);
  if (!$file = db_fetch_object($result)) { ## That filepath doesn't exist! only /Users/webchick/private/path_to_file/image.jpg. You LOSE.
    return; ## Enter 404.
  }
...

Now I am too exhausted to dig further, but this is exactly why this is happening. I'm at least 90% sure this time. :P

After explaining my woes to quicksketch, he came up with the following one-liner patch which fixes this on edit. w00t! Still need to figure out exactly what's happening on initial upload, and the fix should preferably be moved to somewhere in imagefield module since it's responsible for making those thumbnail images.

But, it's a start! :P

Didn't move this to FileField, even though this patch is against it, a) because most people will look in imagefield for this problem, and b) the end place of this code will likely be moved.

Hm. In looking at this more and talking to Senpai tonight, I think we are literally and utterly screwed for previews on new uploads because these files' file IDs will never be found in the {content_type_foo} tables. :( (see #13)

I had an idea to figure out the node the file was attached to, and see if that node was in 'preview' mode, and if so, bypass the logic and just print the headers. However, this won't work... the {files} table no longer stores the files' associated node id. :( Which it can't really, because until a new node is submitted it doesn't have one, so I guess that wouldn't help much anyway. And we have zero other context in hook_file_download() other than the file path. We can't even check arg() because we'll just get system/files/XXX, not node/X/edit.

I thought about saving a record to the content_type_foo table immediately, but this would have two problems:
a) We don't have a node NID/VID on node add, so there's nothing to store in that column.
b) Even on edit, the side-effect would be that the published node's image would change if it were edited, even if that edit was never ultimately submitted. Not acceptable.

Anyone else want to chime in here with some bright ideas? I'd really appreciate it.

Unassigning myself; I give up. :(

We (jvandyk, that is) solved this by implementing hook_file_download() in a custom module and checking to see if a file resides in the files table AND it's temporary AND it resides in one of the directories we configured via filepath's settings to store those images to (ads, etc.). I don't see any way of generalizing this to fix it in FileField module "proper" though.

The patch in #15 still solves this for edits anyway. Not sure how you feel about having imagefield-specific logic in filefield, but I don't really know where else to put it.

Moving over to the FileField issue queue.

Incidentally, this was basically the code, in case it's of interest to someone else:

 /**
+ * Implementation of hook_file_download().
+ */
+function custom_file_download($filepath) {
+  $filepath_parts = explode('/', $filepath);
+  $directory = $filepath_parts[0];
+  if (custom_check_file_directory($directory, $filepath) != -1) {
+      // Remove thumbnail extension prior to checking against the files table.
+    $filepath = preg_replace('/\.thumb\.[a-z]+$/i', '', $filepath);
+    $result = db_query("SELECT f.filename, f.filesize, f.filemime FROM {files} f WHERE f.filepath = '%s' AND status = 0", file_create_path($filepath));
+    if ($file = db_fetch_object($result)) {
+      return array(
+        'Content-Type: ' . $file->filemime,
+        'Content-Length: ' . $file->filesize,
+      );
+    }
+  }
+}

/me applauds webchick's ferocity. I think I can come up with a solution with your contribution...

fixed in imagefield head...

Don't shoot the messenger! ;)

This issue is not resolved correctly. The function hook_file_download() currently stands as this:

function imagefield_file_download($filepath) {
  // return headers for admin thumbnails if private files are enabled.
  $path = str_replace('.thumb.jpg', '', $filepath);
  $path = file_create_path($path);
  if (file_exists($path)) {
    return array(
      'Content-Type: image/jpeg',
      'Content-Length: ' . filesize($path)
    );
  }
}

This is problematic because it doesn't discriminate over which files to modify the headers for! So now, when trying to download a zip file over private download method, the browser thinks it's getting a .jpeg.

In the API documentation for hook_file_download(), it says we need to "check if the file is controlled by the current module"...

// Check if the file is controlled by the current module.
  if ($filemime = db_result(db_query("SELECT filemime FROM {fileupload} WHERE filepath = '%s'", file_create_path($filepath)))) {

One way to do this, I guess, would be to "control" all files that contain the string .thumb.jpg .. it's a bit hackish but it works. Any better solutions?

Patch attached.

Yeah, it *used* to be fixed, now it's back again.

Thanks, I'm renaming the thumbnails in #397578: Uncouple ImageField from FileField Custom Hooks, but this patch would be good to get in regardless. Thanks for accurately identifying the problem. I'll get to it right after the widget refactoring gets in.

Thanks Mark Theunissen, I have to agree it's less than optimal but certainly will get the job done. I've applied the attached patch which improves slightly on the approach and tightens security by checking permissions on the original file (which is checked by FileField).

I am still not getting the preview image to display when the file system is set to private.

I am running
Drupal 6.10
Content Construction Kit (CCK) 6.x-2.1
FileField 6.x-3.0-beta1
ImageAPI 6.x-1.5
ImageField 6.x-3.0-beta1

The preview image has the URL http://example.com/system/files/imagefield_thumbs/example.jpg which does not exist.

The image displays fine when viewing the full node. Let me know what other info I can provide.

With using the development version (shouldn't be too different than beta1), I can't reproduce this problem. I even trashed the entire "imagefield_thumbs" directory from my files directory to check that the thumbnails were regenerated automatically (they were).

What I did:
- Set my file path to /Users/nate/privatefiles and method to "private"
- Created a new image node that has an imagefield set up with all default settings, filepath is field is empty.
- Uploaded a new image, which was saved to /Users/nate/privatefiles/example.jpg. A thumbnail was created at /Users/nate/privatefiles/imagefield_thumbs/example.jpg the URL was http://example.com/system/files/imagefield_thumbs/example.jpg.
- Saved the node and viewed the image.
- Trashed the /Users/nate/privatefiles/imagefield_thumbs directory
- Edited the same node, and the thumbnail was regenerated at /Users/nate/privatefiles/imagefield_thumbs/example.jpg.

So everything is working as far as I can tell. It sounds like the URL to the image is correct on your server. What are the permissions on your files directory? Perhaps ImageField is unable to make the "imagefield_thumbs" directory on the server.

I'm stumped. Drupal has write permissions and the files are getting created. The permissions for the imagefield_thumbs directory and its images are the same as the imagecache directory and its images, and Imagecache is working fine.

I updated the modules and tried on a clean install, but the preview thumbnail does not display and I get a 404 when visiting http://example.com/system/files/imagefield_thumbs/example.jpg

I'm using
Content Construction Kit (CCK) 6.x-2.2
FileField 6.x-3.0-beta2
ImageAPI 6.x-1.5
ImageCache 6.x-2.0-beta8
ImageField 6.x-3.0-beta2

The preview thumbnail works when I switch the file system to public.

Any suggestions on what to try next?

jcruz, thanks for the extensive update. I don't know what I was doing in #29 that made it so this did work, because the logic is currently clearly flawed. I didn't update the private download function when I moved all the thumbnails in #307503: Move *.thumb.jpg files to a separate directory.

Could you try out this patch? I'm pretty sure it will fix (again) the preview problem.

#31 fixes it. Thanks!

Updating status to reviewed and tested.

Thanks for reviewing jcruz (and for your patience)! Committed.

Removed. For some reason I didn't have beta 3 in my updates.

Edit: better yet, now I installed beta 3 but the updates page lists beta 2 as the recommended version. Confused.