Image style previews don't show when using a private file system

Have a development 7.12 that Configuration>Media>File System is set with Default download method set to Private and was doing fine for a couple of weeks. Yesterday I edited a view and saved it and then all the images that were supposed to be served from the private directory were not showing and giving an access denied in the error log. Switching the Default download method to Public seemed to fix the problem just like the OP said.

I have another install that is very similar and the Default download set to private has yet to cause the issue.

@graper, Thanks for filing this issue. It ended some hours of frustration.

I'm also experiencing this but only on Taxonomy terms, changing the default download method to public doesn't resolve it.

Tested on a fresh Drupal 7.10 installation and 7.12

This is also reported in the log just after the access denied is reported:

Warning: Cannot modify header information - headers already sent by (output started at /drupal/common.inc:2575) in drupal_send_headers() (line 1225 of /drupal/includes/bootstrap.inc).

It's happening with a file field on a taxonomy bundle not a node, file fields on a node are working fine for me.

It's also happening with a file field on users.

Files added through a file field set to Private can only be downloaded by the admin. They should be accessible to authenticated users, but they are not - they get a 403.
Default dl method is Public.

Hmmm - seems to work fine now. Must have been a cache mess-up - switching between authorized and unauthorized users a lot.

I've same issue with files uploaded programmatically.

I create a file after an export, with

...

$file = file_save_data($fp, "private://filename.txt", FILE_EXISTS_REPLACE);

and return a correct file object (and the file exist in the directory), then I try to download the file via http://domain.tld/system/files/filename.txt with the admin user and recive access denied in the recent logs ...

I tried to put private directory inside and outside the sites/default/files/ dir but is the same, I'm sure that isn't a filesystem permissions problem's (tried chmod 777 ad apache user as owner).

someone can explain me why?

I think the problem might be in the function file_get_file_reference. My thought process:

• file_file_download uses file_get_file_references to find the entities that reference the file.
• file_get_file_references seems to be returning empty, even when thee are entries in the file_usage table. (In my case, the field is on the COMMENT entity)
• file_get_file_references uses an EntityFieldQuery to find entities that have the file in the reference.
• I think this EntityFieldQuery needs a "->addMetaData('account', user_load(1));" added to it to do an exhaustive search of all entities regardless of permissions.

For me, appending "->addMetaData('account', user_load(1))" at the end of the EntityQuery seemed to fix the issue... but I don't know the core well enough to know if this is a good idea, fix, or just a bad idea (security problem).

Here is my attempt at a patch that fixes this issue for me.

This is my first attempt to contribute, so be kind. I'll leave it up to the Drupal core and file module gurus to determine if this is really a needed fix or just a poor workaround.

The file_get_file_references function is documented as "Retrieves a list of references to a file". With a file field attached to the comment entity, this function was returning an empty set for users other than admin users.

My understanding is that this list of references is then used by the file module to determine if the user has access to any of the parent entities to which the file is attached. Its possible that the real problem is in "EntityFieldQuery", and that it is not properly finding that the user has access to the entity, so it is returning null.

Since the module does subsequent access checks on these references using 'file_download_access', I think that this patch allowing allowing the EntityFieldQuery to return all references regardless of security should be okay.

Comments welcome.

That patch concerns me, as it loads user 1 and somehow attaches it to the query. The query API is so confusing I have no idea what effect that has. I'm just concerned that it will show private files to a user who really shouldn't be seeing them.

That said, I added an image field to my user entities, and made the storage private. Now, users who upload an image cannot view it, they get access denied.

I can confirm that without the patch, file_get_file_references() returns an empty array. With the patch it seems to work. The file remains hidden from anonymous users. I haven't tested much beyond that. Very interested in a fix.

I made several attempts to debug this further. I want to see how that patch in #10 affects the queries sent to the database. I was stymied though. I'm unable to get a log of the database queries either with or without that patch applied. Drupal 7 is truly difficult to debug.

I consider this a pretty serious issue. To sum up my particular case: "Image fields on users using private storage cannot be seen by the user who uploads them." I suspect any file uploaded to any entity (except maybe nodes) can only be seen by administrators, if that field uses private storage.

The original post claims, "If the Drupal system is changed to public, the file is still private, but now downloads properly..." This is not the case for me. I have found no workaround, short of patch #10, to allow the user who uploaded an image to his account to see that image.

Works fine for me with the latest 7.x, added a private filed to user, uploaded a file and was able to view it.

There was an issue with EFQ in 7.14, make sure that you update to 7.15 before testing.

Yeah, I had been using 7.14. Upgrade to 7.15 fixes problem.

The fix in 7.15 is the same concept as the patch in #10. As far as I can tell, in 7.15 field.module does this to the query:
->addTag('DANGEROUS_ACCESS_CHECK_OPT_OUT')

Same issue.. I'm also using Storage API module though, don't know if that could affect.

Just wanted to say me too. D7.23, a config with private filesystem setup and the field set to private as well. I had uploaded a Default image to the Image field, which wasn't displaying. I had uploaded the image using the Content type editor's Browse/Upload widget and it appeared in the sites/default/files_private area as expected.

The images were "access denied" even logged in as admin user, and there was no obvious reason for it. Even the access permissions were setup appropriately (i.e. not relying on user/1 behaviour).

Switching the field to public files results in the deafult image being displayed properly.

...and the same here. Have all the very latest versions, permissions are set correctly and only Admin could see private files...

Jake-B's patch fixed the problem for me, and private files get 403 Forbidden for anonymous, so it still seems secure.

EntityFieldQuery returns empty for private files unless user_load(1) is used, or your logged in as Admin.

Bug in EntityFieldQuery?

I'm seeing the same behaviour as rivimey, default private image isn't displaying, is throwing 403 forbidden even for admin.

eta: on D7.24. Was happening for 7.23 also.

I can confirm it's a bug in 7.23, 7.24 but not in 7.15

Do not change versions please

This module solved it.
http://drupal.org/project/private_files_download_permission

I'm having the same issues and the private_files_download_permission didn't solve the problem.

It happens on term image field. Every time I upload an image and save the term. I've got no access (as admin too!) and I've got this notice message:

Notice: Array to string conversion in drupal_send_headers() (line 1221 of xxxx/includes/bootstrap.inc).

Permissions on private folder seems ok. Files are uploaded but the system/file module returns access denied.

These are the modules installed:

Checked available update data for PayPal Payments Advanced and Payflow Link.                                 [ok]
Checked available update data for Commerce PayLeap.                                                          [ok]
Checked available update data for Commerce Search API.                                                       [ok]
Checked available update data for Shipping.                                                                  [ok]
Checked available update data for Commerce Guys Marketplace.                                                 [ok]
Checked available update data for Connector.                                                                 [ok]
Checked available update data for Countries.                                                                 [ok]
Checked available update data for Crumbs.                                                                    [ok]
Checked available update data for Chaos tools.                                                               [ok]
Checked available update data for Current Search Blocks.                                                     [ok]
Checked available update data for Date.                                                                      [ok]
Checked available update data for Entity API.                                                                [ok]
Checked available update data for Entity Reference.                                                          [ok]
Checked available update data for Eva.                                                                       [ok]
Checked available update data for Features.                                                                  [ok]
Checked available update data for Fences.                                                                    [ok]
Checked available update data for Field Extractor.                                                           [ok]
Checked available update data for Http Client.                                                               [ok]
Checked available update data for Image Delta Formatter.                                                     [ok]
Checked available update data for Inline Conditions.                                                         [ok]
Checked available update data for Inline Entity Form.                                                        [ok]
Checked available update data for Libraries.                                                                 [ok]
Checked available update data for Link.                                                                      [ok]
Checked available update data for Mail System.                                                               [ok]
Checked available update data for Menu attributes.                                                           [ok]
Checked available update data for Message.                                                                   [ok]
Checked available update data for Message notify.                                                            [ok]
Checked available update data for Migrate.                                                                   [ok]
Checked available update data for Migrate Extras.                                                            [ok]
Checked available update data for Mime Mail.                                                                 [ok]
Checked available update data for Module filter.                                                             [ok]
Checked available update data for OAuth2.                                                                    [ok]
Checked available update data for OAuth.                                                                     [ok]
Checked available update data for Pathauto.                                                                  [ok]
Checked available update data for Rules.                                                                     [ok]
Checked available update data for Search API.                                                                [ok]
Checked available update data for Database search.                                                           [ok]
Checked available update data for Search ranges.                                                             [ok]
Checked available update data for Search sorts.                                                              [ok]
Checked available update data for Service links.                                                             [ok]
Checked available update data for Special menu items.                                                        [ok]
Checked available update data for Strongarm.                                                                 [ok]
Checked available update data for Taxonomy menu.                                                             [ok]
Checked available update data for Title.                                                                     [ok]
Checked available update data for Token.                                                                     [ok]
Checked available update data for Views.                                                                     [ok]
Checked available update data for Views Bulk Operations.                                                     [ok]
Checked available update data for Views megarow.                                                             [ok]
Checked available update data for Alpha.                                                                     [ok]
Checked available update data for Omega Kickstart.                                                           [ok]
Checked available update data for Shiny.                                                                     [ok]
Checked available update data for String Overrides.                                                          [ok]
Checked available update data for Address Field.                                                             [ok]
Checked available update data for Advanced help.                                                             [ok]
Checked available update data for Block.                                                                     [ok]
Checked available update data for Chosen.                                                                    [ok]
Checked available update data for Cloud Zoom.                                                                [ok]
Checked available update data for Commerce.                                                                  [ok]
Checked available update data for Commerce add to cart confirmation.                                         [ok]
Checked available update data for Commerce Addressbook.                                                      [ok]
Checked available update data for Commerce American Express.                                                 [ok]
Checked available update data for Omega Tools.                                                               [ok]
Checked available update data for Commerce AutoSKU.                                                          [ok]
Checked available update data for Commerce Backoffice.                                                       [ok]
Checked available update data for Commerce Checkout Progress.                                                [ok]
Checked available update data for Commerce Checkout Redirect.                                                [ok]
Checked available update data for Commerce Discount.                                                         [ok]
Checked available update data for Commerce Extra Price Formatters.                                           [ok]
Checked available update data for Private files download permission.                                         [ok]
Checked available update data for 69 projects.                                                               [status]
Update information last refreshed: Tue, 04/15/2014 - 09:47

Update status information on all installed and enabled Drupal projects:
 Name                                     Installed    Proposed     Status                                             
                                          version      version                                                         
 Drupal                                   7.26         7.26         Up to date                                         
 Commerce Kickstart (commerce_kickstart)  7.x-2.13     7.x-2.13     Up to date                                         
 Omega Tools (omega_tools)                7.x-3.0-rc4  7.x-3.0-rc4  Up to date                                         
 Private files download permission        7.x-2.3      7.x-2.3      Up to date                                         
 (private_files_download_permission)                                                                                   
 String Overrides (stringoverrides)       7.x-1.8      7.x-1.8      Up to date  

Thanks!

@surokeen, there is a dedicated issue for the 'term' image: #1327224: Access denied to taxonomy term image

Hi
I am getting access denied to access private files aswell, private_files_download_permission didn't solve the problem either. Logged in as Super admin/Administrator, still no luck.
Few Drupal settings:
> Drupal version 7.26
>admin/config/media/file-system
--private file system path: /var/www/sitename/sites/default/private
--Default download method set to : Private local files served by Drupal.

>admin/config/media/private-files
--folder_name -> [Logged in Users] selected

>admin/config/media/private-files-download-permission
--Default was: /var/www/sitename/sites/default/private => Added "/private_folder_name" in the box
/private_folder_name -> enabled the user and administrator role
--Folder permission set to 775 recursively

>.htaccess within the private folder:
SetHandler Drupal_Security_Do_Not_Remove_See_SA_2006_006
Deny from all
Options None
Options +FollowSymLink

>Private file URL, eg.: http://sitename/system/files/private-folder-name/filename.pdf

Administrator has full permissions to the site. Still i get 'Access denied' when try to view/download any private files.

Any ideas? Thanks for your time in advance.

is there any new progresses here on #27 ? I have the same issue. When images files are uploaded to a Private Directory, they were prohibited to anonymous (and authentic user) access. Someone suggest that the restriction to images might be caused by an access control module

I can reproduce the original issue on drupal-7.32 I chose "other" as the Component since I don't yet know what core module ImageCache became.

The default image style "large" at /admin/config/media/image-styles/edit/large doesn't seem to even generate.

Both of these return a Access denied. The second one if not even real.
http://localhost/system/files/styles/flex_custom/private/modules/image/s...
or
http://localhost/system/files/styles/flex_custom/private/modules/image/n...

Do change version number please. Stuff is always fixed in dev first and then goes into a release

As per #24 or #25, the private_files_download_permission did solve this for me, but only if

• I used a folder setting of "/" (You can't enter an empty directory name)
• I allowed authenticated users access to this directory

Any other subdirectory kept returning permission denied for authenticated users.
.. Is an acceptable workaround for me.

Confirmed this is broken on 7.32 as well using simplytest.me and op's steps to repeat.

It's still broken in 7.34 and 7.x-dev too. Here's a patch against 7.x-dev to fix the image styles previews at admin/config/media/image-styles.

As for accessing other image styles with the direct URL in to the private file system, this will only work if there's the correct derivative token query. The image download (and therefore the hook_file_download() within image.module) won't actually happen unless this image derivative token is present, as checked for in image_style_deliver():

// Check that the style is defined, the scheme is valid, and the image
  // derivative token is valid. (Sites which require image derivatives to be
  // generated without a token can set the 'image_allow_insecure_derivatives'
  // variable to TRUE to bypass the latter check, but this will increase the
  // site's vulnerability to denial-of-service attacks. To prevent this
  // variable from leaving the site vulnerable to the most serious attacks, a
  // token is always required when a derivative of a derivative is requested.)
  $valid = !empty($style) && file_stream_wrapper_valid_scheme($scheme);
  if (!variable_get('image_allow_insecure_derivatives', FALSE) || strpos(ltrim($target, '\/'), 'styles/') === 0) {
    $valid = $valid && isset($_GET[IMAGE_DERIVATIVE_TOKEN]) && $_GET[IMAGE_DERIVATIVE_TOKEN] === image_style_path_token($style['name'], $scheme . '://' . $target);
  }

The patch does the following:

• Adds the image derivative token in to the url for the preview image.
• Lets image_file_download() deal with the sample image (which wouldn't have had its access granted by any modules due to being outside of the usual private file folder).

Just realised I didn't update the links ('view actual size' and clicking on the image) to use the token. New patch attached.

Hi,
I'm jumping in cause I'm facing a similar problem with Drupal 7.34.

1) As admin, I cannot see (got 403) the generated preview within the image styles admin page. Premissions on the file system are set correctly and the images are generated. I just got Forbidden when apache serve them.
=> patch in #35 corrected this.

2) As authenticated user, I cannot see (403 errors) user profile pictures (style is default "Thumbnail"), while as admin I have no problem. Again, permissions on the server are OK.

My private file system is outside the root.

This is something that is still important and still being overlooked. It has plagued me in development through the entire D7 experience. If D8 is released before this is fixed it'll never get fixed.

Why is user1 being denied seeing any image? I'm going to try the patch in #35. This error makes using a private file system truly damaged and at times hinders productivity (can't see image previews to choose a background image when using dynamic_background for instance) for admins and end users.

Patch in #35 did not clear up my issues, nor flushing cache multiple times, nor running cron multiple times. I have one site that won't show any file previews and another that does in certain areas (like dynamic_background on a node, while image styles preview images still do not show up).

I can reproduce the reported issue "Image style previews doen't show when using a private file system" on a vanilla Drupal 7.x-dev installation.

However, the root cause for access denied on the style previews, may or may not be related to access issues with other files (avatars, taxonomy terms). For example, avatars not showing are likely due to an issue in Media browser plus #2595131: "File (Field) Paths" module broken by "Media Browser Plus" 7.x-3.0-beta3.

Please keep the issue focus on the style previews and open additional issues for other files.

Image style samples do not show up when private files are enabled because:

1. The preview has no itok, and image_style_deliver checks the itok before checking if the file exists in the scheme == private clause
2. There's no module supplying headers for the sample image via hook_file_download, thus denying access

Hiding patch #10 because it is off-topic.

Bump. I have a simple Drupal 7 site...we can see private files, after login, but core image styles and avatars are broken.

I can still (or again?) reproduce this problem in 7.51-dev on Windows (which means it's not an OS file system issue):

1. Install vanilla D7.
2. Go to admin/config/media/file-system, set the Private file system path to be a sibling of the Public file system path, e.g. if the latter is xyz/files, then set it to xyz/files_private, and after saving, select Private local files served by Drupal.
3. Enable the core Image module.
4. Go to admin/config/media/image-styles/edit/thumbnail and you'll see that the right sample image is not showing.

The log says

access denied
http://example.com/system/files/styles/thumbnail/private/modules/image/s...

#35 makes the thumbnail show up. Thanks, LGLC!

I tested the patch #35 above and it didnt work for me, 7.50 as is
BUT
The code changes in the patch worked when i edited the files manually

Running Drupal 7.50, the patch from #35 worked well for me and fixed this, thanks!

Running Drupal 7.52, i have applied patch 35 by editing files manually and refresh cache but still getting access denied message and image is broken

Applied Patch #35 after changing code manually for 7.52 and there were 2 semi colons at the end of the line

+  $token = IMAGE_DERIVATIVE_TOKEN . '=' . image_style_path_token($style['name'], file_default_scheme() . '://' . $sample_image);;
+  $preview_url = file_create_url($preview_file) . '?cache_bypass=' . REQUEST_TIME . '&' . $token;

and an extra line after

+          // By not explicitly setting them here, this uses normal Drupal
+          // Expires, Cache-Control and ETag headers to prevent proxy or
+          // browser caching of private images.
+        );
+      }
+

So removed in the patch attached. And it is getting applied cleanly. Please review the patch

tajinder, many thanks for your contribution. I know it is a pain, but an interdiff.txt file would be helpful in this sort of situation (i.e. a patch that others have worked on and agreed is good that you are proposing a change for). It just makes it easier to spot (and thus easier to approve) the change you made.

An interdiff file is a diff between the original patch state and the new patch state (which is not quite the same as a 'diff' between the two patch files!. If you're using git, and your latest change is 1 commit, then 'git diff HEAD~1' does the trick. ... or perhaps just a 'git diff' if you committed the earlier patch and not your current change.

Hello rivimey,

I am attaching diff.txt file of two patches. I mentioned that patch #35 is working fine but it had an extra line after } and two semi colons at line 291 so i just removed both of these two points and created patch again.

The patch from #35 + changes from #50 together works. I tested with Drupal 7.53 and PHP 7.

I just patched 7.54 with #35 without any problems:

ue4net:/var/www/drupal/7/drupal-7.54 # patch -p1 <drupal-image-styles-1440312-35.patch
patching file modules/image/image.admin.inc
patching file modules/image/image.module
ue4net:/var/www/drupal/7/drupal-7.54 #

I see the two minor issues mentioned by tajinder.minhas, but the patch applies fine.

@salvis, can you explain why you chose #35 rather than #48, which would seem to be the obvious choice? Is there a problem with #48?

Patch applies cleanly and fixes problem.

Attached patch version just fixes minor white space issue and replaces tab with spaces.

Those random test failures are really frustrating...

Patch #35 with changes from #50 works good on drupal 7.54. No errors. I am on php7

@rivimey in #53:

@salvis, can you explain why you chose #35 rather than #48, which would seem to be the obvious choice? Is there a problem with #48?

No, there is no problem with #48 or #54, the three patches are functionally identical except for cosmetic style guide improvements. The people who weren't able to apply #35, won't be able to apply the later ones either, not because of the patch, but because of their own problems.

My reason for checking #35 again is to get those people to look into their patching process, hoping that they'll be able to apply #54 rather than down-voting it.

Ampersands need to be escaped as &. Is the preview URL correct when url rewrites are off?

Shouldn't the elseif clause also check whether the scheme used for the sample image matches the scheme on $uri?

changed & to &

Subscribe.

#61 works for me. Any idea when this will be implemented ?

It needs a review done.

Code looks ok to me, apart from the following:

+++ b/modules/image/image.admin.inc
@@ -821,11 +821,12 @@ function theme_image_style_preview($variables) {
+  $output .= '<a href="' . file_create_url($preview_file) . '?' . time() . '&amp;' . $token . '">' . theme('image', array('path' => $preview_url, 'alt' => t('Sample modified image'), 'title' => '', 'attributes' => $preview_attributes)) . '</a>';

Line is too long / too complex. Please refactor into components and keep lines under ~ 80 chars

I'm putting this back on RTBC.

Although the line @rivimey mentions is long and complex, it *was* long and complex to begin with, as are other lines in the same file. This issue is to fix a bug that breaks functionality, not to clean up only part of the code.

I agree with @cafuego. the 80-char limit is for comments, not for code.

#61 doesn't work with encrypted files.

With #61 I'm seeing

<a href="https://example.com/system/files/styles/medium/private/modules/image/sample.png?1532514739&amp;amp;itok=pBiRzxR1">
                      ^^^^^^^^^

check_plain() already does the escaping.

However, something must have changed in core at some point since 7.54, because even the prior patches have stopped working for me. I'm not getting access denied (as user 1), but the sample derived image is not showing.

I've tested this with Drupal core with or without file_entity and ctools
Drupal core vanilla 7.39 , private thumnails worked
I used sites/default/files/private for a private folder (this is ok by the way)

repeated test with and without the patch on 7.59 and 7.x dev , still works.

I was unable to reproduce the issue, style previews work for me in the node edit screen, the thumbnail 100x100
and I checked the image path, it was generated through system not /sites/files

This is working as designed as far as I am concerned.

Updating issue status accordingly.

I suggest you review your server filesystem permissions, the web server needs to be able to read and write to the private folder, preferably as private folder owner.

In my tests I used sites/default/files/private as the private folder. Drupal will automatically put an .htaccess file and limit read access to this so web users on an apache server will not be able to read this folder.

If you're using nginx or IIS , or some other web server you may have to lock down the permissions , create your own .htaccess equivalent (the file is .htaccess in private)

Thank you for the leads. I'm on Apache, and I'll investigate this further, but for now can we agree that the line

+  $output .= check_plain($style['label']) . ' (' . l(t('view actual size'), file_create_url($preview_file) . '?' . time() . '&' . $token) . ')';

in #61 will deliver HTML with a double-escaped ampersand on the admin/config/media/image-styles/edit/medium page?

@joseph.olstad

I was unable to reproduce the issue

could you please test with the steps at #1438940-14: Private stored default images give access denied ?