Include an admin setting for uploaded files to have Private ACL

I'm not an expert here but how might we get this private S3 files feature done? I'd love to hear the thoughts of the current maintainer on possible routes OR even if it's something we need to build as a separate module or submodule?

Thanks! Killer work on this module...

I'm not quite sure I get how this could be done. Can you suggest what you mean in a bit more detail so I can think through better how to implement this?

http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide...

And http://docs.amazonwebservices.com/AWSSDKforPHP/latest/#m=AmazonCloudFron...

also interested

Just curious if anyone has taken a stab at this yet?

S3 buckets are by default private while uploading (put a object on S3) you set the ACL and my guess is that this module create the object as public.
What you could do is create it private and when Drupal renders the link a signature is added that allows you to download / grep (GET the object from S3) for a limited period of time.

You could say if a user is allowed to see / access the node he should have access to the attachments as well. That is currently the use case i'm interested in i don't know if more people agree with that?

Another possibility is to use cloud front but that will require a extra investment and not always logical for al deployments (low traffic / local sites viewed in one region/country)

regards

I think I'm in the same boat as you Nr. 18.

To put it simply I'd like the files to be treated as "private" by the drupal file system, so that if a user has permission to view/access the file through normal Drupal private file permissions then they can access the file, otherwise they can't.

Specifically in our case I'd like it to work with Commerce File, so that I can upload our product files to S3 and still have them protected from public download.

I am also interested in the data are private. With CloudFront, it is not possible for me to keep the data private.

S3 would be the perfect product, version 6, this is possible. For me, the thought of range for the video modules so that only authorized members of video can be seen. Available and not through the direct URL of the browser.

Unfortunately, my English very bad, I hope you understand me .. :-)

Unfortunately I am not a coder on and I would be glad if this is possible soon.

I think I've got this partially working. All I had to do was change 2 things:

1st edit this line of code in AmazonS3StreamWrapper.inc

line 554 from:

'acl' => AmazonS3::ACL_PUBLIC,

to:

'acl' => AmazonS3::ACL_PRIVATE,

2nd I needed to setup the "Presigned URLs" in the UI, in my case it looks like this:

60|clips/*

Which I think means that it has a 60 second timeout for files uploaded into the "clips" subdirectory - but I'm no expert on how these presigned urls work, it just does work for me.

When I do that the downloads are protected. I can download files I've purchased via Commerce File but if I try to just cut and past the link in a browser it doesn't work.

Also, if I don't set the presigned urls it doesn't work, I get a permissions error.

The one aspect that doesn't work at the moment is "counted" downloads via commerce file. Commerce File usually keeps track of how many times a user has downloaded a file, so you can set limits, this part however doesn't work for me at the moment.

Ideally this PUBLIC vs PRIVATE ACL could be rolled into a toggle in the UI. Also the Presigned URLs should be marked as a "required" field if "private acl" is selected.

There are already some files approaches to deal with privately. am grateful every way. Great modules.

Well I guess this is already in the roadmap for this module considering the "function chmod($mode)" in the wrapper?

that would interest me very..

thanks

Also very interested in #7:

You could say if a user is allowed to see / access the node he should have access to the attachments as well. That is currently the use case i'm interested in i don't know if more people agree with that?

Following.

interested

All of the above issues seem to work fine for me, as long as I've setup Presigned URLs in AmazonS3. For instance to protect all of my files in my bucket with a 60-second lifetime I added this:

60|/*

And now all my URLs to my files are authenticated in the querystring, including files that are referenced in the node via a File field. (Note that I'm using Commerce File which is a special instantiation of the File field, but since AmazonS3 provides stream wrappers, I don't doubt that this should work for any files that have been uploaded to S3.)

the 60|/* did the trick for me as well and i am also using this function with Commerce_file.

The only problem i see Occurring is that after the allotted time, the user will receive the access denied error message from Amazon if he/she has not refreshed the webpage. It would be nice to, upon clicking the link, if time has expired, to have an error message stating "Your link has timed out. please refresh to generate a new link." This warning could always be posted on my site somewhere, not a necessary function just a luxury problem since it all works =-)

An intermediary download location might be an interesting idea. In Ubercart D6, the UC File S3 module created a path at "download-s3/fid/hash" (or similar) that acted as a redirect, allowing you to maintain file permissions, logging, etc. of the download. Something similar might be worthwhile here.

Also, sorry for changing the title of the issue; but I'm changing it back because I do think that the ACL needs to be set to Private as in comment #10. Without that change, files that are uploaded through the AmazonS3 module will be public.

So I'm changing the title back to something more descriptive, and will reiterate what #10 said. A new feature / patch should be added which does the following:

• Adds a setting for "Use a Private ACL on uploaded files"
• Returns an error if "Use a Private ACL" is selected but no "Presigned URLs" are entered into that config field.

We should reopen a new issue as a feature request for the ability to log downloads from S3 (such as through Commerce File) and to check the access control of a download link. In other words instead of providing the raw URL, a field formatter that gives an intermediate path (for instance example.com/download/s3/bucket/fid), which would allow us to check / alter any file permissions, and log it upon successful evaluation of said permissions.

While this is marked as a feature request, there's a fairly significant security issue here since, with ACL_PUBLIC, presigned URLs "leak" enough information to always download the file, even when the link has expired.

For instance, if a presigned URL path is set for:

30/securefiles/*

then the URL returned will be something like:

http://bucketname.s3.amazonaws.com/securefiles/filename.ext?AWSAccessKeyId=XXXXXXXXXXXXXXXX&Expires=1351643941&Signature=XXXXXXXXXXXXXXX

Which, if accessed after the 30 second timeout, will return "expired" instead of the file.

HOWEVER, because the file is set ACL_PUBLIC, it's all too easy to just remove query parameters, and use:

http://bucketname.s3.amazonaws.com/securefiles/filename.ext

This will, unfortunately, allow full access to the file.

Presigned URLs are fairly useless (and even dangerous if you don't understand that you aren't actually protecting the file) given the current implementation unless you follow the steps in comment #10. However a module upgrade will, of course, reset the default, and the administrator may not remember to edit it back.

Possible solutions, with pros and cons:

• torgosPizza suggestion, with modification:
   
  • Add a setting for "Use a Private ACL on uploaded files"
  • Return an error if "Use a Private ACL" is selected but no "Presigned URLs" are entered into that config field.
  • Return an error if "Presigned URLs" are entered but "Use a Private ACL" is not selected.

Pro(s): Straightforward implementation, simple to understand.
Con(s): Cannot have unsigned URLs if desired, because all files will be ACL_PRIVATE and require presigned URLs.

• Set files uploaded to presigned URL paths to ACL_PRIVATE and all others to ACL_PUBLIC
   
  Pro(s): Allows both presigned and public URLs.
  Con(s): Does not gracefully handle changes to presigned URLs list. If a presigned URL path is added, but files already exist in that path, then those files will still remain ACL_PUBLIC.

 

• Monitor changes to presigned URL paths and adjust ACLs (with permission from user) on files in those paths as needed. Files in presigned URL paths would be ACL_PRIVATE, all others would be ACL_PUBLIC.
   
  Pro(s): Seems most robust
  Con(s): I'm sure there are some. Most complex implementation of the three, certainly.

I have just put in a pull request for this feature at https://github.com/justafish/amazons3/pull/1

It adds a "public access" checkbox that switches the ACL for the stream_flush and copy functions.

Cheers,
Dean

#7 Sounds reasonably elegant for general use. Node access = file access.... but #19 hints at functionality that would allow ACL's to be set based on paths rather than at the bucket level (if I understand correct) - which would be particularly flexible.

I am sure this is not what is being looked for above and maybe is too simplistic but all I had done was to put a bucket policy in place that restricted access to a specific referer, my site.

I store videos in there and only wanted members to be able to access them. I am sure this would work similarly for other content types.

Just create a simple bucket policy which allows ONLY yourdomain.com and www.yourdomain.com as the referers. Worked a charm for me. Stops people from emailing each other direct links to my videos and ensures, I hope, that they can only be viewed when embedded in my site. I am sure there are ways to step around the referer but I am happy that 99.9% of my members wouldn't know how to do it.

Hope this helps.

+1 for ACL control optoins

A bucket policy just sounds like a nice security layer.

@pcoghlan Don't suppose you fancy posting that policy here do you....amazon policies fry my brain :)

This is a biggie for us and many people... We need a strictly private upload for some rather sensitive files, if they were made public I'd lose my job.

To summarise this thread and the solutions (especially lego in #19):

1. ACL works, but is hard coded to 'Public'. This makes setting the 'Presigned URLs' (per #10, #16, #19) redundant as it overrides any URL-level timed access grants.
2. A quick hack to lock down perms is to simply edit line 555 of AmazonS3StreamWrapper.inc (current dev version) to 'acl' => AmazonS3::ACL_PRIVATE per #10.
3. Ideally the above point would have an admin setting, and ideally on a per-field basis. Any setting should warn users to use Presigned URLs too when enganged.
4. In #20, lookaflyingdonkey suggested a nice pull request that solves points 2) and 3) above, though the setting is site-wide rather than per-field.
5. FWIW, in #22 it's suggested a bucket policy to allow access to a specific referers only. This is not actually secure though, since referred HTTP headers can easily be spoofed.
6. Lego's summary/suggestions in #19 are great, though I think if the Presigned URLs and Access control setting was per-field, then there would be no need to monitor too much -- if the setting is changed, just change the ACL on all files in the path the file field uses.

This is a big one for me, too. I think there could be a global acl setting as a default, but this really needs to be optional at the field level. Though we need private content, a private ACL on our public content would make this module unusable (i.e. we have PDFs and all sorts of images which are and need to be publicly available directly at their path...sharing something on social media and then having the link expire wouldn't be good).

Also, re: #6 in comment 25, you would have to update all of the files that file field uses. There is no guarantee that two fields don't use the same path (whether or not this is a good idea), so you can't just blanket update all files whose path matches that of the field whose ACL is changing.

Also, re: #6 in comment 25, you would have to update all of the files that file field uses. There is no guarantee that two fields don't use the same path (whether or not this is a good idea), so you can't just blanket update all files whose path matches that of the field whose ACL is changing.

Agreed. djake, I'd go further and say it's no business of this module to muck around with directory/file settings beyond the files directly managed by Drupal, so it should aim for the the simplest secure implementation possible. Once that's in, a few nice help messages when settings are changed can be shown to warn the the user to check the access on the folder (and files within) to ensure they're correct.

This is also something I am interested in. In my case I would need it for the Commerce File module. I would like to make all Commerce File uploads private and hosted on S3, but other uploaded files public.

I really only need all Commerce File's to be hosted on S3 (although I would like other files to be hosted there as well). So I might try to get this to work using some of the methods described above. I will post back my results.

Nothing seems to have happened with the pull request at #20 so here is a patch file I wrote (before I realised this had already been done).

If you've already applied the patch for this module to resolve the cloudfront signed urls issue here: https://drupal.org/node/1837920 then this patch won't apply (I can't figure out how to make it play nicely - I'm a bit new to patching etc......and I'm using Windows which doensn't help life).

Maybe I need to provide a patch file which includes both things?....anyone know?

Hello all, thank you for your work on this. It's just what I've been needing.

I tried to apply the patch in #29 [to 7.x-1.x-dev 2013-Sep-30], but got the following warning: Reversed (or previously applied) patch detected! Assume -R?

... so I left it unapplied and simply changed line 555 as described above.

Is this patch safe to apply, or will it make it's way into the next release?

Thanks.

@madrush - There doesn't seem to have been release for a long time and the pull request was made quite some time ago so I'd be surprised if it'll make it into the code any time soon.

This patch is actually for several files not just one so you should apply it to the directory not to a single file - that might be why you're seeing this error - but I'm not a patch expert, this was my first patch file. Or apply it to each file individually....or if you fancy it then you could look at the patch file to see what I actually changed and add those lines to your own code.

Its not doing anything that magic and I'd say its safe....but then I wrote it so I'm slightly impartial. All its doing is making this a check box option in the admin rather than a fixed behaviour

I thought I'd double check my patch after that last comment from madrush and I had some problems too so try this patch instead

In case it helps anyone, I have cleaned up the code from #32 so that it applies cleanly against the 7.x-1.0 release (as two patches; apologies, my software is being recalcitrant).

New patch for the private setting (I wasn't down with the 2 patch system in #34)

#35 worked for me. Thanks!

The patch in #35 was rolled relative to the web root, rather than the amazons3 folder:

diff --git a/sites/all/modules/amazons3/AmazonS3StreamWrapper.inc b/sites/all/modules/amazons3/AmazonS3StreamWrapper.inc
index b8c9925..64947fc 100644
--- a/sites/all/modules/amazons3/AmazonS3StreamWrapper.inc
+++ b/sites/all/modules/amazons3/AmazonS3StreamWrapper.inc

I have re-rolled.

This patch seems to do the same changes as this issue

https://www.drupal.org/node/2273973

@dwkitchen, yes, that issue combined the work on 2 separate issues into one patch. It was closed as a duplicate.

I don't think leaving the 'presigned URLs' field blank should trigger a validation issue.

In my case, I'm using s3 bucket policies to control access to files by referer, as per #22, so I don't want to use presigned URLs.

Perhaps a warning on form generation saying that some other access method must be employed?

Patch in #35 works for me. Checkbox checked, "Everyone" ACL is unset. Thanks!

Thx for the work on this. This is a very usefully feature.

I decided to extend this to provide the following:
- UI for 'amazons3_acl_object' - Select list of available ACL options minus 'ACL_OPEN' which no one should really use.
'amazons3_acl_timeout' - Default timeout for presigned URLs in seconds.
- protected class variables $acl & $aclS3. The first stores the constant name ("ACL_PUBLIC"), for easy comparison in code, and the second stores the constant value ("public-read") which is sent to Amazon.
- class functions to set and get ACL.
- construct sets default ACL from 'amazons3_acl_object'.
- getExternalUrl() - uses default 'amazons3_acl_object' and 'amazons3_acl_timeout'. Adds presigned querystring if ACL is not publicly readable (Not 'ACL_PUBLIC' or 'ACL_OPEN'). Sets values before hook_amazons3_url_info() is invoked so other modules can modify.

So now you can set the default ACL to ACL_PRIVATE and set a timeout and file URLs will automatically get presigned when required. No need to use "Presigned URLs" input. If you do specify presigned URLs the timeouts will override the default.

There's a disconnect here between how Drupal and S3 handle file permissions:

• Drupal assumes that permissions are handled at the stream wrapper level, and that permissions are entirely controlled through PHP code.
• S3 assigns permissions at the object level, and they can be changed from any S3 client. This means that it would be possible to have a "private" file in the public stream wrapper. We handled that now with the presigned URL requirement, but I agree that forcing the path-based configuration is a pain. It also means that URLs are signed for any user, without checking to see if a given Drupal user has permission to view the code.

If anyone wants to work on this today against the 7.x-2.x branch, feel free to ping me in IRC, otherwise I expect to be getting this done in the next day or so.

I could achieve private file handling by doing this:

function HOOK_amazons3_command_prepare_alter(&$name, &$args) {
  if ($name == 'PutObject') {
    $private_paths = array(
      'whatever',
    );

    if (!empty($args['Key'])) {
      foreach ($private_paths as $private_path) {
        if (!(strpos($args['Key'], $private_path) === FALSE)) {
          $args['ACL'] = 'private';
        }
      }
    }
  }
}

And set all $private_paths as Presigned URLs.

Has this solution any disadvantages?

@szantog, Disadvantage include:
- Manually maintain a list of paths in code
- Manually maintain two lists of paths (one for ACL and the other for presign)

@tinker you are absolutely right, it's not the winner of beauty code award (btw meanwhile $private_paths comes from variable) It wants to be a kind of workaround until the final solution. When I said 'disadvantage', I meant, based on security viewpoint. (I'm using field level access control)

Patch in #43 worked for me against a clean 7.x-1.1. However I got a fatal error about line 246 of amazons3.module, which seems to be related to PHP 5.4.x and the empty() function.

Changing it to the following fixed the fatal error thing:

  $fixthing = intval($form_state['values']['amazons3_acl_timeout']);
  if (empty($fixthing)) {

The existing patch files were for version 1 of the module, I ported and expanded the feature to version 7x-2.0.

Would be great if this could finally make it into the module...

Downgrading priority since this ticket is 6 years old and hasn't been worked on in about a year.