Hi there, i modified some code from this module to support PHP sdk v2, currently i put it on github, it will be great if this project support support php SDK v2 officially.

Let me know if you or anyone need help using S3 over AWS PHP SDK V2

https://github.com/drucloud/drupal-awss3-v2

thanks
Keith

CommentFileSizeAuthor
#26 s3_db_file_managed.png62.64 KBcloudbull
#26 image_s3url_in_field.png86.79 KBcloudbull
#15 2176589.14-sdk2.patch86.35 KBdeviantintegral
#13 2176589.13-sdk2.patch85.37 KBdeviantintegral
#12 2176589.12-sdk2.patch10.46 KBdeviantintegral
#6 supportAWSS3v2-2176589-8474059.patch64.11 KBcloudbull

Comments

paulbeaney’s picture

paulbeaney commented

Hi keithyau,

Just trying out your module, but I haven't managed to get it working yet. The AWS SDK module can't find the library. It seems to look for a file sdk.class.php in /libraries/awssdk, but this file does not exist in the V2 SDK. I downloaded the SDK as a ZIP from the github site linked to from the official Amazon SDK home page: https://github.com/aws/aws-sdk-php.

Any ideas?

Regards,

- Paul

cloudbull’s picture

cloudbull commented

so did u install the awssdk v2 support module ?
You may find support here ~ https://drupal.org/node/1865568

thanks
Keith

paulbeaney’s picture

paulbeaney commented
Status: Active » Closed (fixed)

Hi Keith,

I did indeed install that module, but as it is I ended up getting round the problem altogether by using the S3fs module instead. This is working fine for me.

Regards,

- Paul

paulbeaney’s picture

paulbeaney commented
Status: Closed (fixed) » Needs review
justafish’s picture

justafish
she/her
Primary language English
Location London, UK
commented
Status: Needs review » Needs work

@keithyau Could you make a patch or link me to a pull request please?

cloudbull’s picture

cloudbull commented
StatusFileSize
new supportAWSS3v2-2176589-8474059.patch64.11 KB

Patch attached and thanks :)

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented
Issue tags: +Needs reroll

This patch looks like it has a number of unrelated formatting issues, and may actually not be a patch against the latest dev version.

On another note, the 2.7.x version of AWS SDK for PHP includes its own stream wrapper. I recommend we just subclass this and implement DrupalStreamWrapperInterface.

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented

Also, the SDK now uses composer, and requires guzzle. Maybe work on this should be a 2.x branch?

kim.pepper’s picture

kim.pepper
Primary language English
Location 🏄‍♂️🇦🇺Sydney, Australia
commented

Here's the link to the AWS stream wrapper implementation http://docs.aws.amazon.com/aws-sdk-php/latest/class-Aws.S3.StreamWrapper...

deviantintegral’s picture

deviantintegral commented
Assigned: Unassigned » deviantintegral
deviantintegral’s picture

deviantintegral commented

The last patch has a ton of style and code changes that look like they were automatically applied by an editor. At least some of them go against d.o code standards, so it's pretty difficult to review.

Rather than using Libraries, we should use composer (and yes, a new branch too). I'll be working on this today.

deviantintegral’s picture

deviantintegral commented
Version: 7.x-1.x-dev » 7.x-2.x-dev
Issue tags: -Needs reroll
StatusFileSize
new 2176589.12-sdk2.patch10.46 KB

Here's my WIP as a format-patch, which:

  • Pulls in the AWS SDK using composer.
  • Rips out libraries and the awssdk dependency.
  • Removes our version checks since composer handles that for us.
  • Uses the new SDK to validate API credentials on the config form.
  • Cleans up our validation to split it out from being dependent on the FAPI.
  • Removed the SSL checks because Guzzle now ships with a CA bundle.

Still TODO:

  • Updating the stream wrapper - the SDK now has createPseudoDirectory() so we should be able to use it.
  • Figure out how to handle credentials, since they are shared with other SDK services, but awssdk looks to not be maintained. I'm not even sure that module is needed if everyone just uses Composer.
deviantintegral’s picture

deviantintegral commented
StatusFileSize
new 2176589.13-sdk2.patch85.37 KB

Here's a patch that lets you upload, delete, and view files from S3, as long as the file field is configured to use the S3 bucket name as it's subdirectory. I think this is enough to commit just to keep things reviewable.

Here's a snipped summary of changes since the last patch:

  • 31cd23c Require PHP 5.4 due to PHP bug #40459.
  • 96e247c Use Guzzle\Collection for our config.
  • 54f9e04 Use the AWS SDK's stream wrapper.
  • ec33508 Remove unneeded $uri parameter.

I've filed a ton of new followup issues as well that I won't list here.

deviantintegral’s picture

deviantintegral commented
Status: Needs work » Needs review
deviantintegral’s picture

deviantintegral commented
StatusFileSize
new 2176589.14-sdk2.patch86.35 KB

Update that fixes a missing $this on a variable.

deviantintegral’s picture

deviantintegral commented
Status: Needs review » Fixed

Merged at 082169e.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

cloudbull’s picture

cloudbull commented

So this module is using as the SDK support ?

https://www.drupal.org/project/awssdk2

deviantintegral’s picture

deviantintegral commented

I looked at that, but ended up going with composer manager instead. Composer is used for phpunit support and also for the Drupal doctrine cache adapter, so I figured we may as wall reduce the dependencies.

If the awssdk2 module switches to composer, then it will easily coexist with this module. It might be fine, but I haven't tested it myself.

cloudbull’s picture

cloudbull commented

hi Devianintegral

I had uploaded an image from new amazons3 (aws sdk v2) module which is successful.

https://d3lxutzccbonnd.cloudfront.net/field/image/screen_shot_2015-01-12...

But all this images' images style cannot be generated. (NoSuchKey)

https://d3lxutzccbonnd.cloudfront.net/thumbnail/s3/coconuts4-testing/fie...

Before i upgrade the module from awssdk v1 to awssdk v2, the image path should looks like

xxxxxxxxxxxxx/styles/thumbnail/s3/field/image/xxxxxxxxxxxx
E.g.
http://cdn1.coconutsmedia.com/styles/thumbnail/s3/field/image/derka_0.pn...

If i manually change the url patten of my uploaded image path like
https://d3lxutzccbonnd.cloudfront.net/styles/thumbnail/s3/field/image/sc...

The image still not exists.

Isn't it /styles/thumbnail/s3/ -> /thumbnail/s3/coconuts4-testing/ cause the image style generation not functioning ?

** coconuts4-testing is my s3 bucket name

Please kindly advice and thanks :)

deviantintegral’s picture

deviantintegral commented

The first time you go to an image style, it has to be returned by Drupal. If you load a page with an image style, the initial URL should be http://my-drupal-site/amazons3/image-derivative/..... Once the style is uploaded, it should then use the S3 URL.

Also, did you patch image.module using the patch linked in README.md?

cloudbull’s picture

cloudbull commented

O... I did not patch the image.module, may be this is why ~

thanks, will try again.

Keith

cloudbull’s picture

cloudbull commented

remove ..

cloudbull’s picture

cloudbull commented

One more thing:

do the amazons3 v2 compatible with v1 ? I have a running site using v1, after upgrading to v2, for the old image fields,

the image path changed from cdn-yourdomain.cloudfront.com/field/image/xxxxx.jpg to cdn-yourdomain.cloudfront.com/image/xxxxx.jpg

while in the test, v1 and v2 using the same drupal db and s3 bucket. Just amazons3 and required patches are updated.
For new created nodes, I can see the images uploaded and displayed and images styles can be generated

Attached the uri in DB and the field display in node for better illustration.
s3_db_file_managed
image_s3url_in_field
please kindly advice.

Access denied log for existing images

access warning amazons3/image-derivative/field/styles/teasers_menu_v2/image/ghosthead.png
denied

While in settings.php i already do
$conf['image_allow_insecure_derivatives'] = TRUE;

Keith

cloudbull’s picture

cloudbull commented
StatusFileSize
new image_s3url_in_field.png86.79 KB
new s3_db_file_managed.png62.64 KB
deviantintegral’s picture

deviantintegral commented

You shouldn't need to flip the insecure derivatives setting - we do token validation when a derivative needs to be generated. Of course, you might need it for other reasons.

I just tested an upgrade with cloudfront, and I'm not seeing the same path issues. Is it possible that drush updatedb wasn't completely run? Update hook 7200 was modified in the beta3 release to add the bucket to all URLs, which would explain what you're seeing if 'field' is being used as the bucket name.

cloudbull’s picture

cloudbull commented

After copy 7200 code and run by drush scr, things work ! thanks a lot !
So, if i change to a new bucket name, i need to rerun 7200 ?

For the existing images, now 50% can be regenerate image styles, but the other 50% got this error:

Exception: Amazon S3 was unable to create an image style derivative. Check the temporary directory configuration and permissions. in amazons3_image_deliver() (line 139 of /var/www/html/sites/all/modules/contrib/amazons3/amazons3.module).

And looks like above cause by

The selected image handling toolkit gd can not correctly process image_gd_get_info.

Isnt it actually from PHP 5.5 ?

Do you have any idea ?

thanks
Keith

cloudbull’s picture

cloudbull commented

the new .install file will create variable like amazons3_presigned_urls = ''; which cause below error.

Warning: Invalid argument supplied for foreach() in Drupal\amazons3\StreamWrapperConfiguration::fromDrupalVariables() (line 391 of /var/www/html/sites/all/modules/contrib/amazons3/src/StreamWrapperConfiguration.php).
Recoverable fatal error: Argument 1 passed to Drupal\amazons3\Matchable\BasicPath::factory() must be of the type array, string given, called in /var/www/html/sites/all/modules/contrib/amazons3/src/StreamWrapperConfiguration.php on line 400 and defined in Drupal\amazons3\Matchable\BasicPath::factory() (line 31 of /var/www/html/sites/all/modules/contrib/amazons3/src/Matchable/BasicPath.php).

It require to drush vdel amazons3_presigned_urls to fix the error

deviantintegral’s picture

deviantintegral commented

So, if i change to a new bucket name, i need to rerun 7200 ?

Sort of? We now support multiple buckets, so nothing prevents you from keeping the old bucket around (saving you having to move the files) while uploading new files to the new bucket. But if for some reason you have to change buckets entirely, you'd have to write a (small) update hook in your custom code to update the files_managed table.

Exception: Amazon S3 was unable to create an image style derivative. Check the temporary directory configuration and permissions. in amazons3_image_deliver() (line 139 of /var/www/html/sites/all/modules/contrib/amazons3/amazons3.module).

Yeah... this turns out to be related to EXIF data dependent on the camera that took the image. It's a combination of a PHP issue and an AWS SDK issue. There's an issue over at Github and a pull request that could use some more testing.

I'm going to open a new issue for the amazons3_presigned_urls bug.