I've done a complete re-write of AmazonS3, with maximal performance in mind.

The way that my re-write differs from the existing AmazonS3 module is that relies entirely on the filesystem cache for metadata purposes.

For example, let's say that Drupal core calls file_exists('blahblah.jpg'), and blahblah.jpg doesn't exist. The current version of AmazonS3StreamWrapper checks the cache, sees that the file isn't cached, and then calls out to S3 to ask if the file exists there, and returns S3's response. My re-written AmazonS3StreamWrapper checks the cache, sees that the file isn't cached, and returns false.

This, of course, means that the cache needs to be populated in a totally different way than the existing module does it, which is why I created the "Refresh file metadata cache" button (and drush command).

Performance is vastly improved because the metadata for every file is stored locally, so calls for metadata need only query the database, rather than querying S3. S3 is only called when an operation needs to read or write actual data inside a file, or needs to move or delete that file.

I made a huge mistake while designing the folder system, which I've fixed in the attached version. I had mistakenly assumed that it was important to ensure that the database doesn't get bloated with leftover empty folders after the files inside them get deleted. So I wrote a folder refresh function which cleaned out all the folders in the cache and rebuilt them based on which files exist. This meant that empty folders got deleted, which, in retrospect, was really dumb.

I've gotten rid of the folder refresh function, and re-written the file refresh function to leave folders intact. That function also no longer ignores the "folder" objects in S3 (files with a / on the end), since they may be the only representation of existing empty folders for users who are updating from the original version of AmazonS3

HAve you had a chance to take a look at this, yet? My team's been using it, and have given lots of praise for the performance increase. I'm sure other users of AmazonS3 would be interested.

Hey all, sorry for the late reply. torgos posted his patch request on the day that I left for a week-long hike in Yosemite.

Here's the patch for my re-write, which I made against the latest dev build of AmazonS3. I had originally posted a zip, rather than a patch, because every function is different (or removed, or a new function). This patch is nearly twice the size of the original codebase.

Hi again! I've got some more updates for my re-write of AmazonS3. There is one significant change: the amazons3_file table's collation has been changed to utf8_bin.

By default, MySQL doesn't support string keys with the same words but different capitalization (e.g. s3://panda.jpg cannot be a key alongside s3://PANDA.jpg). However, nearly every filesystem out there (including S3) supports having two separate files with names like this, and AmazonS3's sync code was choking because of that. I had originally overcome this problem (partially) with a slow, clunky workaround. However, I'd thought it was an OSX limitation, rather than a MySQL issue. But when I finally installed my updated AmazonS3 module into a linux machine, the problem persisted, so I realized that I needed to find a different solution.

Unfortunately, MySQL doesn't directly support case sensitive collation in UTF8. However, it is possible to make it collate by binary differences between characters, which is good enough for the amazons3_file table's needs. That's why the collation is now set to utf8_bin rather than the default, utf8_general_ci. I've implemented an update hook to fix this, and changed the hook_schema() implementation.

In addition, I made several comment changes, and added t() around all the remaining user-visible strings.

Attached is the zip file with AmaonzS3 v2.1, as well as a patch for v2.0 to v2.1, and v1.0-dev to v2.1.

Oh, sorry about that. I usually just use git, but I couldn't readily do so in this situation, and I wasn't aware that diff could create git-like patches. I've attached new patches, made with diff -u.

Please post this suggestion to this other issue queue, rather than appending this issue.

Also please be aware that changing the title of an existing issue is strongly discouraged unless you're the module's maintainer or the issue's original poster.

From reading the description of how ImageCache works, I believe your problem is an incompatibility between how AmazonS3 deals with image derivatives and how ImageCache does it.

I'll see if I can find a way to integrate with ImageCache to resolve this incompatibility, but I won't have a chance to look into this until next week at the very earliest.

I've decided to separate my re-written code into my own module, which I've called S3 FIle System. It's currently in sandbox mode, until I get enough time to finish the cleanup, but should be in a fully working state. It uses the same AWS SDK as AmazonS3, which is v1.x.

Hah, whoops. I thought my initial commit had the unmodified code, but it looks like it didn't.

I went ahead and finished the initial cleanup and rename for S3 File System, but I haven't had a chance to test it yet. If you run into any issues, please post them to the S3 File System issue queue. justafish doesn't need any more spam from me. :)

fixed Date iCal link