Migrate from standard File Downloads to S3 File Downloads

There is currently no automatic way to migrate all of the files so you would need to manually change the products to point to the s3 files.

Yeah, you may be able to hire somebody to write a migrate script if you can't do it yourself. If you are able to do it please contribute back to the module!

Good luck.

So that allows you to migrate over all of your file downloads from uc_files to uc_files_s3?

@mattmetzger - Yeah, I did notice that uc_file automatically wipes all its DB records if you disable the module, or even change the file download directory! That seemed like a dumb thing to do, so I made uc_file_s3 not do that.

Thanks for these sql queries. It's basically the same as copying the tables, but much quicker as it's through the CLI.

One note, I had to run these again (and luckily I had a backup of uc_product_features) as, when you first enable the module, you need to immediately run the uc_refresh_files() function - which happens when you either check a user's files (in the user edit form) or go to add a new file feature to a product. When you first do this, you do not have any "bid" fields filled up - these are bucket ids generated from this process. When the uc_file_prune_files() function fires, any files not existing in a bucket (identified by a bid in {uc_files_s3} gets deleted from the database. So, you need to make sure that table has the correct bid listed for each file.

I'm not actually sure a bid is even required - since it seems like you're only allowed to associate one bucket for s3 file downloads in Ubercart, unless this is preparation for a new feature, which would be killer. :)

SO: My point is, keep a backup of ALL of your tables when turning on and scraping your s3 buckets for the first time, as like me, you may need to run these sql insert queries again.

It looks like the bid is needed, since multiple buckets can be referenced from s3 file downloads. In fact, it's impossible to hide files from other buckets attached to the same S3 account. After setting up your S3 credentials (admin/settings/amazon-s3), browse to admin/store/products/files-s3, where you will see all of the buckets for this account. If you click on a bucket name, all of the files in that bucket will be shown.

After that, when you go to attach an S3 file as a feature to a product, files from all buckets will be available. I'm not sure if this was the intention, but it didn't work for my use case. (I had buckets for multiple sites in the same account, but the end user would be adding files. I did not want her to be able to attach files owned by other clients!) I ended up having to create another Amazon account with only one bucket, only for file downloads.

I'd like to second @torgosPizza in thanking @mattmetzger for the basic queries. Since I needed to do this in my development environment for testing and then do it again in production, I wrapped it up in a Drush script. I'm posting it in the hopes that it might be helpful to someone else needing to make this transition (and so that I can find it again if I have to do it for another site). For information about the bid addition, see my previous post.

/**
 * @file
 * Convert site from uc_file module to uc_file_s3.
 * This script must be run from Drush:
 *   drush -r /var/www/drupal6 -l example.com php-script ~/uc_file-convert.php
 */

// Make sure to correctly set the bucket ID (bid) by visiting admin/store/products/files-s3,
// then clicking on the bucket name you wish to use, then looking up the associated
// bid in the uc_file_s3_buckets table.
$bid = 1;

echo 'Copying table uc_files to uc_files_s3: ';
db_query("TRUNCATE TABLE {uc_files_s3}"); // Clear out anything that has been populated first.
db_query("INSERT INTO {uc_files_s3} (fid, bid, filename) SELECT fid, %d, filename FROM {uc_files}", $bid);
echo "done\n";

echo 'Copying table uc_file_products to uc_file_s3_products: ';
db_query("INSERT INTO {uc_file_s3_products}
  (fpid, pfid, fid, model, description, shippable, download_limit, address_limit, time_granularity, time_quantity)
  SELECT fpid, pfid, fid, model, description, shippable, download_limit, address_limit, time_granularity, time_quantity
  FROM {uc_file_products}");
echo "done\n";

echo 'Copying table uc_file_users to uc_file_s3_users: ';
db_query("INSERT INTO {uc_file_s3_users}
  (fuid, fid, uid, pfid, file_key, granted, expiration, accessed, addresses, download_limit, address_limit)
  SELECT fuid, fid, uid, pfid, file_key, granted, expiration, accessed, addresses, download_limit, address_limit
  FROM {uc_file_users}");
echo "done\n";

echo 'Changing uc_product_features to use file_s3 instead of file: ';
db_query("UPDATE {uc_product_features} SET fid='file_s3' WHERE fid='file'");
echo "done\n";

Great stuff, thanks!

I know this is old but I figured something out while working on this that might be helpful in the future.

I migrated a site and wanted to change the file downloads directory. Silly me thought it would be as easy as changing it within the settings within Ubercart. Like you mentioned, this removed all the records from the database.

Luckily, I restored the backup for uc_files, uc_file_products, uc_file_users, and uc_product_features.

THEN, I changed the file download location directly in the SQL dump from the old path to the new path within the "variables" table. I imported it, flushed the caches, and it works! Downloads are now working from a new directory and all the old information is back in the site :)

Now to try and migrate to Amazon S3 downloads...

Can someone PLEASE help me with this...

I seem to have everything set up right, and ran the SQL queries fine. Everything looks fine!

When I click a link under the S3 Files tab, the link looks like this: http://.s3.amazonaws.com/?AWSAccessKeyId=AKIAJCML4XEIHC2532QQ&Expires=13...

It's missing the bucketname at the beginnging. If I manually add the bucketname at the beginning, it still doesn't work though, so is something else going on?

EDIT: If I manually add the bucketname, I get an AWS error message like this:

This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
<StringToSignBytes>
47 45 54 0a 0a 0a 31 33 35 35 35 31 39 34 37 31 0a 2f 64 65 73 69 67 6e 70 61 6e 6f 70 6c 79 66 69 6c 65 73 2f
</StringToSignBytes>
<RequestId>08AEB6EFD4CEDCA0</RequestId>
<HostId>
YLq+qadJBZUfeuOTakKwdUSme1UmL/6+gZ0C3mOmB5ccHZTkH16oS4t0XLYk9Xqb
</HostId>
<SignatureProvided>M0pQPDjMYELMfsuogoDwt/J/2zA=</SignatureProvided>
<StringToSign>GET 1355519471 /designpanoplyfiles/</StringToSign>
<AWSAccessKeyId>AKIAJCML4XEIHC2532QQ</AWSAccessKeyId>
</Error>

The error itself reads "The request signature we calculated does not match the signature you provided. Check your key and signing method" which probably means you are using the wrong key/value pair for your signature. Check that your S3 settings are correct in admin/settings/amazon-s3/

They are correct, I double checked.

If they were incorrect I wouldn't even be able to see my buckets at all correct? I can see the buckets and all the files and everything just fine. It's just the file download links aren't working.

I have the bid column in uc_files_s3 set to 5, because that's the bid of the bucket where all my files are based on the information in the check uc_file_s3_buckets table.

#8 mentions "Ok, it actually worked, I had just not properly set up the s3 bucket yet."

Is there something special I have to do to my bucket regarding permissions or anything?

I have no clue where else to look, why would the links point to a URL without the bucketname in it at all? I thought my bid might be incorrect but it's not.

Oh, sorry. I think I misread your earlier comment. Adding the bucketname won't work because it is using the fully-resolved name (including domain) to generate a signature. So that is fine.

But I still think there's an issue with your JOINs. Check the watchdog for details.

I say this because not only is there no subdomain for amazonaws.com, but there's no filename either - it just has your token/signature params. Since there is no bucket name or filename showing up, my guess is that your columns don't match, but we won't know which ones unless it tells you in the logs.

Thanks, I see what you mean, no filename either.

The only things that show up in the logs when I try to download files are things that seemingly look like the download was completed.

Type uc_file_s3
Date Thursday, December 13, 2012 - 14:01
User usernamehere
Location http://www.domain.com/download-s3/273/28df434c76ba711782f3d36238b95e37
Referrer http://www.domain.com/user/1/purchased-files-s3
Message The user username has started download of the S3 file filename.zip.
Severity notice
Hostname ipaddresshere
Operations

Is there a way I can reset what I did and retry it? I thought I did by truncating the uc_files_s3, uc_file_s3_products, and uc_file_s3_users tables. Then rerunning the 4 MySQL commands in #4 (with 5 as my bid), but that didn't help.

Is there something else I'm missing to completely reset it and try again? All the downloads on the "Files" tab are still there, so all my original uc_file tables are intact still.

Is "5" still the correct bid for that bucket? I have the feeling there is a mismatch there with how it's configured (or at least how the module "discovered" your buckets). Double-check that the bid in the uc_file_s3_buckets table is the correct one, and that your entries in uc_files_s3 use the same bid. I'm fairly confident that's where the issue is.

The log you posted is just from the module logs, but I don't think it's actually indicative of a successful download, but then again if the correct filename is listed in that log entry, the issue might not quite be what I'm assuming it is.

The bid for the bucket I am using is definitely 5, and the bid for everything in uc_files_s3 is 5, so that is correct. I've attached screenshots so you can confirm.

I should have also mentioned when I said the logs showed a successful download...I know it wasn't successful because I was the one trying to download. It says it was, but it wasn't.

Is there anything else I can try? At the moment I am uploading 3GB worth of files to my server so I can take my site live. It would've been way easier if this worked because everything is already on S3.

I'm willing to send database dumps to you if you could find it in your heart to look at them (if it would even help).

By any chance are you using database prefixes in your settings.php? Or is this a multisite setup or something?

Nope. Just the usual regular database, no multisite at all. This really has me stumped :-/

I just tried fully uninstalling the module and then reactivating it. the "bid" was then 1, and I tried running the MySQL commands above with the default "bid" of 1 and it did the exact same thing.

Any other ideas? :-/

Hmmm and still no errors? Anything there related to mysql/database/etc. and not necessarily the module itself?

Yeah, there was no errors at all :-/ I tested it for days from multiple accounts. On the contrary, like I said, the logs were making it seem like everything was fine and dandy, with confirmation messages saying the file was downloaded.

I took the site live and get about 1,000 uniques a day, the only errors I get now are some page not found because I changed a ton of path aliases and am manually redirecting some as I find them. They don't matter anyways because I had to revert to regular UC File downloads to get things working in the meantime.

Is this just a lost cause? :-(

I don't think it's a lost cause. Do you work in an IDE that allows you to debug things locally? If I were you I'd make a local copy of the site and step through the process of downloading in your debugger. The logs saying a file was downloaded is just some kind of false positive but I won't know exactly why.. if you'd like to send me a database dump of your uc_file_s3 tables (there are 4-5 of them?) I could take look. I'd need your key/value pairs too, of course, but you could create one specifically for me and then revoke it when we're done. Or better yet if you wanted to give me access to the site (including command-line access) I could try to resolve it there.

I wish I had more time to actually help, though. (I just had some surgery and am recovering).

In the meantime I'll try to think of some other ways for you to debug without having to go through all that. Mainly if I were you, I'd try to replicate the SQL queries that are fired in the uc_file_s3 module's "download" and "transfer" functions. That should give us a starting point.

Thanks for the help so far.

I'll be honest, I'm not that good at debugging. Apart from what we've done so far, I have almost no experience doing anything more.

I built the site locally and use WAMP. Not sure if there are tools I can use there that might help? I didn't run the MySQL queries until after I took the site online, but I could try it on my local installation and see if I get the same thing, then I could try to debug with you from there.

I would be willing to give you site access if you would be willing to check it out. I messaged you just now with my private info. I'll definitely post the solution if we can figure it out.