By esadot on

Hi,

I want to allow my users to upload very large files, 1G and up. Uploading of such gigantic files might be prone to break – Internet network failure, timeouts, delayed, you name it.

So what I’m looking for is a mechanism to resume file upload from the point the last attempt failed.

Does Drupal employ such mechanism?
Any know technology out there, that can be imported into Drupal?

Any input or pointing me in the right direction, would be appreciated.

Thanks in advance

⋅ Categories: Drupal 5.x

Comments

jasonwhat’s picture

jasonwhat commented

There are a few options for this, but straight drupal is not a good one. The best options are to have the uploads go through a thrid party like Amazon S3. I think some modules are being worked on for this type of thing. I'm also working with a company called Move Digital on Drupal integration. Their stand alone product has good upload and download tiimes.

litwol’s picture

litwol commented

I think this calls for a new module.

A module that will take a huge file. depending on settings will break that huge file into smaller chunks and store them in the server.

that module will give the user 2 options to download the files:
1) by parts (zip or rar format, or other. maybe make this module hotplugable with compression engines)
2) whole file

Sometimes something interesting appears on http://litwol.com

jasonwhat’s picture

jasonwhat commented

My feeling is that this is done well by several third-party services already, and quite cheaply. Let them do the heavy lifting and let Drupal create a node and make the user connection for each upload.

esadot’s picture

esadot commented

Sounds to me as a good plan. Who are those 3rd parties?

litwol’s picture

litwol commented

yes i agree with you completely, but at the same time i would like to see "other" options available to me.

for example if i am running a multi million visitor site and i am running my drupal site on some 10+ server and some of my servers are dedicated specifically for file upload/download then i already have that "3rd" party functionality inside my company.

so again i have to ask some one to push through my suggestion above :-D

Sometimes something interesting appears on http://litwol.com

riemann’s picture

riemann commented

I've looked into that issue a while ago, even though I did not have time to come up with a new module.

These two open source Flash upload tools seemed very useful (I think that's the kind of thing that youtube uses). From experience big uploads turned out much more reliable than through the current upload module:

http://swfupload.mammon.se/

http://www.asfusion.com/blog/entry/file-upload-with-coldfusion-flash-forms (this one uses ColdFusion but maybe it can be ported simply to Flash/PHP).

It would be great if somebody could wrap a module around these upload tools or make them part of the upload module. I'd be willing to help if somebody wants to take it on.

riemann

doc2@drupalfr.org’s picture

doc2@drupalfr.org commented

(For previous' post broken link: http://swfupload.org is the new site. And, about ColdFusion)

I've had the need for DVD diskimage files... I haven't tried the suggestion above yet, but I recently found a workaround which may suit my needs:

I use the video.module in its latest 2007 version, patched a few times. I needed to attached to my video nodes playing some low quality MP4 the diskimages of the original DVD version. As for protecting those copyrighted files, I needed to manage access rights to those files independently from the node.

I first used the filefield.module to be able, thanks to the CCK_field_permissions.module, to manage access rights of files "independently" from my nodes. This works and is implemented on other nodetypes. But the issue #104220: file.inc: file_upload_max_size() silently cuts size in half and the physical limitiations of servers brought me to look for something else.

My workaround will be to attach my diskimage files as video nodes as well. Indeed, the video.module (2007) allows to insert the filepath of a file already stored at its right server's place (if there's no access to the server, this can be done thanks to the suggestions above; FTP, Flash, ThirdParty...). Using the filepath feature of the video.module (2007) avoids the upload limits, and still stores the file data within the database (edit: "server-side detection of dropped files" as Dan says on 29/04/08). The file inherits the node's access rights from drupal core, which is perfect.
To do so, I specified in the video.module and the server's MIME TYPEs that my diskimage files had to be handled as "application/octet-stream", so that nothing would be displayed in the play page of the video node (the node being just a shell to hold the file), but the download would start when getting to the dowload page (instead of displaying the file as txt, which is Safari's default for unrecognised files).
- To tune this up I'll need to created a nodereference field whitin my main video node (the low quality video which is played inline) which will link to the diskimage nodes (the video "shell" nodes). Hopefully, I'll manage to get the referenced nodes pointing directly to the download page in order to mimic the attached files mechanism.
- I'll have to hide these diskimage nodes from my views. But this won't be a problem (through filters such as taxonomy or publishing option).

Hope this helps.

dman’s picture

dman commented

HTTP push from the browser was never designed for huge files, and really can't support it very well at all. No resume, no integrity check, many points where timeouts can break it all.
You need a different protocol, like, um , FTP, and possibly a server-side detection of dropped files. The use of FTP-uploaded files withing Drupal has been on the wishlist for ages, and there's a dozen approaches to it. Dunno what's for you however.

.dan.
How to troubleshoot Drupal | http://www.coders.co.nz/

.dan. is the New Zealand Drupal Developer working on Government Web Standards