By dauphine13 on

We would like to use Drupal for a new beta site and after evaluating all the modules and functionality have decided there is only one thing that stands in our way.
We are posting software builds to 3rd party developers, beta testers and remote QA on a daily basis, some of these files reach close to 3GB. I have successfully used the default upload with files in the hundreds of MB range, but have had no success with Large files. I have tried some of the additional modules, but none are complete or consistently successful.

What I am looking for is someone who would be able to create/ update an upload module that worked on large(3GB) files. Upload Progress display is critical, bulk upload of attachments is desired, but not required.

I currently have 5.7 installed, but am not married to it. I used this version as it had the most selection of modules to evaluate. Some Modules required for the site are:
Organic groups
Email notifications
download_count
Comment upload
Tasklist

optional:
Wishlist
others

I am looking for someone who could first and foremost help solve the large file upload and download solution and then secondly set the structure of the site up by installing and configuring modules so the Product managers and QA can post, manage (upload/download) alert and interact with beta testers, as well as store information that is pertinent to their user, like contact info, nda, etc..

Please contact me at contentdev [at] smithmicro [dot] com
Jason-

Comments

peterx’s picture

peterx commented

3GB is a challenge. 1GB is easy. 2Gb is a limit in some operating systems and file systems because that is all you get from a 31 bit integer. You end up with 31 bit integers because some code treats 32 bit integers as signed instead of unsigned and that removes one bit. I would like to hear the result.

I do not use any file systems or browsers that have the 2GB limit because I upload and download files from 5 to 20GB. There are lots of editors that do not like opening 5 GB files. Anything written in Java chokes on 200 MB which makes 5000MB unlikely. When I upload a 20GB file, I use FTP because it can run quietly in the background and is not upset by browsers choking to death on poorly written Javascript.

An approach I used for one project was to use FTP from PHP. All my desktop workstations contain local applications written in PHP. They all have the latest PHP, MySQL, Apache installed. It is easy to whip up a page to use FTP. Using Drupal and the PHP filter, I can type the PHP FTP code direct into a page.

I can start FTP from a local desktop Web application and upload to or download from any server providing FTP access. FTP is happy with any file the file system can handle. You can use FTP to upload then use a Web service to tell the server to perform work on the uploaded file.

If the Web sites on your local PCs and on your servers are all built on Drupal 6 then you can use the services module in Drupal for the Web services communication.

Local PCs do not normally have fixed public IP addresses which means you can only open FTP to the server from your PC. You could use ADSL with a fixed IP address and make your filewall box assign fixed IP addresses to each internal PC. The firewall can connect FTP to a PC via an alternative port. You can then let your server open FTP to your PC.

You could also leave FTP out and create a Web service to send files in chunks with the recipient Web service appending new chunks to existing files.

petermoulding.com/web_architect

petermoulding.com/web_architect

dauphine13’s picture

dauphine13 commented

peterx,
So thank you, I follow most of what you are saying , but because we have some non-technical project managers, like myself that will be using this tool across multiple machines and sometimes on the road, I am not sure if the some of the later ideas related to fixed IPs and local servers if feasible. I really need this as easy as possible for them to use.

What all this has made me think about though is the possibility of having builds auto-ftp'd to a directory on the Drupal server and then the project manager would just have to attach the file to the project for posting and subsequently user notification and download.

Seems like it would be feasible, but I haven't looked into or ran across any modules that have this functionality yet... is this something that exists in a module now or would it be something that would have to be added to one or created?

I am then assuming that the link to the <3GB would then just be downloaded via the browser with no Drupal interaction or security risks.

Jason-

trailerparkopera’s picture

trailerparkopera commented

but are currently resolving it using SCP clients that have GUIs that work for unsophisticated users (e.g. Fugu), plus some backend scripts that move user uploaded files to predetermined locations on the server. It's ugly, but it works (for the low volume we're currently experiencing).

It would seem that a module that used SCP for upload, that updated a files table and returned a corresponding url for http download (with a decent progress bar for uploads) would be a useful one.