Fast private file transfers for Drupal
| Project: | Drupal |
| Version: | 7.x-dev |
| Component: | file system |
| Category: | feature request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | patch (code needs work) |
Summary
This is a simple & short patch against HEAD that adds a new file download method, "Custom", to the admin/setting/file-systems screen. With "Custom" selected, the "file_transfer_handler" configuration variable is expected to contain the name of a function that will handle the actual file transfer, and file_transfer() (in includes/file.inc) is modified to delegate the transfer process to that function.
Description
This was originally going to be a patch specifically for supporting the X-Sendfile feature provided by modern web servers (including Apache2 and Lighttpd). X-Sendfile provides a significant performance benefit for sites that need to offer large files (audio or video, for instance), but also need Drupal's access controls.
Boris advised me to make the patch more generic so that we could, conceivably, in the future support capabilities such as redirecting file transfers to an FTP server or an Amazon S3-based URL.
So, two file transfer handlers are provided as standard:
- file_transfer_php: the standard Drupal code that writes the file out in 1K blocks; this is the default and works as before.
- file_transfer_xsendfile: uses the X-Sendfile HTTP header to let the web server handle the file transfer in an efficient fashion, and without unnecessarily tying up a PHP process for the duration of the transfer.
At this point, the patch is exceedingly simple: two one-line changes to system.module and upload.module, and less than a dozen new lines added to includes/file.inc.
See screenshot of the modified "Download method" selection.
Installation
To try out the patch, you need a web server that supports X-Sendfile. Lighttpd has this support built in, and a mod_xsendfile module exists for Apache2. (Please see the References section for the relevant URLs.)
Installing the Apache module should be a breeze; it took me less than a minute on OS X with DarwinPorts:
wget http://celebnamer.celebworld.ws/stuff/mod_xsendfile/mod_xsendfile.c
sudo /opt/local/apache2/bin/apxs -cia mod_xsendfile.c
sudo /opt/local/apache2/bin/apachectl restartBe advised that in addition to installing the Apache module, you will need to add the "XSendFile on" directive to your virtual host configuration or the .htaccess file supplied with Drupal. Also be sure to read the limitations of mod_xsendfile: for instance, the file directory must be located under the current directory (i.e. the Drupal web root), implying that you'll likely want to forbid direct HTTP access to the files via Apache config directives.
Once you have X-Sendfile transfer enabled in your web server, enable it for Drupal as by adding the following to your settings.php:
$conf = array('file_transfer_handler' => 'file_transfer_xsendfile');References
- http://celebnamer.celebworld.ws/stuff/mod_xsendfile/
- http://wiki.rubyonrails.org/rails/pages/HowtoSendFilesFast
- http://blog.lighttpd.net/articles/2006/07/02/x-sendfile
- http://lighttpd.net/documentation/fastcgi.html
| Attachment | Size |
|---|---|
| fast_file_transfer.patch | 4.25 KB |
#1
interesting. i will do some reading on xsendfile. meanwhile, some minor administration:
in the fragment above, i would say 'admin defined handler' instead of user defined handler. And the description text implies that access control is only enforced by PRIVATE, but CUSTOM also enforces it.
Also, I find the settings a bit odd. We have a UI for enabling CUSTOM but not for specifying which function is the handler. Perhaps we should handle this custom_url_rewrite(). Thats a magic function name which is used to rewrite incoming or outgoing urls. Drupal will use it if present. See http://api.drupal.org/api/HEAD/function/drupal_get_path_alias. So in this case, we adjust the code to use fe_transfer_custom() is such exists. If not, use the specified transfer methord. This is roughly analogous to the custom SMTP library as well.
Does anyone know of optimizations which will work on typical web hosts that don't allow xsendfile extension or are apache 1.x?
#2
Fantastic write up, Arto. I'm going to ping the dev list with a pointer to this issue as well.
#3
It would be nice to have this in core, but I think we should not have file_transfer_xsendfile in core as it is a non-standard feature.
#4
Interesting, we should reconsider this for Drupal 6
#5
I don't see much point in the UI option as it is now... this feature requires at least some code to support the download method, and the variable for the handler name anyway.
Ideally, we'd have a mechanism where you can have an x-send-file.module which sets the handler variable on hook_enable / disable. By setting this variable, the download method setting would be grayed out (and this x-send-file module could even form_alter it to add the third option saying 'X-Send-File').
#6
Subscribing (after having read http://www.mysqlperformanceblog.com/2007/02/11/content-delivery-system-d...)
#7
subscribing
#8
Great Idea!
Two questions, though:
1) what is this patch for?:
- $header = preg_replace('/\r?\n(?!\t| )/', '', $header);- header($header);
+ header(preg_replace('/\r?\n(?!\t| )/', '', $header));
2) Why not simplify your patch to work based on a setting.php function (i think this is what Moshe ment, but i am not sure). In this case if a function existed in setting.php (or maybe even a module?) it would send all private files with this function. This way you only add a few lines to file_transfer():
<?php//If a user defined a custom private file trasfer function, use it
//The function should exit() if it handled everything.
if (function_exists('custom_file_transfer')) {
custom_file_transfer($source);
}
}
?>
#9
Forgot to mention in my last post, this is also very usefull if you are using file wrapper modules such as imagecache and friends.
#10
subscribing. seems very interesting but i'm not sure it'll make it into 6 this late. i'd be interested in pushing it after some of my other files patches get committed.
#11
#12
Subscribing. We have several clients using Drupal to xfer large files via groups (og). It would be good to see this option available in Drupal core, even if it needs explicit configuration to enable (probably safest option).
Has anyone seen benchmarks comparing x-sendfile to normal Drupal downloads?
While I am here, does anyone know if it is possible to use Partial Content with X-sendfile headers? e.g. http://drupal.org/node/91934
#13
Hey,
People want either "Public" or "Private" file transfers, that's it!
Nobody, and I mean NOBODY wants a third option on that screen
The fact that "Public files" are served by Apache
and that currently "Private files" are served by PHP
is an implementation detail that the users should never have to worry about.
On that note,
the code should probably use X-Sendfile if available, and PHP otherwise for "Private" file transfers.
This would simplify the patch to an if statement and one extra line for the X-sendfile header
Now,
If I only knew how to detect sendfile support on-the-fly
how does Drupal detect mod_rewrite for Clean URL support?
#14
Here is new small patch, which also makes fast private file transfers possible. It just allows overriding the default file transfer handling while the default is the current implementation. Now file_transfer() just invokes the configured handler.
X-Sendfile users can activate it by adding this to settings.php:
$conf['file_transfer_handler'] = 'file_transfer_xsendfile';
/*
* Send the file with the webservers X-Sendfile feature
*/
function file_transfer_xsendfile($source) {
header('X-Sendfile: /path/to/drupal/' . $source);
exit;
}
With this patch one could also implement a small module for activating X-Sendfile file transfers.
#15
didn't test this patch but tried to clean up some of the comments and documentation.
#16
thanks. patch still works as described.
Any testers?
#17
i think it's unreasonable to think this will make it into 6.x at this point. bumping to 7.x.
#18
Subscribing, +1.
#19
doublesubscribe +2!
#20
subscribing, i should be able to test this this week.
#21
What about
in_array('mod_xsendfile',apache_get_modules())?#22
Subscribing.
#23
subscribing
#24
Update
I finally got a chance to try this patch-- I couldn't get that patch to apply properly (against the current head), so I did it manually.
However, I was unable to actually try it because as it turns out I'm having trouble getting apache to use mod_xsendfile-- every time I try to load it apache refuses to start.
#25
I think someone else mentioned this earlier, but should we even display this option to the user. I would think it would be better if things were either public or private, and if you installed a module that offered a new file serving method, it would just take over until it was uninstalled. They should probably be good about when they're enabled and put some checks for which method is the current one in hook_requirements(), but that should be a contrib issue, not a core issue. I think this has the simplicity needed to make things easy for users (with less options) and still offers all the benefits.
#26
subscribing...
#27
#28
Patch still applied against CVS HEAD.
#29
subscribing
#30
it'd be possible to do this totally in contrib now. with the new menu system you could just menu_alter the system/files callback to your own handler, replicate a bit of the code from file_download() and do your own headers.
#31
+1 for a contrib solution w/ hook_menu_alter. This gets overly complicated with more than public/private.
#32
doubt this works now.