Fast private file transfers for Drupal

interesting. i will do some reading on xsendfile. meanwhile, some minor administration:

t('Custom - after applying access controls, files are transferred by a user-defined handler.')),
'#description' => t('If you want any sort of access control on the downloading of files, this needs to be set to private. You can change this at any time, however all download URLs will change and there may be unexpected problems so it is not recommended.')

in the fragment above, i would say 'admin defined handler' instead of user defined handler. And the description text implies that access control is only enforced by PRIVATE, but CUSTOM also enforces it.

Also, I find the settings a bit odd. We have a UI for enabling CUSTOM but not for specifying which function is the handler. Perhaps we should handle this custom_url_rewrite(). Thats a magic function name which is used to rewrite incoming or outgoing urls. Drupal will use it if present. See http://api.drupal.org/api/HEAD/function/drupal_get_path_alias. So in this case, we adjust the code to use fe_transfer_custom() is such exists. If not, use the specified transfer methord. This is roughly analogous to the custom SMTP library as well.

Does anyone know of optimizations which will work on typical web hosts that don't allow xsendfile extension or are apache 1.x?

Fantastic write up, Arto. I'm going to ping the dev list with a pointer to this issue as well.

It would be nice to have this in core, but I think we should not have file_transfer_xsendfile in core as it is a non-standard feature.

Interesting, we should reconsider this for Drupal 6

I don't see much point in the UI option as it is now... this feature requires at least some code to support the download method, and the variable for the handler name anyway.

Ideally, we'd have a mechanism where you can have an x-send-file.module which sets the handler variable on hook_enable / disable. By setting this variable, the download method setting would be grayed out (and this x-send-file module could even form_alter it to add the third option saying 'X-Send-File').

Subscribing (after having read http://www.mysqlperformanceblog.com/2007/02/11/content-delivery-system-d...)

subscribing

Great Idea!

Two questions, though:

1) what is this patch for?:

-    $header = preg_replace('/\r?\n(?!\t| )/', '', $header);
-    header($header);
+    header(preg_replace('/\r?\n(?!\t| )/', '', $header));

2) Why not simplify your patch to work based on a setting.php function (i think this is what Moshe ment, but i am not sure). In this case if a function existed in setting.php (or maybe even a module?) it would send all private files with this function. This way you only add a few lines to file_transfer():

  //If a user defined a custom private file trasfer function, use it
  //The function should exit() if it handled everything.
  if (function_exists('custom_file_transfer')) {
    custom_file_transfer($source);
    }
  }

Forgot to mention in my last post, this is also very usefull if you are using file wrapper modules such as imagecache and friends.

subscribing. seems very interesting but i'm not sure it'll make it into 6 this late. i'd be interested in pushing it after some of my other files patches get committed.

Subscribing. We have several clients using Drupal to xfer large files via groups (og). It would be good to see this option available in Drupal core, even if it needs explicit configuration to enable (probably safest option).

Has anyone seen benchmarks comparing x-sendfile to normal Drupal downloads?

While I am here, does anyone know if it is possible to use Partial Content with X-sendfile headers? e.g. http://drupal.org/node/91934

Hey,

People want either "Public" or "Private" file transfers, that's it!
Nobody, and I mean NOBODY wants a third option on that screen

The fact that "Public files" are served by Apache
and that currently "Private files" are served by PHP
is an implementation detail that the users should never have to worry about.

On that note,
the code should probably use X-Sendfile if available, and PHP otherwise for "Private" file transfers.
This would simplify the patch to an if statement and one extra line for the X-sendfile header

Now,
If I only knew how to detect sendfile support on-the-fly
how does Drupal detect mod_rewrite for Clean URL support?

Here is new small patch, which also makes fast private file transfers possible. It just allows overriding the default file transfer handling while the default is the current implementation. Now file_transfer() just invokes the configured handler.

X-Sendfile users can activate it by adding this to settings.php:

$conf['file_transfer_handler'] = 'file_transfer_xsendfile';

/*
 * Send the file with the webservers X-Sendfile feature
 */
function file_transfer_xsendfile($source) {
  header('X-Sendfile: /path/to/drupal/' . $source);
  exit;
}

With this patch one could also implement a small module for activating X-Sendfile file transfers.

didn't test this patch but tried to clean up some of the comments and documentation.

thanks. patch still works as described.
Any testers?

i think it's unreasonable to think this will make it into 6.x at this point. bumping to 7.x.

Subscribing, +1.

doublesubscribe +2!

subscribing, i should be able to test this this week.

If I only knew how to detect sendfile support on-the-fly

What about in_array('mod_xsendfile', apache_get_modules())?

Subscribing.

subscribing

Update

I finally got a chance to try this patch-- I couldn't get that patch to apply properly (against the current head), so I did it manually.

However, I was unable to actually try it because as it turns out I'm having trouble getting apache to use mod_xsendfile-- every time I try to load it apache refuses to start.

I think someone else mentioned this earlier, but should we even display this option to the user. I would think it would be better if things were either public or private, and if you installed a module that offered a new file serving method, it would just take over until it was uninstalled. They should probably be good about when they're enabled and put some checks for which method is the current one in hook_requirements(), but that should be a contrib issue, not a core issue. I think this has the simplicity needed to make things easy for users (with less options) and still offers all the benefits.

subscribing...

Patch still applied against CVS HEAD.

subscribing

it'd be possible to do this totally in contrib now. with the new menu system you could just menu_alter the system/files callback to your own handler, replicate a bit of the code from file_download() and do your own headers.

+1 for a contrib solution w/ hook_menu_alter. This gets overly complicated with more than public/private.

doubt this works now.

Core currently provides enough flexibility for modules to provide this functionality. I say we call it won't fix.

Can I use this patch for drupal 6.x?

I have created module for x-send file, included small patch against HEAD http://drupal.org/project/xsend.
Welcome your comments!!!

subscribe

I am a newbie on the site. Just wanted to follow up with regards to the status of this patch.