Resumable & access controlled file download system
zenic - September 29, 2009 - 04:46
Hi Everyone,
Just got started with Drupal and was looking for the ability to support access controlled files (i.e. some users can and some users can't access them) as well as resumable downloads. I didn't find what I was looking for so I tried making my own module. I am a drupal beginner and I've almost certainly missed (or misused) a whole lot of important stuff, so suggestions very welcome! And just maybe someone else will find this interesting.
I wanted to be able to:
1. have private files, separate to the drupal-wide setting
2. be able to control the access to the files via hooks
3. be able to resume file downloads
the module is called "fileget". It's not complete, but should be functional. I'll continue to improve it for my own purposes and post updates depending on interest and feedback.
Quick howto:
1. enable the module
2. go to /admin/settings/fileget and set up your private folder. (You'll also want to manually create a folder on your webserver and do a "deny all" in a .htaccess file)
3. upload files to the folder in question
4. go to /get/file/add and add them, much like normal nodes I suppose - I chose to do this to ensure clean separation, but maybe I should have just reused nodes somehow? guidance here appreciated!
5. from then on you'll be able to direct users to /get/file/fid where fid is the numeric id of the file. NOTE: they will need at minimum "access fileget content" privilege.
6. go to /get/file/fid/edit to change details.
NOTE: there is still tidying up to be done (e.g. using l() to create links and t() for all the translatable text) but I want to establish if I'm on the right track here or not first...
fileget.info:
; $Id$
name = "FileGet"
description = "Access controlled resumable file download system"
core = 6.xfileget.module:
<?php
// $Id$
DEFINE('KB_PER_READ', 2); // min = 1, max = 8
// don't touch the defines below, they are automatically calculated:
DEFINE('SLEEP_NUMERATOR', KB_PER_READ*1000000);
DEFINE('MINIMUM_SLEEP_TIME', KB_PER_READ*100);
DEFINE('BYTES_PER_READ', KB_PER_READ*1024);
/**
* Display help and module information
* @param path which path of the site we're displaying help
* @param arg array that holds the current path as would be returned from arg() function
* @return help text for the path
*/
function fileget_help($path, $arg) {
switch ($path) {
case "admin/help/fileget":
$output = '<p>'. t("fileget enables downloads that are resumable and access controlled") .'</p>';
break;
}
return $output;
} // function fileget_help
/**
* Valid permissions for this module
* @return array An array of valid permissions for the fileget module
*/
function fileget_perm() {
return array('administer fileget', 'access fileget reports', 'access fileget content');
} // function fileget_perm()
/**
* Menu for this module
* @return array An array with this module's settings.
*/
function fileget_menu() {
$items = array();
$items['get/file/%fileget'] = array(
'title callback' => 'fileget_file_title',
'title arguments' => array(2),
'page callback' => 'fileget_file_download',
'page arguments' => array(2),
'access callback' => 'fileget_file_access',
'access arguments' => array('view', 2),
'type' => MENU_CALLBACK);
$items['get/file/%fileget/download'] = array(
'title' => 'Download',
'type' => MENU_DEFAULT_LOCAL_TASK,
'weight' => -10);
$items['get/file/%fileget/edit'] = array(
'title' => 'Edit',
'page callback' => 'fileget_file_edit',
'page arguments' => array(2),
'access callback' => 'fileget_file_access',
'access arguments' => array('update', 2),
'weight' => 1,
'file' => 'fileget.pages.inc',
'type' => MENU_LOCAL_TASK);
$items['get/file/add'] = array(
'title' => 'Add',
'page callback' => 'fileget_file_add',
'access arguments' => array('administer fileget'),
'weight' => 1,
'file' => 'fileget.pages.inc',
'type' => MENU_LOCAL_TASK);
//Link to the fileget admin page:
$items['admin/settings/fileget'] = array(
'title' => 'FileGet Settings',
'description' => 'Admin Files for Download',
'page callback' => 'fileget_admin_page',
'access arguments' => array('administer fileget'),
'file' => 'fileget.pages.inc',
'type' => MENU_NORMAL_ITEM);
return $items;
}
/**
* Load the file details from the database
*/
function fileget_load($param) {
if (is_numeric($param)) {
$fields = drupal_schema_fields_sql('fileget_files');
$fields = implode(', ', $fields);
$cond = 'fid = %d';
$arguments[] = $param;
$fileobj = db_fetch_object(db_query('SELECT '. $fields .' FROM {fileget_files} WHERE '. $cond, $arguments));
return $fileobj;
}
}
/**
* Save the file details into the database.
*/
function fileget_save(&$fileget) {
// Generate the node table query and the node_revisions table query.
if (empty($fileget->fid)) {
drupal_write_record('fileget_files', $fileget);
}
else {
drupal_write_record('fileget_files', $fileget, 'fid');
}
}
/**
* return the file title (the short name)
*/
function fileget_file_title($fileobj) {
return $fileobj->title;
}
/**
* download called (file/download menu)
*/
function fileget_file_download($fileobj) {
// $ip=$_SERVER['REMOTE_ADDR'] // log the ip address
$filereq = _fileget_downloadFile( fileget_createpath($fileobj->filepath), variable_get('fileget_max_speed', 40), variable_get('fileget_part_speed', 10) );
module_invoke_all('fileget_result', $filereq);
}
function fileget_fileget_result($filereq)
{
if (variable_get('fileget_debug_logging', 0))
watchdog('fileget', json_encode($filereq), null, WATCHDOG_NOTICE, null);
}
/**
* Determine access rights to download a file
*/
function fileget_file_access($op, $fileobj, $account=NULL) {
global $user;
if (!$fileobj || !in_array($op, array('view', 'update', 'delete', 'create'), TRUE)) {
// If there was no file to check against, or the $op was not one of the
// supported ones, we return access denied.
return FALSE;
}
// If no user object is supplied, the access check is for the current user.
if (empty($account)) {
$account = $user;
}
if (user_access('administer fileget', $account)) {
return TRUE;
}
else if (in_array($op, array('update', 'delete', 'create'), TRUE)) {
return FALSE;
}
if (!user_access('access fileget content', $account)) {
return FALSE;
}
// if any module returns false, then disallow
// if all modules return true (or null) then allow
$perm_overrides = module_invoke_all('fileget_perms', $fileobj);
foreach ($perm_overrides as $override) {
if ($override == false)
return false;
}
return true;
}
// hook file permisions
//function fileget_fileget_perms($fileobj) {
//return false;
//}
function fileget_createpath($rawpath) {
$basePath = variable_get('fileget_path', 'sites/default/private');
return $basePath . (empty($rawpath)?'':'/'.$rawpath);
}
/**
* downloadFile function
*
* implements resumable file downloads, returning the results of the operation
*/
function _fileget_downloadFile($filePath, $maxSpeed = 80, $partSpeed = 20)
{
ignore_user_abort(true); // if users drop the connection, we want to know about it so we can log the failure.
$fileName = basename($filePath);
if (strstr($_SERVER['HTTP_USER_AGENT'], "MSIE")) {
$fileName= preg_replace('/\./', '%2e', $fileName, substr_count($fileName, '.') - 1);
}
$filereq['path'] = $filePath;
$filereq['file'] = $fileName;
$filereq['size'] = filesize($filePath);
if ($filereq['size'] <= 0) {
$filereq['error'] = 'zero length file';
return $filereq;
}
$filereq['header'][] = "Cache-Control: public";
$filereq['header'][] = "Content-Transfer-Encoding: binary";
$filereq['header'][] = 'Content-Type: application/octet-stream';
$filereq['header'][] = sprintf('Last-modified: %s GMT', gmdate("D, d M Y H:i:s", filemtime($filePath)) );
$filereq['header'][] = sprintf('Content-Disposition: attachment; filename="%s"', $fileName);
$filereq['header'][] = 'Accept-Ranges: bytes';
//http://tools.ietf.org/id/draft-ietf-http-range-retrieval-00.txt
if(isset($_SERVER['HTTP_RANGE'])) {
list($size_unit, $range_orig) = explode('=', $_SERVER['HTTP_RANGE']);
if ($size_unit == 'bytes') {
list($firstrange, $otherranges) = explode(',', $range_orig);
$filereq['range'] = explode('-', str_replace(' ','',$firstrange) );
}
}
if (empty($range[0])) $filereq['range'][0] = 0;
if (empty($range[1])) $filereq['range'][1] = $filereq['size']-1;
// do we have a partial file download?
$filereq['partial'] = ($filereq['range'][0]!=0) || ($filereq['range'][1]!=($filereq['size']-1));
if (true == $filereq['partial']) {
$filereq['header'][] = "HTTP/1.1 206 Partial Content";
$maxSpeed = $partSpeed; // use the partial speed
}
$filereq['length'] = ($filereq['range'][1]-$filereq['range'][0])+1;
$filereq['header'][] = sprintf('Content-Length: %u', $filereq['length']);
$filereq['header'][] = sprintf('Content-Range: bytes %u-%u/%u', $filereq['range'][0], $filereq['range'][1], $filereq['size']);
// open & seek to position
set_magic_quotes_runtime(0);
$fp=fopen("$filePath","rb");
if (fseek($fp,$range[0]) != 0) {
$filereq['error'] = 'unable to seek';
return $filereq;
}
/*
* emit the accumulated headers now
*/
foreach ($filereq['header'] as $hdr)
header($hdr);
/*
* The following code attempts to balance the needs of:
* - large files don't overly degrade server performance (max flush once per second)
* - download rate limits can be as low as 1kb/s
* - browsers with poor connections don't time out easily
*/
$sleepTime = SLEEP_NUMERATOR / $maxSpeed;
$sleepTime = max($sleepTime, MINIMUM_SLEEP_TIME); // limit stress on server
$filereq['expectedreads'] = ceil($filereq['length'] / BYTES_PER_READ);
$reads_per_second = $maxSpeed / KB_PER_READ;
$reads_until_flush = $reads_per_second;
set_time_limit(0); // we do not want to time out from our end
for (
$filereq['numreads'] = $filereq['expectedreads'];
(connection_status()==0) && ($filereq['numreads']>0);
--$filereq['numreads'])
{
print(fread($fp, BYTES_PER_READ)); // transfer chunk of data (note: fread max is 8k)
usleep($sleepTime); // limit the output rate (give the web server a rest)
--$reads_until_flush;
if ($reads_until_flush <= 0) {
flush(); // push it to the browser (avoids timeouts)
$reads_until_flush = $reads_per_second;
}
}
$filereq['endposition'] = ftell($fp);
fclose($fp);
$filereq['status'] = connection_status();
switch ($filereq['status']) {
case 1: $filereq['error'] = 'aborted'; break;
case 2: $filereq['error'] = 'timeout'; break;
}
//watchdog('fileget', json_encode($filereq), null, WATCHDOG_NOTICE, null);
return $filereq;
}
?>fileget.pages.inc:
<?php
// $Id$
/**
* @file
* Page callbacks for adding, editing, deleting, and revisions management for content.
*/
function _fileget_fsize($fname) {
$sz = filesize($fname);
if ($sz < 1024)
return "${sz} bytes";
$sz/=1024;
if ($sz < 1024)
return ceil($sz)." Kb";
$sz/=1024;
if ($sz < 1024)
return (ceil($sz*10)/10)." Mb";
$sz/=1024;
if ($sz < 1024)
return (ceil($sz*100)/100)." Gb";
$sz/=1024;
return (ceil($sz*1000)/1000)." Tb";
}
function _fileget_dirArray($path) {
$dir = fileget_createpath($path);
$r = array();
if (is_dir($dir)) {
if ($dh = opendir($dir)) {
while (($file = readdir($dh)) !== false) {
if ("file"==filetype($dir . $file)
&& !eregi( "^\.", $file ) // security check
&& !eregi( "p?html?", $file )
&& !eregi( "inc", $file )
&& !eregi( "php3?", $file ) ){
$r[] = array("filename" => $file, "size" => _fileget_fsize($dir.$file),
"modified" => date ("Y-m-d H:i:s", filemtime($dir.$file)));
}
}
}
closedir($dh);
}
return $r;
}
function _fileget_sorted_files($files) {
foreach ($files as $key => $row) {
$modified[$key] = $row['modified'];
$filename[$key] = $row['filename'];
}
if (is_array($modified) && is_array($filename))
array_multisort($modified, SORT_DESC, $filename, SORT_ASC, $files);
return $files;
}
function _fileget_create_mapping($files) {
$r = array();
foreach ($files as $row)
$r[$row['filename']] = sprintf("%s (%s)", $row['filename'], $row['size']);
return $r;
}
/**
* Menu callback; presents the fileget editing form, or redirects to delete confirmation.
*/
function fileget_file_edit($fileget) {
drupal_set_title(check_plain($fileget->title));
return drupal_get_form('fileget_form', $fileget);
}
function fileget_file_add() {
$fileget->title = '';
drupal_set_title(t('Add file'));
return drupal_get_form('fileget_form', $fileget);
}
function fileget_form($form_state, $fileget) {
$form['title'] = array(
'#type' => 'textfield',
'#title' => t('File title'),
'#default_value' => t($fileget->title),
'#size' => 60,
'#maxlength' => 60,
'#description' => t('The short name of the file (60 chars max)'));
$filelist = _fileget_create_mapping(_fileget_sorted_files(_fileget_dirArray('/')));
if ( isset($fileget->filepath) ) {
$filelist = array_merge(
array(t($fileget->filepath) => sprintf("%s (%s)", t($fileget->filepath), 'not found!')),
$filelist);
}
else
$fileget->filepath = '';
$form['filepath'] = array(
'#type' => 'select',
'#title' => t('File'),
'#default_value' => t($fileget->filepath),
'#options' => $filelist,
'#description' => t('files are stored in '.fileget_createpath('')) );
$form['fid'] = array(
'#type' => 'value',
'#value' => $fileget->fid );
$form['buttons'] = array();
$form['buttons']['submit'] = array(
'#type' => 'submit',
'#value' => t('Save'),
'#weight' => 5,
'#submit' => array('fileget_form_submit') );
if (!empty($fileget->fid) && fileget_file_access('delete', $fileget)) {
$form['buttons']['delete'] = array(
'#type' => 'submit',
'#value' => t('Delete'),
'#weight' => 15,
'#submit' => array('fileget_form_delete_submit'), );
}
return $form;
}
function fileget_form_submit($form, &$form_state) {
global $user;
$fileget = (object)$form_state['values'];
$insert = empty($fileget->fid);
fileget_save($fileget);
$link = sprintf('get/file/%u/edit', $fileget->fid); // BAD: Hardcoded, fix later
drupal_goto($link);
}
/**
*
*/
function fileget_admin_page() {
$page_content = '';
$page_content .= drupal_get_form('fileget_admin_form');
return $page_content;
}
/**
* The callback function (form constructor) that creates the HTML form for fileget_message().
* @return form an array of form data.
*/
function fileget_admin_form() {
$currentPath = fileget_createpath(''); //variable_get('fileget_path', 'sites/default/private');
$form['fileget_path'] = array(
'#type' => 'textfield',
'#title' => t('Private file path'),
'#default_value' => $currentPath,
'#description' => t("Enter the path where the controlled files will be stored."),
);
// do some tests on the current path to see if it works
if (file_exists($currentPath)) {
if (is_dir($currentPath)) {
if (is_file(fileget_createpath('.htaccess'))) {
$htaccess = file_get_contents(fileget_createpath('.htaccess'));
$shortaccess = str_replace(array(' ', '\t', chr(13), chr(10), '\0', '\x0B'), array(), $htaccess);
if ( strtolower($shortaccess) == 'orderdeny,allowdenyfromall' ) {
$form['exists'] = array(
'#type' => 'fieldset',
'#title' => t('Congratulations! This directory exists and is protected by a .htaccess file'),
'#collapsible' => true,
'#collapsed' => true );
} else {
$form['exists'] = array(
'#type' => 'fieldset',
'#title' => t('This directory exists, but please check the .htaccess file:'));
}
$form['exists']['htaccess'] = array ('#value' => '<div><pre>' . $htaccess . '</pre></div>');
} else {
$form['exists'] = array('#value' => '<p>WARNING: The path exists, but isn\'t protected by a .htaccess file!</p>');
}
} else {
$form['exists'] = array('#value' => '<p>ERROR: This isn\'t a directory!</p>');
}
} else {
$form['exists'] = array('#value' => '<p>ERROR: This path does not exist!</p>');
}
$form['fileget_max_speed'] = array(
'#type' => 'textfield',
'#title' => t('maximum download speed'),
'#default_value' => variable_get('fileget_max_speed', 40),
'#description' => t('limit download speed in kb/s') );
$form['fileget_part_speed'] = array(
'#type' => 'textfield',
'#title' => t('maximum download speed for partial requests'),
'#default_value' => variable_get('fileget_part_speed', 10),
'#description' => t('partial requests are made when people resume downloads. However, they are also made by download accelerators, so sometimes you get a lot of partial requests. this lets you set a separate speed for parital requests (in kb/s)') );
$form['fileget_debug_logging'] = array(
'#type' => 'checkbox',
'#title' => t('enable debug logging?'),
'#default_value' => variable_get('fileget_debug_logging', 0),
'#description' => t('when enabled, will log all information about file download attempts') );
//Submit button:
$form['submit'] = array(
'#type' => 'submit',
'#value' => t('Save Settings'),
'#submit' => array('fileget_admin_form_submit')
);
return $form;
}
function fileget_admin_form_submit($form, &$form_state) {
// set the fileget_path varaible
$newPath = $form_state['values']['fileget_path'];
if (empty($newPath)) $newPath = 'sites/default/private'; // default
variable_set('fileget_path', $newPath);// NOTE: changes to this variable need to be reflected in fileget.module:fileget_createpath()
if (is_numeric($form_state['values']['fileget_max_speed']) && $form_state['values']['fileget_max_speed'] > 0)
variable_set('fileget_max_speed', $form_state['values']['fileget_max_speed']);
if (is_numeric($form_state['values']['fileget_part_speed']) && $form_state['values']['fileget_part_speed'] > 0)
variable_set('fileget_part_speed', $form_state['values']['fileget_part_speed']);
variable_set('fileget_debug_logging', $form_state['values']['fileget_debug_logging']);
}
?>fileget.install:
<?php
function fileget_schema() {
$schema['fileget_files'] = array(
'description' => 'File download permissions and details',
'fields' => array(
'fid' => array('type'=>'serial', 'unsigned'=>true, 'not null'=>true),
'title' => array('type' => 'varchar', 'length' => 60, 'not null' => TRUE, 'default' => '', 'description' => 'title of the file'),
'filepath' => array('type' => 'varchar', 'length' => 255, 'not null' => TRUE, 'default' => '', 'description' => "Path to the file"),
),
'primary key' => array('fid'),
);
return $schema;
}
function fileget_install() {
// Create my tables.
drupal_install_schema('fileget');
}
function fileget_uninstall() {
// Drop my tables.
drupal_uninstall_schema('fileget');
}
?>
You should bundle this up and
You should bundle this up and submit it as a module, in a beta version. Others will then help you test it.
Thanks for the feedback!
Thanks for the feedback! I applied to put it up as a module but was denied on the basis that no one had looked at the application for 7 days.
Just for reference, here is the application I made:
Motivation message:
Module: "FileGet"
Summary: Resumable, Private, Speed-limited File Downloads
Target: Drupal 6.x
I originally posted this in the forums (http://drupal.org/node/590774)
and got a reply that suggested I submit it as a module.
FileGet as its own module will let you have:
- Private files (independent of drupal's setting).
- Similar to the FileForce module, it will force the browser to
download the file.
- It will also ensure that the receiving browser uses the correct file
name.
- It also supports the main use for the byte range retrieval
(http://tools.ietf.org/id/draft-ietf-http-range-retrieval-00.txt) http
request, which lets download accelerators and file download resumes
work.
- Control the download rate (in kb/s) to ensure a webserver doesn't get
overloaded
FileGet will also provide an API for:
- hook for fine-grained permissions on downloads (e.g. per user access
control or by ip)
- hook for logging/processing of results (including aborted or failed
downloads)
- a function (e.g. fileget_download_file()) that will let other modules
force their own file downloads with all the features such as byte range
support.
The reason I created FileGet is because we distribute our software to
our customers via a web login system (and I'm trying to port our website
to drupal). These files are typically about 30-50 mb in size and several
parts of the world have issues with disconnects when downloading. We
need to have fine grained access control over files and want to keep
track of who is downloading what (which I do in our own custom module)
I will be motivated to maintain this module as we will be using it
ourselves in our production website.
Current Status:
it is currently usable and functional. There is house-keeping to be
done (using t() and l()) and possibly some security checks. I'm very new
to drupal and hope to get feedback on any obvious mistakes I've made. It
is certainly still developmental and I expect to improve things a lot
before a stable release.
Similar Modules:
File Node
(http://drupal.org/project/filenode) This module is the closest thing
to what I wanted with FileGet. But, I wanted FileGet to not rely on
IMCE. Also, I wanted to provide fine-grained control over the access to
the files from my other module and treating the files as nodes seemed to
make that less straight forward. I essentially see FileGet as a
lower-level module and I would suggest that in the future FileNode could
rely on FileGet for providing multi-session download abilities. The
thing with that is that FileGet creates its own table, which some people
might not like, but I guess we'll find out over time. Integration with
FileNode would be something to look at, although I don't need it yet.
File Force
provides the ability to force downloads, but within drupal's existing
file download ability. I wouldn't want to complicate it - it does what
it does well and simply.