Really make this thing scalable for large media sites -- add volumes

lutegrass - June 26, 2006 - 22:31
Project:Filemanager
Version:HEAD
Component:Code
Category:feature request
Priority:normal
Assigned:Unassigned
Status:active
Description

Ok, I love this module and am glad it exists. The auto creation of file areas is great for scalability and inode restrictions. I can't believe the core upload module doesn't have something similar. Anyway, I think it might be better to define "volumes" where the user files can exist. Volumes could map to physical or virtual disk areas. Right now, you only support one volume area (one for private and one for public). This could be striped across a raid array to get a huge disk farm, but it would also be good to be able to define alternate volumes that could be added to the round-robin of assigning working directories. So, it might be something like this in the settings/filemanager screeen ...

Private File Areas (volumes)...
1. /mnt/volume1
2. /mnt/volume2
3. /mnt/volume3

(option list to be selected by user)
__ Use all active volumes sequentially (spill-over when full)
__ Use all active volumes randomly (round-robin)

I think this type of setup would allow an admin to quickly add a disk to react to a spike in uploads or when running out of space. It could also be a way to spread the io load, especially using the round-robin configuration.

Basically, the current implementation is very good. Just adding the ability to specify more than one file location or volume would make it really scale for large media sites. There's no way the current implementation would work for a large media sharing site unless you wanted to put out the cash for some expensive network attached raid array. Yes, disk space is cheap, but flexibility is key to solving io and scalability problems. What do you think. Shouldn't be too much of a hack?

#1

jmanico - June 21, 2007 - 22:10

This is a brilliant (or at least prudent) suggestion. I like where your head is. Is there any progress on this feature? Anyone else interested in seeing this happen? I'm getting more requests to Drupal for *very* large sites.

 
 

Drupal is a registered trademark of Dries Buytaert.