making upload.module more scalable with a better directory structure [#28769]

the biggest drawback of upload.module, IMHO, is that it stores all files in a single directory. This is fine for small sites that aren't going to be doing a lot of uploading or if all the uploading is going to be managed by a single person. But if you want to allow users to upload files, it's not very practical for them to all be uploading to the same directory.

I tried using filemanager.module, but I'm just not impressed with it. My primary issue with it is that it creates a new database table called 'file', which is mostly a duplicate of 'files' table. There's absolutely no reason they couldn't have used the existing files table and added an additional table that only held values that didn't exist in 'files'. If you use filemanager, you cannot use any other modules that deal with uploaded files unless they point to the filemanager version of 'files'. I've already been bitten by this as I wrote my own custom module.

Anyway, I'd rather try to implement the 'missing features' in the upload.module and then go on to add my own modules and tables as required by my site. I do not want to duplicate existing functionality.

With that, I'm going to modify upload.module with a single new option, and that option will be whether to store uploaded files in a unique directory based on user id and node id. that is, files would go into $files/USERID/NODEID/filename.ext

This will allow users to upload files with the same names as other users or even as other nodes. It will also prevent a massive number of files in any single dir, which should allow readdir calls to be more efficient (assumming there are no recursive readdirs hidden in the code). also, users should be able to turn this feature on and off without breaking any existing file pointers on the site, as long as they don't mess with the contents of the files dir :)

wanted to know if there was any general interest in this type of feature, and if so, do you think the directory structure proposed above is acceptable?

Comments

Comment #1

Harry Slaughter

Omaha NE USA

CreditAttribution: Harry Slaughter commented 14 August 2005 at 05:22

well, this is one of those cases where the problem is orders of magnitude less complex than the solution (so it appears to me).

i figured there would be an obvious location in the code of upload.module where a destination path was being determined, and i could implement this "feature" with a few lines of code.

but after a day of trying to trace calls by hand and hours spent with debug_backtrace() output, i just can't follow the logic here. there's a vast labrynth of file checking, prepending, modifying... of directory paths. ambiguously named calls nested in calls nested in.... a function named "upload_download()" almost caused my brain to meld against my headrest. this simple looking, 7 line function was the dangling thread on the sweater, and i pulled it. makes me actually miss objects, which is hard to do :)

definitely not criticizing anyone here. it was obviously written by someone adroit with the drupal api and not worried about noobs like me wanting to tinker with it. i see why filemanager was born :)

and apparently i'm the only one with any interest in creating more private namespace for file uploads, so i'm left SOL.

Comment #2

robertDouglass CreditAttribution: robertDouglass commented 14 August 2005 at 08:30

re: upload_download. That's a naming convention that says this is the "download" function in the "upload" module. We do it to avoid namespace confilcts (yay PHP).

You'd be interested in knowing that the upload module and the filemanager module were "competing" to get into core at one point, which is why they seem to fill the same space but not overlap at all. They were developed in parallel.

From a file system point of view we've researched and found that there shouldn't be a problem putting all files in one dir, at least not from the computer's point of view. It makes it hard to go in and find a particular file, to be sure. Everyone agrees there is lots of room for improvement, so any energies you are able to spend on it are welcome.

Comment #3

Harry Slaughter

Omaha NE USA

CreditAttribution: Harry Slaughter commented 15 August 2005 at 17:53

ah, that clears up quite a bit of confusion. i had assummed that filemanager was written as a replacement for upload.module. so the 'duplicate' db table was not a poor design decision after all :)

as far as storing all uploads in a single dir, i'm not familiar with various file system restrictions on different OSs. but i do know they exist. and i do believe that multiple users sharing a single directory is asking for problems (what if two users upload files with the same name at the same time?). and there's probably no drawbacks to using "private" directory space for users/nodes.

judging from upload.module/issues page, it looks like that module isn't getting much attention. and judging from the threads i've read, there's not much need/interest right now for fixing any potential problems with it.

so i'm adding it as a low-pri "bug" to my own project for now and will review the issue when needed.

Comment #4

sito-1 CreditAttribution: sito-1 commented 23 November 2005 at 02:18

Hi,

Since no one else seems to have commented on this since August, I want to put up my hand and say that I am interested in this feature, although I don't feel confident enough to offer my own programming skills for this. :-)

Comment #5

exhuma CreditAttribution: exhuma commented 26 January 2006 at 10:17

Wohoo. Someone has a similar idea as me :)

I was too thinking about modifying the upload code in order to get a cleaner directory structure. I was however thinking about simply putting the node id as subdirectory, so you would have

/files/nid/filename.ext

I beleivethat suffices as abstraction. If you add the user id as well you will (on a system with many users) end up with a quite opaque directory structure.

The reason I am interested in this feature as well is because I have to set up Drupal for our company's intranet site. We have quite a high number of editors so if things really go crazy you might end up having lots of "userid" subdirectories with each one file in it.
You might however add an option to the upload module, let's say 2 chackboxes of the form "Create a subdirectory per node" and "Create a subdirectory per user". This would allow quite a flexible setup. To disambiguate, you can then prepend the name of the subdirectories with either "node" or "user"

This would also solve another request of the editors for which I am configuring Drupal. They want an "easy" way to uplaod files. This might then be done using a samba share or via ftp. Nodes could then display all files contained in the directory "node_nid" instead of passing through the databse (which should still be kept for consistency and compatibility).

Any houghts on this?

Comment #6

raintonr CreditAttribution: raintonr commented 25 March 2006 at 08:30

My 2c:

- Once a file is uploaded, it would be nice to be able to replace it while re-using the same path to that file while keeping revisions just like the other parts of a node.

- Storing files as blob in the DB would be nice (if you have access through system/files?file=yada turned on that is).

I tried to search for these but didn't find anything but this so figured it was where such a comment belongs. Appologies if this is a duplicate.

Rob

Comment #7

robertDouglass CreditAttribution: robertDouglass commented 27 March 2006 at 14:37

Harry, make sure you explore filemanager and attachment modules. The combination of the two provides a much more powerful and technically superior alternative to the upload module (an I vote that they replace upload in core). They provide at at least some of the features you're looking for.

Comment #8

bradlis7 CreditAttribution: bradlis7 commented 18 April 2006 at 16:54

At first, I didn't like this idea, but after thinking about it, this would be useful if we ever get the much-needed comment attachments. I kind of think it should be optional for now though, so that the low-traffic/single-admin sites don't have to mess with it.

> This would also solve another request of the editors for which
> I am configuring Drupal. They want an "easy" way to upload
> files. This might then be done using a samba share or via ftp.
> Nodes could then display all files contained in the directory
> "node_nid" instead of passing through the databse (which
> should still be kept for consistency and compatibility).

If you're website is on a local intranet, I don't see why there's a problem with uploading files via the web interface, unless someone wants to upload more than 3 files at a time. Maybe there could be an option to allow more than one upload field? Since file attachments is collapsed starting in 4.7, there's no need to keep it short, so there might as well be 5 fields (if it can be implemented easily).

> - Storing files as blob in the DB would be nice (if you have
> access through system/files?file=yada turned on that is).

Storing files as blob is not a good thing from what I've read. It could be an option of last resort for those whose servers do not allow them to write to the directory though. Major problems include that it will be stored in memory, and also has a size limit.