[discussion] Ground-up rewrite of WebFM

The first three items are de rigueur, however as the inventor of the module with over three years of support (albeit fitful) you'd think an offline query might have been in order wrt a complete rewrite.

Abstracting the UI to a cloud or local backend sounds good - Andre and I talked about it in the past but were too busy to pursue. Taking the file handling outside of Drupal will require a clean break from file.inc. webfm_file.inc doesn't reference file.inc, however webfm.module still uses file_transfer(), file_directory_path(), file_check_directory(), file_save_upload(), file_move() and file_delete(). These calls will need to be abstracted. Not trivial overall.

The javascript is ad hoc and could use a framework (so far I've had a 'if it ain't broke...' attitude). I don't recommend jquery due to the heavy 'object' nature of the code. I think MooTools is a better fit though I haven't invested any time investigating. If I continue any relationship to the module it is in this area I'd be most interested.

One scheme that I saw elsewhere that I thought was a great security enhancement for local storage was base-64 encoding file content and adding a .php extension with a added die header at the start of the file. This way all files are made inherently non-executable. Not sure what latencies would be introduced by the encode/decode operations.

Another opportunity is the semantic web stuff whereby file content metadata should be in rdf. I cannot intelligently comment on this since I haven't looked much into that world.

One potential source of opposition to this are some who would rather see the module mesh closer with Drupal - reuse of the files table, leverage hooks to other modules, etc.

The reason for the first three items is to carry the point of: "Clean up the issue queue, commit what's needed, and call it good for this branch". Yes, I suppose I should have emailed or something, but my initial thought was that users of this module should have some input too, since...you know...they use the module.

For the Javascript, isn't jQuery being loaded on the page anyways? I think that if core uses jQuery, it's only natural to assume that contrib modules should as well (and it makes one less step for the user -- not having to go download Mootools separately is a big plus in my book)

I like that base-64 idea, though I share your concern with the latency that might be added.

I would guess that most of the proposed functionality is already available from other modules, such as these:
http://drupal.org/project/rdf
http://drupal.org/project/amazon_s3

I think that it would still be feasible to use the files table for locally stored files. Maybe with some clever coding, it might even be possible to use the files table for remote files.

The users of the module (of which I am one) definitely get more than a say since they have contributed many essential patches over the past couple of years. My concern is keeping 2.x working and supported until a 3.x branch is ready.

jQuery is fantastic for what it was intended for - enriching presentation, however my javascript code does much more. A JQuery port would improve some cross browser issues and reduce the amount of code transferred at page load, however manipulating complex file and directory objects makes much more sense in an OO architecture. The bias at Drupal was for years against javascript (modules are supposed to degrade to php only functionality) and it figures they would pick a framework that is primarily just window dressing.

Hi Cameron,

thank you very much for your time and passion. I'm really glad to read that somebody's working on this module.
I think PHP5 dependence shouldn't be a problem. The most hosting providers have switched to PHP5 already. And IMO it's not efficient to limit the development progress because of a minority group of PHP4 users.

You asked for input:
1. A must-have we REALLY urgently need is a multiple-file upload. At the moment it's frustrating that the core functionality is not given. Due to security reasons by far not every end user has ftp access but sometimes has to upload umpteen files to WebFM. I think I don't have to illustrate that it's a pain.
2. I think it would be great to have the possibility to create blocks wether of the whole WebFM repository or an instance (i.e. folder "xyz"). That would provide much more flexibility -> we could embed/print this blocks individually into QuickTabs, regions, nodes etc. with appropriate permissions.

Kind regards,
Stefan

I would like a lot to see this feature in the oficial code:

http://drupal.org/node/555528

With this improvement you are opening a lot of doors to the use of this module.

About PHP5, I don't think it will be a problem. The actual code works on PHP4 and when you release the version 3.x almost everybody will have the PHP updated

And, by the way, a lot of thanks for mantain this code :-)

I was wondering when this discussion was going to start...I've been waiting for this a long time :-)

Actually I oppose a the complete rewrite: It will take a long time to be done - the normal tree will hurt from this. Also the usual reason for a complete rewrite - chaotic code - is not a problem in webfm.
My suggestion is to gradually improve parts of webfm (small tasks) instead of taking on the whole thing at once: This will ensure quality because we can test whether the aims where actually reached and squash bugs faster as well.

These are my thoughts (no specific order):

User Interface - Refreh Look

Maybe we can rework this a little? There are a lot of file browsers out there - and I don't think there is a reason why webfm can't have a refreshed look.

User Interface - Remove Right Click Menu

My pet peeve - it would be nice to get rid of this. We could have a little icon (gearwheel) next to the file names that opens a dropdown list with the tasks. It would also be nice if one could define the default task through the settings. I would suspect "Attach to Node" to be the default, but usually it is "open file in new window". We could make it possible to (de)activate actions through the settings as well.

User Interface - Rework Meta Data Interface

An other thing which always bothered me is the MetaData dialogue: not very intuitive to use, options aren’t accessible from the outside and/or not needed. This should also somehow be connected to token.

Remove Index this directory

This button should be removed – why isn’t the directory automatically indexed or why do we need to index it? It is almost impossible to explain this reasonable to any user.

Views/Token integration

In order to allow users to work with webfm files across modules we should provide more tokens and views integration

wyswyig-api integration

To allow users to use webfm with there favorite wyswyig –editor we should support wyswyig –api for the editors supported with wyswyig –api in webfm core.

Rework the rights management.

There are lot of questions about this here - the main message appears to be the same: I don't understand. In my opinion the way rights management is current implemented is also a break within drupal: Usually you would expect it to be at admin/user/permissions. I have some more thoughts on this – if there is interest in this I would explain further.

Side notes:

Breaking away from Drupals file handling

I am sorry, but this doesn’t sound too good to me – we should be more oriented in facilitating drupals methods – only this way other modules can plugin into webfm files as well

base64

Sounds strange to me: If I would want to move away from webfm all my files would be broken. Also one would have to decrypt on the fly – this takes place in webserver memory.

I'm not an existing user, although I'm considering trying this out as I've got a client who's having difficulty with the document management I've set up using filefield, CCK etc.

I'd argue that WebFM would benefit from closer integration with the rest of Drupal. Currently, core and popular contributed modules can make up about 90% of a well integrated document management system. A ground-up system would be better off focusing on filling in the gaps with a set of smaller contributed modules and creating a recipe to bring the various modules into a unified solution, rather than rebuilding a separate all-in-one application. It may also be better off doing so from a Drupal 7 base, as that has fields, RDF etc. in core.

Some of these modules might include:

• Instead of abstracting file handling to the cloud just for WebFM, why not contribute a module that enables e.g. the entire files directory or various subdirectories to be stored in the cloud?
• A module that gives finer-grained user permissions for file manipulation, including per-user or per-role directory creation and access, and implements a revision system for files in specified directories.
• A browser module (possibly integrated with Views, and with file and image field editing in node edit screens) that integrates user permissions and directory access, RDF metadata entry and display, revisions handling, and file or image operations, including multiple upload, deletion, copying, moving etc. Ideally, when using the browser from Views rather than node editing, include a link to view/edit the node attached to each file or image. There are some potentially hairy issues around the idea of allowing multiple fields to point to the same content, but that doesn't need to be tackled if it's too complex. Having a joint file and image browser may also make it easier to find other developers to get involved.
• A separate module to add base-64/php-die protection to file storage. Not sure the encoding is a good idea in general though, for the reasons other people mention.

Hope that helps!

Hi Niels

I agree that a complete rewrite isn't necessary, however a proper abstraction of client and server sides would allow for alternative file stores. A proper client/server API wouldn't be a waste of time.

Wrt to some of your points:

• User Interface:
  The module always lacked proper themeing (I'm a systems programmer, not a UI designer). Drupal themes should propagate into the client side better.

  I have no issue with context menus other than the usual cross browser problems - a javascript framework would likely help solve this. Enabling/disabling the menu task list in the admin settings is a good suggestion.

  The metadata structure was built to satisfy the requirements of the original client of the module. It was never a meant to be a general solution but I didn't have the time and resources to properly implement a metadata layer. It definitely deserves rework. I'd be thrilled if a rdf person stepped forward to tackle this.

• Remove Index this directory:
  Not a clue what you are talking about. The module has no 'indexing' though it would be fantastic if a proper content search was integrated with the module. Currently the search is for file title only.
• Views/Token integration:
  I've always been suspicious of Views because of my concern for file security. Hopefully a views hook can be constructed that respects this.
• Rework the rights management:
  I'm not opposed to reviewing this but one must understand that file rights in the module are dictated by directory topology as much as by user id. It is similar to any file system whereby files and subdirectories inherit rights from parent directories. Of course this is a break from Drupal. Drupal doesn't have a filesystem other than the one it is encapsulated within by the server. WebFM uses a combination of the server filesystem and Drupal user security to define file rights.

  The current rights implementation is a loose approximation of the linux group-owner-world concept:

  • Group file rights are determined by a combination of directory topology (role root directory) and the file permissions set in the 'perm' column of the webfm_file table.
  • Owner rights are determined by the Drupal uid column of the webfm_file table. Owners have full rights for file operations except the for ability to move it outside of the group root directory or to remove it from the database. Only module admins can change the uid of a webfm_file record, move the file to a different role group or remove/add it to the database.
  • World rights are controlled by the 'WEBFM_FILE_ACCESS_PUBLIC_VIEW' bit in the 'perm' field of a webfm_file record (World users get no other right than read).

  This brings us to the 'perm' bitfield values near the top of webfm.module which is an approximation of the unix rwx:
  

  define('WEBFM_FILE_ACCESS_PUBLIC_VIEW', 1); //publicly viewable via webfm_send link
define('WEBFM_FILE_ACCESS_ROLE_VIEW', 2);   //role can view/download file (owner/admin can always view)
define('WEBFM_FILE_ACCESS_ROLE_ATTACH', 4); //role attach permitted (owner/admin can always attach)
define('WEBFM_FILE_ACCESS_ROLE_FULL', 8);   //role modification of file permitted (owner/admin can always modify)
  
  These bits determine how the uid of a Drupal user affect file access. I'm open to debating these control bits and how they are applied. Perhaps a traditional rwx triad would be better.

• base64:
  Not so strange. Only real tests will determine how inefficient this scheme is - I suspect that packetization is more of a bottleneck. I'm not recommending this for a cloud scenario - they have their own file execution protections I'm sure.

Another point that I've made a few times in the past is in regards to file/directory contention. Php's System V IPC should be investigated as a way of preventing multiple users from operating on a file/directory simultaneously. For instance, if I attempt to access a file that has is being moved to another directory by the file owner or admin, there should be feedback that states that the file is locked by another user and that a refresh is required. Likewise an admin who wants to rename a directory while users are operating on files inside that directory might be denied access (I've actually never done tests on this). The stumbling block here is the recompile of php required for System V semaphore support since it isn't enabled by default. Administrators who want to use the module as a 'sharepoint' replacement will insist on this feature.

Wrt comment #7, breaking the module into multiple modules doesn't seem like a simplifying exercise. Integration is a good thing but sometimes it just isn't perfect - Civic crm is a case in point. A D7 guru should step in to comment on how changes to the file api can be leveraged by WebFM.

A missing feature that keeps us from using webfm for certain pages with dozens of files is a multi-attach and (and maybe a multi-href-insert).

Attaching 30 files one-by-one is nothing you would like to to because for every file you need to scroll back and forth if the list is too long.

And I still hope for http://drupal.org/node/285455 to get included.

Except from that, webfm has all I need ;-)

I am afraid Frank has a point: 1-3 are going to take a long time :-)

I think a useful feature would be an alternative way to attach files, that is having a 2-pane page with the list of the nodes on the left e webfm on the right. The files are attached by dragging the files from webfm to the right node in the left pane, more or less the same that now happens with folders...

I am not an expert of javascript: do you think such a feature is implementable?

I would like to thank robmilne for a great module and I am very glad this module looks like it will be supported in the future by cweagans.

I agree that #1-3 should be the priority and are not trivial.

I also share concerns expressed by Niels Hackius when he says "Actually I oppose a the complete rewrite: It will take a long time to be done - the normal tree will hurt from this... My suggestion is to gradually improve parts of webfm (small tasks) instead of taking on the whole thing at once: "

I would also like to mention that in the planning for this project by the time #1-3 are done Drupal 7 will likely be breathing down on us all. I am concerned if a complete rewrite is tackled that we will all be waiting until long after D7 is released before we have the rewrite of webfm for either D6 or D7.

Planning for the future of the product is great, but if it leaves the existing branch of the code high and dry without support on D6 and without a working port for D7 (and I think this is very likely to happen with a rewrite), the community of exiting users will really suffer.

Please be wise in considering how much you tackle at once, and include a useful port to D7 in the planning for early next year.

help~

webfm the file upload, file names will lead to problems in Chinese

webfm can add [Default permitted file extensions ]

Posted some thoughts over here: #604018: Going Drupal 7 and changed my mind:

If we are going for D7 this module must be rewritten in huge parts.

Some thoughts about WebFM rewrite, from a user.

1) I only want my files on my server, nowhere else. As it is is fine.
2) Indexing is bothersome, but being able to upload to a directory via FTP and then having WebFM index things effortless is priceless.
3) I wish it were possible to download all files in a range of directories via zip. Maybe there could be a feature that would add files to a queue that could then be downloaded in a file manager. I know, pipedream.
4) The #1 request actually… it's difficult for users to drill down a tree and then get deposited back up to the top of the tree.

It's a great module though. Many, many thanks from my students and myself.

I like the look of this.
Any chance this could be connected to a hosted Sharepoint service ??

I'm building an Intranet for a firm that uses Sharepoint.
having the ability to include in nodes to the sharepoint server would be highly beneficial.

Hi Cameron,

First of all, I should thank you for the work already done on this module. It really is an excellent piece of code.

I was wondering if there is any way (existing or that could be integrate) to make a record of who downloads what? I would quite like to be able to monitor the activity of the files that I am hosting.

Thanks in advance for your time and good luck with all future endeavors,

Dan

I'd like to echo the thanks and appreciation for those who have worked on this module! Great work! I look forward to the changes discussed.

@devondan - http://drupal.org/project/webfm_statistics is great for seeing who downloaded what...

My $0.02:

I seem to have a lot of download failures or partial downloads. I don't know if that could be something to do with the module, or if it is my host/ISP etc. If there would be anyway or tricks to help down loaders make sure they got the whole file it would be great.

I would also like to see Amazon s3 integration in such a way that the files can be protected behind registration and/or one time (emailed or displayed) links provided for the hard addressed s3 files.

The goals of your rewrite are mostly achieved by http://drupal.org/project/media already in D7, so maybe a better approach would be to work on a migration path from D6 WebFM to D7 Media module, getting WebFM plugins ported, etc.

D7 offers proper support for other file sources (S3, etc.) in core (pretty sure anyway), so at this point I don't think it's really worth trying to write all that again. If nothing else, you could use the Resource module from the D6 Media branch.

Media isn't ready for general use yet, but looks like it will be by the time D7 is usable.

SUBSCRIBE :O)

project file browser has download of directory as zip. I use webFM for upload and offer file browser for download. Both work together nicely.

Hi,
Changing to php5 is not a problem, my hosting provider runs php 5.1.9 and don't want to update for now, I hope it is anough.
I'd like to have the capability to define a default action when I make a left click to a file, attach to node will be great.
The counter should be visible not only into the metadata.
I really thank all authors and contributors to this module.

+100 on wysiwyg-api integration.

+100 on multi-attach.

Thanks for the right-click comment. Never occurred to me to look for anything other than cute little icons...

Is there a way to do plugins for the storage media, with the typical contribs directory? Might make for a more incremental approach than complete rewrite?

Thanks for this module, all. It's clean, rugged, simple, and easy to use. Please don't change that.