Save two identical file on the server is a waste of storage, it's good to keep the data/file overall size compact.

Comments

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Active » Postponed (maintainer needs more info)

We have already made steps towards doing this in 7.x-2.x but you can't do anything if the user uploads the same file twice as they could have the same name but could technically be two different files.

Do you have anything specific where we need to address?

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Also, why wouldn't you be using the Media Browser at the point that the file has already been uploaded?

comicat’s picture

comicat commented

My thoery is : If two files have the same MD5 hash, they must(almostly) be identical in content regardless what file names they got.

Refer to this function:

http://www.php.net/manual/en/function.md5-file.php

which could generate a MD5 hash of a file.

We could use that to avoid file duplication.

aaron’s picture

aaron commented

that will work fine until you're looking at a video delivery site where they're working with 8GB files...

Jackinloadup’s picture

Jackinloadup commented

@aaron I suppose a remedy to the huge file size issue would be an option to allow fields to be optionally marked as "require unique" content. This seems really valuable to sites with content editors who "dont care" or "dont know better" and just upload the same images over and over. Though I see your point with the video. At the same time it seems that would only be more valuable for a video site. Maybe if the MD5 was done in the background?

I'm not sure what the effect of that would be. From the name I provided it would sound like it will error if the user has already uploaded the file but maybe it should be more like "alias duplicate content"

Then making some kind of way to alias one to another. Possibly with multiple file_managed entries with the "newer" content stating its usage of the original in the file_usage table.

Or maybe when aliasing needs to be done the original fid (A) could be duplicated to create (B) then (A) and the new "file" (C) will be aliased to (B). If any more duplicates show up they will easily be aliased to (B) and you can delete (A), (C) or others with out affecting the each other.

Be warned, I have no idea as to the plausibility of this idea nor do I have the time to dedicate to it. Long live the do-acracy. ^_^

P.S. Thanks for all the hard work you guys do!

comicat’s picture

comicat commented

Or we maintain a table to dedicatedly store the file entity status, any usage of the file in other table will always reference the file_id/file_md5_hash:

file_id/file_md5_hash(unique) | file_info... | file_status (value may be 'in use[file_id is being reference by other party]' / 'orphan[literally pending to delete]')

l0calh0rst’s picture

l0calh0rst commented

I'd second the need of file uniqueness. Make it optional so the admin can configure what files the uniqueness applies to: mime type, filesize, file extensions etc. The argument with large files is valid but I'd say that the overall majority will not face that problem so why not make them happy?

Upon upload the md5 hash could be calculated and if a file exists just reference the existing file instead of stopping with an error. Clearly the user intended to reference that file to the content so this should be done for him. Optionally an info message could be given.

aaron’s picture

aaron commented
Project: D7 Media » File Entity (fieldable files)

moving to file entity

mfer’s picture

mfer commented

I've got a few thoughts on this...

  1. Take a weak web server and a large audio file (like a podcast) and you may start to run into performance problems. Or on shared hosting. It's not just video files that are gigs in size. This needs to be written smartly. Maybe on by default for images and nothing else. But, this is a media settings that can be enabled per type.
  2. If someone uses cloud storage (think Rackspace or Amazon) and stream wrappers this performance hit could be huge and even costly. Imagine pulling down (using bandwidth) these files to local copies to get he hashes. This could effect images as well. So, this needs to be written to take this into account.
  3. Looking for duplicate content thinks very much about using this as a media library. What if media or files are used differently. For example, a flickr clone. Different users may upload the same image multiple times. This could happen with fields as well. file entities are not just for the media browser. That's why there is a separate module.
  4. Does someone want to write a patch or another module to get the code rolling? I wonder if http://api.drupal.org/api/drupal/modules--system--system.api.php/functio... could be used as a start.
dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented

Yeah this is why people shouldn't be uploading - they should just be referencing the file externally/remotely whether it's a large audio file uploaded to S3, or a picture on Flickr. We can easily prevent duplicates if people don't actually 'upload'. Otherwise we're kinda forced into a bad performance situation if this feature is added.

dave reid’s picture

dave reid
he/him
Primary language English
Location Nebraska USA
commented
Status: Postponed (maintainer needs more info) » Closed (won't fix)

As this could likely be implemented in a separate contrib module since it's not an appropriate "core" feature for File entity, I'm marking this as won't fix.

PatchRanger’s picture

PatchRanger commented
Issue summary: View changes

Looks like there is already a module for that: https://www.drupal.org/project/filehash . According to its README, it is supposed to be used together with https://www.drupal.org/project/fe_paths .