file.inc: file_upload_max_size() silently cuts size in half

Agreed, this was added by http://drupal.org/node/72545 and acknowledged to be arbitrary.

http://drupal.org/node/72545

Seems to be closed without this issue being addressed. It's also discussed here:

http://drupal.org/node/100373#comment-228710

As far as I can tell, this is still a bug in 5.1.

I think it should do a sanity check at 90% of max post size, not 50%. Or, at a minimum, the error message should report what it is doing.

Yes, this assumes too much for a sanity check.

Also, using a percentage for image vs (images+text) and even making it 50% does not seem so sane to me. It is fairly typical for Images to take up most of the size. It would be more reasonable to *subtract* 0.0001M (50-100 Unicode characters) from the post size. Or just -1M or 0.

If the intention was to allow uploading two almost-max-size images in a post, this is pretty arbitrary as well.

Since I am not a php developer, I am also curious to know whether attaching multiple files to a node is a single POST operation or not.

Also, In addition to my objection to using a percentage

according to http://www.php.net/ini.core
- a sanity check should also include php memory limit
- there may be a way to check for an error and report it

Check this issue I posted a while back for upload module silently failing: http://drupal.org/node/30520.

We *definitely* need to check ini_get settins and provide an accurate error. Silently failing is just so lame.

Here's a patch to remove the 50% limitation.

Patch works as described. It's also the correct behavior. We're being way too conservative with the /2. It'd always annoyed me that core got this wrong.

Hm, I am trying to understand what this limit is trying to prevent at the first place. I see pwolanin said this when he created the original patch:

My chopped off post above tried to say that I picked an arbitray cutoff of 50% of post_max_size as the upper limit for the size of a single upload. FYI, the PHP defualt values are post_max_size = "8M" and upload_max_filesize = "2M" (corresponding to 25%). However, a value of 80% or 90% might be fine too- please review and provide feedback so this can get finalized.

How should these relate? I don't see.

The only restrictions mentioned are:

http://gr2.php.net/ini.core

Sets max size of post data allowed. This setting also affects file upload. To upload large files, this value must be larger than upload_max_filesize.

If memory limit is enabled by your configure script, memory_limit also affects file uploading. Generally speaking, memory_limit should be larger than post_max_size.

When an integer is used, the value is measured in bytes. You may also use shorthand notation as described in this FAQ.

If the size of post data is greater than post_max_size, the $_POST and $_FILES superglobals are empty. This can be tracked in various ways, e.g. by passing the $_GET variable to the script processing the data, i.e. , and then checking if $_GET['processed'] is set.

I assume that post_max_size refers to a POST operation.

That patch was supposed to be a sanity check. I am unsure whether it prevents or reports any error, for example when you upload 3 files, each one 50% (is that a single POST operation).

My feeling is that "-1MB" would be sane enough, but error reporting would be even better.

The sanity check also confuses users who have set both to 8M and then look at /admin/settings/uploads and see: "Your PHP settings limit the maximum file size per upload to 4 MB."

When I first saw that I was not sure what I needed to do. I thought I was all set for 8MB.

as rusty.cog points out the current formula is very confusing and incorrect. the post_max_size setting affects the total amount of data posted including the file. the upload_max_filesize setting limits the size of each individual file.

we can't provide an exact value of the maximum upload size because there are many factors that play into it. if more than one file is being submitted then each one must be less than upload_max_filesize, and the total size of the files and other data that's being POSTed must be less than the post_max_size.

i'll provide a few examples using the following settings:

you could submit:
a ~1mb node body and ~1mb file.
a 0.1mb node body and two ~1mb files

you could not submit:
a 1.5mb file (upload_max_filesize - would report an error on the file)
a 0.5mb file and a 1.5mb file (upload_max_filesize - would report an error on the second file)
a 3.5mb node body (post_max_size - which would discard the entire POST)
four 1mb files (post_max_size - which would discard the entire POST)

now these numbers aren't exact because there's some encoding overhead in the POST that can fudge it.

#6 still applies and IMHO is the correct way to treat the upload limit.

Agreed that this size restriction was arbitrary. Committed #6 to Drupal 6-dev. Needs to be backported.

backport

Committed to 5.x.

note - see my quote in #9 - it went in as 50% since no one reviewed it and suggested otherwise...

The main bug being fixed in that issue was that no information about the PHP limit was being shown in Drupal.

pwolanin, it sounds like you're looking for: http://drupal.org/node/30520

Automatically closed -- issue fixed for two weeks with no activity.

How come this is closed?

EDIT for #21: Precisions:

I would have liked to upload files as large as DVDs, for archival purposes, as strange as this might sound. And I can't.
My config is:

- No particular restriction in .htaccess nor settings.php in my drupal directory regarding memory, post and upload limits. Time limits where pushed to about 50 minutes.

- php.ini :
upload : 5000 MB
post : 10000 MB
memory : 512 MB

With a 872 Mo file, upload is OK, but no more.

I've already read related:
#104220: file.inc: file_upload_max_size() silently cuts size in half
#72545: upload sizes limits are not validated
#30520: file_save_upload() not notifying user when PHP upload limit is exceeded.

But no luck!

EDIT for #22: Thanks, great topic! I think I found a workaround. Follow the link of that post.

doc2@drupalfr.org, it's pretty obvious why it's closed if you read the comments. the patch was committed to D6 and D5... there's nothing more to do so the issue gets closed.

Uploading very large files is a very good request, but I am afraid it is unrelated to this issue, which was about fixing a problem. There are applications out there which slice files and upload them in chunks, or other methods (third party services, ftp solutions etc). See also this forum thread:

Upload very large files
http://drupal.org/node/129138

You could start a new issue with a feature request, but experience says that the way things are done in Drupal, this kind of new functionality would start as a contributed module if someone got interested. So, the best chance is to somehow stir interest in that direction.

A small comment to #20: A reason that those php.ini settings didn't work may be that, according to what http://php.net/ini.core says about post_max_size, "Generally speaking, memory_limit should be larger than post_max_size", which of course is unrealistic.

Yes it is unrealistic!

My memory_limit was 256 MB and is now 512 (2 GB of physical RAM on our server, still 1 GB free), but I've still been able to upload files up to about 700 to 800 MB respectively. This throws troubles on relations between my specs and possibilities according to those PHP ini.core rules... We just set 5 GB to upload_max and 10 GB to post_max for not being as less restricted as possible.

Moreover, our server (OS X dual G5) uses about 70 GB of virtual memory, but I guess files can't be "cached" there. I was wondering if having 4 to 6 GB of server RAM and almost as much dedicated to my scripts in memory_limit would change anything, if not completely unrealistic!

Anyways, I think I found a workaround for my videos.

I've been wondering about this for ages and thought it was a PHP configuration problem. Thanks for fixing this, folks.