We've implemented a stack of filters on some images for a project we're working on and have noticed that suddenly the alpha transparency filter was no longer being applied to the images. After following the above message found in our error logs, I see that on line 196 of coloractions/transparency.inc there is a comparison of ($width * $height) > (1200 * 1200). If the image is larger than 1,440,000 pixels, the processing is aborted.

I was wondering why the limit of 1.5MP was chosen; if it is arbitrary or necessary based on functions that follow? Incidentally, when I swapped it out for 2400 * 2400, everything worked as expected again, so it does feel like the 1.5 megapixel limit was chosen as an arbitrary safe ceiling.

Any insight would be appreciated!

Comments

kaidjohnson’s picture

kaidjohnson commented

To put the issue into context a bit more, we're also using the retina images (http://drupal.org/project/retina_images) module. Our images are targeted to display at 940x444 but retinafication crops them at twice that, 1880x888 or approx 1.6 MP, just above the 1.5 MP limit.

Here's the dev instance of the site with image filters working properly (scroll to the accordion near the bottom) http://www.stsarc.org.php53-13.ord1-1.websitetestlink.com/

Thanks!

fietserwin’s picture

fietserwin
Location Le Mont-Dore
commented

This indeed looks like an arbitrary limit. The real limit here is probably in memory usage which leads to a nasty wsod. Nasty as this also leads to many support requests.

I'm in favor of removing the limit, or (@dman) are there any reasons for this limit (that then should be documented in the code)?

dman’s picture

dman commented

Indeed, the number is arbitrary and just there as sanity-protection, and was set in the days Drupal could be expected to run on 8MB
#1087570: D7 status? Brightness and Alpha not working?

However, the failure case was that IF a image was uploaded that was too big for the resources of a shared hosting account, it would try the expensive action again repeatedly, which badly hurt the small players who can't diagnose why their site keeps crashing. I didn't like having a module that could demonstrably and trivially kill someones live server, based on somone using an image just a bit bigger than another...

Retina things are a new field, and we've got more mem to play with, so I guess it should be either OPTIONALLY throttled (on by default) or maybe parameterized if we care enough.

FWIW, 1200x1200 was the largest sane screen size in those days, and putting anything higher on a website was extravagant...

kaidjohnson’s picture

kaidjohnson commented

That all makes a lot of sense. At the very least, implementing a drupal_set_message() when the process decides to abort would be useful so power users can be made aware of the issue and adjust the settings for the image output.

I'm not convinced that simply removing the limit is a good idea - if there's a risk of OOM errors, there's a responsibility to take reasonable care in avoiding that circumstance.

Most threads I've read recently point to a minimum of 32MB for D7 sites, with a recommendation of 128MB (see http://drupal.org/requirements and http://drupal.stackexchange.com/questions/29546/what-is-actual-drupal-7-...). I'm not sure what the usage implications are of the function in question, but if 1200 * 1200 was a safe threshold for keeping within a target of 8MB, I would be comfortable saying 1920 * 1080 (a now-common upper limit of screen resolutions) would be fine for 32MB and even venture as far as saying 3840 * 2160 (a soon-to-be popular resolution, aka "4k") would be adequate with 128MB.

I would propose adding a hook_requirements() (http://api.drupal.org/api/drupal/modules!system!system.api.php/function/...) to this module to define a minimum memory limit of at least 32MB. I don't think its unrealistic given the server requirements many fully-packed D7 sites have and that image processing, which this module does a ton of (and does it well, I might add), is resource intensive under the best of circumstances. Then resolution limit(s) can be safely increased.

Another solution may be to inject a warning when building out the image style if no crop function has been implemented above it to notify the user that the use of the transparency filter on overly large files may not work or fail.

Is it possible to use http://php.net/manual/en/function.memory-get-usage.php along with ini_get("memory_limit") to only abort the processing as the system approaches the upper memory limit of the server?

I'll play around with some of these ideas as I find some time in the next few weeks.

Thoughts?

fietserwin’s picture

fietserwin
Location Le Mont-Dore
commented

If you could try to get insight into the memory usage of this algorithm, that would be nice. E.g. do a get memory usage in the outer loop and see how it increases.

BTW: a "4K" image has about 12M pixels, which is (8 bit color depth + alpha) = 36M bytes. During processing there is the old an new image, so 72M should be available during image processing.

another option is to see if Imagemagick offers a similar operation. Imagemagick runs in a separate process and is not bound by the PHP memory limit.

kaidjohnson’s picture

kaidjohnson commented

There are some interesting variations on 4k (http://en.wikipedia.org/wiki/4K_resolution), depending on how you define it. This megapixel calculator offers some insight into what types of filesizes we could expect (http://web.forret.com/tools/megapixel.asp?title=Red+One+(4K)&width=4520&height=2540) on the high end of images in the foreseeable future. Obviously, filesize largely depends on the image itself, but reasonable estimates should help establish reasonable limitations.

I'll do some testing over the next week or so and see if I can come up with some concrete numbers to work with.

fietserwin’s picture

fietserwin
Location Le Mont-Dore
commented

But when processing an image in a program it is not compressed and can have an alpha channel, so use 4/3 * BMP size for estimations of internal memory usage...

dman’s picture

dman commented

Clearly it shouldn't be our problem to decide the optimal size for everyone, and back-of-the-envelope calculations only give us ballpark figures.
We don't know what memory size every user will have, and we shouldn't have to care most of the time. We don't need to guess what the upper limit is and then push for that, the vast majority of uses have fit inside the limits so far. In short - the actual numbers don't matter, and will matter less next year.

What we need to do (for this OP) is just provide the ability for an admin to lift or ignore that restriction if they want to - if they've got a dedicated machine and understand the issue. But leave the training wheels on for the majority.

Historically, any power user that really wanted to spend time crunching hi-res images *chose* to switch from GD to imagemagick for performance reasons. That was a sane choice. Imagemagick processing should NOT have been throttled by us with the size check. If it was, that was accidental and should be an allowed exception.