Core interfaces can go over max_input_vars

This problem (or one very like it) drove me mad for a whole day. Eventually I tracked the problem down, using Firebug and lots of Google. The series of "is not a function" messages turned out to be something of a red herring. In actual fact, the problem was caused by a truncated Ajax request. The fact that the Ajax data was being truncated was causing Drupal to try to load an incomplete page; hence apparently missing files.

There could be several reasons for an incomplete Ajax request. In MY PARTICULAR CASE, fixing these settings in my php.ini solved the problem:

max_input_nesting_level = 1000
max_input_vars = 100000

This is because the form that the Ajax was being called from was huge, with lots of field collections within field groups. So I think it hit a limit either with its nesting or with the number of variables it could handle. Setting these to larger values allowed all the Ajax data to get through.

Various other settings that could possibly cause this problem include settings like post_max_size, publish_max_size (but not in my case).

EDIT: important note - initially I set the above variables to 0 assuming that this would remove the limit as is the normal convention. This was not in fact the case and caused ALL forms on ALL sites on my server to fail on submission.

I should also note, for anyone using Suhosin, you'll also want to set the following variables in your php.ini:

suhosin.post.max_vars = 100000
suhosin.request.max_vars = 100000
suhosin.post.max_value_length = 1000000
suhosin.request.max_value_length = 1000000

Obviously your exact values don't have to be the same as mine. The point is, unless you do something about it, by default Suhosin is going to chop off your form submissions if they're too long, and not tell you about it. This is a problem I've seen with other very large forms, e.g. the User Permissions form on a site with a very large number of modules.

This seems like more of a Drupal core bug. Should those values be included in our default settings.php file? With many more modules opting to make use of ajax, it would be nice to accomodate them out of the box. Does this need documentation somewhere too?

Related issues:
#1517092: admin/structure/menu/manage/MENU_NAME cannot be submitted on PHP 5.3.9+ for large menus (>~1000 items)
#1870014: Admin Interface Fail: PHP max_input_vars & suhosin post variable limit with a node with lots of fields (50+).
#329056: Unable to save changes to menu

There's lots more in contrib regarding this issue.

Tagging.

Edit:
V--- not sure if you intended to remove the tag, when you bumped priority. Would be useful to go through contrib and find similar issues.

Need steps to replicate. It sounds like it's this particular setup that causes an issue. If it's a php limit it's probably because something is going overboard with the fields/settings/whatever.

I wouldn't say it's a core issue, but if it can be reproduced, why not.

Actually, is that related/duplicate of #1517092: admin/structure/menu/manage/MENU_NAME cannot be submitted on PHP 5.3.9+ for large menus (>~1000 items)?

@_nod: its easily reproducable. Please check out my.explanation into why this is happening here.
#1870014: Admin Interface Fail: PHP max_input_vars & suhosin post variable limit with a node with lots of fields (50+).

Yes, you are most likely going to need to go a little overboard with something before you run in to this problem.

If you created a page with 1001 unique HTML id tags, then attempt to perform any Ajax request and your limits were set to 1000, my bet is query would get included twice and cause problems in the request.

In chrome inspector check the post fields to /system/ajax . Once the number of those goes over 1000 you'll start running into ajax issues.

While this is a configuration issue, if possible we should try and reduce the Ajax post fields if possible. It would help with speed as well. Otherwise we need to document the problem and make people aware what's going on. Gonna be pretty embarrassing for drupal to have to tell people to bump up their max post cars to over 1000 because drupal can't figure out a more elegant solution to this problem. For me I've had this issue for months and its very hard to debug.

Sorry, didn't mean to remove the issue tag!

Thanks for the great documentation so far. Is there a way to reproduce this bug using only the latest Drupal core? That may help clarify where the problem exists.

Well, in D8 the ids are in 1 variable only. It's a space separated string of ids so that shouldn't happen because of it (and I think there is a D7 patch waiting in the issue).

So I guess it's something to add to this: #956186: Allow AJAX to use GET requests.

@nod_ There's no mention of suhosin in issue #956186, so it's something that should get brought up there. The suhosin log is usually added to /var/log/syslog, which is probably why most people don't notice these problems. I would assume they happen more frequently, but are not getting reported because this is not a log a PHP developer usually checks.

For D7 a long path to a javascript file could potentially also trigger something like post.max_name_length as well. In my install for instance, I believe these should of been getting triggered as well...but it seems to get passed them despite vars like
"ajax_page_state[js][sites/all/modules/contrib/context/plugins/context_reaction_block.js] : 1" being 88 chars long.

If you search a default suhosin.ini for 'post.max_' you'll find some other variables which we need to keep in mind.

;suhosin.post.max_array_depth = 100
;suhosin.post.max_array_index_length = 64
;suhosin.post.max_name_length = 64
;suhosin.post.max_totalname_length = 256
;suhosin.post.max_value_length = 1000000
;suhosin.post.max_vars = 1000
;suhosin.post.disallow_nul = on

@nod_ do you think it would be wise to write a test for core, which would test the values in $options against suhosin defaults in `ajax.beforeSend` in misc/ajax.js

Would be a good test, or at very least a warning to see if an ajax request would ever overflow suhosin / php defaults.

Yeah that sounds like a plan. I'm happy to make the ajax call fail and throw an error if it goes over the limit.

@nod_, we'll this is technically a PHP configuration issue, so a hard fail would cause more harm than good. Except if you were detecting the settings and they were in fact going to mess up the request.

But some kind of notice needs to happen so this doesn't go under the radar.

I've seen a Drupal 6 permissions screen go over max_input_vars as well, in the UI it just failed silently (i.e. any vars over the limit just get ignored but the form still submits).

We could potentially do something in PHP to detect if the number of vars == max_input_vars then throw a warning although it's a shame that PHP itself doesn't throw an exception or similar.

Configuration issue though so downgrading from major.

Additional data

I believe I just ran into this problem on D7 using the Menu Editor module #1032270: Menu not changed, no error message displayed (max_input_vars). I tried to make some changes to my main menu and the form appeared to submit as expected, but none of the changes were saved. I realize this is technically a configuration issue, but what concerns me is that there is no error, or visual indication of why it isn't working. If I hadn't stumbled upon some existing issues, I never would have figured this out.

We could potentially do something in PHP to detect if the number of vars == max_input_vars then throw a warning although it's a shame that PHP itself doesn't throw an exception or similar.

I think this would help, yes. The counting could happen after all hook_form_alter() stuff has been applied.
Then maybe show a message and write to watchdog? Or remember the form ids and url of big forms, and show this on the requirements page?
In my experience, the watchdog easily gets easily clogged up with PHP strict notices, or file permission stuff. So for things where a straightforward solution is available (change your PHP configuration), a report on the status/requirements page seems more useful.

The following line in misc.js will cause a huge request array:
options.data['ajax_html_ids[]'].push(this.id);

It can be reproduced by creating a multiple field on a node with 1000 instances of the field (or a number of more complex fields like ckeditor textarea).

This is a real issue, especially if you run a site with huge forms, which is sometimes a necessity. In a perfect world 'ajax_html_ids' should be killed with fire, along with the dev that came up with such a horrible solution.

But since stuff has grown dependant on this it might actually make more sense to bypass the max_vars settings simply by letting drupal pass the post data from the input stream instead, something like:

$_POST = parse_str(file_get_contents('php://input'));

in the bootstrap would bypass max_vars for posts entirely.

@EJTH developers come up with horrible solutions all the time. Especially when they're working for free on a volunteer open source project I don't think that deserves the death penalty.

Killing of drupal_html_id() though is happening in #1305882: drupal_html_id() considered harmful; remove ajax_html_ids to use GET (not POST) AJAX requests.

Adding #1203766: With large number of permissions /admin/people/permissions becomes unusable

Here's a hasty patch (without tests) that provides feedback for the core permissions form.

It's perhaps possible to generalize this in FormBuilder::doBuildForm; after any #after_builds have run, one could in principal determine how many form elements would be generated. It might not always be accurate, especially after contrib gets involved, and it won't be beneficial in the vast majority of cases. I think it's better to address this on a per-form basis. Maybe add a helper function on FormBase to keep things DRY.

Bumping this to critical, it's a long standing bug, but it's silent data loss with no workaround.

I'm posting a patch here for D7 based on work we've done to solve this issue in Backdrop CMS. It will need to be reworked for Drupal 8+, but I'm hoping it might still be helpful.

Attached patch converts @Jenlamptom's d7 backdrop patch to 9.2.x.

I first tried to move this to form validation, before I realized that it needs to happen here, before form validation, so that we check the input variables and display a warning /BEFORE/ the form is submitted.

I simplified some of the messaging, so that we can further discuss it messaging here. I think it would be nice to link to a d.o. page about why/how to increase this, the backdrop patch had that. Should we add logging?

I'm not sure if adding StringTranslationTrait, MessengerTrait to FormBuilder has any drawbacks...

This still needs a test :(

Updated patch, fixes PHP fatals using SubformState (errors noticed during manual block placement tests)

These errors highlight even more so, that this still needs a test...

Updated patch fixes missing trait in test.

Updated patch adds the new methods to FormStateDecoratorBase, although I'm not sure what this is used for.

Patch from #35 still applies to 7.80.

Sadly the return from TestBot Build Successful doesn't mean all went well, in fact, there are more errors then could be shown at once.

If you click through you can see them, like for the last test..

Back to Needs work.

Attached patch is a minor tweak, to limit the hook_requirements() check to runtime, because most systems use different settings for commands line and runtime, and this check is really meaningless for the command line (which doesn't have form variables).

This patch contains a proof of concept for another approach: allow forms to be marked as "large", if a post for a large form contains more input vars than max_input_vars, parse the raw payload of the request to retrieve all input vars.

What do you think?

#45 did seem to stop the max_input_vars error from appearing when editing the Management menu.

I'm not sure about #45 as we're silently bypassing a security feature added by site admins.

The backdrop approach of showing a warning seems much better

The long term fix here would be to redesign the forms so they don't go over this limit, but that was already being discussed before 2012 when this issue was opened and it doesn't look likely to be resolved imminently.

Given how intractable this is, and also how silent the failure is when it's a problem, I think it would be a good idea to add warnings in a couple of different places:

1. In hook_requirements() when max_input_vars is quite low. We could either multiply roles * permissions and warn if it's more than 80% of max_input_vars, or set some arbitrary amount. This could link to a change record.

2. Similar checks on the user permissions and/or menu links forms themselves, not sure what the form-specific wording should look like, it might want to link to the status report maybe?

This would help to resolve the data loss issue, we can then have a follow-up to remove the warnings when we've resolved the form design issues themselves. For example with the user permissions system, this could be a hierarchical permissions system, or AJAX-ifying the permissions form to write to temp store + a confirm changes button instead of submitting everything at once.

#1 and #2 could be split to their own issues and maybe convert this to a meta/plan.

Tagging for issue summary update, a bit pre-occupied at the moment but might be able to tackle that +spinning the issues out later if no-one beats me to it.

Updated the issue summary with new plan