drupal_process_form() deletes cached form + form_state despite still needed for later POSTs with enabled page caching

Just came across the very same problem and looked around on how I could fix it. Nothing in ajax_register turned out to be the culprit for this so I tried it on a site without ajax_register installed and found the very same problem on the user login page. That looks to me like a Drupal core bug.

Now loocked a bit deeper into this and this is most certainly a Drupal core issue in relation to forms being submitted through ajax.

Here is what happens:

When page caching is turned on and if we grab the form for user login, we always get a form for each anonymous user with the same form_build_id. This can be reproduced when you use two different browsers on the same PC and call the same page www.example.com/user. You can now have a look into the html source and will see that both forms come with the same form_build_id.

If both users from both browsers are using that form to login, this seems to be working fine - which puzzels me, but that's a separate issue.

However, if the form was configured to be submitted through ajax, then the second one (and all subsequent ones) will fail. This is because the form will be deleted from cache as soon as it has been submitted and then the subsequent submission from user 2 will be rejected in ajax_get_form() because the form can't be found in cache anymore.

Consequently I ask myself two questions:

1) Should any form ever be delivered twice with the same form_build_id? I guess not.

2) How can we avoid delivery of the login form twice with the same form_build_id?

Looking for advise, this is really serious I guess.

Reassigning the issue as I believe this is Drupal core.

Can you write an issue summary? It's not clear how this is a core bug.

Just updated the Issue summary

Any feedback on this one? It is causing real pain!

Same problem here.

Two identical drupal installations, ajax enabled login form, site A no caching, site B caching, on site B when i log-in->log-out i can't login back, the ajax page is called but no response, just watchdog error popping.

Help...

I'm also encountering this issue, not fun.

I think that this problem deserve a critical categorization.

This still needs a clear issue summary.

Ok I'll try to be clearer.

Drupal 7 installation with an AJAX enabled login form.
Starting situation (just cleared the cache).

Login form_build_id "form-sqXHwJympdAcZGTGIeHaZgsWGBfcE1ubCVH_-m1CsDo"
Form working as expected, ajax request sent e reply received.

Then I logout, refresh the login page.

Login form_build_id "form-sqXHwJympdAcZGTGIeHaZgsWGBfcE1ubCVH_-m1CsDo" (the same as before?)
when i click on login an ajax request call /system/ajax but an empty response is given.

I put some breakpoints in validation and submit functions, but the code didn't pass here. The problem is earlier.
I guess that the issue is related to the fact that the form_build_id doesn't change even if I refresh the page if I enable caching on admin performance page.

If I disable the caching, everything works as expected, the form_build_id change every page refresh.

I tried to put the $form_state['cache'] = FALSE or $form_state['no_cache'] = TRUE key but it seem not to work.

To be more specific, everything works if I disable "cache pages for anonymous users"

The login form is in a block, on the navigation region, so it's on every page.
On the cache form table sometimes I find the cached form, that disappear on the first submit, and never pop again.
Sometimes instead the cached form is still there after the submit, but the valid token is always the same (so the validation check fails).

The problem is that on form submission Drupal run the ajax_get_form function (http://api.drupal.org/api/drupal/includes!ajax.inc/function/ajax_get_form/7) sometimes he can't find the cached forms, sometimes he found it but the form_get_cache fails on valid token check (http://api.drupal.org/api/drupal/includes!form.inc/function/form_get_cac...)
if ((isset($form['#cache_token']) && drupal_valid_token($form['#cache_token'])) || (!isset($form['#cache_token']) && !$user->uid))

Help still needed :(

Updated the issue summary to underline the fact that this seems to be a core issue.

I've just written an issue summary according to the template and hope that tim.plunkett can now take this forward.

make the login form being submitted via Ajax

That's not possible with a stock Drupal 7 install. Please either post steps to reproduce from a clean install of Drupal 7, or indicate the contrib or custom code that's contributing to this.

I've created a block, available in top region on all pages:

function mymodule_block_info(){
	$blocks['mymodule_login'] = array(
		'info'		=> t('My User Login'),
		'status'	=> FALSE,
		'weight'	=> 0,
		'cache'		=> DRUPAL_NO_CACHE,
	);
	return $blocks;
}

function mymodule_block_view($delta){
	switch($delta){
		case 'mymodule_login':
			$block['content'] = mymodule_block_userlogin();
			return $block;
			break;
	}
}

function mymodule_block_userlogin(){
	return drupal_get_form('user_login_block');
}

Then i've altered the login form

function mymodule_form_alter(&$form, &$form_state, $form_id){
	switch($form_id){
		case 'user_login_block':
			$form["#validate"][2]="mymodule_user_login_final_validate";
			$form['actions']['submitlogin'] = $form['actions']['submit'];
			unset($form['actions']['submit']);
			$form['login_error'] = array(
				'#type' => 'markup',
				'#prefix' => '<div id="login_error">',
				'#suffix' => '</div>',
				'#markup' => '',
			  );
			$form['name']['#attributes']['onkeypress'][]='if(event.keyCode==13){jQuery(\'#edit-submitlogin\').trigger(\'trigLogin\');}';
			$form['pass']['#attributes']['onkeypress'][]='if(event.keyCode==13){jQuery(\'#edit-submitlogin\').trigger(\'trigLogin\');}';
			$form['actions']['submitlogin']['#ajax'] = array(
				  'callback' => 'mymodule_form_login_callback',
				  'event' => 'trigLogin',
				);
			$form['actions']['submitlogin']['#attributes']['onkeypress'][]='if(event.keyCode==13){jQuery(\'#edit-submitlogin\').trigger(\'trigLogin\');}';
			$form['actions']['submitlogin']['#attributes']['onmousedown'][]='jQuery(\'#edit-submitlogin\').trigger(\'trigLogin\');';

		break;
	}
}

When I deactivate the page caching everything works, but if I activate page chaching the form works only once as said before.
Neither the validate neither the callback functions are called, the workflow exits earlier (https://drupal.org/node/1694574#comment-6546788)

This is not criticial unless there is prove that this is a core bug and not caused by your custom code or a contrib module. Please to not set it back to critical without confirming that.

Well, I can't really agree to your assesment here, @Berdir. The form API is from Drupal core and the Ajax framework is from Drupal core too. And both behave fairly different with regard to form submission and they cause severe issues as described in the issue summary: a form with the same form_build_id is delievered many times and as soon as it is submitted the first time, this makes all the other delivered form unusable.

This is at least a major issue if not critical.

Agreed, core doesn't come with a way to submit the login form via Ajax but it provides all the credentials and they don't seem to be throught through properly. Yes, it only happens with some custom code or a contrib module but as clearly pointed out, the faulty behavior is in core.

So what else can we do to get this dealt with other then what we have done already? I'm happy to work on this, so I don't want to put more work on anyone'S shoulders that are carrying so much already. But I need assistance in final analysis and decision on how to resolve this.

Yeah, as said by jurgenhaas, I want to collaborate to solve this issue, but all I can do is post the custom code I've created to submit via Ajax and as I said before, neither the form validation, nor the form callback are called, the problem pops before, on the form_get_cache core function ( http://drupal.org/node/1694574#comment-6546788 ).

I can't go forward than this...

Thank you for your work :)

I can't switch to a non-AJAX login, the customer do want a ajax response on the login form with a popup alert.... :(

I've published the website for my customer without chaching enabled...I hope the server can carry the load...

The reported bug sounds legit and sensible to me. Trying to clarify the issue title.

Attached patch implements the second solution proposal... but just now I realize that this does not fully solve problem, because:

The cache items will still be deleted when they are older than the expiration being set in form_set_cache():

  // 6 hours cache life time for forms should be plenty.
  $expire = 21600;

Thus, even with this patch, the first user who tries to submit the form after 6 hours will still see the error message, since the form cache items were pruned / garbage collected, but the page containing the form (including the form_build_id) is still cached; perhaps even by a reverse-proxy.

That inherently means we have a much larger architectural problem with the current form cache.

However, this patch should at least fix the discrete problem that has been reported here - which apparently leads to much more frequent broken form submissions than 6 hours. Therefore, we might want to move forward with this independently.

Writing a test for this is going to be tough, since WebTestBase still lacks proper support for concurrent user sessions. (@see #1378126: Simplify tests needing multiple, concurrent user sessions / logins)

I'm also attaching a patch for D7, but only for facilitating tests of the proposed solution with your current code. Please do not attach any further D7 patches until this issue has been fixed for HEAD/D8 first.

Closely related (and apparently conflicting): #343415: Form cache is not cleared on submit when page cache is activated

I encountered what I thought was this bug... I was submitting a user login form via AJAX with caching turned on. The first time it was submitted was fine, but from that point on it failed, which sounds the same as described above. I tried applying the d7 patch from #25, but that didn't solve my problem so I dug deeper...

Turns out in my case the problem wasn't the cache getting cleared (which made sense due to the following line in around the caching):

if (!variable_get('cache', 0) && !empty($form_state['values']['form_build_id'])) { ... }

It turns out in my case the problem was in form_get_cache(), specifically the $form['#cache_token'] was only valid for the first person to submit the form.

I'm not sure if this is more of the same or if it's a completely separate issue. Would love to help resolve it, but not really sure where to start...

Done a little more digging...

form_set_cache() has the following:

    if ($GLOBALS['user']->uid) {
      $form['#cache_token'] = drupal_get_token();
    }

When the form is rebuilt after a login, $GLOABLS['user']->uid contains the freshly logged in user, meaning #cache_token is set. drupal_get_token() builds a token involving session_id() which then invalidates the form for any anonymous requests that follow.

The only check form_set_cache() uses for whether it sets a #cache_token is $GLOABLS['user']->uid... Perhaps we need to bring in an ability for a $form/$form_state to stop it being set, which login forms make use of? I wonder if this is also an issue for registration forms.

This only happens with AJAX requests, I've not quite figured out why that's the case though...

Ok, so done a bit of playing. Here are patches that resolve the issue I was having with the user login via AJAX. I'm a bit unconvinced that this is actually the right way to resolve the bug, but it definitely confirms what the bug is...

In terms of tests, I don't think the problems with concurrency will be an issue. If caching is enabled, you can log in, log out and then try to log in again and that's when it'll fail. I'm not sure where the best place to put the tests are? I can't decide if the tests fall under ajax, cache, form or user... If you let me know what you think I'll write some tests for D8 and D7 which demonstrate the failure.

I'm also happy to update the attached patches for a better solution if you've got any pointers?

Oops, that was supposed to be do-not-test... I've attached a module (dependent on ctools for the AJAX redirect command) which demonstrates the problem in Drupal 7... You need to turn anonymous page caching on.

Sorry, forgot to attach...

Apologies for posting up d7 patches rather than d8 but a client needed the d7 meaning I had to write them and I couldn't get d8 working locally to try and get it all working in d8. I will try again when I've got time, but in the mean time hopefully the patches can get some discussion as to whether this approach is the right approach at all... I didn't want to mess with the issue title/summary until someone who knows the Form API better than me confirms what I think is happening...

Possible solutions

The problem is in form_set_cache() which, when there is a logged in user, sets #cache_token on the form. This then invalidates the form cache for any other requests, as ajax_get_form() rightly tells the rebuild to copy the #build_id.

I can see three possible solutions:

• Use a flag in $form_state to allow login submission handlers to prevent form_set_cache() from storing the #cache_token in the rebuilt form.
• Form API should detect a change in user logged in and only set the token appropriately.
• AJAX forms should not be rebuilt if they've been successfully submitted as it is unnecessary.

I've got a for the first in d7 along with a test which flags up the issue. I think I'm now as far as I can take this without having someone who knows Form API better than me giving some input.

This one made us think for a bit, heres my recap.

Problem: cached forms may reach a state where the build id is no longer valid.
Cause: The form build id no longer exists in the caching table because it is older than 6 hours and we've performed garbage collection via cron, removing it from the form_cache table.
Solution (not really): set max page cache lifetime to 6 hours so that we regenerate form build ids around the same time our build ids become invalid and are set to be deleted by cron.

Timeline of a broken form:
0 hour: generate page and form id, cache both.
+6 hours: form_cache's build id is now expired, but still valid (Drupal just checks if it exists in the cache table)
Sometime after 6 hours: cron runs, deletes expired build ids and breaks cached pages.

Note: on form render, Drupal will create a new build id if the current id is expired.

DaneMacaulay, there is actually another thing going on.

When the form is rebuilt after an AJAX submission, the form token gets changed, meaning even within the 6 hour time frame, any submission of the form will invalidate the cache as the form token is based off of the user (specifically anon & session id) which has changed.

it should works if you enable "cache pages for anonymous users".

it should works if you enable "cache pages for anonymous users".

dev.firoza unfortunately that's specifically what breaks it with AJAX requests...

If any user is submitting a form data using Ajax, form will not be cache if 'cache pages for anonymous users' is not enabled. after submitting user want to reset form data then he will get an error message Invalid form POST data.

I have written and deployed a form that makes extensive use of the form ajax api's on D7, and I have now started getting complaints in from users all over the world that they get an ajax popup that simply shows an ajax error with the response code as "200 OK". Looking in the logs, I also get "Invalid form POST data.". I traced the request / response using wireshark, and can confirm that the request is identical in both cases where it works and were it doesn't, however where it doesn't the response carries code 200 but with no body.

Any advice for a desperate D7 user who is now in deep trouble?

Ps, disabling the anonymous cache has significantly improved the situation, but I still get the odd error in the logs and the odd user calling. Add to that the fact that my server now works a bit harder, I really would love to see a fix here soon.

Same as the user above, I have been running into this problem in several areas.

200 OK Parse Errors, logs showing "Invalid form POST data."

This has caused me hours of headaches, I haven't experienced with previous installs.

I am experiencing this error if I have any ajax form on a node.

Example: 1-Multi-value field on node, can add unlimited fields and populate.
2-Save Node
3-Go To Edit, try to add or remove a field from multi-field. Get ajax popup error.

This also happens with addressfield when changing countries, thus refreshing the fields (after node saved).
This is when logged in as admin.

"cache pages for anonymous users" should not be applied to non cached page (ie : user/login) and so, form cache should act like when this option is disabled.
Am i wrong ?

The problem is when you have a login block that submits via AJAX. The submission rebuilds the cache with a new build id but the correctly cached page still has the old build id.

I have the same problem in user/login page too.

This is at least a major bug, for the reasons given by jurgenhaas, in post #20:

The form API is from Drupal core and the Ajax framework is from Drupal core too.

This functionality can be replicated with very little code (D7):

function mymodule_form_alter($form, &$form_state, $form_id)
{
  if($form_id == 'user_login')
  {
    $form['#prefix'] = '<div id="mymodule_user_login_form_wrapper">';
    $form['#suffix'] = '</div>';
    $form['buttons']['submit']['#ajax'] = array
    (
      'wrapper' => 'mymodule_user_login_form_wrapper',
      'callback' => 'my_module_user_login_ajax_callback',
    );
  }
}

function my_module_user_login_ajax_callback($form, &$form_state)
{
  $commands = array();
  $commands[] = ajax_command_replace(NULL, render($form));
  return array('#type' => 'ajax', '#commands' => $commands);
}

This ajaxifies the login form (note: the user login block can be used by changing the form ID from user_login to user_login_block). It works as long as page caching for anonymous users is turned off, but if it is turned on, the above code will not work after the first submission. The above code uses a core API, and fails when core functionality is enabled.

It's an AJAX internet, and an AJAX login is a commonly requested and required feature from clients. The above example is for the login form, but actually any AJAX enabled form will not work when page caching is turned on as a result of this bug, not just the login form. I myself have multiple multi-step ajax enabled forms on my sites for anonymous users, and none of them will work as a result of this bug. And it's not just me as seen by the number of responses in this thread.

This bug isn't critical, as it doesn't directly impede usage for core Drupal. However, it does represent a conflict/bug between a core API and core functionality, on a core-provided form. It is also AJAX related, and with AJAX being a major requirement for many sites these days, this bug should rate major. I've changed it accordingly.

Wrote a proper issue summary.

Today I got the same trouble as described in comment #40, I have a code in my module like that:

function Mymodule_register_form($form, &$form_state) {
  $form['userinfo'] = array(
    '#type' => 'fieldset',
    '#title' => t('User'),
  );  
  $form['userinfo']['email'] = array(
    '#type' => 'textfield',
    '#title' => t('E-mail'),
    '#required' => TRUE,
  );
  $form['userinfo']['password'] = array(
    '#type' => 'password',
    '#title' => t('Password'),
    '#required' => TRUE,
  );
  $form['userinfo']['passwordconfirm'] = array(
    '#type' => 'password',
    '#title' => t('Repeat password'),
    '#required' => TRUE,
  );  
  $form['actions'] = array('#type' => 'actions');
  $form['actions']['submit'] = array(
    '#type' => 'submit',
    '#value' => t('Register'),
    '#ajax' => array(
       'callback' => 'Mymodule_register_form_ajax_submit',
       'wrapper' => 'Mymodule-register-form',
       'method' => 'replace',
       'effect' => 'fade',
    ),    
  );
  return $form;    
}

function Mymodule_register_form_ajax_submit($form, &$form_state) {
  $element = $form;
  if(!form_get_errors()) { 
    //some code here never executes  
    $element = t('You have registered to system');
  }
  return $element;
}

Tried to submit this form as admin user (uid=1) so the page was not cached.
When cache for anonymous users is enabled:
Always get ajax error message on page with code 200 OK.
Looking in the logs, I get "Invalid form POST data."
But when cache for anonymous users is disabled everything works, ajax form submit is executed.

Welcoming this bug into the new year...

This is hitting me in a somewhat different way in D7:

* Load up a page with an Ajax-based login form like those described above.
* Let the page sit there until the form's entry in cache_form expires and is removed.
* Try to log in with the existing form -- without refreshing the page.
* The "Invalid form POST data" dialog will appear.

This is independent of the setting of "Cache pages for anonymous users".

Any thoughts about this might be addressed in the short term? I suppose I could add some javascript to the page that forces a page reload just before the form cache timeout, but that's pretty awful...

By the way, it should perhaps be mentioned that this way of handling the error, while helpful for developers, is pretty terrible from a user experience perspective. I'm trying to imagine one of my site's users encountering this dialog and trying to figure out what it means and what they're supposed to do, and it's not a fun exercise.

@DanT1252: FWIW, in the absence of any other solution, I put together the javascript-based "wait until just before the cache_form entry expires and force a browser refresh of the page" thing, and, for my situation, it seems to be working reasonably well. In case it's generally helpful, I did this, as part of some code that creates the page with the form that times out:

	// Set up the refresh to happen 60 seconds before the cache_form expires in 6 hours
	$timeout = 21600 - 60;

	// Make the meta tag
	$refresh_page_meta_tag = array(
		'#type' => 'html_tag',
		'#tag' => 'meta',
		'#attributes' => array(
		  'content' =>  $timeout,
		  'http-equiv' => 'refresh',
		)
	);

	// Put it into the page header
	drupal_add_html_head($refresh_page_meta_tag, 'refresh_page_meta_tag');

Better ways of doing this are of course encouraged...

https://drupal.org/project/cacheable_csrf has an approach to fix this - it replaces ajax_form_callback() for cacheable_csrf enabled forms to handle the cache differently.

This will work OK as long as there's not a multistep form, did not need to figure that out yet.

Hello, Still no solution to this?

There is no solution for this because no one has worked on it. If there was, it would be posted here.

@DanT1252

In the world of Drupal there isn't really a "core team". If you write a patch for this issue, and it get's in - surprize! - you are on the "core team".

In terms of the "assigned" field here - that's really only used when someone starts working on this issue, and they choose to assign it to themselves. It's useful because then you don't get two people working on the same issue at the same time, and wasting time or effort.

If you really want to have this issue resolved, you have a few options.

1) Add your support here and wait. You can chime in and say "man, I lost a Drupal site because of this, I wish it will get resolved" and someone - when they have time or interest - may solve it. They also may not.

2) Fix it yourself. This is how most of Drupal development works. If you or one of your sites has a problem, you fix it. When it's fixed you post a patch here, and change the status to "Needs review". As others have the same problem, they will test your patch and post feedback, or perhaps improve the patch along the way. That process continues until everyone agrees the fix is good and the issue is marked "Reviewed and tested by the community". Then, the patch gets committed.

3) Hire someone to fix it for you. If you don't have the time, skills, or interest to fix this issue, you can often hire someone to do it for you. if you have a favorite development team or contractor, you can call them up. If you don't know where to turn for help, you can sometimes reach out to people who are active in the issue you want resolved. Look at people here who have posted patches previously (since you know they have the skills), contact them privately, and ask if they are available to fix this issue. You'll find that people who are having the same problem as you are often willing to help fix it for an extremely reduced rates, since they are working for Drupal, and helping resolve one of their own issues too.

The one thing we really don't like is when people show up in issue queues and say "why isn't this fixed yet!". There are a lot of critical issues in the Drupal queue - it's not fixed yet because we're busy! Most of us are working on Drupal for free, and often in what little spare time we have. When we get that kind of reaction, it hurts. It feels like we're taken for granted, and everything we already do give - has gone unappreciated.

Some people may react badly or with snarky comments (ahem @tim.plunkett), but its usually those people who give the most. If you start by saying thank-you, you'll find that you get a better reaction from this already over-taxed developer community. :)

https://drupal.org/project/cacheable_csrf deals with this, including simple AJAX-enabled forms, but you need to enable that for each form that's going to be render cached.

Also #1191278: Simplify DX of removing form_id, form_build_id, and token from RESTful GET forms is related.

It seems the latest update is related to this (and in some extent maybe fixes it?) - am I right?

Correct, https://drupal.org/SA-CORE-2014-002 is the reason for the "silence" and why this issue did not see further progress/updates.

The change still has to be forward-ported: #2242749: Port Form API security fix SA-CORE-2014-002 to Drupal 8

I'm not sure whether that change fully fixed this issue though, because it's not 100% the same problem. Best way to verify is to re-start by writing an automated test.

Not sure this is the same issue but I upgraded to 7.27 after having the same issue and when this didn't resolve my issue I narrowed it down to a Firefox issue. My Ajax form works ok on chrome and safari (didn't test on IE) but not on Firefox (for anonymous users)
I have added the following to my ajax submit for testing:

<?php
  list($form, $form_state, $form_id, $form_build_id, $commands) = ajax_get_form();
  drupal_set_message($form_build_id);
?>

and on chrome and safari I get a new $form_build_id each time I refresh the page but on Firefox its the same ID each time and therfor the $form_state is preserved.
I may role back to the previous core version to check if this was the same before the upgrade.

Reroll of #33 against latest 7.x-dev.
Tests only, should break. Feed the testbot.

Reroll of #33 against latest 7.x-dev.
Tests plus fixes, should pass.

Wrong patch submitted in #62.
Reroll of #33 against latest 7.x-dev.
Tests plus fixes, should pass.

Please don't change the version.

I was facing a similar issue. reysharks replay in comment #17 works for me.

$form['actions']['submitregister'] = $form['actions']['submit'];
unset($form['actions']['submit']);

Thanks All.

For anyone who needs to fix this issue on a Drupal 7 site, cacheable_csrf doesn't work for anonymous users, so if you are trying to cache an ajax form for anonymous users, cacheable_csrf won't do the trick.

cacheable_csrf may work for the ajax login form (haven't tested it). The way I understand it, cacheable_csrf will let you properly handle cached forms served to logged in users because it changes the form token handling. It won't work for the issue where the form cache expires after 6 hours.

See #30 for a very clear explanation of what's going on.

The proposed resolution is thus:

1. Either do cache forms always individually, so that each delivered instance of a form has its own form-build-id
2. Or do not delete the form from cache when going through ajax_get_form().

Please correct me if I'm wrong, but the first option will require each form to be added to the cache for each user on the site. Having to add each form to the cache for every user on every page the form appears on defeats the purpose of having the cache in the first place and makes the memory requirements go way up. Leaving the form in the cache seems to make the most sense, so every user can use the same cached form. It would make sense if ajax_get_form() had the same behavior as drupal_get_form() since the functions perform similar tasks.

I think that updating ajax_get_form() is the best solution and would give us the highest performance in the long run. The alternative of creating a copy of the form for every user would have it's own consequences and would likely require an update to drupal_get_form() in addition to ajax_get_form(). The ajax functions should work like their static counter parts so they don't conflict with one another. At the moment these two functions are in conflict and that's what creates this bug.

After reviewing ajax_get_form() and drupal_get_form() I noticed an inconsistency. ajax_get_form() calls form_get_cache() directly, but drupal_get_form calls drupal_build_form() instead. drupal_build_form() calls form_get_cache() and and both functions return the form. I made a small adjustment to Drupal 7 in includes/ajax.inc:ajax_get_form() lines 319 to 324 change from:

  $form_state = form_state_defaults();
  
  $form_build_id = $_POST['form_build_id'];
  
  // Get the form from the cache.
  $form = form_get_cache($form_build_id, $form_state);

to:

  $form_state = form_state_defaults();
  $form_state['input'] = $_POST;
  $form_id = $_POST['form_id'];
  
  // Get the form from the cache.
  $form = drupal_build_form($form_id, $form_state);

Line 357 "$form_state['input'] = $_POST;" had to be moved to the top of the function. Everything seems to be in working order so far, but the build_id gets updated with every ajax request. This might not be optimal behavior, but it might resolve the problem in the short term. Can someone confirm this?

Upon further testing this fix did not work. My apologies.

#2263569: Bypass form caching by default for forms using #ajax. and various related issues completely removed use of the form cache on GET requests.

Moving this back to 7.x.

We really need to get this fixed for D7 as well.

While that happens, I found a workaround that can help get over this issue.

@clecidor suggests implementing hook_exit and then attempt submitting the form again when ajax POST fails:

https://www.drupal.org/node/1939254#comment-10413657

/**
 * Implements hook_exit().
 */
function example_exit($destination = NULL) {
  if (arg(0) == 'system' && arg(1) == 'ajax') {
    $is_user_login_form_submission = isset($_POST) && isset($_POST['name']) && isset($_POST['pass']) && isset($_POST['form_build_id']);
    if ($is_user_login_form_submission && user_is_anonymous()) {
      $form_build_id = $_POST['form_build_id'];
      $form_state = form_state_defaults();
      $form_state['values'] = $_POST; // Important!
      $form = form_get_cache($form_build_id, $form_state);

      if (!$form) {
        watchdog(__FUNCTION__, 'User login AJAX form submission failed. Trying again...', array(), WATCHDOG_WARNING);

        $form = drupal_rebuild_form('user_login', $form_state);
        $form['#build_id_old'] = $form['#build_id']; // Important!

        // Try form submission again after it is rebuilt above
        $commands[] = ajax_command_update_build_id($form);
        $commands[] = ajax_command_invoke('form#user-login', 'trigger', array('submit'));

        print ajax_render($commands);
      }
    }
  }
}

I can confirm this problem should really be fixed in D7 too, it problematically still exists there.

I found a nice(er) workaround to this issue, for anyone who really needs issue addressed.

It basically consists of unleashing the power of ajax_command_update_build_id by means of a tiny Javascript file which requests a new build_id for the form if it isn't found in the cache table and updates the form 'on the fly' so that any ajax interactions actually work.

You can find the complete solution and a full explanation in this post:

http://bmunslow.com/2015/12/28/solving-invalid-post-error-drupal-7/

Hi bmunslow! Your solution looks quite straightforward. Have you used it in any sites yet? I'm eager to get this fixed for my site which is heavily using caching for anonymous users. I can temporarily disable caching for the pages with the forms but this has some negative implication on performance.

@ngocketit Absolutely!

I have it up and running in production environment in a large website powered by commerce and it works like a charm!

Commerce + Drupal Commerce AJAX Cart + Page caching (Boost in my case) = Anonymous users weren't able to add products to cart (on occasions).

After applying this solution, problem is gone!

By the way, I'm working on a full featured module which implements this solutions in a more generic way. I will report back when it is ready (hopefully soon!).

My workaround is now available as a sandbox module:

https://www.drupal.org/sandbox/bmunslow/2682059

Checkout the README for more information on how to implement the fix.

Feedback welcome!

@bmunslow: Tried your sandbox module and I can confirm that the form_build_id gets replaced. However, the form is not cached (there is no record for form with new build id in cache_form table) and therefore, the issue is not solved, form submission still fails. Following is the code:

function my_module_form_webform_client_form_alter(&$form, &$form_state) {
  $node = menu_get_object();

  if (!$node && isset($form_state['values']['node'])) {
    $node = node_load($form_state['values']['node']);
  }

  if (my_module_is_contact_page($node)) {
    $webform_node = $form['#node'];
    $rebuild_callback = 'my_module_rebuild_ajax_forms_callback';
    rebuild_ajax_forms_initialize($form, $rebuild_callback, array('nid' => $webform_node->nid));
  }
  ....
}

function my_module_rebuild_ajax_forms_callback($form_id, $args) {
  $webform_node = node_load($args['nid']);
  $form = drupal_get_form('webform_client_form_' . $args['nid'], $webform_node, array());
  return $form;
}

Any ideas?

@ngocketit Your code looks good, that is indeed the right way to retrieve a webform.

If the form_build_id gets replaced, it means the form was re-built and the record was generated and it should be available in the cache_form table.

The only explanation I can come up with is that some hook or function might be clearing cache again after Drupal has-rebuilt the form.

Are there any bits of code in the rest of hook_form_FORM_ID_alter which clear cache?

BTW, perhaps we should continue this discussion in the issue queue of Rebuild Ajax Forms...

I got the form saved to cache_form with following piece of code:

function my_module_rebuild_ajax_forms_callback($form_id, $args) {
  $form_state = array(
    'cache' => TRUE,
  );

  $build_args = array(
    $args['webform_nid'],
    $webform_node,
    array(),
  );

  $form_state['build_info']['args'] = $build_args;
  $form = drupal_build_form('webform_client_form_' . $args['webform_nid'], $form_state);
  return $form;
}

The form returned from above function will be passed thru hook_form_alter(). However, in hook_form_alter(), I used some contextual data to alter the form, something like so:

function my_module_form_webform_client_form_alter(&$form, &$form_state) {
  $node = menu_get_object();

  if (!$node && isset($form_state['values']['node'])) {
    $node = node_load($form_state['values']['node']);
  }

  if (my_module_is_contact_page($node)) {
    $webform_node = $form['#node'];
    $rebuild_callback = 'my_module_rebuild_ajax_forms_callback';
    rebuild_ajax_forms_initialize($form, $rebuild_callback, array('nid' => $webform_node->nid));

    // ALTER THE FORM FOR THE CONTACT PAGE.
  }

So the code inside if (my_module_is_contact_page($node)) is not executed for the second time when the form is regenerated because it doesn't have all the contextual data as when the page is freshly loaded. As a result, the form regenerated is not similar to the one created previously. So I think your solution may work in some cases but not all the cases because regenerating the form in a Ajax request is far different from that in a normal page load request.

I'm glad you got it working.

You are absolutely right, rebuilding the form in an AJAX request can vary wildly for every scenario, that is why properly writing a custom 'callback' is required in order for the module to work correctly.

The callback examples provided in the README file are just generic examples which need to be tailored to every specific case.

Your code looks very good though, I might include it as an example in the README file, for others to use it if you agree.

@bmunslow: Feel free to include it if you see helpful. Like I said, I still need to find a way to get the regenerated form altered properly.

Heads up that some 3rd party modules can cause issues like what is described in here
#1270986: Make BOTCHA work with AJAX

Wow, what a difficult issue to solve!

In case anyone needs a pseudo-solution, you could add this to a custom module to bypass page caching on any page that contains a form with an ajax element.

/**
 * Implements hook_form_alter().
 */
function mymodule_form_alter(&$form, &$form_state, $form_id) {
  // Bypass page cache if form contains an ajax element. This prevents "very
  // difficult to solve" issues surrounding expired form caches.
  // https://www.drupal.org/node/1694574
  $function = function($element) use (&$function) {
    if (!empty($element['#ajax'])) {
      drupal_page_is_cacheable(FALSE);
    }
    foreach (element_children($element) as $key) {
      $function($element[$key]);
    }
  };
  $function($form);
}

#2263569: Bypass form caching by default for forms using #ajax. and various related issues completely removed use of the form cache on GET requests.

There is a functionally-equivalent port of this now for Drupal 7 that I wrote at #2819375: Do not make entries in "cache_form" when viewing forms that use #ajax['callback'] (Drupal 7 port). So in situations where #ajax['callback'] is used, the anonymous pages would become indefinitely cacheable without needing cache_form entries in the first place.

Nice work! I hope this that is passed. I have page caching turned off on all sites, as I create an AJAX login on all my sites.

I need cache on my site, but problem still there. Is it working for anyone with caching enabled or disabling cache is the only solution?

Hello joelstein, so #85 prevents any pages with ajax elements from caching or just ajax elements in the pages not to be cached?
I deleted Boost cache folder before installing your module from #85, Boost is not generating cached page for any Ubercart product pages with attributes. If Ubercart product doesn't have any attribute, cached pages are generated by Boost.

Workaround on #85 works for me. Many thanks!!!

hi
Many thanks to all for everyday's headache NOT EASY work .. this thread help's with @joelstein #85 solution.

thanks

@bmunslow, thanks for a good idea (#75)! We used it for our work
BUT note, your sandbox has a critical security vulnerability
@see https://www.drupal.org/node/2908604

+1

I am currently using Modal forms (with ctools) on the login, register, and password reset forms which appear on every page.

Looking at the above solutions suggested, I can see:

1. Disabling caches on pages with the forms (not viable)
2. Rebuild Ajax Forms sandbox module
3. Cacheable CSRF protection contrib module (dev only, only supports authenticated caches)

What is the current best approach for this issue?

We just had the very same issue as described at #47. Very frustrating.

After many, many... many hours of debugging, we found the issue to be Redis.
cache_form was cleared too fast/too often, causing this bug.

We solved it through having cache_form being written to Drupal instead.
$conf['cache_class_cache_form'] = 'DrupalDatabaseCache';

A catch with this is the eventual enormous table size, but for that we used the contrib module safe_cache_form_clear as solution.

Caching of product pages with ajax was not essential for my low-traffic Drupal Commerce shop - so "pseudo-solution" #85 worked nicely in my case.

Thnx @joelstein

Greetings!
I have created small module with only one hook_init, which creates new cache for ajax form if it doesn't exist:
https://www.drupal.org/sandbox/sergey-serov/3216053