Batch update fails with access denied message

Did you try to delete the batch database table, and force Drupal to run again the update that failed?

Oops; I meant to empty the table.

This is a won't fix. Please tweak your mysql configuration or get a better hosting plan.

I am updating the report for the latest Drupal version. I would try installing Drupal 6.10, and see if the issue is still present.

I am not sure the issue depends on the maximum allowed packet size, as in my case it's 1,048,576 and I don't get any errors like the reported one.

I am also getting the same symptoms in another area (not the above mentioned modules) but we have traced it to batch API handling of huge job sets.

Got a packet bigger than max_allowed_packet bytes  query: UPDATE batch ...

This is causing WSOD, and cascading errors as the warning tries to get logged to watchdog ... which itself can't handle the size of the message and triggers an error ...

For other cases, we also see an "Access denied" on the batch page.
This is caused when there are no batch jobs in the queue to process
- the message could/should say as much rather than 'denied'
There are no jobs in the queue because the previous DB insertion into the queue failed.

Anyway. The problem is mostly mine, because I'm using import_html to queue up thousands of jobs using batch API
and this is working freaking awesome on dev machines and I guess batch API wasn't exactly engineered for that.
but
It could be.

Previously, (taxonomy_xml - tens of thousands of terms) I worked around batch job size limitations by breaking my batches into smaller batches and having them queue one after the other.
Looking at the current D6 batch internals I see that this work-around is useless, because although I submit several discrete jobs, batch_set just gathers them all into a static array anyway.
Damn. That's why my jobs don't scale. And batch_process blows the memory/packet stack even though I tried to keep it down.

I'm not sure how to progress here, short of building my own parallal daemon job queue in the database - which I REALLY don't want to do if we can get batch api to scale better.
Isn't it possible to get batch_set() to actually put the batch instructions into the queue rather than into a static, do-later huge array?

Acknowledged that adjusting my own packet limit is a work-around for me, for now, but may not be a solution for other folk who want to use my module.

.dan.

dman: instead of one batch op per item to process, which won't scale well with 1000s of items, the recommended way is to use a single, multistep op, storing its progress information in its $context param (see example on http://api.drupal.org/api/group/batch/6)

Thanks for the suggestion, I'll look into that!
I feel may run into a similar issue as it might just push the temporary storage into another place in the DB - but at least I can (probably) make that storage terser that the current function signatures that batch stores.
I'm already using multistep, so we'll see.

I was experiencing the same issue - access denied errors during batch api calls, breaking my batches. I finally traced the problem to batch.inc. The access denied error is coming from here:
system.admin.inc line 1800:

function system_batch_page() {
  require_once './includes/batch.inc';
  $output = _batch_page();
  if ($output === FALSE) {
    drupal_access_denied();
  }
  elseif (isset($output)) {
    // Force a page without blocks or messages to
    // display a list of collected messages later.
    print theme('page', $output, FALSE, FALSE);
  }
}

Line 1804 evaluates to true, therefore returning drupal_access_denied(). So if we look at _batch_page() function we see
batch.inc line 10:

function _batch_page() {
  $batch =& batch_get();

  // Retrieve the current state of batch from db.
  if (isset($_GET['id']) && $data = db_result(db_query("SELECT batch FROM {batch} WHERE bid = %d AND token = '%s'", $_GET['id'], drupal_get_token($_GET['id'])))) {
    $batch = unserialize($data);
  }
  else {
    return FALSE;
  }

  // Register database update for end of processing.
  register_shutdown_function('_batch_shutdown');

  $op = isset($_GET['op']) ? $_GET['op'] : '';
  $output = NULL;

The problem I noticed here is that for some reason (probably something related to my environment) the $_REQUEST superglobal is not being populated. It appears to be rather random, I'm attaching a watchdog screenshot to better display its characteristics (I replaced the $_REQUEST['x']s (x being 'id', 'op') with $_GET['x'] before taking this screenshot). So line 15 in batch.inc where the function evaluates to false. $_GET on the other hand has the proper values, therefore preventing the process from 403ing.
Attached is also the patch that fixed the problem for me. Hope this helps.

I got the same issue whit D.6.12. When enabeling a module for the first time, I get a access denied message whit a warning in /includes/form.inc on line 2524.

In relation to http://drupal.org/node/292060
#15 I don't think I could do anything about the max_allowed_packet setting...
#19 I have FR enabled as default language along whit EN.

Thx

Dadaisme

Bumping to 6.x-dev so the testbot tests the patch.

The changes in #13 worked like a charm. Thank you so much.

Tim

The fix in #13 does not work for me.

I am trying to install the Translation Overview module and get the access denied error.

I would like to provide some debug info, but I don't know how?

Let me ask,

What is the purpose of the : http://www.yoursite.com/batch url?

I have installed the xmlsitemap module, and I have created a xml sitemap which automatically submits to the google/yahoo/msn search engines.

For some reason, the google search engine continues to try and come to my site to view the /batch area, but it continues to show in my logs that they are getting "Access Denied" @ /batch area on my site.

So, I was wondering... what is so important about the batch url? Is it bad that batch url comes up with "You are not authorized to access this page"?

On a side note, I visit : http://www.yoursite.com/sitemap.xml
and the sitemap shows up just fine.

Is this thread even related to the issue that I am experiencing?

Also, I am logged in as the Admin on my site.

The URL /batch is used internally by Drupal to execute batch operations; Google, or any other search engines should not have such links.

Oh, ok, so as per my question on #18, so if I "the admin of my own drupal site" goes to the http://www.yoursite.com/batch url
should it actually show something on that page or is it okay that it says, "You are not authorized to access this page"?

As @KiamLaLuno pointed out to me in #19 (after my post) that the /batch operations is internal, but does that mean that drupal handles everything for the /batch process own its own automatically and that page will not show up to normal users? Is something screwed up.. or is this how it is suppose to work?

I wish I could figure out why Google is trying to go to the /batch url? I was just messing with the xmlsitemap in google account for my site... but, why is google interested in my sites /batch url?

Once that the batch operation is concluded, every people (including the first user) receive an access denied error message.

#1 you get that (unhelpful and frankly incorrect) message when there is no batch activity to process. Imagine if cron.php told you to go away if there wasn't actually any pending jobs. You (your browser) do have access to it in the course of submitting complex forms and running ajax callbacks when that progressbar ticks over. Then it reverts to saying '401 Access denied'.
Dammit should just say "4?? No batch operations pending".

Hm, maybe it's just because I can't think of a better error code number that they re-use the 401 boilerplate response...

No idea how or when Google would have found out about that URL. That's not normal. But no big deal either. as it's a 4xx, it won't be indexing it or anything.

/batch URL is always used as /batch/[batch_id].
- Without the id,
- or with an id that doesn't exist,
- or if the corresponding batch record couldn't be retrieved because of a MySQL error (as seems to be the case for most/all reports of this bug, batch record too big, 'max allowed packet' error),
'Access denied' is the expected behavior.

So, gensuperman, your own issue is to find why google tries to access the /batch URL directly. But that's for a different thread (or most probably the forums), this would only add confusion here.

re #22: cron is not a good analogy. There's no 'let's go to /batch see if there is any batch work to process', batch API doesn't work like that.

Run in the same problem. I see there was no token set in DB.

The same error, shit)

UPD: Changing max_allowed_packet to 2M successfully fixed error while applying new module.

Isn't it the purpose of the batch function to downsize the packets?

How are you going to fix this UdarEC?

I don`t now)
Actually, I think it`s not a big problem, to set max_allowed_packet to 2M.

I got the same issue (reported http://drupal.org/node/564420).

UPDATE statement in function batch_process() in includes/form.inc fails with "Got a packet bigger than max_allowed_packet bytes" warning. It's because mysql settings max_allowed_packet = 1M (my hosting plan) and serialized batch string exceed 1MB. So batch column is empty and this leads (function _batch_page() and function system_batch_page() in include/batch.inc) to Access denied page.

Well, I thing that settings max_allowed_packet to 2M is not the right solution. It's possible only on dedicated or virtual servers, not on shared hosting (most of hosting plans).

using image beta3
I found this problem also and edited my /etc/my.cnf and restarted,
Since it was my server, I set it
max_allowed_packet = 16M

reference
http://drupal.org/node/360463
and
http://dev.mysql.com/doc/refman/5.1/en/packet-too-large.html

Most of Drupal users are using shared hosting. They can't change mysql configuration.

Am I correct to assume then, that the issue exists in my max_allowed_packet setting?
If this is a problem outside of drupal, at least drupal should let me know that it is not drupal and preferably tell me what it is.
If anyone can tell me the minimum max_allowed_packet setting for mysql I can perhaps tell my provider. Ifdrupal is distributed with a mention of this minimum it would even be easier to persuade providers.
At the moment drupal isn't telling me anything and it took me half an hour to find this issue, which in my case on a drupal multisite installation on shared hosting, occurred with every translation batch that is run after an update:
/batch?op=start&id=23
But try searching on access denied and even with the new solr search on drupal.org (very nice) loads of irrelevant info pops up.

Sorry, it's just the nth time drupal isn't telling me what's going on. Really starting to dislike that. Looking forward to d7.

I found this:
http://drupal.org/node/542890

http://drupal.org/node/542890 (drupal_tweaks module) will not work on servers, where your account don't have proper privileges. I'm afraid that in most of shared hosting you don't have proper privileges to change mysql settings. Drupal is trying to insert huge serialized batch string (exceeding 1MB) into database, that's the problem! It's not about max_allowed_packet setting, 1 MB is enough to "normal" work with database.

@milosnemec So what is causing these serialized 1mb+ batch strings. Is it just the translation files? Because if only those who need native translations are suffering from this issue, it would explain why many people have no issues with shared hosting at all and it's perceived as a minor bug, when basicly it causes a significant drupal adoption issue outside of english speaking countries.
Does #simpletest check for that?

@whatdoesitwant I think it's batch API issue, but don't know much about batch API. It's not just about translation files. I got this problem with image module (image import) and there are dozen users reporting the same problem with other modules.

I think I have a workaround for this issue. This is my first post on the Drupal site, so please forgive me for not giving patches, but I'm sure it will help someone in the right direction.

I am on a shared hosting so cannot update max_allowed_packet size. I think this is a very common situation. I noticed the problem when trying to run image import.

I got the idea from a discussion about similar problems with big queries in Views.

My solution is to base64_encode(gzcompress()) the batch string before inserting, then reversing the process when read from the database, thus:

form.inc:
(around line 2529): in the UPDATE batch code, change serialize($batch) to base64_encode(gzcompress(serialize($batch)))

batch.inc:
(around line 352): as above, change serialize($batch) to base64_encode(gzcompress(serialize($batch)))
(around line 16): change $batch = unserialize($data); to $batch = unserialize(gzuncompress(base64_decode($data)));

Now the sql statements are smaller :) No idea what this might do to server load or anything like that. This fixes the image import for me.

@milosnemec.cz You're absolutely right. The key problem is with batch API.
I checked if simpletest tests for the overflow. It doesn't, because batch API's behaviour has not been addressed. But it seems that the people working on simpletest have had the same problem and solved it just for simpletest instead. This issue que thread describes the problematic behaviour by batch API and the work around for simpletest:
http://drupal.org/node/320374
Maybe a similar approach is possible for translation files if fixin batch API is such a problem. Maybe somebody who knows how to patch can look at this? (It's really not my cup of tea.)
I don't know about solving the image import though, because mysql throws up a hard limit at 41 files as well, regardless of size considerations.

I had the same problem as describe higher... I tried all the solutions except the patches ... the max_allowed packet, the memory_limit were good... and I just find the solution for me.
When I connect my database I put "localhost" to use the locale socket of the server. Normally it would work fine, but I don't know why, that will create all the problems describe. So we put the real address of the database and it works...
Hope that will be usefull for you !

Thanks flumo, your patch fixed also mine Image import!

I supply the patch of #37 as a patch because I think the real problem is caused the batch API and not by the system configuration.

In a shared environment you can't just reset the configuration variables. Their values have to be as tiny as possible, for performance and security reasons.

The batch API is not really scalable. It just coarsely serializes the whole batch into a single database field (table: batch, field: batch). When one has millions
(or a couple of thousands are already sufficient) operations, it will exceed the the maximum packet size of MySQL (variable max_allowed_packet). The work around
here is to compress this veeeeeeery long batch string.

It's good enough for my purposes, but maybe it would be more wise to refactor the "batch" table into "batch" and "batch_operations".

Note that #629794: Scaling issues with batch API should fix those issues in D7. No easy backport, though.

*Also note* that a batch with "millions (or a couple of thousands)" operations is terrible design to begin with - and this will still be true in D7. Use a multipass op instead.

Subscribing.

I had that error many times on last months... http://acquia.com/node/36765

Any toughts on backporting?

Thanks.

Subscribing

#37 Worked for me on 6.17 to fix the image import access denied issue.

Updated /etc/my.cnf on one of our hosting platforms and changed:

max_allowed_packet = 1M

to

max_allowed_packet = 4M

which seems to have fixed my problem.

subscribing

subscribing

Got the same problem after enabling the simplenews module, after this the translations where not imported.
Changed max_allowed_packet to 16M, deinstalled simplenews, and after enabling it again all translations where fine.

I'm having a similar problem, but increasing max_allowed_packet to 4M did not seem to solve it, and it happens even when the batch table contains only one item.

I first noticed the problem when I rebuilt permissions after installing ACL and other access control modules that require the permissions to be rebuilt. It goes to the /batch page and gives the "access denied" message (I've tried this using both the garland theme and my own theme, with the same results).

What I discovered is that I can look at the bid in the batch table and *manually* run the batch page (/batch?op=start&id=41) and it works correctly. However, it never works automatically, and it seems unusual to have to intervene every single time this wants to run (for example, uploading an image with node gallery).

Any ideas what could be causing this or how to fix it?

subscribing

#37 (and #41) worked for me.

subscribing

For the records:
On my hosting plan, max_allowed_packet=1M and I'm not allowed to change it via database_tweaks etc.
I get the error (packet bigger than 'max_allowed_packet') frequently when installing modules, apparently when translations are imported.

Patch #41 worked, du to an issue occured #564420: 'Access denied' after image import and tried to import images via the component import_image with a tree folder that contains up to 3 subfolders:

+-- Folder 1
|   |-- Folder 1-1
|   |-- Folder 1-2
|       |-- Folder 1-2-1
|       |-- Folder 1-2-2
|       |-- Fodler 1-2-3
|   |-- Folder 1-3
+-- Folder 2

In this case, there's a "Access Denied" page generated by the batch example.com/batch?op=start&id=xxx

If there is only :

+-- Folder 1
|   |-- Folder 1-1
|   |-- Folder 1-2
|   |-- Folder 1-3
+-- Folder 2

It works fine

max_allowed_packet = 2M fixed my errors... tnx!

I get this error with fastcgi php5-fpm apache2 postgresql. Update.php works though, since it's not an internal url.

After discovering this myself, I went back to see what I changed. Fresh install; it worked fine, but after configuring secure pages, this is where I ran into the issue of ACCESS DENIED. The answer is within your secure pages setting.

I added "batch* to the "pages that will be secure" section. It flew.

Enjoy - hope this is what will solve everyones issue. BTW - Drupal Rocks!

Zig

I was getting a 401 error when trying to enable Webform 3.0 I just added..

max_allowed_packet = 16M

to /etc/my.cnf and then restarted mysql to resolve the issue.

Can you please elaborate on 'added "batch* to the "pages that will be secure" section'?

I installed l10_update module a bit too late, now it tries to update translation for 200+ modules in one hit. So I get the ACCESS DENIED message. I managed to convince my hosting provider to increase max_allowed_packet to 16M! Didn't help.

How about that, Damien Tournoud (http://drupal.org/node/434032#comment-1477502)?

I have no idea what you are talking about here, nor why you changed the title of this issue. Anyway, this is won't fix for core.

Sorry for changing title, I thought it would be a title for my comment.

I wanted to ask zigiboi , comment #59 what he meant by

I added "batch* to the "pages that will be secure" section. It flew.

because for him it seems to fix the problem.

And why do you say "no fix"? I have the ACCESS DENIED problem and I need to fix it.

[#63] Damien Tournoud, no need to be rude! Drupal is a pain, I just hope for some help.

I found that the problem I'm experiencing is caused by the theme. I use aquia_commons theme, but I checked PushButton has the same issue.

So for the batch operation I have to switch theme to Garland, then back to my theme. Not very convenient :(

At least I'm glad I found a workaround, so I can now publish my website!

HobbyBlob.com

I am having a similar issue, although I cannot for the life of me figure out what is causing it.

I cannot run any batch item that I am trying to. Namely a batch to import a feed, or rebuild the content permissions. I have tried #13s solution to no avail. What I can tell you is that when I try to rebuild the content access permissions, it truncates the {node_access} table (doesn't even put back a default value). When I disable all my node-access modules (Taxonomy Access Control and Organic Groups) it works. After disabling / uninstalling both, and enabling them one at a time, it still does not work. This leads me to think it is not directly related to them, but more of a core issue. If I disable both, I still cannot import news feeds from uploading an XML file, for the same access denied error.

I have changed the max_allowed_packet to 2M then 16M, cleared the batch tables, and cleared all the caches, etc. I have no idea what is causing this, nor how to stop it. I have done this on both my admin account with all privileges, and user 1.

Thoughts?

Surprisingly it works now, I don't know what's changed.

#41 saved my day, works now on the customer's hosting plan with max_allowed_packet = 1M. I attached the patch adapted for Drupal 6.19 which is currently running there.

I have the same issue on D7

subscribing

I had also 16Mb of allowed paket size but here is still the same issue.

I was also getting "access denied" on authorize.php, but discovered that a batch update will force an https connection if available. From system.module:

function system_authorized_get_url(array $options = array()) {
  global $base_url;
  // Force https if available, regardless of what the caller specifies.
  $options['https'] = TRUE;
  // We prefix with $base_url so we get a full path even if clean URLs are
  // disabled.
  return url($base_url . '/authorize.php', $options);
}

I then remembered I had just installed an ssl cert, so logging in via https was the solution.

anybody have solutions with batch max size? What killed in drupal for zeroing "batch" table ? What`s modul make batch longblob binary ?

subscribing, same problem every time i enable a new module as user #1

same here

Same here still in 2013

Thank you zigiboi ! your comment #59 is working perfect !

And more, I had the issue also with Worbench moderation. Just following the idea, I added node/*/moderation* in the same configuration field.
Now everything is working fine !

In details :
Go to : admin/config/system/securepages

In the "Pages" textarea, add for worbench to work :
node/*/moderation*

For batch operations
batch*

Hope it helps others :)
Works for me on d7 version 7.x-1.0-beta1, but I'm pretty sure it's the same for other versions.

Liaz.

I don't have secure pages installed, and still have this problem.

The same proble, I solve it adding "max_allowed_packet=16M" to "/etc/my.cnf" with "vi /etc/my.cnf" at SSH, after "user=mysql" and them "service mysqld restart".

I have to report that the issue remains for me, even though my MySql settings are:
max_allowed_packet = 32M
Drupal version: 7.34

I don't have any of the modules mentioned above installed, and the patch obviously no longer applies.

EDIT: Solved
In my case, the culprit turned out to be a rewrite rule I had in my .htaccess file which stripped 'id' GET parameters from the URL, which thus didn't allow the batch to be processed accordingly.

#82 solved the issue for me. Except instead of .htaccess it was my nginx try_files statement had a typo that was stripping the parameters.

#82 reference to GET parameters been disallowed in htaccess helped me resolved. thanks a lot