Download & Extend

add docs for troubleshooting large imports

Project:Image
Version:6.x-1.0-beta5
Component:image_import
Category:support request
Priority:normal
Assigned:Unassigned
Status:closed (duplicate)

Issue Summary

I have 149 images of 3264x2448. I want to import them to a gallery.
I can't even import 1 image... Memory limit is set to 32MB while the total used when NOT importing is 8~9MB
The images are 1.30~1.40MB

Error when inserting all on the first page:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 4046675 bytes) in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 323

Error when inserting 1 image:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 3478254 bytes) in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/bootstrap.inc on line 735

Comments

#1

Category:bug report» support request
Status:active» closed (fixed)

32MB is not enough for image operations -- your status report should tell you that.
Try 96.

#2

No that does not tell me that. I will try to up it (external host)

K done:

Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:1662615:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE batch SET token = 'ad2703ea95c8801e7af9023363fb11d7', batch = 'a:10:{s:4:\\"sets\\";a:1:{i:0;a:11:{s:7:\\"sandbox\\";a:0:{}s:7:\\"results\\";a:0:{}s:7:\\"success\\";b:0;s:5:\\"title\\";s:29:\\"Afbeelding aan het importeren\\";s:16:\\"progress_message\\";s:29:\\"Importing @current of @total.\\";s:10:\\"operations\\";a:1:{i:0;a:2:{i:0;s:22:\\"_image_import_batch_op in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 128

Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:2044255:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE sessions SET uid = 1, cache = 0, hostname = '77.250.191.93', session = 'messages|a:1:{s:5:\\"error\\";a:1:{i:0;s:1662717:\\"user warning: Got a packet bigger than 'max_allowed_packet' bytes\\nquery: UPDATE batch SET token = 'ad2703ea95c8801e7af9023363fb11d7', batch = 'a:10:{s:4:\\\\"sets\\\\";a:1:{i:0;a:11:{s:7:\\\\"sandbox\\\\";a:0:{}s:7:\\\\"results\\\\";a:0:{}s in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 128

#3

Status:closed (fixed)» active

#4

Your upload_max_filesize needs to be larger than your post_max_size which both need to be large enough to handle your uploads.

Also it doesn't matter how large your filesize is when it comes to manipulating your images, it's the resolution that counts because it uncompresses it to manipulate it. So for those sizes, you'll need at least 128 MB and possibly as high as 256 MB. Mine is set at 256 MB in order to handle the occasional huge resolution image.

#5

Status:active» fixed

Sounds like it's all memory stuff.
TBH 3264x2448 is an INSANE resolution to be importing. Better to run some sort of batch operation on them offline first.

#6

Status:fixed» active

I've batched 'em, the error posted above is with 800x600 with 1 image from the list. Uploading works fine, importing not... Please don't set to fixed when it's not fixed.

#7

Try googling "Warning: Got a packet bigger than 'max_allowed_packet' bytes query"

#8

I increased my mysql max_allowed_packet=32M to fix it for me.

edit etc/my.conf
under [mysqld] add the above line. then restart mysql

I get no more errors on import.

Not sure of how you can that with shared hosting.

#9

Title:importing a large number of big images fails» add docs for troubleshooting large imports
Component:image_import» documentation
Category:support request» bug report

Could someone add a something about this to the troubleshooting page in the documentation please?

Then I can mark this fixed.

#11

Component:documentation» image_import
Category:bug report» support request

I am still getting the same three errors after trying all the suggestions. See http://drupal.org/node/610954

- I am on shared hosting (STRATO).
- I've created a php.ini file in the root of the Drupal site with memory_limit = 128M, upload_max_filesize = 16M and post_max_size = 8. I know these settings work (checked with an info.php file).
- I've also created a my.cnf file with and uploaded it to the root (next to php.ini), but I don't think it's working. Joachim advised me to rename it to my.conf but no difference.

I called STRATO and they told me I have to place php.ini and my.cnf in each folder, rather than just the root. True? I've tried a few folders but no luck. It would be insane to upload to all the folders.

I believe it's something much simpler and I'm missing something. Importing worked on an initial test site on the same server (with less modules installed though).

#12

I tried changing max_allowed_packet using Drupal tweaks but again no changes. Still the same three annoying errors... very frustrating.

I have tried tons of stuff and searched for hours so if someone has any clue please help me... I'm coming close to the release date of my website and I'd very much appreciate any help.

*Update*

If I remove the pnp.ini and my.cnf file I am left with only one error:

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 3195658 bytes) in /mnt/web3/13/95/51999495/htdocs/includes/bootstrap.inc on line 741

So upping the memory_limit only makes it worse. For a small site with a few modules isn't 32M enough? All the pictures I import have already been processed in Photoshop and are 600x338 and around 100KB.

#13

As I am still researching my import problem: could it have something to do with "Your server is not capable of displaying file upload progress. File upload progress requires PHP be run with mod_php and not as FastCGI."?

#14

Version:6.x-1.0-beta2» 6.x-1.0-beta5

I run into the same problem. I have uploaded some images, but if I go to
http://example.com/admin/content/image_import
and try to import one small image (600x800 / 40kB) from the list with I get an error like

Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:1535406:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE batch SET token = '2341a5a9221871ac8e8ee628da8f0abe', batch = 'a:10:{s:4:\\"sets\\";a:1:{i:0;a:11:{s:7:\\"sandbox\\";a:0:{}s:7:\\"results\\";a:0:{}s:7:\\"success\\";b:0;s:5:\\"title\\";s:20:\\"Bild wird importiert\\";s:16:\\"progress_message\\";s:29:\\"Importing @current of @total.\\";s:10:\\"operations\\";a:1:{i:0;a:2:{i:0;s:22:\\"_image_import_batch_op\\&qu in /kunden/229719_22926/webseiten/drupal/includes/database.mysql.inc  on line 128

Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:1915345:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE sessions SET uid = 1, cache = 1263726427, hostname = '92.226.33.195', session = 'updates_remaining|a:0:{}dblog_overview_filter|a:0:{}messages|a:1:{s:5:\\"error\\";a:1:{i:0;s:1535493:\\"user warning: Got a packet bigger than 'max_allowed_packet' bytes\\nquery: UPDATE batch SET token = '2341a5a9221871ac8e8ee628da8f0abe', batch = 'a:10:{s:4:\\\\"sets\\\\";a:1:{i:0;a:11:{s:7:\\\\"sandbox\\\\"; in /kunden/229719_22926/webseiten/drupal/includes/database.mysql.inc on line 128

PHP settings are
memory_limit=128M
upload_max_filesize = 16M
post_max_size = 8M

and MySQL settings are
max_allowed_packet=1*1024*1024

As I am on shared hosting (domaingo.de), I cannot change the MySQL settings and as this probably applies to many users - I think the import should also work with MySQL default settings, which are max_allowed_Packet=1M

I have seen #610954: Image import error's and warnings..., which is marked to be a duplicate of this thread here.
I have read all pages linked to from this thread and I tried almost all soultions proposed, but none could solve my problem.

I also tried the mini module described here: http://drupal.org/node/379976
which as I understand does the same as the Drupal tweaks module.
According to http://example.com/admin/help/max_packet the max_allowed_packet was increased (for each session respectively) to 16M, but the problem with image_import persists.

The error logged in the dblog report ( http://example.com/admin/reports/dblog ) is a bit different:

Cannot modify header information - headers already sent by (output started at /kunden/229719_22926/webseiten/drupal/includes/database.mysql.inc:128)  in /kunden/229719_22926/webseiten/drupal/includes/common.inc in Zeile 335.

Maybe there is a php closing tag where it should not belong?

I don't know what else I could try, any help appreciated...

#15

This is all to do with Drupal core's batch system and nothing that can be fixed from within Image module.

#16

Status:active» closed (duplicate)

Ok, thanks. I thought it had something to do with the way image_import SQL commands were generated, similar to this issue here:
#320374: Simpletest now fails with max_allowed_packet = 1M
but I don't know how image_import and batch API work.

As you say it is a Drupal issue, I marked this issue here as duplicate (but maybe it can be left in the image_import issue queue in case someone is experiencing the same problem) and re-opened an issue from the Drupal issue queue instead:
#355076: Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog

nobody click here