add docs for troubleshooting large imports
kjarli - September 7, 2009 - 15:06
| Project: | Image |
| Version: | 6.x-1.0-beta2 |
| Component: | image_import |
| Category: | support request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Jump to:
Description
I have 149 images of 3264x2448. I want to import them to a gallery.
I can't even import 1 image... Memory limit is set to 32MB while the total used when NOT importing is 8~9MB
The images are 1.30~1.40MB
Error when inserting all on the first page:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 4046675 bytes) in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 323
Error when inserting 1 image:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 3478254 bytes) in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/bootstrap.inc on line 735

#1
32MB is not enough for image operations -- your status report should tell you that.
Try 96.
#2
No that does not tell me that. I will try to up it (external host)
K done:
Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:1662615:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE batch SET token = 'ad2703ea95c8801e7af9023363fb11d7', batch = 'a:10:{s:4:\\"sets\\";a:1:{i:0;a:11:{s:7:\\"sandbox\\";a:0:{}s:7:\\"results\\";a:0:{}s:7:\\"success\\";b:0;s:5:\\"title\\";s:29:\\"Afbeelding aan het importeren\\";s:16:\\"progress_message\\";s:29:\\"Importing @current of @total.\\";s:10:\\"operations\\";a:1:{i:0;a:2:{i:0;s:22:\\"_image_import_batch_op in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 128
Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:2044255:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE sessions SET uid = 1, cache = 0, hostname = '77.250.191.93', session = 'messages|a:1:{s:5:\\"error\\";a:1:{i:0;s:1662717:\\"user warning: Got a packet bigger than 'max_allowed_packet' bytes\\nquery: UPDATE batch SET token = 'ad2703ea95c8801e7af9023363fb11d7', batch = 'a:10:{s:4:\\\\"sets\\\\";a:1:{i:0;a:11:{s:7:\\\\"sandbox\\\\";a:0:{}s:7:\\\\"results\\\\";a:0:{}s in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 128
#3
#4
Your upload_max_filesize needs to be larger than your post_max_size which both need to be large enough to handle your uploads.
Also it doesn't matter how large your filesize is when it comes to manipulating your images, it's the resolution that counts because it uncompresses it to manipulate it. So for those sizes, you'll need at least 128 MB and possibly as high as 256 MB. Mine is set at 256 MB in order to handle the occasional huge resolution image.
#5
Sounds like it's all memory stuff.
TBH 3264x2448 is an INSANE resolution to be importing. Better to run some sort of batch operation on them offline first.
#6
I've batched 'em, the error posted above is with 800x600 with 1 image from the list. Uploading works fine, importing not... Please don't set to fixed when it's not fixed.
#7
Try googling "Warning: Got a packet bigger than 'max_allowed_packet' bytes query"
#8
I increased my mysql
max_allowed_packet=32Mto fix it for me.edit etc/my.conf
under [mysqld] add the above line. then restart mysql
I get no more errors on import.
Not sure of how you can that with shared hosting.
#9
Could someone add a something about this to the troubleshooting page in the documentation please?
Then I can mark this fixed.
#10
See also:
http://drupal.org/node/360463
and
http://dev.mysql.com/doc/refman/5.1/en/packet-too-large.html
and
http://drupal.org/node/434032
#11
I am still getting the same three errors after trying all the suggestions. See http://drupal.org/node/610954
- I am on shared hosting (STRATO).
- I've created a php.ini file in the root of the Drupal site with memory_limit = 128M, upload_max_filesize = 16M and post_max_size = 8. I know these settings work (checked with an info.php file).
- I've also created a my.cnf file with and uploaded it to the root (next to php.ini), but I don't think it's working. Joachim advised me to rename it to my.conf but no difference.
I called STRATO and they told me I have to place php.ini and my.cnf in each folder, rather than just the root. True? I've tried a few folders but no luck. It would be insane to upload to all the folders.
I believe it's something much simpler and I'm missing something. Importing worked on an initial test site on the same server (with less modules installed though).
#12
I tried changing max_allowed_packet using Drupal tweaks but again no changes. Still the same three annoying errors... very frustrating.
I have tried tons of stuff and searched for hours so if someone has any clue please help me... I'm coming close to the release date of my website and I'd very much appreciate any help.
*Update*
If I remove the pnp.ini and my.cnf file I am left with only one error:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 3195658 bytes) in /mnt/web3/13/95/51999495/htdocs/includes/bootstrap.inc on line 741
So upping the memory_limit only makes it worse. For a small site with a few modules isn't 32M enough? All the pictures I import have already been processed in Photoshop and are 600x338 and around 100KB.
#13
As I am still researching my import problem: could it have something to do with "Your server is not capable of displaying file upload progress. File upload progress requires PHP be run with mod_php and not as FastCGI."?