add docs for troubleshooting large imports

kjarli - September 7, 2009 - 15:06
Project:Image
Version:6.x-1.0-beta2
Component:image_import
Category:support request
Priority:normal
Assigned:Unassigned
Status:active
Description

I have 149 images of 3264x2448. I want to import them to a gallery.
I can't even import 1 image... Memory limit is set to 32MB while the total used when NOT importing is 8~9MB
The images are 1.30~1.40MB

Error when inserting all on the first page:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 4046675 bytes) in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 323

Error when inserting 1 image:
Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 3478254 bytes) in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/bootstrap.inc on line 735

#1

joachim - September 8, 2009 - 12:04
Category:bug report» support request
Status:active» closed

32MB is not enough for image operations -- your status report should tell you that.
Try 96.

#2

kjarli - September 8, 2009 - 16:45

No that does not tell me that. I will try to up it (external host)

K done:

Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:1662615:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE batch SET token = 'ad2703ea95c8801e7af9023363fb11d7', batch = 'a:10:{s:4:\\"sets\\";a:1:{i:0;a:11:{s:7:\\"sandbox\\";a:0:{}s:7:\\"results\\";a:0:{}s:7:\\"success\\";b:0;s:5:\\"title\\";s:29:\\"Afbeelding aan het importeren\\";s:16:\\"progress_message\\";s:29:\\"Importing @current of @total.\\";s:10:\\"operations\\";a:1:{i:0;a:2:{i:0;s:22:\\"_image_import_batch_op in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 128

Warning: Got a packet bigger than 'max_allowed_packet' bytes query: INSERT INTO watchdog (uid, type, message, variables, severity, link, location, referer, hostname, timestamp) VALUES (1, 'php', '%message in %file on line %line.', 'a:4:{s:6:\"%error\";s:12:\"user warning\";s:8:\"%message\";s:2044255:\"Got a packet bigger than 'max_allowed_packet' bytes\nquery: UPDATE sessions SET uid = 1, cache = 0, hostname = '77.250.191.93', session = 'messages|a:1:{s:5:\\"error\\";a:1:{i:0;s:1662717:\\"user warning: Got a packet bigger than 'max_allowed_packet' bytes\\nquery: UPDATE batch SET token = 'ad2703ea95c8801e7af9023363fb11d7', batch = 'a:10:{s:4:\\\\"sets\\\\";a:1:{i:0;a:11:{s:7:\\\\"sandbox\\\\";a:0:{}s:7:\\\\"results\\\\";a:0:{}s in /home/mosterdz/domains/mosterdzaadje.com/public_html/includes/database.mysqli.inc on line 128

#3

kjarli - September 8, 2009 - 16:45
Status:closed» active

#4

2noame - September 8, 2009 - 19:33

Your upload_max_filesize needs to be larger than your post_max_size which both need to be large enough to handle your uploads.

Also it doesn't matter how large your filesize is when it comes to manipulating your images, it's the resolution that counts because it uncompresses it to manipulate it. So for those sizes, you'll need at least 128 MB and possibly as high as 256 MB. Mine is set at 256 MB in order to handle the occasional huge resolution image.

#5

joachim - September 9, 2009 - 12:42
Status:active» fixed

Sounds like it's all memory stuff.
TBH 3264x2448 is an INSANE resolution to be importing. Better to run some sort of batch operation on them offline first.

#6

kjarli - September 9, 2009 - 22:02
Status:fixed» active

I've batched 'em, the error posted above is with 800x600 with 1 image from the list. Uploading works fine, importing not... Please don't set to fixed when it's not fixed.

#7

joachim - September 9, 2009 - 22:14

Try googling "Warning: Got a packet bigger than 'max_allowed_packet' bytes query"

#8

milanod - September 14, 2009 - 00:15

I increased my mysql max_allowed_packet=32M to fix it for me.

edit etc/my.conf
under [mysqld] add the above line. then restart mysql

I get no more errors on import.

Not sure of how you can that with shared hosting.

#9

joachim - September 14, 2009 - 06:43
Title:importing a large number of big images fails» add docs for troubleshooting large imports
Component:image_import» documentation
Category:support request» bug report

Could someone add a something about this to the troubleshooting page in the documentation please?

Then I can mark this fixed.

#11

thealchemist - October 22, 2009 - 20:44
Component:documentation» image_import
Category:bug report» support request

I am still getting the same three errors after trying all the suggestions. See http://drupal.org/node/610954

- I am on shared hosting (STRATO).
- I've created a php.ini file in the root of the Drupal site with memory_limit = 128M, upload_max_filesize = 16M and post_max_size = 8. I know these settings work (checked with an info.php file).
- I've also created a my.cnf file with and uploaded it to the root (next to php.ini), but I don't think it's working. Joachim advised me to rename it to my.conf but no difference.

I called STRATO and they told me I have to place php.ini and my.cnf in each folder, rather than just the root. True? I've tried a few folders but no luck. It would be insane to upload to all the folders.

I believe it's something much simpler and I'm missing something. Importing worked on an initial test site on the same server (with less modules installed though).

#12

thealchemist - October 24, 2009 - 20:46

I tried changing max_allowed_packet using Drupal tweaks but again no changes. Still the same three annoying errors... very frustrating.

I have tried tons of stuff and searched for hours so if someone has any clue please help me... I'm coming close to the release date of my website and I'd very much appreciate any help.

*Update*

If I remove the pnp.ini and my.cnf file I am left with only one error:

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 3195658 bytes) in /mnt/web3/13/95/51999495/htdocs/includes/bootstrap.inc on line 741

So upping the memory_limit only makes it worse. For a small site with a few modules isn't 32M enough? All the pictures I import have already been processed in Photoshop and are 600x338 and around 100KB.

#13

thealchemist - October 25, 2009 - 23:57

As I am still researching my import problem: could it have something to do with "Your server is not capable of displaying file upload progress. File upload progress requires PHP be run with mod_php and not as FastCGI."?

 
 

Drupal is a registered trademark of Dries Buytaert.