Failing to import file with a lot of lines [#245483]

I've tried 6 times to import the file that's attached at the end of this comment. I obviously have it in .csv format, not .txt

I create it and then save it as UTF-8 because there are accents on many of the Italian town names. Those that do get imported look fine and the accents are showing, so no problems there.

But it's only a 300k file and I should be able to go up to many mB but each time I go to import, I wait about 30 seconds and then it ends up on the 404 page (the logs say Message:500.shtml Severity:Warning). And each time, I see through phpMyAdmin that somewhere around 550-700 lines have been imported and the other 7000 not. It's never exactly the same amount, so it's not tripping over any weird characters that I can tell.

what could be the issue...the utf-8 encoding or maybe something related to my hosting/server...or the module itself?

Comment	File	Size	Author
	ItalyALL2.txt	298.93 KB	esllou

Comments

Comment #1

esllou commented 11 April 2008 at 21:55

I've just tried it on a fresh install and it all went in perfectly - in about 20 seconds too!

Any idea on the sort of things that could have caused problems first time round?

Comment #2

esllou commented 12 April 2008 at 09:43

I'm continuing to have big problems with other csv files. The first time I import, it gets to about 700 lines, then all the following times, it's adding about 5 terms in 20-30 seconds before timing out. I wonder if it's a server issue?

Comment #3

decafdennis commented 19 April 2008 at 18:01

I tried to import the file you posted, and it appears to fail on my installation too. I'll investigate.

Comment #4

decafdennis commented 19 April 2008 at 22:27

Component:

User interface

» Code

On my side the import failed (hang indefinitely) because I didn't enable the hierarchy option for the vocabulary. Are you sure you set the hierarchy option for the vocabulary you're trying to import into?

Is the import always failing after exactly 30 seconds? In that case, the module is unable to override the default PHP timeout value (30s), and so the script time outs. This might be caused by your PHP running in safe mode. A solution is to split the file into multiple files of say 500 lines. You can also try upgrading to Drupal 6, as the 6.x version of the module uses the batch API to split the import over multiple requests (to give you a nice progress bar, and to prevent timeouts).

On a side note, while the import is working over here, using the 5.x version, it did take many minutes before the import was complete.

Comment #5

esllou commented 20 April 2008 at 10:28

thanks for the testing. FYI, me and my host have been doing some other testing on a separate server away from the usual shared server environment and I was still running into some problems, albeit at a higher level. He seemed to think it was quite a resource-intensive operation I was undertaking, especially with a 3- or 4-level hierarchy.

As you say, large multi-hierarchy imports need to be broken up a little to ensure they get through. That's not the end of the world as once imported, they're in for good so it's not something that will need to be done on a regular basis.

Comment #6

decafdennis commented 20 April 2008 at 14:12

I think the main issue is the number of lines (>8000), the depth of the hierarchy is less of a problem.

So if you break up the file into smaller files of 500 lines, does it fix your problem?

Comment #7

esllou commented 20 April 2008 at 17:14

I haven't had problems with files of that size...just that I've read about other users getting multi-mB files imported and this one is less than a mB.