Doesn't import large csv files? [#81505]

I'm having a problem where a CSV file with 1900 items (1.9MB) will import fine, but with 2000 items (2.15MB) it gets ignored: node_import just says "You must select a file to import.".

The php.ini limits seem big enough (post_max_size and upload_max_filesize) and there's nothing in Apache's error.log, so I'm not sure where the problem might be at the moment. Is there a PHP limit I might be missing somewhere?

Comments

Comment #1

Robrecht Jacques CreditAttribution: Robrecht Jacques commented 31 August 2006 at 19:30

It might be the execution time limit and/or the memory limit.

Although I think this should only hit you when you go to the last page (the actual import).

Please investigate and let me know. I'll try to look at it too.

Comment #2

hickory CreditAttribution: hickory commented 1 September 2006 at 10:14

Status:

Active

» Closed (fixed)

It seems to have resolved itself after all, and I think max_post_size was responsible - perhaps Apache didn't restart properly when I tried it yesterday so didn't pick up the changed setting.

Comment #3

hickory CreditAttribution: hickory commented 1 September 2006 at 10:23

post_max_size, I mean.

Comment #4

Vacilando CreditAttribution: Vacilando commented 11 February 2008 at 13:56

Status:

Closed (fixed)

» Active

I try to import 96 MB of data (some 90 thousand little profiles) and, because of the size and timeouts I have to upload in batches (per 10,000 profiles).

Is it possible to upload the file to the server and have it parsed from there, rather than have to upload the file at each import? If so, where can I upload the file and how should I point to it from Node Import?

Thanks!

Comment #5

petya_vulcheva CreditAttribution: petya_vulcheva commented 2 June 2008 at 13:27

Hi,

I think I may have the same problem. When I export the csv file it is working at some point and then it doen's fill the data from the profile fields, but only for some users. My csv file is from around 9000 users. When I run it with small number everything is OK. Does anyone have some explanation?
Thanks!

Comment #6

zeezhao CreditAttribution: zeezhao commented 13 June 2008 at 08:52

See also some tips to improve MySql for large imports here: http://drupal.org/node/259580

and also use the set_time_limit() tip in this thread: http://drupal.org/node/191197

I am experimenting with importing a 250MB file with over 1 million records...

It now works without breaking after using suggestions from those two links. But I can't complete it in one shot, as it is going at the rate of about 30,000 rows/hr so will need days to complete....

So for me, the outstanding question is how to speed up node_import.