Doesn't import large csv files?
hickory - August 30, 2006 - 13:27
| Project: | Node import |
| Version: | HEAD |
| Component: | Code |
| Category: | support request |
| Priority: | normal |
| Assigned: | Unassigned |
| Status: | active |
Jump to:
Description
I'm having a problem where a CSV file with 1900 items (1.9MB) will import fine, but with 2000 items (2.15MB) it gets ignored: node_import just says "You must select a file to import.".
The php.ini limits seem big enough (post_max_size and upload_max_filesize) and there's nothing in Apache's error.log, so I'm not sure where the problem might be at the moment. Is there a PHP limit I might be missing somewhere?
#1
It might be the execution time limit and/or the memory limit.
Although I think this should only hit you when you go to the last page (the actual import).
Please investigate and let me know. I'll try to look at it too.
#2
It seems to have resolved itself after all, and I think max_post_size was responsible - perhaps Apache didn't restart properly when I tried it yesterday so didn't pick up the changed setting.
#3
post_max_size, I mean.
#4
I try to import 96 MB of data (some 90 thousand little profiles) and, because of the size and timeouts I have to upload in batches (per 10,000 profiles).
Is it possible to upload the file to the server and have it parsed from there, rather than have to upload the file at each import? If so, where can I upload the file and how should I point to it from Node Import?
Thanks!
#5
Hi,
I think I may have the same problem. When I export the csv file it is working at some point and then it doen's fill the data from the profile fields, but only for some users. My csv file is from around 9000 users. When I run it with small number everything is OK. Does anyone have some explanation?
Thanks!
#6
See also some tips to improve MySql for large imports here: http://drupal.org/node/259580
and also use the set_time_limit() tip in this thread: http://drupal.org/node/191197
I am experimenting with importing a 250MB file with over 1 million records...
It now works without breaking after using suggestions from those two links. But I can't complete it in one shot, as it is going at the rate of about 30,000 rows/hr so will need days to complete....
So for me, the outstanding question is how to speed up node_import.