I have a list of cities in CSV Ancestry format, which has almost 30 000 rows.
When I try to import it, it takes from one to three seconds on one row and sometimes just stops with

PHP Fatal error: Maximum execution time of 30 seconds exceeded in includes/database.mysqli.inc on line 128

Comments

int_ua’s picture

int_ua commented

I tried to increase max_execution_time, but it just loops on the line with new region name.

int_ua’s picture

int_ua commented

Oh, that region name was already in the database.

int_ua’s picture

int_ua commented

And it also loops if third level name equals the second one.

dman’s picture

dman commented

Yeah, well some of that is inevitable.
A single CSV file with 30,000 rows is going to be limited by the capabilities of the PHP process that needs to be able to read it all into memory before even starting to process. It's not stream-based.
I wouldn't expect *seconds* per row, but I don't know about the php-native CSV reader. Maybe it's maxing out on the file read.
Plus, as it seems you've found, there is no provision for string-match duplicates if the source data has the same name more than once in the same file. That's not really allowed in any examples I've used so far.

CSV Ancestry is going to be limited by those sorts of issues.
You MAY be able to get better results when doing a huge job like that from the commandline... there are ways to do that but they are not really documented.

When it comes to huge input imports, we need to go into batch mode.
And ... if your only source of data is one huge file, then batch mode cannot help.

Can you try splitting your input into chunks (say 1000 items) and doing them? CSV Ancestry *may* be able to manage that ... but still has not been built to manage string-duplicates. To avoid string-duplicate issues (and also solve the batch problem) is why I tend to use RDF with GUIDs for mega-imports. CSV is deliberately simpler than that. But that is why it can't scale.

For your task .. I don't really know what to suggest.
If you skim the approach described here #460920: HOWTO: Import the MeSH taxonomy database. Or a subset. you may see the difficulties I've had to work around on big stuff