So I've got a hierarchical taxonomy to import of about 70K entries to start with. I'm using taxonomy csv import to attempt to get this done (D7).

I can start it up and get about 9K entries in before it pukes because of DB timeouts. If I split it up into 5K chunks, I can get a little over 3 of those in before it pukes (DB timeout errors).

I am in the process of attempting to try it from scratch again, and can't just delete the taxonomy because of DB timeouts.

hints from people who might be experienced in such head banging would be greatly appreciated.

Comments

Daniel_KM’s picture

Daniel_KM commented

Hi,

Huge import is always a problem. Some hints:

  • increase memory of your php server
  • remove checks of lines in the third tab
  • choose "Ignore existing items" in the fourth tab (if you haven't any duplicates or if duplicates are true ones, what is normally the case in a tree)
  • remove any information in the last tab (no basic stats, no list, and only first warning and notice)
  • disable any module which uses taxonomy directly or indirectly (with hook) and reenable them after import
  • use Drush import, with same options, which use less memory than Drupal UI.

As you have probably try all of these hints, you can wait a few days for the next dev release: I've just removed a compatibility layer with Taxonomy CSV for Drupal 6 and it results in a lower usage of memory and cpu.

Other solutions:

  • If you have a simple taxonomy, you can use Taxonomy Builder API too.
  • If you have a very simple taxonomy (without parents and relations), you can import it directly in your database in the table taxonomy_term_data, and after, add your tids with 0 as parent in the table taxonomy_term_hierarchy. It is not recommended, but it works fine if you have no modules related to taxonomy, directly or indirectly. Taxonomy CSV module uses only core taxonomy functions to allow invocation of all hooked modules.

Sincerely,

Daniel Berthereau
Infodoc & Knowledge management