I am working on an module that is handling importing relatively large amounts of data using the Drupal 6 batch API.

The files that I am importing are small CSV's that contain up to 16000 rows of data. The files themselves are small (less than 2mb) but my routine is white-screening on files that have these large amounts of rows.

I have spent all day researching and the best I can figure is that the batch routine is running out of memory. I have tried many things, including trying to split the batch up into multiple batches. But, apparently the batch doesn't start until all batches have been processed.

Doesn't anybody have any suggestions on how to handle this??

Example:

This is called from a form submit:

>
function example_submit($form, &$form_state) {
// setup file...
  $batch = array(
    'operations' => array(),
			'finished' => 'example_finished',
			'title' => t('Importing records from CSV file'),
			'init_message' => t('Starting import...'),
			'progress_message' => t('Imported @current out of @total lines.'),
			'error_message' => t('An error occurred during the import.'),
  );
	
// more file handling...
    $batch['operations'][] = array('example_import_line', array(array_map('base64_encode', $line), $options));
	  batch_set($batch);
}
<

Note: the example_import_line() function is semi-complex and handles parsing the row from the CSV, building the record and inserting it into a table.

Any ideas as to how to handle these long CSV files will be greatly appreciated!!

Comments

beautifulmind’s picture

I have similar situation but I didn't calling the batch from the form submit. Finally, I ended up with normal routine.

By the way, did you figure out the issue and was able to fix it? Would you please share it?

Regards.

Regards.
🪷 Beautifulmind

begun’s picture

I would also like to know if you found a solution.