Using CSV export with 27,000 nodeprofile nodes resulted in "Allowed memory size .. exhausted" errors, even with the memory limit set to 544MB.

My solution is relatively simple and should be considered by anyone with a large amount of data needing export.

In views_bonus_export.module, replace this function with the code below..


/**
 *  Main Function to export a view as CSV
 */
function theme_views_bonus_export_csv($view, $nodes) {
  if (!user_access('export views')) {
    return;
  }
  $fields = _views_get_fields();

  // headings row
  $headings = array();
  foreach ($view->field as $field) {
    if ($fields[$field['id']]['visible'] !== false) {
      $headings[] = $field['label'] ? $field['label'] : $fields[$field['fullname']]['name'];
    }
  }

  drupal_set_header('Content-Type: text/x-comma-separated-values');
  drupal_set_header('Content-Disposition: attachment; filename="view-'. $view->name .'.csv"');
  print implode(',', $headings) ."\r\n";

  // one row for each node
  foreach ($nodes as $node) {
    $values = array();
    foreach ($view->field as $field) {
      if ($fields[$field['id']]['visible'] !== false) {
        // I don't believe this was really necessary.
        // $value = $field;
        $value = views_theme_field('views_handle_field', $field['queryname'], $fields, $field, $node, $view);
        // Replace preg_replace with strip_tags.  My output generates at 40KB/s instead of 20KB/s with this change alone.
        $value = strip_tags($value);
        $value = str_replace(array("\r", "\n", ','), ' ', $value); // strip line breaks and commas
        $value = str_replace('"', '""', $value); // escape " characters
        $value = decode_entities($value);
        $values[] = '"' . $value . '"';
      }
    }

    // Output immediately instead of using $output .=
    print implode(',', $values) . "\r\n";
    // Flush immediately so the user can start downloading and gets an immediate response.
    @ob_flush(); @flush();
  }

  module_invoke_all('exit');
  exit;
}


FYI -- I am posting this to http://drupal.org/node/114115 also, but the patch should be discussed here.

Comments

jason.fisher’s picture

Sorry, I discovered that we need to flush the node_load static $nodes array() cache manually to keep memory usage low. The problem is that in exporting thousands of nodes with relations and terms, each stays in memory forever and drives the cache up -- I would have needed to ini_set the memory limit to 1.5GB in my case.

The functionality of flushing this cache could be improved if we did it less often, possibly by monitoring memory usage or number of iterations -- not flushing it after every load, basically.

Unfortunately I cannot edit the original above, and should have kept the code in a comment. The version you should use is here:

/**
*  Main Function to export a view as CSV
*/
function theme_views_bonus_export_csv($view, $nodes) {
  if (!user_access('export views')) {
    return;
  }
  $fields = _views_get_fields();

  // headings row
  $headings = array();
  foreach ($view->field as $field) {
    if ($fields[$field['id']]['visible'] !== false) {
      $headings[] = $field['label'] ? $field['label'] : $fields[$field['fullname']]['name'];
    }
  }

  drupal_set_header('Content-Type: text/x-comma-separated-values');
  drupal_set_header('Content-Disposition: attachment; filename="view-'. $view->name .'.csv"');
  print implode(',', $headings) ."\r\n";

  // one row for each node
  foreach ($nodes as $node) {
    $values = array();
    foreach ($view->field as $field) {
      if ($fields[$field['id']]['visible'] !== false) {
        // I don't believe this was really necessary.
        // $value = $field;
        $value = views_theme_field('views_handle_field', $field['queryname'], $fields, $field, $node, $view);
        // Replace preg_replace with strip_tags.  My output generates at 40KB/s instead of 20KB/s with this change alone.
        $value = strip_tags($value);
        $value = str_replace(array("\r", "\n", ','), ' ', $value); // strip line breaks and commas
        $value = str_replace('"', '""', $value); // escape " characters
        $value = decode_entities($value);
        $values[] = '"' . $value . '"';
      }
    }

    // Output immediately instead of using $output .=
    print implode(',', $values) . "\r\n";
    // Flush immediately so the user can start downloading and gets an immediate response.
    @ob_flush(); @flush();
   // This flushes the node_load static $nodes array() cache, fixing the memory limit problem.
   node_load(array(), NULL, TRUE);


  }

  module_invoke_all('exit');
  exit;
}
dmitrig01’s picture

Status: Needs review » Fixed

Comitted. I hope my fix will work.

Anonymous’s picture

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for two weeks with no activity.

bryancasler’s picture

#1 worked for me just had to increase my memory limit.