Improve webform.report.inc abstraction to support drush/batch processing

I can't see anything of a Drush script (at least from the patch), but could you make that a separate issue anyway? Seems like it'd be dependent on this patch but it should still be a separate issue nonetheless.

The patch itself looks pretty great. I appreciate your going through and documenting all those functions too.

A few of these changes were handled in a separate patch #1292252: Update the exporting mechanism to make it extendable by other modules, which I imagine would be somewhat helpful for your purposes also. This patch unfortunately now needs a reroll for compatibility.

Patch attached for 7.x-3.17.

This brings Drush integration to webform. @Owen Barton did the original d6 patch and I just modified it to work with d7.

Has this patch been incorporated anywhere? It looks like it is for a version pre 6x-3.17 Thanks.

No it hasn't been implemented. New features (and especially API changes) are only being added to 4.x at this point. At least this eliminates the need for a D6 backport, though it probably still needs a reroll for 7.x-4.x.

Here's a reroll against 7.x-4.x

Thanks @eporama! This patch looks pretty good but I've got a few reservations:

- Shouldn't the drush command be able to set options that match the interface? Or at the very least, be able to choose the exporter (delimited or Excel for example)?
- The calls to webform_results_download_options($node, $options, $submission_information) are confusing (taking two empty variables by reference and populating them). For clarity it would be preferable to have separate functions with a single return value each, one that initializes $options and one that initializes $submission_information. Calls by reference are always a little difficult to follow, but initializing empty variables through them is more confusing yet.

Looks great overall, I just fear maintaining this :P

I'm in the same situation as others (large submission sets, results downloads time out) and I've read through this thread (and the one about running Webform downloads through batch - awesome idea). It seems like my best option for a fast solution is to apply the #5 patch and use Drush to get the CSVs.

Can anyone confirm that this patch is safe to apply against 3.18 in a multisite env. with several sites sharing the module and several active webforms on each? I'll test on my own, of course. Just interested in any feedback I can find in advance.

Thanks!
-J

For anyone looking into this, I applied the patch from #5 to Webform 3.18 and it appears to work as advertised. One chunk in the patch did fail, but it was a nice clean block (a whole function), so it was easy to add it manually. After that, I ran drush wfx [nid] > dumpfile.csv and was able to dump over 7000 submissions without error to create a 3.5MB CSV file (it's tab-separated, btw, not comma). Using the GUI, this dump failed every time. Definitely a step in the right direction!

Patch adapted for 7.x-3.18 version

When I want to export submissions from a webform using the command drush webform-export nid=7343, the following error occurs:
Invalid argument supplied for foreach() webform.components.inc:935
Thanks for your help

Thank's @Owen Barton. It's working like a charm :-)

Assigning to version 7.x-4.x as according to #8:

New features (and especially API changes) are only being added to 4.x at this point.

I don't mind that the drush option only exports csv and doesn't allow other options - those can always be added in later (and csv is easy to change into any other format by Excel).

I agree with quicksketch at #10: the calls by reference should be eliminated in favour of separate functions.

Version change.

Thanks @burningdog. I'd still love to see this patch move forward. Batching our CSV files would be a great improvement for forms with a lot of submissions. The Drush handling is a nice-to-have on top of that.

Here's a reroll of this patch that makes my requested changes plus some additional refactoring:

• No variables are instantiated in a pass-by-reference approach, instead separate function calls are used to return a single value. This results in some cleaner code. There's a lot less nesting in this version and all of the control logic is in a single place. See the new webform_results_export() for example:
  

    $handle = @fopen($file_name, 'w'); // The @ suppresses errors.
    $exporter->bof($handle);
  
    $headers = webform_results_download_headers($node, $options);
  
    // Add headers to the file.
    foreach ($headers as $row) {
      $exporter->add_row($handle, $row);
    }
  
    $rows = webform_results_download_rows($node, $options);
  
    // Write data from submissions.
    foreach ($rows as $row) {
      $exporter->add_row($handle, $row);
    }
  
    // Add the closing bytes.
    $exporter->eof($handle);
  
    // Close the file.
    @fclose($handle);
  

  Ahh, much better. :)

• All the options supported by the Webform 4.x UI are now supported via Drush. i.e. drush wfx --range-latest=10 or drush wfx --delimiter="\t" --range-start=10 range-end=100. However there's a followup I'd like to do here to change how start/end ranges work: #2041221: Submission range options should be by serial number, not by SID.
• In order to make the --range-latest option work, I had to make a new function webform_download_latest_start_sid() to determine the start submission ID. Previously we were doing a "range" when getting the latest submissions, but since the batch process utilizes a "range" method and we can only have one range per query, I had to alter the query to determine the starting SID first, then let the batch process add its range on top of the new query.

I've committed #20 to the project's 7.x-4.x. Thanks Owen Barton for your work on this long, long ago. Let's finish up #1327186: Use BatchAPI to Export very large data sets to CSV/Excel now, that's the more important one in my mind.

The patch from #20 applies nearly unmodified to webform-7.x-3.x. Here is a re-roll that fixes the only rejected hunk and makes it apply again.

We have been using a modified version of this patch on the 7.x-3.x release for some time, and realised others would benefit from our work. We would love to see the enclosed patch, which applies against the 7.x-3.x branch, included in a release.

Apologies to all, this should've been 7.x-3.x-dev.

Hi there. Thanks for publishing your work in the issue queue. 7.x-3.x will receive only critical fixes. Your patch might still be useful for others though.

Cheers @torotil, I suspected that would be the case, but figured our patch could be beneficial to others so I should post it. It also aids out internal workflow when we can refer to a patch on d.o.