With a solr-index view where the Data export display is configured to use batched export, I consistently get the following error when trying to export the file:

Fatal error: Call to a member function addMetaData() on a non-object in .../sites/all/modules/contrib/views_data_export/plugins/views_data_export_plugin_display_export.inc on line 734

I started to look a bit closer; $view->build_info is in it's initial state, as though $view->build() had not executed. However calling $view->build() does not turn $view->build_info['query'] or $view->build_info['count_query'] into SelectQuery objects. They are still empty strings.

I did not look further.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Steven Jones’s picture

Title: Fatal error: addMetaData() on non-object » Restrict to adding to SQL base table
Priority: Normal » Critical

Yeah...this module isn't going to work with a Solr backend, as it needs the SQL to do the export magic. We should have made this more clear, and not let you use it on a non-SQL base table, but when we originally wrote the module this part of views wasn't stable and we couldn't reliably detect base tables we could be applied to safely.

We should.

Bevan’s picture

Most of the module works with the solr backend. Only some features are broken. Perhaps just those features can be disabled with non-SQL views backends?

rooby’s picture

Status: Needs work » Active

So is it only with batched that it breaks?

Bevan’s picture

rooby: There may be other features which are broken with non-SQL views backends such as Solr-indexes. I did not find any, but I did not look hard either.

rooby’s picture

Yeah, for my usage it is working OK without batching.

GaëlG’s picture

As Crell said in #1532472: Hard-coded dependency on SQL Views query plugin, batched export should be made available for any query back-end. I could try to work on it. Has someone any idea on where and how to start?

GaëlG’s picture

Currently, it only works when the query plugin is views_plugin_query_default. Search API declares its own query plugin: SearchApiViewsQuery.
I don't really know what would be the best way to do what I want. As far as I understand, the query is done at the beginning of the batch process, and the result is stored in a database table. Then, each batch iteration renders an amount of results.
This logic could be kept, but the implementation might be pretty different. If I'm right, results storing is done with this statement:

$insert_query = 'CREATE TABLE {' . $display_handler->index_tablename() . '} SELECT @row := @row + 1 AS ' . $display_handler->batched_execution_state->sandbox['weight_field_alias'] . ', ' . implode(', ', $select_aliases) . ' FROM (' . (string)$query . ') AS cl, (SELECT @row := 0) AS r';

This works because initial query is SQL. But of course this won't work for a Solr query.
And if we manage to get the results in the database in the same format, will that be OK with no changes to the rest of the code? Will rendering work? I don't know well how Views code works, which confuse me a bit actually. I don't want to lose much time in wrong ways. So, any help from a maintainer would be greatly appreciated.

Steven Jones’s picture

So, any help from a maintainer would be greatly appreciated.

Really sorry, the code is amazingly messy, and we've been always meaning to refactor the whole thing. We're attempting to carve out some time so that we can give this a couple of weeks of proper development.

GaëlG’s picture

OK, thank you for the answer. I suppose this means we just have to wait. Any idea about when it would be available (at least in dev)? For me to know if I have to find another way right now (custom code, switch to SQL backend, tell the customer it won't work for big exports,...) or if it will be done before project deadline.

GaëlG’s picture

Status: Active » Needs work
FileSize
4.65 KB

I could not wait more. Here's the hack I did to have it work with Search API Views. Not easy.

ptitwolfy’s picture

Thanks GaelG for your patch, seems to be working well on latest dev.

Sneakyvv’s picture

Status: Needs work » Needs review
FileSize
5.62 KB

The post from #1258390-10: Add support for batched exports of Search API views didn't work for me, particularly this part

@@ -586,13 +597,17 @@
     
       // Replace the query object.
       $view->query = $query;
+      
+      $view->execute();
+    }
+    elseif ($query_plugin == 'SearchApiViewsQuery') {
+      $this->store_search_api_result(clone($view));
     }
     else {
       exit(0);
       //TODO Make it work for other query plugins.      
     }
     
-    $view->execute();
   }

as the latest dev version doesn't have an if which should before line 586, nor could I find the else.

Therefore I attached an updated patch.

Status: Needs review » Needs work

The last submitted patch, views_data_export-solr_export-1258390-12.patch, failed testing.

Sneakyvv’s picture

Status: Needs work » Needs review
FileSize
5.77 KB

ignore patch from #12... I was switching my git repo between dev and beta and accidentally posted the beta-patch here.

Correct patch attached

rerooting’s picture

Just used this, appears to work just fine with search api with both batch export mode and the normal export.

Strangely, I did not need to use the patch you provided in https://drupal.org/node/2080297#comment-7822017 for my batch export... using views_data_export development (not git)

guillaumev’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

I can confirm this patch works

milesw’s picture

Great patch. I can confirm this allows batching to work with Search API Views with a database backend.

MichaelGow’s picture

The patch certainly did fix the "Call to a member function addMetaData() on a non-object" for View based on Multi-index searches 7.x-1.1 for Search API 7.x-1.11 using Solr search 7.x-1.4 , but there still remains error:

PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table '...views_data_export_index_x' doesn't exist: SELECT COUNT(*) FROM {views_data_export_index_x}; Array ( ) in views_data_export_plugin_display_export->execute_initial() (line 301 of ../sites/all/modules/views_data_export/plugins/views_data_export_plugin_display_export.inc).

caco13’s picture

#14 worked for me too.

Great patch!

opi’s picture

Confirm that #14 ' patch allows me to do batched export using SearchAPI and a database backend (not a Solr one). Thanks for this patch !

vikramy’s picture

#14 works for me too. Thank you.

opi’s picture

#14 works for me again on last release : 7.x-3.0-beta9

joelhsmith’s picture

I was using Faceted Search, SearchAPI (not with Solr). I can confirm #14 works in this case also.

ignat.volosky’s picture

I have applied patch from #14 and it helped me. But when I try to start another export from database (not SOLR), I get next:
"PDOException : SQLSTATE[42S02]: Base table or view not found: 1146 Table 'blabla.views_data_export_index_16' doesn't exist" on line 301 in plugins/views_data_export_plugin_display_export.inc

This export worked good before patching. I use beta9 version.

ignat.volosky’s picture

I have found a problem. Patch add the condition by query plugin class name in initialize_index method - 'SearchApiViewsQuery' for search_api based queries, and 'views_plugin_query_default' for others. But my views use 'views_contextual_filters_or_query' and that was a problem, index for view wasn't initialized.

drumm’s picture

Issue tags: +affects drupal.org

We ran into this on Jobs.Drupal.org, which uses Search API and has reports.

The patch still applies and works for us.

opi’s picture

Patch from #14 still apply fine on latest stable release 7.x-3.1, and allows me to do batched export with a SQL backend.

Loparev’s picture

Patch from #14 still apply fine on latest stable release 7.x-3.1, and allows me to do batched export with a SQL backend.

+

Steven Jones’s picture

Title: Restrict to adding to SQL base table » Add support for batched exports of Search API views

Thanks for the work on this.

I'm really torn between committing this, and sending this back for more work, specifically because adding switch statements just seems messy.

However, as this issue shows we're rubbish at getting around to committing things, so I'm just going to throw it into the dev branch anyway.

  • Steven Jones committed 06aca43 on 7.x-3.x authored by Sneakyvv
    Issue #1258390 by Sneakyvv, GaëlG: Add support for batched exports of...
Steven Jones’s picture

Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.