Index bundles Export/Import

Is this going to integrate the Profile2 entity?
Thanks.

-Youssef

Well done recidive. Please keep us updated and if possible also provide an export file so we can test this functionality?
It would be even better if there would be simpletests that can test this functionality.

is this a duplicate?

We all love the testbot ;-)

+++ b/apachesolr.moduleundefined
@@ -1035,6 +1035,12 @@ function apachesolr_get_solr($env_id = NULL) {
+  if (!db_table_exists('apachesolr_index_bundles')) {
+    return array();
+  }

Can't we solve the apachesolr_load_all_environments problem in a deeper stage? As in : Should we modify the module so it does not call this load all environments if the tables do not exists?

+++ b/apachesolr.moduleundefined
@@ -2429,6 +2426,60 @@ function theme_apachesolr_settings_title($vars) {
+function apachesolr_environment_load_subrecords(&$environments) {

Subrecords? Is there a more descriptive name for this? Subrecords seem very confusing for people that will go through the API/function calls
It also seems that we should be able to do this using the main functions? Looks like duplication

Correct, we do not want ctools as a dependency since lots of high-end websites still prefer to have maximum control.
However, nothing stops us from moving forward and build ctools integration (as we are doing). There might come a moment where ctools will be accepted globally (Maybe even part of Drupal 8) and then we can utilise those functions natively.

Let's find out where this call comes from. Afaik I don't see a function call before any of the tables are created.

Regarding the ctools issue, i think that nearly all people use ctools in d7 and compared to, for example apachesolr, it's actually a small amount of code if you don't use other plugins then /includes/export.inc.

The current patch did't worked for me because of this code:

+    $index_bundles = !empty($all_index_bundles_keyed[$env_id]) ? $all_index_bundles_keyed[$env_id] : array();
+    $conf = !empty($variables[$env_id]) ? $variables[$env_id] : array();

This overrides the config settings everytime for index_bundles and conf, because either the value from the db is used: which is fine, or array() is used, which isn't fine, because we want to use the default value in the code.

So i added some additional checks:

+    if (is_array($environment)) {
+      // Environment is an array.
+      $environment['index_bundles'] = (empty($environment['index_bundles']) || !empty($index_bundles)) ? $index_bundles : $environment['index_bundles'];
+      $environment['conf'] = (empty($environment['conf']) || !empty($conf)) ? $conf : $environment['conf'];
+    }
+    elseif (is_object($environment)) {
+      // Environment is an object.
+      $environment->index_bundles = (empty($environment->index_bundles) || !empty($index_bundles)) ? $index_bundles : $environment->index_bundles;
+      $environment->conf = (empty($environment->conf) || !empty($conf)) ? $conf : $environment->conf;

Regarding the install table problematic: I guess entity_get_info could be called really early, though some backtrace in the if() didn't got fired.

#12: 1439564.patch queued for re-testing.

I think this is a duplicate of #1498490: Menu paths for custom search pages are broken when exported as features - please correct me if I'm wrong.
The only thing that is not exportable right now is the index bundles I think, would you be so kind and open up a new topic for that?

- Nick

Why not keep this issue just for Index bundles?

I've tested #17 and it works just fine. As for the code, the apachesolr_environment_features_revert() can be removed in favor of converting all exportable solr things (environments, pages) to objects. This will leverage using standard ctools export functions. I can prepare the patch, if maintainer thinks that this can be potentially commited. This will be pretty huge patch, so I need to be sure before proceeding to avoid wasting of time.

Re-roll of #21 agains current dev version.

Also, included a fix of 2 errors:

• If you try to save bundle and don't see update after refresh (because of cache).
• If you try to add new node bundle, it throws musql duplicate key error, since there are already records in index for all node bundles after reindex.

Items in #21 are still relevant.

Attached wrong file, sorry. This one is good to go.

Another re-roll.

#26: solr-index-bundles-export-1439564.patch queued for re-testing.

The patch in #26 no longer applies cleanly, and downright fails in one part due to some refactoring done in #1519900: Error in apachesolr_index_get_entities_to_index() function (commit c5bd5c7e1f). Here's a reroll that *skips* this code change:

index d1e7de6..9f685db 100644
--- a/apachesolr.index.inc
+++ b/apachesolr.index.inc
@@ -700,6 +700,10 @@ function apachesolr_index_node_bundles_changed($env_id, $existing_bundles, $new_
         ->execute();
     }
     if ($added_bundles) {
+      db_delete($indexer_table)
+        ->condition('entity_type', 'node')
+        ->condition('bundle', $added_bundles)
+        ->execute();
       $select = db_select('node', 'n');
       $select->addExpression("'node'", 'entity_type');
       $select->condition('type', $added_bundles);

It seems to work w/o this in my local testing.

I had a hard time reviewing this code change because there is some code that is being moved and mainly the subrecords function is a bit strange

I came up with a revert link, similar to the search pages revert link but then for the environments

When you revert an environment, it will delete the DB copy it has and use the code. This now also works for the index bundles. Before, if you reverted it did not correctly override the settings you had in the database.

Only missing part now is when you disable all the indexed bundles, it will revert to the code change since disabling all indexed bundles = removing the database entries.

cleaner, the features revert hook was not necessary

Committed to 7.x-1.x-dev.
Thanks to all!

Huge patch for Apache Solr 6.x-3.x

- Converted all search pages to objects because of caching troubles and ctools troubles in D6
- Fixed the delete link for more like this blocks
- Search pages can be exported/imported/reverted
- Search Environments + Bundles can be exported/imported/reverted
- Fixed solr index tests for drupal 6

obviously the main thing is consistency - this needs to be applied to 7.x also?

Committed this to 6.x-3.x dev. Closing this one and continuing in #1702526: apachesolr_search_page_load is quirky

Hi Nick,

Can I ask why the features revert hook was removed? (in comment #31 http://drupal.org/node/1439564#comment-6272474).

I have been having trouble with websites whereby the index bundles would export fine (into features) but never revert correctly on each deployment (causing a complete reindex to be necessary on each deploy).

The features revert hook solves the problem.

As this functionality should leverage standard ctools import/export tools and not features hooks. I prefer not to include features hooks in the module.
What is it that breaks? Could we solve it differently?

Sorry to resurrect this issue...

I am having the same issue as described in #40. I havent touched my apachesolr settings and I noticed my_search module (created with features 7.x-2.x) is marked as overridden. So I ran this:

$ drush fd my_search
Legend:
Code:       drush features-revert will remove the overrides.
Overrides:  drush features-update will update the exported feature with the displayed overrides


Component type: apachesolr_environment
      ),
      'index_bundles' => array(
<       'drealty_listing' => array(
<         0 => 'basic_drealty_listing',
<       ),
        'node' => array(
          0 => 'aggregation_page_detail',
          1 => 'listing',
          2 => 'page',
>         3 => 'webform',
        ),
      ),
    ),
  )

running drush fr my_search does nothing...

The GUI confirms these setting and if I fix it in the gui then I can update the feature and all is well, however this happened again a few weeks later. Again I made no changes to these settings.

I'm not sure is this is related but without the features revert hook, on revert the environment delete callback is called but the environment save callback - not. And after feature revert a have empty both apachesolr_environment and apachesolr_environment_variable table. To fix that I had to add 'apachesolr_environment_features_revert' function in my module and save apachesolr environment there. In this case save callback called, as I'm using it in 'apachesolr_environment_features_revert' function and delete callback is not.

I think I'm experiencing the same issue as described in #40, #42, #43.

Here's what I have observed:

I have a feature in code with an apachesolr environment. There are no overrides initially, so the feature is completely in-code, i.e. the database table `apachesolr_environment_variable` is empty.

Now, when I edit e.g. the environment settings, some variable values are stored in the database table (namely apachesolr_access_always_add_grants, apachesolr_direct_commit, apachesolr_multilingual_index_settings, apachesolr_read_only, apachesolr_soft_commit). Other variables, e.g. for bias configuration (apachesolr_search_changed_boost, apachesolr_search_comment_boost, apachesolr_search_date_boost, apachesolr_search_promote_boost, apachesolr_search_sticky_boost, apachesolr_search_type_boosts, field_bias) are NOT saved in the database table. The same thing happens vice-versa, when I save bias configuration instead.

In either case, my feature now appears overridden, and the values which haven't been saved to the database (e.g. bias configuration) appear as "deleted" even though I didn't touch them.

Now, reverting the feature only works if I edited the environment settings before; if I changed the bias settings instead, the feature also appears as overridden but reverting has no effect. I think this is because "saving bias settings" only actually saves respective "environment variables", but does not insert the "environment" itself into the database, which means that ctools cannot find the exported object in the database, and therefore cannot delete it (=revert it back to code).

The problem with reverting index "index_bundles" (as described in #42) most likely has the very same issue.

I can imagine two solutions to this problem:
a) Make ctools exportables work with having "partial objects" in the database. I'm not sure this is a good idea though, because, e.g., I can't imagine how ctools would be able to distinguish between values which were "not imported" versus deleted ones.
b) Make sure that the exported object is fully saved to the database, before trying to make changes to it or any data related to it. In other words: make sure we don't accidentially get "partial objects" in the database.

I have tried to implemented b) in attached patch; luckily ctools keeps track off whether an objects is entirely "in code only" via a respective flag, so we can simply check for that flag and, if it is set, save the complete environment to the database.
I haven't tested the patch extensively yet, but it does seem to fix the problem for me. One thing I'm not sure about is whether drupal_merge_array_deep() will work properly in all cases, so more testers would be welcome, I guess :-)

Tested #44 + the patch found at https://drupal.org/node/1984502#comment-8726587, and it seems I can properly export / revert apachesolr features, to deploy them across environments, thanks!

The only remark I have here is that we're copying the same code 3 times. Can someone make a patch that puts this code in a function so we only need to call a function?

Can someone make a patch that puts this code in a function so we only need to call a function?

Sure, here you go. :)

+++ b/apachesolr.module
@@ -1433,6 +1443,22 @@ function apachesolr_environment_variable_del($env_id, $name) {
 /**
+ * Makes sure that the given environment has been saved to the database.
+ *
+ * This is a required step before any environment-related data is modified or
+ * deleted. It ensures that ctools exportables can properly determine whether
+ * something has been overridden.
+ *
+ * @see https://www.drupal.org/node/1439564#comment-8727467
+ */

Missing the @param parameter. Almost!

Right, should've thought of that. ;-)

Committed. Please close this if this is impossible to backport to 6.x-3.x but due to the possibility I'm setting it to needs backport.

Thanks for your contribution. It's very much appreciated and I hope you continue to contribute to our issue queue. We can use all the help we can get.