Deploy Apache Solr Search Integration 6.x-3.x

The tentative schedule is test this week, and deploy live early next week.

Fresh dev sites for solr-drupal.redesign.devdrupal.org and solr-groups.redesign.devdrupal.org are now building, they will take about an hour.

Likely 7.x issues we night use for testing checklists #1700572: Manually test all existing ApacheSolr functionality in D6 to determine what, if anything, is still broken and #1699164: [Meta] Port drupal.org's solr search functionality to D7

Add another missing issue.

Found another issue

The current status is that the dev sites are still fresh, they need their codebases updated and need to be pointed at a blank core.

updatedb is kinda rocky:

Executing apachesolr_update_6301                                                                                                                                                                           [success]
Table 'solr_drupal.apachesolr_index_entities' doesn't exist                                                                                                                                 [warning]
query: /* 127.0.0.1 home */ ALTER TABLE apachesolr_index_entities DROP INDEX changed database.mysql.inc:172
Table 'solr_drupal.apachesolr_index_entities' doesn't exist                                                                                                                                 [warning]
query: /* 127.0.0.1 home */ ALTER TABLE apachesolr_index_entities ADD INDEX bundle_changed (bundle, changed) database.mysql.inc:172
Table 'solr_drupal.apachesolr_index_entities_node' doesn't exist                                                                                                                            [warning]
query: /* 127.0.0.1 home */ ALTER TABLE apachesolr_index_entities_node DROP INDEX changed database.mysql.inc:172
Table 'solr_drupal.apachesolr_index_entities_node' doesn't exist                                                                                                                            [warning]
query: /* 127.0.0.1 home */ ALTER TABLE apachesolr_index_entities_node ADD INDEX bundle_changed (bundle, changed) database.mysql.inc:172
ALTER TABLE {apachesolr_index_entities} DROP INDEX changed                                                                                                                                                 [error]
ALTER TABLE {apachesolr_index_entities} ADD INDEX bundle_changed (bundle, changed)                                                                                                                         [error]
ALTER TABLE {apachesolr_index_entities_node} DROP INDEX changed                                                                                                                                            [error]
ALTER TABLE {apachesolr_index_entities_node} ADD INDEX bundle_changed (bundle, changed)                                                                                                                    [error]
Executing apachesolr_search_update_7006                                                                                                                                                                    [success]
Invalid argument supplied for foreach() apachesolr_search.install:188                                                                                                                                      [warning]

We can either ignore these errors, and/or patch apachesolr to be more forgiving. Or maybe we need to do a complete uninstall of the previous version.

Executing drupalorg_search_update_6301                                                                                                                                                                     [success]
Table 'cache_apachesolr' already exists                                                                                                                                                          [warning]
query: /* 127.0.0.1 home */ CREATE TABLE cache_apachesolr (
`cid` VARCHAR(255) NOT NULL DEFAULT '',
`data` LONGBLOB DEFAULT NULL,
`expire` INT NOT NULL DEFAULT 0,
`created` INT NOT NULL DEFAULT 0,
`headers` TEXT DEFAULT NULL,
`serialized` SMALLINT NOT NULL DEFAULT 0,
PRIMARY KEY (cid),
INDEX expire (expire)
) /*!40100 DEFAULT CHARACTER SET utf8 */ database.mysql.inc:172

This is our custom code, we need to fix it.

More deployment details.

Quick todo

another todo

I updated the code on http://solr-drupal.redesign.devdrupal.org/ and configured solr. Running drush solr-mark-all, then solr-index says 114241 items successfully processed. 0 documents successfully sent to Solr.

Next step is figuring out why it is not sending documents.

another step

new dependencu

For staging - the new core is named 'do-staging' and will become active when jetty is restarted.

More issues

Filter by block config

more config

The staging core is ready to go when staging is.

I found another deployment step, updating project_solr, and added it to the issue summary. The dev site is working on reindexing projects. The is still uncommitted code in #1549496: Upgrade project to support 6.x-3.x version of Apachesolr (prep for d.o. upgrade to D7). I requested maintainership to do the commits, #1805376: Maintainership request for drumm.

Found another dependency

Update apachesolr version

Add version numbers

We already have cron limit in settings.php

More issues

More specific versions.

Can't forget the blue cheese.

Attached is a patch needed for bluecheese on deployment. I'm moving bluecheese_preprocess_search_result() into drupalorg_search.

fixed that

Correction, moved into drupalorg_crosssite so groups gets it too.

bluecheese patch

I've been looking into restoring the blocks on pages like http://drupal.org/project/modules. On 1.x, these blocks are made by drupalorg_order_facet.

drupalorg_order_facet in general hasn't been working well. drupalorg_order_facet_apachesolr_facets() had been throwing errors before I committed http://drupalcode.org/project/drupalorg.git/commitdiff/35be32c34d41f48ae... and http://drupalcode.org/project/drupalorg.git/commitdiff/5506830ceeea7e380.... Currently, inside drupalorg_order_facet_block(), both $enabled_facets and $facets are empty arrays. I was able to fill $enabled_facets with

-      $enabled_facets = facetapi_get_enabled_facets($env_id);
+      $enabled_facets = facetapi_get_enabled_facets('apachesolr@' . $env_id);

If this was working on 6.x-3.x, I'm thinking it could not have used drupalorg_order_facet module. facetapi itself only provides facets for drilling down into search results, not search results from a facet. Are we missing another module? How should this be working?

I went ahead and rewrote drupalorg_order_facet for #17. It has a lot less flexibility, and is a lot more simple. Adding new sorted result blocks requires editing the $blocks array and could be made more generic if someone else has a use for it.
http://drupalcode.org/project/drupalorg.git/commitdiff/dd1e628?hp=90d81c...

I also fixed up project_solr's category search. A value key was mismatched.
http://drupalcode.org/project/project_solr.git/commitdiff/97c1c22?hp=15b...

http://solr-drupal.redesign.devdrupal.org/project/themes and http://solr-drupal.redesign.devdrupal.org/project/modules seem good now.

Those seem unused, so I removed them, http://drupalcode.org/project/project_solr.git/commitdiff/10c6527?hp=990...

All dev sites have a reduced node table. Only old issues and forum posts are removed.

I'm going to go ahead with deploying this on staging for Drupal.org.

The groups deployment may lag a bit after Drupal.org. We'll have to keep the old index up for that, which is also our rollback plan.

Updated issue summary.

Handled with -dev

Branches merged

Done and done

Updating for staging deployment

Merges done.

For Drupal.org, we are going to try out http://drupal.org/project/apachesolr_url_switch on staging only. I have it configured with

$conf['apachesolr_url_switch_start_url'] = 'http://staging.drupal.org';
$conf['apachesolr_url_switch_end_url'] = 'http://drupal.org';

in settings.local.php.

I indexed http://staging.devdrupal.org/ overnight, it took about 8 hours with 4 threads, which is good news.

I got the hostname wrong in the above comment, so we definitely can't use that index directly. Worst case is that we have to do a live reindexing, that's faster than 8 hours or even faster if we can use more threads. The production DB is nicer than staging. MySQL is the bottleneck.

I spot checked most things and have noticed something funny with meta facets, for example http://staging.devdrupal.org/search/site/unicorn?f[0]=ss_meta_type%3Afor....

Please test http://staging.devdrupal.org/ for other issues.

After updating for #1808072: Backport D7-specific render array elements, I edited the facet and reverted it. It snapped back into place.

#25 isn't fixed. FacetAPI's ability to deal with block caching in D6 is non-existant.

Specifically, there is some D7 code commented out that might handle it.

This is the second holdup from incomplete backports in FacetAPI. Combine this with QA finding some problems and me writing the downtime announcement, http://drupal.org/node/1816068, too late to get reviewed, I think this has to be moved to early/mid-day PDT.

• I wrote #1817008: Work with block caching and went ahead and deployed to staging.
• I merged for #1792104: Handle redirecting /search/apachesolr_search to /search/site and that is deploying.
• I started re-indexing.

Next, I need to get groups indexing.

Attached is a screenshot of http://drupal.org/admin/settings/apachesolr/content-bias. We'll want to start with the same biases. I made a commit to exclude releases from indexing.

Groups: http://groups.drupal.org/admin/settings/apachesolr/content-bias doesn't bias any content types. We do exclude book pages, KDI proposals, and image node types from indexing.

Att 77% indexed, #1819312: Gracefully handle invalid UTF-8 happened. I edited the blocking node and it is continuing.

Groups indexing has started.

Indexing completed on both staging.devdrupal.org and groups.staging.devdrupal.org.

To-do:
- Revise deployment instructions to include transferring solr core and associated tracking tables from staging.
- Double check content bias settings, see #35. Automate if practical.
- Test more
- See about getting association.staging.devdrupal.org indexing
- More testing couldn't hurt

I merged facetapi 6.x-3.0-beta2 and apachesolr 6.x-3.0-rc1 to drupal.org and groups.drupal.org. And apachesolr_og 6.x-3.0 to groups only.

#1819542: D7 Upgrade QA Bug: Search - The link to first result in the solr search results page is wrong is a blocker.

Updated issue summary to include latest details regarding gdo deployment.

I trimmed the deployment instructions and checked them on staging.

More settings to copy over.

Deployed and 8% reindexed.

Indexing live is done and took 11 hours.