Core search index not complete

Our core search index, which is used for the issue searches at the moment says "96% of the site has been indexed. There are 11854 items left to index.". That would give reasons to why the recent issues you cite do not appear. We are either not running the core search indexing cron yet, or it did not complete yet.

As an example without having to think of a manual search, compare this search http://drupal.org/project/issues/search/webform?text=useremai&assigned=&... with this issue: http://drupal.org/node/357687.

Gabor mailed to the infrastructure list about this. In short: due to a variety of reasons the search index had to be rebuilt.

8% of the site has been indexed. There are 335335 items left to index.

We're going to have at least a few hours (maybe days) before it's all done, folks...

Possibly related to the original bug reported here: #373760: Views 2 Search Term Exposed Filter does not provide results when searched 2+ multiple words

@greggles: that might be unrelated. The Views use the core search index to look up stuff, while the rest of the site (ApacheSolr search and project browsing) use the ApacheSolr index. Both needs to be up to date. Interestingly, however, both core search and ApacheSolr search reports "66% of the site has been indexed. There are 123778 items left to index". Not sure how it is that both says the same, but we will see later as the indexer crawls forward.

The core search index says it is fully up to date now.

Something is broken, then. Behold:

mysql> select * from search_index where word = 'stale';
+-------+--------+------+----------+
| word  | sid    | type | score    |
+-------+--------+------+----------+
| stale | 147310 | node |  0.28248 | 
| stale | 196070 | node |        1 | 
| stale | 206820 | node |  4.60242 | 
| stale | 231190 | node |        1 | 
| stale | 235035 | node |  1.90974 | 
| stale | 262502 | node |  1.70252 | 
| stale | 267783 | node | 0.264083 | 
| stale | 276111 | node | 0.766757 | 
| stale | 285247 | node |        1 | 
| stale | 304388 | node | 0.473576 | 
| stale | 311472 | node | 0.646943 | 
| stale | 323925 | node |        1 | 
+-------+--------+------+----------+
12 rows in set (0.00 sec)

Why isn't #184418: update(_status) results stale for up to 6 hours in there?

I noticed this in the FeedAPI queue, too.

A search for cron (http://drupal.org/project/issues/search/feedapi?text=cron&assigned=&subm...) reveals a mere 3 issues:

#311039: Use simplepie project instead of including own version
#321422: Feed Nodes are not getting deleted after the set time
#320712: Large Volume of Error Messages on Cron Run

Yet, when you're on the front page of the results page you can clearly see:

#380020: Cron fails to import new items
#375119: Possible to update Automatically without cron, because one of mine does
#366267: Feed item nodes are recreated everytime when cron runs
#344381: Wrong date for items created by cron
#370318: Keep exceeding cron time with FeedAPI

..and several others.

Also, bumping to critical. The advanced search not working is vastly increasing the number of duplicate issues being created.

The apachesolr_solr module responds to search module hooks, so it's indexing stars are part of the combined total displayed by the search module.

Ok, we are running with the core searches patches and the node_search module was not enabled. I've enabled it, but commented out its exposed "Content" tab for the search page, which would confuse the hell out of people at least. Now, the search indexing meter for solr is still 99%, but the overall search index meter says 49% at http://drupal.org/admin/settings/search so we need to wait some time until this is fully indexed again.

I've also set the number of nodes to index per cron run to 500. If that is too big of a task for the server, we can turn that back to 200 as it was, or even lower.

FYI: that views issue was moved into #362830: Search filter breaks for multiple word searches when view has a sort, which is potentially still relevant for the original point of this issue... Anyone who can test out that patch is encouraged to do so.

Ok, looks like the core search index is not growing in any way, the number of items to reindex just keeps growing. Hrm.

Is this also why user search is returning "no results" on every name I've tried, or should I open a separate issue for that?

@webchick: we found bugs in node_search.module as deployed on d.o. Gabor commented out a line of code that ended up breaking core search indexing entirely. ;) Anyway, please open a separate issue about user search. This issue is about the core node search index for the advanced search issue views....

Based on a fruitful discussion in #drupal-infrastructure today, we:

A) Identified and fixed bugs in apachesolr_search that were messing up things with how the core search index was displayed.

B) Identified bugs in node_search.

C) Decided it's silly for us to run node_search on d.o -- we're only indexing nodes for issue advanced search, so, we're only going to index issue nodes and their comments. #381554: Create a submodule to only index issue nodes and their comments to the core search index is now written, tested, committed, and deployed. Therefore, I just turned off node_search entirely on d.o. We can probably remove it from SVN, too.

D) Before I turned on project_issue_search_index (and with nnewton's blessings) I truncated all the core search index tables, since they were fubar to begin with and we wanted to start with a clean slate so we can reap the benefits of the reduced size in the index. I also bumped the index/cron batch size to 1000 so we can more quickly rebuild the index and get the issue search working again.

So, we're *finally* on track to have a working search index for the issue views. We're up to 1% indexed, we're doing 6000 nodes/hour, and we've got 179162 nodes to go, so we should be done in ~30 hours.

The core search index is just now completely rebuilt with issue nodes. Behold:

http://drupal.org/project/issues/search/infrastructure?text=stale

Sweet! ;) Therefore, I removed the warning message that was being generated on the advanced search pages.