duplicate search results

I do see that on occasion, whenever the search term is in both the title of the node and the node body. It ends up returning 2 results.

There's some discussion of a similar problem at http://drupal.org/node/143160 . Though the cause isn't firmly identified, it may have something to do with overlapping or improperly terminated cron jobs.

That issue discusses duplicates in the search_index table, but fixing those alone didn't solve the problem. The thing that affected search results for me was duplicates in the search_dataset table.

Here is the SQL code I used to remove duplicate rows from both search_index and search_dataset without rebuilding the entire index. (Have a backup handy before doing anything like this, of course.)

create table search_index_temp select distinct * from search_index;
select (select count(*) from search_index)
 - (select count(*) from search_index_temp) as index_duplicates_removed;
truncate search_index;
insert into search_index select * from search_index_temp;
drop table search_index_temp;

create table search_dataset_temp select distinct * from search_dataset;
select (select count(*) from search_dataset)
  -(select count(*) from search_dataset_temp) as dataset_duplicates_removed;
truncate search_dataset;
insert into search_dataset select * from search_dataset_temp;
drop table search_dataset_temp;

#218403: Duplicate entry errors in search indexer