I have a big number of nodes (~2400). Indexing is done in slices of 500 (Items to index per cron), which yields ~5 cron runs to reindex all.
Steps to reproduce:

  1. I deleted the Solr Search index in solr/data/index.
  2. Request Re-indexing via Drupal > admin/settings/search > Button Re-index
  3. Waited for the multiple cron-runs to complete

I discovered that a couple of nodes were not indexed, although they should have been (State published). To analyze the problem, i logged the Node-IDs when indexing was done. I found that the same nodes were multiple times indexed whereas others were omitted.

A short extract from the log (Items to index per cron decreased to 10)

1st Cron run
[Sat Aug 09 16:10:45 2008] Adding nids 1,3,13,24,27,29,30,36,37,38
2nd Cron run 
[Sat Aug 09 16:26:52 2008] Adding nids 1,3,13,24,27,30,39,40,41,42
3rd Cron run 
[Sat Aug 09 16:27:46 2008] Adding nids 1,3,13,24,27,30,53,54,55,56

The nodes 1,3,13,24,27 were indexed again and again (without having changed)... and some nodes were omitted.

The function that defines the indexing todo list is apachesolr.module > ApacheSolrUpdate::getNodesToIndex. I guess there error is in that.

As a workaround i made a path there to re-index everything in one step:

  static function getNodesToIndex($namespace) {
    register_shutdown_function('apachesolr_shutdown');

    $cron_change = self::get_change($namespace);
    $cron_last = self::get_last($namespace);
    $cron_limit = variable_get('search_cron_limit', 100);
    
    // PATCH: Indexing without Limit 
    $cron_limit = 5000;
    set_time_limit(0 ); // no time Limit

Comments

janusman’s picture

janusman commented

I have not had these problems, but I normally do 2 extra steps you did not mention. So the complete list (IMO) is:

  1. ** Stop Solr
  2. I deleted the Solr Search index in solr/data/index.
  3. ** Restart Solr
  4. Request Re-indexing via Drupal > admin/settings/search > Button Re-index
  5. Waited for the multiple cron-runs to complete

Perhaps this helps?

ducdebreme’s picture

ducdebreme commented

I am sorry, but it did not help.
I forgot to mention the stopping and starting of Solr.
How many node do you have? The problem only occured, when having more nodes than a cron run can handle in one rush.

brownjord’s picture

brownjord commented

I am getting the same problem, for a site with a large number of nodes (> 200,000). Something seems to be wrong with the database query in apachesolr.module > ApacheSolrUpdate::getNodesToIndex.

As mentioned above, with multiple cron runs, the same nids are being returned by the getNodesToIndex function. The first 500 nodes are indexed properly, but with subsequent cron runs I can see that nids are being repeated.

JacobSingh’s picture

JacobSingh commented

I think more detail is needed here before it can be acted on. I too have seen getNodesToIndex get "stuck"... but I forget why, and it was a long time ago. Can anyone still having the problem give a reproducible case? or at least some kinda clue to start with?

robertdouglass’s picture

robertdouglass commented

One way that indexing gets stuck is if there is PHP code in the node body that gets executed and emits a drupal_goto or outputs a redirect header. Don't know if that is relevant.

robertdouglass’s picture

robertdouglass commented
Priority: Normal » Critical
JacobSingh’s picture

JacobSingh commented
Status: Active » Closed (duplicate)

I'm guessing this is a dupe of:
http://drupal.org/node/302378

We need to do some work on the patch from:
http://drupal.org/node/42277#comment-846833

If you're keen, please go ahead, if you think this is a separate issue, please close.

Thanks!
Jacob