Allow modules to modify query used to select nodes for reindexing

I have done quite a bit of research on my particular issue. I've also reviewed http://drupal.org/node/1828014 to see if my idea was superceded by work done there, which it is not. So...

The Problem

I come from a business where there's a lot of aggregated news content placed on our sites. This content is indexed in our SOLR index and in particular (this "news" content) accounts for a large percentage of our content online across 20+ sites. The thing is with news content it goes old -- quick. We have decided to exclude all news content older than 1.5 years from our search index. All other content should remain.

The way Apache SOLR reindexes content doesn't allow customization of the query that selects nodes from the node table. Within apachesolr_index_node_solr_reindex, we have:

 ...
   $insert = db_insert($indexer_table)
      ->fields(array('entity_id', 'bundle', 'status', 'entity_type', 'changed'))
      ->from($select)
      ->execute();
 ...

Furthermore, you cannot use addTag() here because DBTNG does not allow that for insert queries, nor will addTag() fire when this particular query uses from($select) as the text for that query is used as-is.

The Solution

Within apachesolr_index_node_solr_reindex, add this line of code before the above query is executed to allow other modules to modify the query:

  drupal_alter('apachesolr_index_node_reindex_query', $select);

I have attached a patch that I've tested locally (along with our custom rules to exclude content over a certain age).

Peter, good suggestions - thanks. I will look over those two alternatives and report back. I was hesitant to propose a change to "changed" since there seemed like a lot of work surrounding that in another issue, but I'm sure one of these two alternatives will work fine :)

Re: #4, I feel this is overkill for my situation. I don't care to reimplement the whole reindexing logic -- I just need to control whether a node should be indexed base on some of it's native criteria. This is a very nice addition, though, to the API which may prove very helpful later :) Thanks for the suggestion.

Peter I did some research on your suggestions. hook_apachesolr_ENTITY_TYPE_exclude() would fit the bill but like you said it would scan all rows and still not give me the node's created date.

Doing the join worked nicely in apachesolr_index_get_entities_to_index. However, there's a side effect on the apache solr status page. Because I am excluding items within this hook callback, the status query assumes there's no filtering and it saying for my case 9% was already sent to the server. For example:

before_add_tag.png - status page after I cleared my local index (verified no docs via solr status page) with my custom hook callback
after_add_tag.png - status page after I cleared my local index WITH the attached patch applied with my custom hook callback

Is this patch a good idea?

#7: 1925026-count-query-tag.patch queued for re-testing.

I am finding that the patch (or the concept) I originally created in #1 limits the population of the apachesolr_index_entities_node table based on custom criteria (node age for example). While I did find a workaround to reduce my selection set for indexing using the join pwolanin suggested, I'm running into another issue when implementing my own custom cron hook to remove old items on a rolling basis because there is no way to find out exactly what nodes are in SOLR without querying solr directly.

Attached is a revised patch to solve an error when running cron when implementing hook_apachesolr_index_node_query_alter. For any queries using addTag() they all should use consistent table aliases (which is what caused the error).

Re: #11, the reason for the table alias changes (aien to aie) is to make the alias in this function more consistent with other parts of the contrib code that are calling addTag() on the same value; any queries tagged with the same string all should use the same aliases.

With this patch, the following custom code becomes possible (which we're using with good results in production):

// custom code to exclude content of a certain type over a certain age from the solr index
function hook_query_apachesolr_index_node_alter(QueryAlterableInterface &$query)
{
	$timestamp_min = strtotime('-18 months', REQUEST_TIME);
	$query->join('node', 'n_custom', 'n_custom.nid = aie.entity_id');
	$query->condition(db_or()
		->condition('n_custom.type', 'news', '!=')
		->condition('n_custom.created', $timestamp_min, '>'));
}

Without this patch, you'd need some extra conditionals to find the table alias for apachesolr_index_entities_node. Does that help?

Flipping to needs review since maintainer was given info :)