I'm setting up Solr search on a site that needs more granular control of which documents are indexed than content type alone. So I've implemented a hook_apachesolr_node_exclude which checks the criteria for each node as specified in the documentation.
However, the limit of documents per cron run doesn't take that hook into account.

The flow is as follows:

  • Cron calls apachesolr_search_update_index()
  • That runs apachesolr_get_nodes_to_index() with the limit.
  • apachesolr_get_nodes_to_index() does a database query of nodes with the limit.
  • It passes the results to apachesolr_index_nodes() which (a few levels deep) calls hook_apachesolr_node_exclude.
  • That hook then excludes nodes on an individual basis, but if it excludes every node retrieved by the database query - as seems to be happening now - then it indexes nothing.

So either I'm using the wrong method to control the per-node indexing, or the module needs to check that limit somewhere else. But I realize loading every nid in the system in the query and then only using a fraction of them (until the limit of actually indexed nodes is met) isn't great, either.

Any suggestions?
Thank you!

CommentFileSizeAuthor
#1 apachesolr-count-actual-documents-1548978-1.patch4.88 KBthebuckst0p

Comments

thebuckst0p’s picture

thebuckst0p commented
Status: Active » Needs work
StatusFileSize
new apachesolr-count-actual-documents-1548978-1.patch4.88 KB

This patch attempts to shift the limit from the database query to the number of actual documents indexed.
I'm going to test it on the site I'm working on and see if anything breaks.
Could someone with more familiarity with this module take a look and check for red flags?
Thanks

thebuckst0p’s picture

thebuckst0p commented
  1. Potential problem: If a site has tens of thousands of nodes and the exclusion hook excludes all of them, cron will time out trying them anyway.
  2. The problem we're having that led me to this patch (index getting stuck at 245 documents) doesn't seem to be resolved with the patch. So maybe I mis-diagnosed the problem and this is completely unnecessary?
thebuckst0p’s picture

thebuckst0p commented

So it turned out my problem was two-fold, there was a bug in my exclusion hook that excluded too many, but this patch is still necessary on top of that. Without it, it would run through the apachesolr_cron_limit number of records, they'd all be excluded by the hook, and it wouldn't add anything to the index.

Further review would be great. Does this affect anyone else? (It seems at the very least, this should be an option, given that the hook exists, is documented, and is useful, but isn't currently compatible with the limit.)

Thanks

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented
Status: Needs work » Closed (won't fix)

Closing in favor of 6.x-3.x and because nobody seems to care (time of last response) or cannot reproduce. If you think this is still an issue in 6.x-3.x, feel free to re-open!

I really do appreciate the time you've put in this, but I'm hoping we can see if this issue still exists in 6.x-3.x and work to a better future ;-)