I'm running a drupal site pushing 50,000 nodes and I'm starting to notice that the cron job to update the search index can take about 90 minutes. This is mostly due to importing nodes by the thousand, but ...

I noticed that the search_update_totals() function calls a killer of a query when it checks the search_total table for words that no longer appear in the search_index table.

$result = db_query("SELECT t.word AS realword, i.word FROM {search_total} t LEFT JOIN {search_index} i ON t.word = i.word WHERE i.word IS NULL");

I'm on 4.7, but the code appears the same in CVS.

This query doesn't account for my 90 minutes all by itself, but left joining 250,000 rows to 5 million rows takes a good 4 minutes every time.

I could optimize MySQL a little better, ok.

But, I still wonder if the task of maintaining search_total might be better handled by search_wipe as called from node_delete. Instead of checking the whole table, check only the words found in the node being deleting (cache them from search_index before calling on a delete there).

My assumptions are 1) this is a heavy query and 2) most sites run cron more often than they delete a node.

I can see the advantage of running this kind of maintenance, but if takes a lot of resources, maybe it doesn't belong in cron.

Or, am I missing some point of philosophy or other that would help me understand why it's better to keep it the way it is now? Does search really break if search_total has words that search_index does not?

Comments

calebtr’s picture

I just thought of the fact that we could edit nodes too, and a word might disappear that way also - but my final question stands.

ricabrantes’s picture

Status: Active » Closed (fixed)

Close, No activity..