Here is patch. I found that using the original query, some nodes would come up higher in the list than their actual last_updated value because of the NULL comparison on the comments table. This seems to resolve that.

Comments

JacobSingh’s picture

Priority: Normal » Critical
robertdouglass’s picture

Yup. This is a confirmed issue for Drupal, too. Need to check and see if they resolved it in core, and if so, how.

robertdouglass’s picture

Version: 5.x-1.x-dev » 6.x-1.x-dev
robertdouglass’s picture

TODO before commit: track this issue down in core and see if it has been committed and what the solution is. Look here for clues: http://groups.drupal.org/node/10569

JacobSingh’s picture

Assigned: Unassigned » JacobSingh
JacobSingh’s picture

Status: Active » Needs work

Okay, this bug has been fixed through a re-factoring in Drupal-6...

However, it seems it is still there in Drupal-5.

Here is an apt description;
http://drupal.org/node/42277#comment-846833

There is a patch attached to that node. I suggest we rip that query from there and test and apply.

For the D6 branch, I'm thinking we might want to look into refactoring it to use the new method used in node.module... Although I haven't a clue how it works. I can commit to fixing the D5 branch, can someone else more knowledge about D6 tackle the other one?

If you're curious about the difference, here it is:

node.module in D5:

function node_update_index() {
  global $last_change, $last_nid;

  register_shutdown_function('node_update_shutdown');

  $last = variable_get('node_cron_last', 0);
  $last_nid = variable_get('node_cron_last_nid', 0);
  $limit = (int)variable_get('search_cron_limit', 100);

  // Store the maximum possible comments per thread (used for ranking by reply count)
  variable_set('node_cron_comments_scale', 1.0 / max(1, db_result(db_query('SELECT MAX(comment_count) FROM {node_comment_statistics}'))));
  variable_set('node_cron_views_scale', 1.0 / max(1, db_result(db_query('SELECT MAX(totalcount) FROM {node_counter}'))));

  $result = db_query_range('SELECT GREATEST(IF(c.last_comment_timestamp IS NULL, 0, c.last_comment_timestamp), n.changed) as last_change, n.nid FROM {node} n LEFT JOIN {node_comment_statistics} c ON n.nid = c.nid WHERE n.status = 1 AND ((GREATEST(n.changed, c.last_comment_timestamp) = %d AND n.nid > %d) OR (n.changed > %d OR c.last_comment_timestamp > %d)) ORDER BY GREATEST(n.changed, c.last_comment_timestamp) ASC, n.nid ASC', $last, $last_nid, $last, $last, $last, 0, $limit);

In particular see the last bit:

ORDER BY GREATEST(n.changed, c.last_comment_timestamp) ASC, n.nid ASC', $last, $last_nid, $last, $last, $last, 0, $limit);

node.module in D6:

function node_update_index() {
  $limit = (int)variable_get('search_cron_limit', 100);

  // Store the maximum possible comments per thread (used for ranking by reply count)
  variable_set('node_cron_comments_scale', 1.0 / max(1, db_result(db_query('SELECT MAX(comment_count) FROM {node_comment_statistics}'))));
  variable_set('node_cron_views_scale', 1.0 / max(1, db_result(db_query('SELECT MAX(totalcount) FROM {node_counter}'))));

  $result = db_query_range("SELECT n.nid FROM {node} n LEFT JOIN {search_dataset} d ON d.type = 'node' AND d.sid = n.nid WHERE d.sid IS NULL OR d.reindex <> 0 ORDER BY d.reindex ASC, n.nid ASC", 0, $limit);
robertdouglass’s picture

Yeah, there is basically a "reindex" flag in the search_dataset table that lets us mark a node as needing re-indexing. It works much better, but begs the question, are we dependent on using the core search index? A lot of people are attracted to ApacheSolr so that they can stop using the core search index.

Please move ahead with the fix to D5, and forward port it to D6 (if it works) until we come up with a better strategy for D6.

john.money’s picture

StatusFileSize
new1.44 KB

Missed this issue and posted a dupe because I filter by 5.x.

Here's my patch from #309878 for 5.x-dev (0917). Of particular note, there is no need to recalculate last_change when ordering.

edit: patch has been tested against a database of 50k+ nodes and there is 100% indexing coverage.

JacobSingh’s picture

Status: Needs work » Needs review

Works for me. I tested against 6.x too and it works.

Can I get a witness? If it is good, I'll commit it.

robertdouglass’s picture

go for it.

robertdouglass’s picture

Status: Needs review » Postponed (maintainer needs more info)

I'm committed this. BUT:

Can anyone answer me the question - is IF() cross database compatible? Or are we now officially only MySQL compatible?

john.money’s picture

Given that the original code came out of node.module (with the IF), I would assume that it is a compatible function. Otherwise, parts of core are also only MySQL compatible.

john.money’s picture

bump...

We've got Solr humming along on ConsumerSearch. Now I have some time (a little at least) to work on more patches we have on our implementation.

pwolanin’s picture

Note that the tracking mechanism has been totaly rewritten already and now lives in a separate table. Might be bug or improvements to be made, but the ugly query is reall no longer an issue for 6.x

JacobSingh’s picture

Status: Postponed (maintainer needs more info) » Closed (won't fix)