As possible we should not rely on "on demand" index updates. Here is the scenario:

I unpublish a node or change access

The Solr Server has a problem handling my request. Maybe it has a locked file, maybe it is down, maybe there is a network issue. Perhaps another module uses nodeapi and redirects or craps something out... I know it's not going to happen 95% of the time, but it's quite possible.

How can we then update the solr index to reflect this change? There is no way to know that the operation failed (without looking in watchdog, and even then the error is totally ambiguous to what the user was doing). Next time we index the content, it won't remove it.

I can only see a couple ways to solve this:

(along with all of these, we explicitly give the user an error message so they know)

1). We actually block the operation with a confirmation dialog if Solr rejects the request
- I realize this will require a menu or init hack and some hacking to figure out if solr is supposed to be acting to preempt node.module, and probably impossible.
2). We store a list of pending index updates which failed, and give the user a place to process the list in a batch mode.
3). We provide a "node sync" operation (which will be expensive and must be run in batches). This would just be removing variable entries for last node updated and re-indexing from scratch, but not using cron, providing an interface to do it if it has to be run in emergency (this would also be useful in the case of schema change / reset).

Am I just being paranoid? I know it seems like work for something which is "not supposed to happen" however I feel that the consequences of unavailable causing unpublished or access restricted content to the public and NO way to fix it or even know about it is pretty bad.

Proposal

We create a status message if the update fails on submission.

Drupal 5

We go with #2. Every time a request to update an index is made in real time (not cron) and it fails, we add it do a queue of "out of date" nodes
The next time cron runs, we run these nodes first.

This will be a separate operation to the standard update_index because we don't want to modify the cron times or complicate the logic. However it will run from uptate_index.

Drupal 6

We just set the re-index bit on the node which failed. And wait for cron to pick it up... This means it might take a cron run or two on a busy site to get found, but it's still probably a good idea.

Comments? Is this 1.0 material?

CommentFileSizeAuthor
#11 handle-failed-deleted-305370-11.patch2.26 KBpwolanin
#7 check-delete-305370-7.patch1.41 KBpwolanin

Comments

JacobSingh’s picture

JacobSingh commented

We still need to handle this. Bumping up the list to get some attention.

pwolanin’s picture

pwolanin commented

We should set up a queue (DB table) of nodes to-be-deleted, issue a delete request for each one on cron, and then look for the response to indicate success before removing it from the queue.

pwolanin’s picture

pwolanin commented

still important...

pwolanin’s picture

pwolanin commented
Priority: Normal » Critical

this is pretty important for long-term stability

JacobSingh’s picture

JacobSingh commented

Okay, how about this:

we chuck the nid in apachesolr_index_nodes table.

We modify the update_index to find nodes with a NULL right join and then send delete requests against them. This process will also get fired every time a node is deleted. so in 

#!/bin/psuedocode

nodeapi ($op == 'delete') {
 db_query('insert $nid into apachesolr_index_nodes)
 _apachesolr_cleanup_deleted_nodes() 
}


function  _apachesolr_cleanup_deleted_nodes() {
  db_query('select $nid from apachesolr_index_nodes ain RIGHT join node n on ain.nid = n.nid AND n.nid IS NULL')
  for each one() {
   if ($solr->deleteById(...)) {
    db_query(delete from apachesolr_index_nodes...).
  }
 }
}


function apachesolr_search_update_index() {
  _apachesolr_cleanup_deleted_nodes()
  //do the normal stuff.
}
pwolanin’s picture

pwolanin commented

Sure that seems like a pretty robust approach

pwolanin’s picture

pwolanin commented
StatusFileSize
new check-delete-305370-7.patch1.41 KB

Here's a patch to at least not delete from our table if the query fails.

JacobSingh’s picture

JacobSingh commented

Looks good, but we should set up our exception handling (and chaining) better in the future.

pwolanin’s picture

pwolanin commented

committed this to 6.x, but still needs a follow-on solution

pwolanin’s picture

pwolanin commented
Title: Provide some functionality for notifying user on failed index requests and handling it later » Handle failed delete requests so unpublised/deleted content doesn't stay in the index
Category: feature » bug
pwolanin’s picture

pwolanin commented
Status: Active » Needs review
StatusFileSize
new handle-failed-deleted-305370-11.patch2.26 KB
pwolanin’s picture

pwolanin commented
Status: Needs review » Fixed

committed to 6.x

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.