Deleted attachments not being removed from index

This has been fixed a while ago in the main apachesolr module

With last dev versions bug exists still. As wrote tauno inserted attachment have other ID (file/81-62 in my log) than deleted attachment (file/81). I debug this in Drupal_Apache_Solr_Service.php in function update. Attached debug output contains values of $rawPost.

Has next change of apachesolr_attachments.module (only partial solution) any serious bugs?
Instead file_delete can be used entity_delete hook also (or to remove apachesolr_attachments_entity_delete ?).
Full solution needs for deleted file to request parent entity - I don´t know how.
Please excuse me for missing patch file.

// in apachesolr_attachments_solr_document add field
// for our file_delete hook
$filedocument->sm_file_entity_id = apachesolr_document_id($file->fid, $entity_type);

// file delete hook is called when file reference count is 0
function apachesolr_attachments_file_delete($file) {
static $failed = FALSE;
if ($failed) {
return FALSE;
}
try {
$sm_file_entity_id = apachesolr_document_id($file->fid, 'file');
$solr = apachesolr_get_solr();
$solr->deleteByQuery("sm_file_entity_id:$sm_file_entity_id AND entity_type:file AND hash:" . apachesolr_site_hash());
$solr->commit();
return TRUE;
}
catch (Exception $e) {
watchdog('Apache Solr Attachments', "On file delete: " . nl2br(check_plain($e->getMessage())), NULL, WATCHDOG_ERROR);
// Don't keep trying queries if they are failing.
$failed = TRUE;
return FALSE;
}
}

Would be great to have this fixed as attachments that are often updaten result in many versions in the index.

Any news? thanks a lot!

Problem still exists with apachesolr-7.x-1.2 and latest apachesolr_attachments-7.x-dev

Still there for me with apachesolr-7.x-1.4 and apachesolr_attachments-7.x-1.3.

Comment #2 implies the issue may be to do with the apachesolr module rather than apachesolr_attachments, which module should we be debugging?

Could this be due to a mismatch in the id sent in the delete query and the id that exists in solr?
The apachesolr module defines the document id as follows:

function apachesolr_document_id($id, $entity_type = 'node') {
  return apachesolr_site_hash() . "/{$entity_type}/" . $id;
}

However the id for the indexed file has the following structure

[site_hash]/[entity_type]/[id]-[parent_id]

See apachesolr_attachments_solr_document().

  ...
  // Build our separate document and overwrite basic information
  $filedocument->id = apachesolr_document_id($file->fid . '-' . $parent_entity_id, $entity_type);
  ...

OK, it's not the most elegant solution but this is working for me.
I couldn't use hook_entity_delete as the parent node information required for the solr document id was not available.

Please ignore the previous patch, trying to access $parent_entity->original causes a fatal error when creating new nodes.

I have a similar problem.
I remove a private file from a node, but this file appears in result of solr search.

I just look in DB for Solr Attachment, I see this one, apachesolr_index_entities_file but even I try to "Clear the attachment text extraction cache", this table is not empty.

When I look into the associated code, I see :

$indexer_table = apachesolr_get_indexer_table('file');
  $transaction = db_transaction();
  $env_id = apachesolr_default_environment();
  try {
    // Clean the table
    db_delete($indexer_table)
      ->condition('entity_type', 'file')
      ->isNull('body')
      ->execute();

as I understand it, body must be null for the raw to be deleted ?

In my case, body is always fill in.

Thank you for any help

I get undefined function apachesolr_attachments_index_delete_attachment_from_index() with #10 when I delete the attachment but leave the node. I'm using Apache Solr search attachments 7.x-1.3.

Deleting the node with the attachment also leaves the attachment in the search index and breaks any search with a keyword in the attachment.

The patch in this issue will stop the search from breaking but it doesn't address the issue where the attachments are not removed from the index.

Based on ls206's comments in #8 and the watchdog messages listing the queries used for delete as

id:"bapxga/file/52" OR sm_parent_document_id:"bapxga/file/52"

this patch creates the attachment id in the index as [type]/[id] instead of [type]/[id]-[parent id].

I'm not too sure about this because I don't know if anything else depends on the [id]-[parent id] combo. However, apachesolr_attachments seems to expect apachesolr to delete these and it has no way of knowing about the compound id. It doesn't appear in the apachesolr_index_entities_file table but I believe the record is removed from the db before it's removed from the index anyway. There used to be a hook_cron implementation which deleted from the index using [id]-[parent id] but this was removed in this commit in April 2012.

With this patch, you will get an error after a node with an attachment is deleted and before cron runs plus the delay between items being sent to solr and when the index is updated. See this issue for more info and patch. Other than this, the attached patch is working for me.

Is this problem resolved? I have the same exact issue: Delete file using file_delete, but file still remains in Apachesolr index even after re-indexing. The only way to remove from index is to delete the index then reindex.

I tried this to no avail:

// Remove file from apachesolr index
$env_id = apachesolr_default_environment();
$entity_type = 'file';
$entity_id = apachesolr_document_id($file->fid, 'file');
apachesolr_remove_entity($env_id, $entity_type, $entity_id);
drupal_set_message('Deleting '.$file->uri.' from private attachments.');
file_delete($file);

The file is physically removed, but not removed from index.

Any suggestions?

Since that change would kill all existing indexes, I don't think it's acceptable.

Either in this module or the parent, we could add a delete which handles the combo ID

This causes problems for me too. Deleted and unpublished nodes gets properly removed from the search index but their attachments remain indexed. I'm using Apache Solr 4.10.3, Apache Solr Search 7.x-1.7 and Apache Solr Attachments 7.x-1.4.

With a little fiddling with the patches in #9 and #10 I managed to find a solution that works for me. Attachments for unpublished and deleted nodes gets deleted from the Solr index.

Might not be the prettiest solution but it works. I attach a patch but it's my first so I'm not really sure if I managed to create it in the right way.

Setting to 'needs review'.

I believe the patch in #2606214: Not all files get indexed for multilingual file field resolves this issue as well. Files no longer associated with an entity get removed from the indexer table, similar to patch #16 here. However, #16 assumes that files removed from an entity have been deleted, which is not always the case.

I confirm the problem, and honestly I don't understand the reason for using [type]/[id]-[parent id] . It breaks apachesolr_index_delete_entity_from_index() and it does not add any useful information as the parent id is anyway saved somewhere else ("zm_parent_entity", etc).

In other words I concord with @Reuben Unruh's approach. @pwolanin I see your point but if the index are created incorrectly (from the point of view of compatibility with main apachesolr module), it's better to fix the source of the problem and to clean our indexes. I also tried the approach you suggest: "we could add a delete which handles the combo ID", but only to clean up the current mess. Now I would prefer to follow Reuben's approach and to use [type]/[id] as per apachesolr module.
To clean up the mess I found all the entity that are in solr but not in Drupal, then deleted them invoking apachesolr_index_delete_entity_from_index() adding a wildcard in the document id like this: [type]/[id]-* . But before to do so you need to remove quotes around $document_id in the query, otherwise the wildcard is interpreted literally:

in ../sites/all/modules/contrib/apachesolr/apachesolr.index.inc line 715 or so...
from

$query = "id:\"$document_id\" OR sm_parent_document_id:\"$document_id\"";

to:

$query = "id:$document_id OR sm_parent_document_id:$document_id";

It worked for me.

@Reuben and @nislas, if I understand correctly, your 2 patches are conflicting, or not?

Reuben's one brings back the original behaviour of setting the id = [type]/[id] (without [parent id])
While nislas's one, apart from doing other checks, deletes records still querying with [type]/[id]-[parent id]

nislas can you confirm? Unless I'm missing something ...

Perhaps we should set on one behaviour, or we will having patched that don't work together.

We found a solution for this problem by adding an extra parent field to our filedocument object - sm_parent_document_id (function apachesolr_attachments_solr_document in apachesolr_attachments.module):

// Add all to one field because if it is spread out over
// multiple fields there is no way of knowing which multifield value
// belongs to which entity
// It does not load the complete entity in to the index because that
// would dramatically increase the index size and processing time
$filedocument->zm_parent_entity = drupal_json_encode($small_parent_entity);
$filedocument->sm_parent_entity_bundle = $parent_entity_type . "-" . $parent_entity_bundle;
$filedocument->sm_parent_entity_type = $parent_entity_type;
// PATCH - add parent document id
$filedocument->sm_parent_document_id = apachesolr_document_id($parent_entity_id, $parent_entity_type);

If we take a look in apachesolr.index.inc, function apachesolr_index_delete_entity_from_index you can see the query where we search for all entities with the specified id or where the specified id is the parent:

$query = "id:\"$document_id\" OR sm_parent_document_id:\"$document_id\"";

So a node with one or multiple files is set unpublished. The node itself will be deleted from the index (by id) and the files which now have a sm_parent_document_id will also be deleted from the index (by sm_parent_document_id).

Looks like an "elegant" solution to me. And it works how the apachesolr module intends to do it.

PS: I'm not able to make a patch (no clue how :) ), but if this gets accepted, maybe someone can make a patch to get this commited and added in one of the following releases.

I can confirm patch in #2606214: Not all files get indexed for multilingual file field fixes this issue too.