Search results: possible to return node and not attachment?

this is pretty easy - you jsut need to alter each result to make a link from the nid, rather than using the link to the file.

Other people are taking the option of indexing all the extracted attachment text directly in a single document with the node text, so there is never more than one result.

The main person I know trying this is EclipseGC. I'm not sure about others - you might also talk to janusman.

Fell free to add - then we can at least improve the documentation, etc.

Subscribing

subscribe

Hi guys -- checking to see if there's any progress here or if you can provide more pointers on how to do indexing differently or somesuch. The problem I have on the site I'm developing is the client has different content types (say Article, Report, Editorial for instance) where the "full" versions of the items are actually in attached PDFs. They would like users to be able to narrow by type while searching. Unfortunately, if you choose the Report content type filter, for example, while searching for a given keyword, the PDFs aren't included. Any help, ideas for things to look at, etc. would be greatly appreciated.

Thanks!

In the interest of being clear, here's an example:

I have an "Article" item with an attached file in a field called Content PDF. The title of the Article includes the word "foo" as does the PDF file. However, only the PDF file contains the word "bar".

If I search for "foo", I'll see two results -- one links to the Article item, and the other to the PDF (because both contain foo, and presumably they are indexed separately). While this is technically a duplicate result in our case (and I think the sort of thing that started this thread), we can live with that for now if we must.

However, if at this point we look at the facet links on the results page to "Filter by type", the "Article" filter link only shows a count of 1. If indeed we click that link, we'll only see the Article item (the PDF result is no longer included). This problem gets worse when we search for "bar" and choose to filter by the "Article" type: we get 0 results because the term isn't part of the non-attached item.

I hope this makes sense and better illustrates the problem in our particular case.

subscribe

For what it's worth, I worked around the issue by writing a module that implements hook_nodeapi() and responds to the "update index" operation to allow the contents of my attached files to be indexed as part of the node they're attached to.

From the manual (http://api.drupal.org/api/function/hook_nodeapi):

"'update index': The node is being indexed. If you want additional
information to be indexed which is not already visible through
nodeapi "view", then you should return it here."

I borrowed code from apachesolr_attachments_add_documents so that, in theory, I could disable indexing the attachments separately and resolve the "duplicate" issue as well (though we've not chosen to implement that as yet).

I suppose if we were looking at solving the problem generically, we could decide how attachments would be indexed on a per item type or even per field basis. I don't know if there's a better way to handle it in order to give more control over the weight the attachments would carry in determining search results.

In any case -- I hope this helps, and would appreciate feedback/concerns with this approach.

/*
 * Implementation of hook_nodeapi() that returns contents of file attachments
 * borrowed from apachesolr_attachments_add_documents
 */
function my_attachments_nodeapi( &$node, $op, $a3 = NULL, $a4  = NULL ){
  $rv = '';
  if( $op == 'update index' ){
    include_once(drupal_get_path('module', 'apachesolr') .'/apachesolr.index.inc');
    
    // Since there is no notification for an attachment being unassociated with a
    // node (but that action will trigger it to be indexed again), we check for
    // fids that were added before but no longer present on this node.
  
    $fids = array();
    $result = db_query("SELECT fid FROM {apachesolr_attachments_files} WHERE nid = %d", $node->nid);
    while ($row = db_fetch_array($result)) {
      $fids[$row['fid']] = $row['fid'];
    }
  
    $files = _asa_get_indexable_files($node);
  
    // Find deleted files.
    $missing_fids = array_diff_key($fids, $files);
    if ($missing_fids) {
      db_query("UPDATE {apachesolr_attachments_files} SET removed = 1 WHERE fid IN (". db_placeholders($missing_fids) .")", $missing_fids);
    }
    $new_files = array_diff_key($files, $fids);
    // Add new files.
    foreach ($new_files as $file) {
      db_query("INSERT INTO {apachesolr_attachments_files} (fid, nid, removed, sha1) VALUES (%d, %d, 0, '')", $file->fid, $node->nid);
    }
    foreach ($files as $file) {
      // error_log( "my_attachments_nodeapi extracting content from: " . $file->filepath );
      $rv .= "\r\n " . _asa_get_attachment_text($file);  
    }
    // $breakpoint = strlen($rv);
    // if( $breakpoint > 100 ){
    //   $breakpoint = 100;
    // }
    // error_log( "my_attachments_nodeapi returning for node " . $node->nid . ": " . substr($rv, 0, $breakpoint) . "..." );
  }
  return $rv;
}

I'm also looking into a drupal-friendly way of having a single solr document to cover both the parent node and attached files.

Last night, I had to very quickly solve the problem outlined above where attachments disappear from search results when filtering by content type by using the modify query hook. I was already using it to provide OR functionality for type filters. I've left that in as it may also be useful to people. It's pretty rough and ready as I'm only using it so I can search using a querystring like this:

filters=tid:32&or_filters=type:news_article type:press_release

function modulename_apachesolr_modify_query(&$query, &$params, $caller) {
  $or_filters = $_GET['or_filters'];
  
  if (!empty($or_filters)) {
    // this should be more generic rather than specifiying 'type' explicitly
    $filters = Solr_Base_Query::filter_extract($or_filters, 'type');
    
    $subquery = apachesolr_drupal_query();
    
    foreach($filters as $filter) {
      list($key, $val) = explode(':', $filter['#query']);
      $subquery->add_filter($key, $val);
      
      // this bit adds a new OR filter based on the mime facet which is used by solr documents created for attachments
      if ($val == 'document') {
        $subquery->add_filter('ss_filemime', '[* TO *]');
      }
    }
 
    if (!empty($subquery)) {
      $query->add_subquery($subquery, 'OR');
    }
  }
}

On the subject of consolidating attachments and parent nodes in the index, I can see how Brant's code above would work -- i'm considering something similar myself -- but what is a 'nice' way of suppressing the attachments module so it doesn't add the attachment to the index too but keeps the other bits?

It would be handy to have an admin setting (or maybe one per content type) to dictate whether attachments should be indexed separately or combined with the parent. Do you guys reckon this would be a good idea and worth me developing a patch for?

We needed to add the attachments to each content type, not as separate entities, so here's an initial stab at adding a 'per content type indexing' setting.

This is against 6.x-1.0

opops - posted to wrong issue

Indexing will fail on cron because the apachesolr_clean_text() function is needed. Attached patch sorts this by including apachesolr.index.inc.

I had the same requirement as the OP, but we 'remodeled' the returned attachment result to resemble a node result but with the attachment details beneath. We now have the duplicate result challenge, so really it makes sense in our use case to filter out the node results where an attachment results is present, or to index the attachment content with the node and return a result that contains a link and description of the attachment as it currently does (so that the process is shortcut from search to attachment download).

Thoughts?

@Dave the Brave - patch #19 provides the means to index the attachments 'onto' the nodes. I'm not sure if it possible to ask solr to return the attachments to a node so that they could just be themed onto the search results, but that would be a way to give a direct link to the attachment, in the same way that you get a direct link to the parent node on attachment results today.

I've been playing with this some and have cleaned it up some.

1. Combined update file listing into a single function so we don't have duplicate logic
2. Cleanup content type settings some and move it to its own tab. I could be quite large an unwieldy with a lot of content types. Also, we where showing content types that weren't being indexed which was confusing.

Thanks for pushing this forward.

I'm not so fond of APACHESOLR_ATTACHMENTS_MODE_SEPARATE_ENTITY type constants
given that they are only used a couple times - maybe readable strings would make more sense

also, almost seems like people might want a per-node option. Not in the UI necessarily, but a hook so it would be possible?

also please fix up:

 $rv .= "\r\n " . apachesolr_attachments_get_attachment_text($file);

variable naming and why the \r?

Ok, this should address the issues brought up ind #23.

Talked this through with pwolanin on in IRC there was some discussion of maybe storing it differently on the solr objects so there may be more work to come.

oh and uninstall hook.

@swati_patel_8497 it looks like maybe you have confused modules. This issue is for apachesolr_attachments and it looks like you're using search_files . Also, you question sounds like a support request unrelated to this issue which is devloping a new feature.

I applied the patch in #25 to version 6.x-1.0-beta2 and it didn't work - still displaying attachment file instead of node or both attachment and parent node.

I applied the patch manually on apachesolr_attachments.admin.inc due to conflicts but pretty sure applied it in right places. I deleted files from index and also deleted cached file text before reindexing.

Any idea?

I'll see about providing an updated patch a see if it helps.

I applied the patch in #25 and selected and set "Attachments as part of parent node" in admin/settings/apachesolr/attachments/content_type for a CCK content type and then I reindexed all the file attachments by clicking the button in admin/settings/apachesolr/attachments.

When I run cron manually I get a blank page (not even a redirect) and the warning message "Cron run exceeded the time limit and was aborted." appears in the logs. This happens very quickly, before the cron run should timeout.

I'm using tiki-0.3-standalone.jar to index files, although i've also tried tika-0.3.jar and tika-app-0.7.jar with the same results.

@neclimdul it will be a great help! Thanks.

Patch in #25 works great for me with 1.0-beta2.

@#28, After applying the patch, did you clear your caches, and configure the per-content type settings?

@#30, I don't think it's a problem with the patch, per se. You'll need to increase you server's PHP time limits, and/or reduce the number of items to index per cron run at admin/settings/apachesolr

I found a solution to this with theming functions, and documented it here:
https://foss.stat.ubc.ca/ubc-dug/blog/mjoyce/customizing-apache-solr-sea...

If there's a better way to accomplish this please let me know.

I am indexing nodes and attachments. Unfortunately I get duplicate results because (I believe) the hits are often for attachments AND nodes because they contain those matching attachments. Its 2 lines of code to cull the search results in the theme, but I am convinced this is the wrong place to do this since the filters still show the wrong counts. It needs to happen in a Solr-related module. I have installed the patch and it did not change anything... because I believe my issue is different from the one that is the basis for this thread. Is that correct? If so, should I start another issue?

No, that does sound like the issue. My guess is you probably didn't rebuild your index. To clarify, the patch basically adds options on how solr indexes your attachments. One is to attach the documents to the node so you're searching both at the same time and only have one result. This sounds like what you want. You'll have to change this in the settings. After you do this change you'll probably want to drop and rebuild your index. You at /least/ want to rebuild but I don't remember what would happen to the dangling attachment items so a full drop of the index might be best.

Is there a patch for the 6x.2x-dev branch to return the the node where the file is attached without displaying duplicate result for node and or attachement?

Reroll

@michael121 - no I haven't looked at doing that
@jpmckinney was there a reason you moved it down to needs review or just because?

@neclimdul It takes more than one person to get from "needs work" to "reviewed & tested by the community" :) However, it looks like it had been incorrectly left as "needs work", when it should have been "needs review", so maybe it is "reviewed & tested by the community".

Hi,

Is there a version of this patch for the 2011-May-26 6.x.1.0-beta3 release or what do you recommend ?

thanks!

this isn't a bug with -beta3. it really needs to be rerolled against the latest -dev.

Rerolled for 6.x-2.

This feature is a must. I've ported it to Drupal 7, might need more testing, but it seems to be working.

@franz when making a patch against an issue that's for a different branch, could you name the patch accordingly so its clear? http://drupal.org/patch/submit#patch_naming

When I thought about it, it was too late...

Had to fix some things. I tested using tika 0.10 pre-built, it is working fine so far.

subscribe

I created a solution for this some time ago, though I did not think I was solving an actual problem. I overrode theme_preprocess_search_results() to find when a result was not a bona fide node, and ignored it if it wasn't. Some may not think of this is a wholly correct solution, but it worked very well on quite a large (in the thousands) database of documents.

Just my 2 cents.

It doesn't really work because as with anything when trimming at the theme layer you end up with odd page lengths and paging functionality.

Yeah, I'm back here because d I spoke to soon. I just realized what you wrote above, went back to an old project and found the d6 patch I had applied. Thankfully, someone's already working on a d7 version :)

nevermind...

I realized that the D7 patch gives the admin options on how to select what combination of node and/or attaches files are indexed.

If you apply this patch and its still not working, see: admin/config/search/apachesolr/attachments/content_type

nevermind...

Re rolled the patch for 6.x-1.x-dev if anyone is interested.

Oops, missed a line that's sort of important. Round two.

I don't see a 6.x-2.x-dev version of this module, so not sure why this thread is geared toward that. Maybe I'm missing something and someone can point it out for me.

I installed the patch (apachesolr-attachments-attach-to-node-56182-6x1x-01.patch) on the 6.x-1.x-dev version and everything appears to have run correctly but I still don't see an option to show the node that the attachment pertains to. Where can I find that setting or is there something else I need to do?

As mentioned on the Apache Solr Search Integration module page, 2.x as been abandoned some time ago.

6.x-2.x: Previous experimental branch. Unsupported and deprecated except for looking at code examples.

Thanks, I guess I overlooked that.

Still, any idea why the patch might not be working for 6.x-1.x-dev version?

That is that exact patch I'm using on my site right now, so not really. I applied it against the git branch 6.x-1.x, so if 6.x-1.x-dev is not the same as the current git repo then you may have issues. The other thing that I did (but shouldn't effect you) is make the changes discussed here: #1387240: File attached to multiple nodes causes failure.. I can provide that code if you'd like, but again, it shouldn't cause any errors for you unless you have one file attached to two or more nodes.

The only thing I'd suggest is to make sure that you clear your file AND node index. Now, in order to index a file, you must be indexing it's corresponding node, so you'll need to re-index all your nodes.

The latest D7 dev release of apachesolr has undergone a lot of changes #966796: Separate indexer for multiple entity types

There has been some work here #1393540: Upgrade apachesolr_attachments to co-operate with latest beta of apachesolr get get apachesolr_attachments compatible with those changes.

Attached is a patch to also provide a solution to this issue that is compatible with the above. It is based partly on @franz's patch in comment #46 above.

This patch is for drupal 7 and is dependent on #1393540: Upgrade apachesolr_attachments to co-operate with latest beta of apachesolr (Hopefully soon to be committed. tested with patch from comment #51 of that thread. )

attached is an update to my patch posted above. This time compatible with apachesolr_attachments 7.x-1.x-dev

I see a bunch of tabs ;-)

edit: tabs instead of spaces. Check coder for code standards

Thanks drasgardian; Patch applied ok but the file attachments don't seem to be being submitted to solr when 'attachments as part of parent entity' is set. e.g. On my test site, there is always 17 documents remaining (the attachments) - if you select 'index all remaining' the progress bar says 'Indexed 17 items, 0 items submitted to solr'.

If the bundle setting is set to 'attachments as single entities' they are indexed and available to search as per the existing functionality of the module.

I don't see any problems with tabs in my testing.

Thanks drasgardian for this patch.

I have just grabbed the latest modules for solr:

• apachesolr-7.x-1.x-dev from May 13th, 2012
• apachesolr_attachments-7.x-1.x-dev.tar.gz from May 14th, 2012
• facetapi-7.x-1.x-dev.tar.gz from April 7th, 2012

The patch doesn't apply clean to these latest dev module files. Here is the output of the patch command:

$ nbpatch < 561862-61.patch
Hmm... Looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/apachesolr_attachments.admin.inc b/apachesolr_attachments.admin.inc
|index aa408b0..8f0095a 100644
|--- a/apachesolr_attachments.admin.inc
|+++ b/apachesolr_attachments.admin.inc
--------------------------
Patching file b/apachesolr_attachments.admin.inc using Plan A...
Hunk #1 succeeded at 95.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/apachesolr_attachments.index.inc b/apachesolr_attachments.index.inc
|index 180d274..7c6bb69 100644
|--- a/apachesolr_attachments.index.inc
|+++ b/apachesolr_attachments.index.inc
--------------------------
Patching file b/apachesolr_attachments.index.inc using Plan A...
Hunk #1 succeeded at 88 (offset 1 line).
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/apachesolr_attachments.install b/apachesolr_attachments.install
|index 1bd0592..15a3b30 100644
|--- a/apachesolr_attachments.install
|+++ b/apachesolr_attachments.install
--------------------------
Patching file b/apachesolr_attachments.install using Plan A...
Hunk #1 succeeded at 20.
Hmm... The next patch looks like a unified diff to me...
The text leading up to this was:
--------------------------
|diff --git a/apachesolr_attachments.module b/apachesolr_attachments.module
|index 7824904..6f5a4f9 100644
|--- a/apachesolr_attachments.module
|+++ b/apachesolr_attachments.module
--------------------------
Patching file b/apachesolr_attachments.module using Plan A...
Hunk #1 succeeded at 42 (offset -1 lines).
Hunk #2 succeeded at 71 (offset -1 lines).
Hunk #3 failed at 115.
Hunk #4 succeeded at 503 (offset 60 lines).
Hunk #5 succeeded at 458 (offset -1 lines).
1 out of 5 hunks failed--saving rejects to b/apachesolr_attachments.module.rej
done

(Note: this is using the NetBSD patch command)

So I took the liberty to update the patch...the attached patch is for the 7.x-1.x-dev versions released on the respective module pages listed above, not against the HEAD in git....sorry.

I was able to get it working on drupal 7.14 with the listed modules above using this updated patch.

Thanks!
Nathan

subscribe

Applies cleanly and the indexing as a part of the node works as expected.

Updated patched to fix some notices when _apachesolr_attachments_update_parent_entity() was being run for non-file entities.

Some remarks, but overall it looks like a nice solution. Almost there! Hopefully ready for the 7.x-1.3 release

+++ b/apachesolr_attachments.moduleundefined
@@ -57,6 +71,8 @@ function apachesolr_attachments_apachesolr_entity_info_alter(&$entity_info) {
+  ¶

Add a comment like :
// used whenever we do not want duplicate content in the search results

+++ b/apachesolr_attachments.moduleundefined
@@ -103,66 +119,127 @@ function apachesolr_attachments_solr_document(ApacheSolrDocument $document, $fil
+    //indexing attachments with parent entity or not indexing attachements

space after the //

+++ b/apachesolr_attachments.moduleundefined
@@ -103,66 +119,127 @@ function apachesolr_attachments_solr_document(ApacheSolrDocument $document, $fil
+      foreach ($callbacks as $callback) {

you still need to check if the callbacks array exists and is an array

+++ b/apachesolr_attachments.moduleundefined
@@ -103,66 +119,127 @@ function apachesolr_attachments_solr_document(ApacheSolrDocument $document, $fil
+      $filedocument->id = apachesolr_document_id($file->fid . '-' . $parent_entity_id, 'file');

should not be 'file', should be $entity_type

+++ b/apachesolr_attachments.moduleundefined
@@ -103,66 +119,127 @@ function apachesolr_attachments_solr_document(ApacheSolrDocument $document, $fil
+      $filedocument->zm_parent_entity = serialize($small_parent_entity);

this wasn't serialize, this is drupal_json_decode

+++ b/apachesolr_attachments.moduleundefined
@@ -103,66 +119,127 @@ function apachesolr_attachments_solr_document(ApacheSolrDocument $document, $fil
+    foreach ($fields as $field_id => $field_info) {

check if fields is an array and not empty

+++ b/apachesolr_attachments.moduleundefined
@@ -432,10 +509,18 @@ function apachesolr_attachments_apachesolr_query_alter(DrupalSolrQueryInterface
+  if ($type == 'file') {

not valid anymore, since media adds different bundles to the file entity and becomes audio/video/file/default/...

+++ b/apachesolr_attachments.moduleundefined
@@ -432,10 +509,18 @@ function apachesolr_attachments_apachesolr_query_alter(DrupalSolrQueryInterface
+  if ($type == 'file') {

same as above

Attached is a new patch against the latest dev version. It addresses most of the points in #68 and also uses bundle labels as requested in #69.

@Nick_vh - I'm not really sure why + if ($type == 'file') { isn't valid any more. $type is the entity type rather than the bundle.

Attached is another patch to address errors like this on saving nodes with empty filefields

Warning: Invalid argument supplied for foreach() in apachesolr_attachments_field_attach_update() (line 592 of [path_to_modules]/apachesolr_attachments/apachesolr_attachments.module).

Holy smokes, drasgardian, thanks for this patch. It applied cleanly to 7.x-1.x-dev and provided exactly the functionality that I needed. It is a thing of beauty!

Please, please apply this to the next release.

This is essential functionality and really should be applied to the next release.

Using this in production on a couple sites with no issues found yet.

Committed this to dev, let's see if people are happy :)

You committed this? I'm still concerned since especially with several attachments on one node, the content is likely to be truncated unless people tewak the solrconfig.xml

There is more risk than there is with just an individual file, but wouldn't a really large single file run into the same problem when indexing files separately from nodes? I guess it's a question of how much the risk of truncation increases.

hey all,
in search result i added the file that is attached to content and i have to highlight it if keywords match the file content.
how i can know if keywords ar matching node content or file content or both ?

many thanx.

Closing due to inactivity - Has been in dev for a while now

Is this ported to 6.x-3.x?

It doesn't seem so, given the issue history...

Hello Franz!

I would like to sponsor the port of this module to 6.x-3.x. Are you interested?

Would be great if someone could port this to 6.x-3.x. I'm willing to coach where needed.

Thanks for your support @nick_vh! I just talked with Franz and unfortunately he can not do it. He is super busy (as you) with thousands of Drupal projects (that's good!).

Anyone else interested in this job?

here's a port of the patch from #54 to 6.x-2.0-alpha3 (the version my client's using), in case someone still wants to port this to 6.x-3.x and it's helpful.