apachesolr Attachments: solr (remote server) option extraction configuration [#1205190]

is there a installation procedure for a recent ubuntu tomcat / solr 1.4 installation ?

I was reading about a patch for solrconfig.xml ? Do I need Tika ?

The file extraction is working great for .txt files but for .pdf, .doc and .xls I'm getting messages like this:
On the Drupal log I see:
1) ApacheSolrAttach 2011-06-29 17:36 Could not extract any indexable text from ...Anonymous
Location http://mysite.com:81/admin/reports/status/run-cron?destination=admin%2Fs...
Referrer http://mysite.com/admin/settings/apachesolr/index
Message Could not extract any indexable text from sites/default/files/Web-work-daily-sheet.xls
Severity: warning

2) ApacheSolrAttach 2011-06-29 17:36 Exception occured sending ...Anonymous
Location http://mysite.com:81/admin/reports/status/run-cron?destination=admin%2Fs...
Referrer http://www.mysite.com:81/admin/settings/apachesolr/index MessageException occured sending sites/default/files/Web-work-daily-sheet.xls to Solr "404" Status: Not Found: Not Found
Severity: error

thanks!

Comments

Comment #1

toby53 commented 7 July 2011 at 21:43

I implemented apache tika and it works well.
I added the tika patch in the module directory to solrconfig.xml.
Added the Tika java .jar to the tomcat6 lib .jars.
Reconfigured the solr attatchments configuration page to use tika.
Reindexed the content again.
Is working great for .doc and .pdf .

Comment #2

flatcircle commented 14 July 2011 at 14:20

Could you describe in a bit more detail how you configured Tika on a remote Solr server?

In my setup I select 'Extract using Solr (remote server)'

Tika directory path => leave blank (because Tika will run on remote server)
Tika jar file => leave blank (because Tika will run on remote server)

In solrconfig.xml, I included the necessary Tika.jar:

<config name="drupal-3.0-beta7">
<lib path="../../tika.jar" /> 
...

This doesn't work :-(
Any advice?

Regards.

Comment #3

toby53 commented 2 August 2011 at 15:04

Hi,
I did not notice your post !
I'm running solr on ubuntu 10.04 tomcat6.
1) to install tika
I downloaded the tika-app-0.9.jar from the apache tika project and placed in /usr/share/tomcat6/lib dir and restarted tomcat6
2) To update solrconfig.xml
(located in /usr/share/tomcat6/solr/conf/solrconfig.xml)
I ran the patch in the solr module directory on solrconfig.xml. I believe it currently adds this code to the solrconfig.xml:

 <!-- An extract-only path for accessing the tika utility -->
  <requestHandler name="/extract/tika" class="org.apache.solr.handler.extraction.ExtractingRequestHandler" startup="lazy">

    <lst name="defaults">
    </lst>
    <!-- This path only extracts - never updates -->
    <lst name="invariants">
      <bool name="extractOnly">true</bool>
    </lst>
  </requestHandler>

Comment #4

flatcircle commented 2 August 2011 at 19:30

Thank you for your help.

So select "Extract using Solr (remote server)" in my Solr configuration. (after of course applying the patch)

Do I have to enter something in the fields:

Tika directory path
&
Tika jar file

Comment #5

toby53 commented 4 August 2011 at 19:48

I wound up using Tika.

on the Drupal admin solr attachments tab. I used:

Excluded file extensions:
aif art avi bmp gif ico jpg mov mp3 mp4 mpg oga ogv png psd ra ram rgb tif flv

Exclude files attached to a node of a type excluded by Apache Solr Search:
Yes

Extract using:
Tika (local java application)

Tika directory path:
/usr/share/tomcat6/lib

Tika jar file:
tika-app-0.9.jar

Comment #6

Jānis Bebrītis commented 25 September 2012 at 08:08

Assigned:	Unassigned	» Jānis Bebrītis
Status:	Active	» Closed (works as designed)

in short: remote server option lets you extract documents using solr. for that you need to configure solr accordingly.

first, edit solrconfig.xml (in my case, example/solr/conf/solrconfig.xml) to have these lines:

  <lib dir="../../dist/" regex="apache-solr-cell-\d.*\.jar" />
   <lib dir="../../contrib/extraction/lib" regex=".*\.jar" />
   
  <!-- An extract-only path for accessing the tika utility -->
    <requestHandler name="/extract/tika" class="org.apache.solr.handler.extraction.ExtractingRequestHandler" startup="lazy">
      
      <lst name="defaults">
      </lst>
      <!-- This path only extracts - never updates -->
      <lst name="invariants">
        <bool name="extractOnly">true</bool>
      </lst>
    </requestHandler>

put them somewhere along other requestHandlers. Restart solr instance.

Then, configure drupal as follows:

Extract using: Solr (remote server)
Tika directory path: [empty]
Tika jar file: [also empty]

Save configuration

That's it, "Test Your tika extraction" should work now.

Comment #7

amontero

Barcelona

commented 17 February 2015 at 17:08

Issue summary:	View changes
Issue tags:		+solrconfig.xml

Comment #8

swati sharma commented 11 May 2016 at 22:30

I tried to follow the same steps Mentioned by Janis
Added the details in solrconfig.xml and restarted solr
Extract using: Solr (remote server)
Tika directory path: [empty]
Tika jar file: [also empty].

Rebuilt the index after doing this. But still this is not working.
I am trying to use Solr 5.5 with Drupal 7.x Apache Solr Search and Apache Solr Attachment module.
Am i missing anything? Solr 5.x comes with inbuilt Tikka. I want to use the same with
Extract using: Solr (remote server) option.

Thanks

Comment #9

swati sharma commented 11 May 2016 at 22:33

Version:	6.x-1.0-beta3	» 7.x-1.4
Status:	Closed (works as designed)	» Active

Comment #10

mausolos commented 23 November 2016 at 00:34

@Swati Sharma

Did you figure it out? I'm using Solr 5.4.1 and have tried Tika 1.7 on the remote server as well as 1.14 (latest stable Tika).

When I test, Drupal test page says "Text can not be succesfully extracted. Please check your settings".
Drupal Watchdog has two messages. The first says:

HTTP Status: 404; 
Message: Not Found: Not Found; 
Response: <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404</h2> <p>Problem accessing /solr/apache_drupal/extract/tika. Reason: <pre> Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/> </body> </html> ; 
Request: ; 
Caller: apachesolr_attachments_extract_using_solr() (line 169 of /apps/drupal/inside-dev.wv.mentorg.com/sites/all/modules/contrib/apachesolr_attachments/apachesolr_attachments.index.inc)

The second says:
Exception occurred sending <full_local_server_base_path>/sites/all/modules/contrib/apachesolr_attachments/tests/test-tika.pdf to Solr HTTP 404; Not Found: Not Found

This last bit is weird because the test file is precisely there. (EDIT: with 777 permissions, just to be sure)

Lastly, there's an error message from SOLR logs:

Level: WARN
Core: null
Logger: HttpParser
Message: badMessage: java.lang.IllegalStateException: too much data after closed for HttpChannelOverHttp@31a69774{r=1,c=false,a=IDLE,uri=-}

I'll keep trying different settings and solr configs, but if you did figure out something, do let me know! Thanks!

Comment #11

mausolos commented 23 November 2016 at 06:13

@Swati Sharma

I solved it by just giving up and using a local instance of tika to parse the documents before sending to the remote server.

Comment #12

amontero

Barcelona

commented 6 December 2016 at 17:08

This might be useful for those who don't want to hack the solr config: #2427473: Allow customization of remote server Tika extraction servlet

Reviews are welcome to help push forward the change into the module.

Comment #13

mattjbondi commented 10 August 2017 at 00:21

I can confirm that #6 worked for me!

The main point of difference though was to use absolute paths to the libraries and to specify solr-cell not apache-solr-cell in the regex:

<lib dir="/opt/solr/dist/" regex="solr-cell-\d.*\.jar" />
   <lib dir="/opt/solr/contrib/extraction/lib" regex=".*\.jar" />

And restart your solr instance. You should see no errors in the solr logs. When I tried using relative paths I could see that it was not loading in the necessary .jar files at /dist and /contrib

Hope that helps.