Use Solr Cell, aka ExtractingRequestHandler

Yes, the goal is to have that as an option to do the extraction remotely - so your suggestion is very much on the roadmap!

I think it needs to be stored so that the highlighter can work on it.

in terms of remote extract, oe thing to add will probably be local caching (in DB or files) of the extracted text, so that one doesn't need to send the file over again for extraction if just the meta-data like taxonomy is updated.

I don't think we want to index directly into Solr via the request handler, since then we'd have to rewrite the PHP library in terms of sending over other POST variables.

I strongly disagree - imagine you send a 10 MB PDF (e.g. with lots of images) and get ~5 kB of text back. It would be very silly not to locally cache the extracted text vs. waiting for the upload and processing time any time the related node or meta data was updated.

Here's a start on this - just setting the groundwork.

playing with this today - looks like we relly need to post multi-part for data like described:

http://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1

Here's the solrconfi.xml stanza I'm using (need to update the patch).

  <!-- An extract-only path for accessing the tika utility -->
   <requestHandler name="/extract/tika" class="org.apache.solr.handler.extraction.ExtractingRequestHandler" startup="lazy">
      
    <lst name="defaults">        
    </lst>
    <!-- This path only extracts - never updates -->
    <lst name="invariants">
      <bool name="extractOnly">true</bool>
    </lst>
  </requestHandler>

I had to copy jar files around to get it to work with multi-core

see: http://wiki.apache.org/solr/ExtractingRequestHandler

Argh - this is not really working, but a start.

requires this patch to work: http://drupal.org/node/612024#comment-2190226

with revised solrconfig patch and some other refactoring.

finally figuring out how to format the POST body correctly required resorting to running tcpdump and posting a document to extract with curl following the solr wiki example. For future ref, I did a tcpdump on the localhost will running solr on my laptop:

$ sudo tcpdump -i lo0 -s0 -w dump.tcp
tcpdump: listening on lo0, link-type NULL (BSD loopback), capture size 65535 bytes
^C
23 packets captured
23 packets received by filter
0 packets dropped by kernel

$ tcpdump -r dump.tcp -A
...

committed to HEAD, leaving issue open for any minor follow-up

minor cleanup patch committed. let's call this done for now.