I made a simple patch (against 7.x-1.0) to allow extraction/indexing of remote URLs in file fields. I tested by manually changing a row in the file_managed table to point to a Word doc on the Internet, and it worked (though of course that's only very basic testing).

Is there any chance of this functionality making it into the main codebase?

EDIT: See also #2652920: working with s3 file system

CommentFileSizeAuthor
#12 apachesolr_attachments-remote_stream_wrapper-1430710-12.patch4.43 KBOliverColeman
#10 apachesolr_attachments-remote_stream_wrapper-1430710-10-D7.patch2.07 KBTheOptimizer
#2 apachesolr_attachments-remote_files-1430710-3.patch1.37 KBfeedbackloop
apachesolr_attachments-remote_files.patch872 bytesfeedbackloop

Comments

feedbackloop’s picture

feedbackloop commented
Title: Integrate with Remote Stream Wrapper module » Index file fields with http:// URIs

see next comment

feedbackloop’s picture

feedbackloop commented
Title: Index file fields with http:// URIs » Integrate with Remote Stream Wrapper module
StatusFileSize
new apachesolr_attachments-remote_files-1430710-3.patch1.37 KB

Second pass: I realized I really needed the remote stream wrapper module. This new patch at least maintains the current behavior UNLESS you have that module installed, in which case it fetches the data into a temporary file and passes *that* to Tika/Solr.

I did testing with the remote file source module to plug in PDFs and DOCs from the Internet.

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented
Title: Index file fields with http:// URIs » Integrate with Remote Stream Wrapper module
Status: Needs review » Needs work

The patch is not in the correct format, could you re-roll that?

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented
Version: 7.x-1.0 » 7.x-1.x-dev
pwolanin’s picture

pwolanin commented

Also, this is not the correct behavior. The stream wrapper should have a flag itself indicatiing whether it's local. See: http://api.drupal.org/api/drupal/modules!system!system.api.php/function/...

http://api.drupal.org/api/drupal/includes!stream_wrappers.inc/constant/S...

We should copy to tmp based on that flag for a given scheme (i.e.is/t not local), NOT on this particular module name.

omaster’s picture

omaster commented

What is the status on this. I really could use this working. And the patch files really make no sense from what I can see. Not in the right format.

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented

You could start by trying to see what that patch changes and make a decent patch for it?

omaster’s picture

omaster commented

Yeah I probably will do that. I was just hoping that seeing as it was sooo many months ago something had happened with it. But I guess not.

omaster’s picture

omaster commented

Definitely not easy. Not only do I have to work out where they go but have to figure how the latest version has changed from 1.0.

Would it be easy if I worked out where they used to be in 1.0. And then we can work out how to re factor them for the current version by where the processes have moved to?

TheOptimizer’s picture

TheOptimizer commented
Issue summary: View changes
StatusFileSize
new apachesolr_attachments-remote_stream_wrapper-1430710-10-D7.patch2.07 KB

Based on previous patches, I have made a patch that integrates Apache Solr Attachments with Remote stream wrapper files as well. It simply pulls in the remote file to a local temporary managed file. This local file can then be indexed. The garbage collector afterwards takes care of removing this temporary file.
This patch should work against the latest version.

TheOptimizer’s picture

TheOptimizer commented
Status: Needs work » Needs review
OliverColeman’s picture

OliverColeman commented
StatusFileSize
new apachesolr_attachments-remote_stream_wrapper-1430710-12.patch4.43 KB

Attached is a patch that allows indexing local or remote files, doesn't require the Remote Stream Wrapper module, and makes sure the file is only downloaded once from the remote storage (whether tika is used locally or via solr). It borrows code from the patch in #10 (thanks!).

pwolanin’s picture

pwolanin commented
Status: Needs review » Needs work

Some code style problems like:

$file_contents=NULL

Could use some more comments, and I'm wondering if the some of the logic should be split out into separate functions for readability.

janusman’s picture

janusman commented

Another problem is that this fetches the remote file into memory, which will cause PHP out of memory errors under many cases. Ideally this might need a way to fetch chunks of that data directly into the temporary file instead of keeping all in memory.

Perhaps using cURL (http://php.net/manual/en/book.curl.php) and saving to the file from there (similar to http://php.net/manual/en/function.curl-setopt.php#92522 ) could work?

amontero’s picture

amontero
Location Barcelona
commented
Issue summary: View changes
Related issues: +#2652920: working with s3 file system

See related issue: #2652920: working with s3 file system