Hi,

We have a content type that has large pdfs that often fail to get indexed. For this content type, we created an extra filefield for a text file version of the pdf that is only there for indexing. We set that field to display only in the "search index" custom display settings so an actual user viewing the node would never see it. And we also set the PDF file field to NOT display for "search index".

My hope was that apachesolr_attachments would get its attachments from the display. But the pdfs are still being accessed for indexing and the indexing still fails. We've worked around this in a cumbersome way but it is not sustainable long term.

Is this a good way to go do you think?

Thanks,
Travis McCauley
Toronto

Comments

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented

What tika version do you use?

Please give us some more information on how you've set it up :-) It could be that if you POST it to solr, the attachment is too large for a post to handle in a successful upload.

travismccauley’s picture

travismccauley commented

We're using tika-app-1.1 and solr 3.5.

solrconfig.xml has: <requestParsers enableRemoteStreaming="false" multipartUploadLimitInKB="2048" />

The problem pdf is 13mb. Could this be our culprit?

Thanks,
Travis

nick_vh’s picture

nick_vh
he/him
Location Ghent
commented
Status: Active » Fixed

similar to #1648016: Allowed memory exhausted

you could add an exclude hook that removes it from the indexing process after a specific check, please see regular apachesolr docs for that. If you feel adventurous, you could make a patch so that we have a setting where you disable certain attachments above a certain size from indexing?

pwolanin’s picture

pwolanin commented
Status: Fixed » Active

Actually - this brings up an interesting point - whether the file should be excluded if it's not supposed to be "displayed" in the search index.

This might have been easier to implement in the 6.x-1.x since we were looking at the node and its attachments together. However, for 7.x, seems like we might consider this condition in the default exclude callback, or you could implement hook_apachesolr_ENTITY_TYPE_exclude

coreycondardo’s picture

coreycondardo commented

Does a somewhat "easy" solution to this exist? I'm trying to index 2000+ nodes all of which have a pdf attached and the whole index is choking. It will just spin for a while and eventually the screen will reload with no error message. It prevents everything from being indexed.

--Corey