html entities are not properly handled on search/display

I got the same problem. I have customized polls results using FusionCharts (showing poll results on a flash chart) and the object HTML code is shown on search results.

Like this: "= new FusionCharts("/sites/all/themes/my_theme/flash/poll.swf", "FusionCharts_1", "680", "340", "0", ...".

Is there a way to filter HTML / JavaScript code?

This is still a problem in recent versions of 6.x-1.x-dev. I've attached a screenshot and the Solr response that references this particular node. In many cases, the entities are double encoded (&) and in some cases they're triple encoded ("). Should apachesolr be handling these entities better, and if so, at which layer (i.e. indexing, preprocessing of search results, presentation of search results)?

We make text safe before we index is in apachesolr.index.inc

So - we should make the same transformation to search terms, and make sure not to encode again the result.

some of the double encoding may come from: http://api.drupal.org/api/function/template_preprocess_search_result/6

Note -the dev version of the module pushes all the encoding into the PHP client.

Please try this patch.

Patch above is incorrect - see discussion on upstream issue.

This patch should be more correct, but still safe. Requires reindexing.

check_markup already ran (we assume) when the node body was rendered for indexing.

The situation here is that we now decode entities before we put that in the search index - that means I can inject XSS via encoded entities like <script> (I tried it - it works absent the check_plain() in the patch)

check_plain just calls htmlspecialchars() and does utf-8 validation. We could just call htmlspecialchars() directly I think, but for consistency doing it this way since the preprocess function does check_plain() for the title.

With this patch these entities no longer interfere with search, but I'm not sure they are actually being matched.

This tries to make all relevant keys safe - wish we had some better way...

Ugh, yeah. That's a nasty hack. I don't have a better idea at the moment, though, so +0.5 in favor of committing.

committed after discussion w/ Robert

rolled back the prior patch and committed #22 as less likely to cause problems for existing users

Fixed in 5.x-.2.x-dev... Not so sure if is OK for D5.

Patch attached.

In 6.x-2.x-dev quotes in snippets are shown as &quot;. Is it possible to fix this?
And, searching of single character & produces the URL like search/apachesolr_search/%2526 and the query box on the result page contains %26 (and the result is irrelevant).

HTML entities are showing up in search results in 6.2 and 6.1 branches.

  exists in our database due to a large amount of programming code content (esp. withing code blocks). We believe   is showing up in our search snippets as a result of extra escaping which is being performed by the Acquia Search module (Apache Solr).

Filter_xss and htmlspecialchars are being run on rendered node content as part of the indexing process which which we believe is causing most of the extra escaping.

Hi,

Just to add some detail to the issue I can post some sample data which will hopefully help with figuring out this issue.

From our database:

The search snippet:

In this case we would not want the &nbsp; to show up since this is affecting how Google ranks keywords on our site. If you view the node itself the &nbsp; is not being escaped.

My gut feeling is the snippet needs to have extra processing done to it in apachesolr_process_response().

Note: We're using 1.0RC5 of this module.

Thanks.

Hi pwolanin,

As far as I can tell these are the fixes for 1.0:

I don't see any reference to escaping fixes. Were the additional changes to escaping not included as part of the changelog or am I looking at the wrong set of changes?

Thanks.

Actually the fix I'm thinking of went into RC4:

We are using the codefilter module (http://drupal.org/project/codefilter).

There are a few possibilities - including that this issue is specific to your theme.

Can you reproduce this in garland theme?

Can you attach a file containing text that when indexed exhibits the problem?

@malex you are running 6.x-2.x or 6.x-1.x?

Until someone posts the SOURCE text (i.e. what you see when editing the node, preferably as an attachment) that's indexed and exhibits the issue, with full info on the input format, exact apachesolr module version info, and with confirmation as to whether the problem appears in Garland I cannot possibly debug it and this will continue to be postponed.

Ok, confirmed it on a clean install today

Apachesolr module:
version = "6.x-1.2"

Node body:

Solr result:

Definitely repeatable in Garland.

I can also give you access to the amazon instance I used in testing this if it would help.

I can confirm this issue. And as with #42, we really don't want to filter out the returned HTML.

I guess so. Wrote a quick test script that called apachesolr_clean_text qith a quote entity and I get the double escaped quote entity out.

Looks to me like html_entity_decode doesn't decode the quote entity because of ENT_NOQUOTES and then htmlspecialchars encodes the & in the entity because that's what it does. We aren't setting the php 5.2.3 double_encode parameter so that's doing exactly what its told but I'm guessing its not what we want.

What is the reason its not decoding quote entities?

I *think* the fix might be as simple as changing line 25 in apachesolr.index.inc to:

so this seems to be close to a full fix in combination with the change in #50 for the indexing code. You have to re-index again to make it effective.

So, the fact that you can even have a protected words file was missing form the Solr wiki (I just updated it), but is included in the LucidWorks Solr reference PDF, and indeed seems to work - here's a highlighted snippet example:

search: http://localhost:8983/solr/ad/select/?q=%26amp%3B

So you see we avoid the problem of getting the highlighting as &<strong>amp</strong>;

The relevant change to the drupal-1.1 schema.xml is attached. The contents of delim-protwords.txt is simply:

Here's full patch for 7.x-1.x

committed to 6.x-1.x and 7.x-1.x

I think we have to double-pass since htmlspecialchars() has no option for doing single not double quotes.

patch above has the 2 string transformations in the wrong order so caused double encoding.

committing this to 7.x

Looks good for 6.1 (committed)

Looks like a clean port from the diff.

Great! Works in 6.2.

I have my protwords.txt file with the words above and I already re-index my data, but if I search for "quots" i get results with ... received &<strong>quot</string>;Sending....
Because we have '... received "Sending...' in our searchable data.

thanks

We need much more information for a thread this old

What version?
What is the reason you are using protwords.txt for these replacements?
How can we replicate (as in add this content to a node, index the node, search for this word, voila)

Thanks!