Search module should keep language information on snippet content.

Would be good if the snippet could contain specific tags, like <span> and <a>.

Then the snippet would become:

<p class="search-snippet" xml:lang="nl"> De titel van ons nieuwste <strong>boek</strong>: <span lang="en">The Late Roman Cemeteries; Stray Finds and Excavations</span> ...</p>

Which is correct HTML again.

This is an interesting issue. The patch would need to be ported to Drupal 8 and go in there first, before we could consider it for Drupal 7. And it's pretty hard to review now, due to lack of comments and explanations. Plus it needs a test.

In any case, there are some major reworkings of the search_excerpt() function going on in #916086: search_excerpt() doesn't highlight words that are matched via search_simplify() right now, so we should probably postpone this issue until that one goes in.

#916086: search_excerpt() doesn't highlight words that are matched via search_simplify() has been committed

I took a look at the patch that was posted a while back. I think that, at least in Drupal 8, we actually need to approach this a little differently.

First of all, the function search_simplify(), which is used in search indexing, searching, and search highlighting, is now language-aware. Functions search_index_split() and search_excerpt(), which call this function, are also language-aware. And search_index(), which calls search_index_split(), is also language-aware. All of these functions take the language of the text as a parameter, and assume that the entire text is in a single language.

Second, it appears that in HTML5, the correct way to specify language is with a "lang" attribute, as "xml:lang" is not supported any more. This attribute may go on any tag, and they can be nested. It looks like the patch is doing this part mostly right -- detecting it on any tag, and then in the output making spans, but the difference is we should output "lang" not "xml:lang".

Third, we cannot rely on the Filter module being turned on. The patch uses the function filter_dom_load() to load the DOM object. This doesn't exist in Drupal 8... maybe we can use this class instead though:
https://api.drupal.org/api/drupal/core!vendor!masterminds!html5!src!HTML...

So... It looks like if we want to do this right in Drupal 8, before calling search_simplify(), both search_index_split() and search_excerpt() would need to look for lang attributes on HTML tags that specify areas in the text that have different languages, and they should then call search_simplify() on each language part separately. Then in the case of search_excerpt(), you'd want to reassemble the tags after excerpting, similar to what is done in the patch.

Yeah search did not consider yet that parts of the content may be in different language. If we can split them and process them per language that would be amazing. Adding more D8MI tags.

Talked to pwolanin about this at DDD today...

The point of this issue is that since the NodeSearch plugin and/or the Search module is putting an xml:lang attribute on the result snippet, it needs to respect the xml:lang attributes in the text.

So I think we won't try to do the snippet processing really carefully -- it's just too complicated to break the text up and preprocess it. But we can fix the snippet creator to put out spans with xml:lang attributes.

We triaged this issue as part of the Bug Smash Initiative in the #bugsmash Drupal Slack channel.

I think I reproduced the issue with the following steps:

1. Install Drupal 10.1.x with Umami profile (Demo: Umami Food Magazine (Experimental))
2. From the Manage menu, navigate to Content, search for "pasta" in the Title field filter.
3. Edit the node with the title "Pasta vegetariana al horno súper fácil". (es/node/3/edit)
4. For the field, Resumen, update the Formato de texto) to HTML completo. Click the blue Continuar button when the Change text format pop-up warning displays.
5. On the Resumen field, in the CKEditor toolbar, click Origen to edit the HTML.
6. Added the following inside the <p></p> tags: <span lang="en">Pasta is delicious!</span> <span xml:lang="en">XML Pasta is tasty!</span> so that the contents of the field are now:

   <p>
   	Una pasta al horno es la comida más fácil y saludable. Este delicioso plato es súper rápido de preparar y una comida ideal entre semana para toda la familia. <span lang="en">Pasta is delicious!</span> <span xml:lang="en">XML Pasta is tasty!</span>
   </p>
   

7. Save the changes (at the bottom of the Edit (Editar) form, click the blue button Guardar (esta traducción).)
8. Return to the site and ensure that you're on the Español version (which should be the case since you just edited a Spanish language node).
9. Use the search field (Buscar) to search for the text "pasta".
10. In the search result snippet for "Pasta vegetariana al horno súper fácil" (the 2nd result), the search result snippet displays: "Pasta vegetariana al horno súper fácil Una pasta al horno es la comida más fácil y saludable. Este … y una comida ideal entre semana para toda la familia. Pasta is delicious! XML Pasta is tasty! …"
11. Inspect the HTML for this snippet. Notice that both <span> tags have been stripped.

Note to re-rollers! The code that the last patch modifies still exists in core/modules/search/search.module, but that file has been massively refactored, so I would be very surprised if a regular re-roll would work. I think what probably needs to happen is someone needs to take the latest patch's changes and manually apply them to core/modules/search/search.module and create a new patch.

Tests will also be needed.

I updated the issue summary with my steps to reproduce and also added remaining tasks.

Updated issue summary with:

As noted in #6, the solution should handle the use of lang attributes on any tag, and probably doesn't need to handle xml:lang attributes.