Strip content of tags like SCRIPT that shoud not go into the search index.

The jQuery is specific to that page. It's not being used anywhere else. Although I technically can put it in a block, I would much prefer to leave it in the page itself and avoid the clutter.

Is it possible to have a tag between which everything is omitted? Something like:

<noindex>
jQuery goes here
</noindex>

That would provide some flexibility in terms of giving the editor ability to exclude things from index.

Odd about filter_xss() and comments. It seems to not be working.

Thanks,
Andrey.

This shouldn't be too hard to implement since there is already built-in functionality to weigh certain tags higher than others, right? There is already a way to omit them entirely, so it sounds like this can be as simple as just adding another tag to the list of already existing ones.

I'll tinker around with it.

Odd about the comment thing. Here's the code that leaks through:

<p>Untitled Document</p><!-- .style4 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 12px; } .style10 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 14px; 	font-weight: bold; } --> <table border="0" cellspacing="0" cellpadding="0" width="691"><tbody><tr><td colspan="3"><strong class="style10">Jason Crandell Yoga Workshop at Lulu Bandha's</strong></td></tr> <tr><td class="style4" colspan="3">This footage is from a 3 day Workshop in April 2007 at Lulu Bandha's with<a href="http://jasonyoga.com/" target="_blank"> Jason Crandell.</a></td></tr> <tr><td colspan="3"><span class="style4"><img src="/images/LBO/09_1201_spacer.jpg" alt="" width="700" height="20" /></span></td></tr> <tr><td colspan="3">
...

I didn't write it and looks like it was copied from a Dreamweaver with embed style setting enabled.

nope, one line.

No, this would be new, since filter_xss strips just the tags themselves, not the content inside the tags.

On admin/settings/apachesolr/query-fields there is a setting to omit some of the tags. This wouldn't work the same way?

Sure.

Input format is Full HTML.

Here is the node body content (it's all on one line):

<p>Untitled Document</p><!-- .style4 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 12px; } .style10 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 14px; 	font-weight: bold; } --> <table border="0" cellspacing="0" cellpadding="0" width="691"><tbody><tr><td colspan="3"><strong class="style10">Jason Crandell Yoga Workshop at Lulu Bandha's</strong></td></tr> <tr><td class="style4" colspan="3">This footage is from a 3 day Workshop in April 2007 at Lulu Bandha's with<a href="http://jasonyoga.com/" target="_blank"> Jason Crandell.</a></td></tr> <tr><td colspan="3"><span class="style4"><img src="/images/LBO/09_1201_spacer.jpg" alt="" width="700" height="20" /></span></td></tr> <tr><td colspan="3"><p class="style4">In this clip, <a href="http://jasonyoga.com/" target="_blank">Jason Crandell </a>demonstrates a method of helping each other play towards Viparita Chakrasana.</p></td></tr> <tr><td colspan="3"><span class="style4"><img src="/images/LBO/09_1201_spacer.jpg" alt="" width="700" height="20" /></span></td></tr> <tr><td class="style11" colspan="3"><strong class="style10">Workshop Description:</strong></td></tr> <tr><td colspan="3"><strong class="style10">Walking into Difficulty</strong></td></tr> <tr><td colspan="3"><p class="style4">I&rsquo;ll be honest with you. I want to be able to do scorpion pose. You know the one where you&rsquo;re on your forearms (or hands) and you do such an enormously deep backbend that your feet touch the top of your head and (assumeably) you feel unbelievably skillful and extraordinary, like you&rsquo;re Richard Freeman or Ana Forrest? Well, I can&rsquo;t do it just yet. In fact, I have about 3 feet to go, which, for a six-foot tall person is pretty far. BUT, (and this is why I want you to keep reading) I can move towards it. In fact, I can approach any pose because I understand the process of tackling difficult postures. And that, I tell you, is a really great and interesting thing.</p> <p class="style4">So, this is what will happen in this workshop. We will practice a whole slew of advanced poses (please keep reading if this sounds horribly intimidating), like scorpion, yogi nidrasana, yogi dandasana, pasasana and others with the intention of moving into difficulty with poise, confidence, awareness, and skill. Most of you mortals will not ace these poses, so don&rsquo;t worry. You will however, get to have fun, move beyond your current plateau, and empower yourself to face difficult scenarios with grace and resilience.</p></td></tr> <tr><td colspan="3"><span class="style4"><img src="/images/LBO/09_1201_spacer.jpg" alt="" width="700" height="20" /></span></td></tr> <tr><td colspan="3"><p class="style4"><strong class="style10">About Jason Crandell</strong></p></td></tr> <tr valign="top"><td width="151"><img src="/images/LBO/09_1124_feet_Jason_Crandell.jpg" alt="Jason_Crandell" width="150" height="223" /></td> <td width="7">&nbsp;</td> <td width="543"><p class="style4"><a href="http://jasonyoga.com/" target="_blank">Jason Crandell</a> was recently named &ldquo;one of the next generation of teachers shaping yoga's future&rdquo; by Yoga Journal for his skillful, unique approach to vinyasa yoga. Jason's steady pace, creative sequencing, and attention to detail encourage students to move slowly, deeply, and mindfully into their bodies.</p> <p class="style4">Based in San Francisco, Jason teaches around the world and is a regular contributor for <a href="http://www.yogajournal.com/" target="_blank">Yoga Journal</a>. You can read more about him on his site, <a href="http://jasonyoga.com/" target="_blank">jasonyoga.com</a>, and watch more of his videos <a href="http://www.lulubandhas.com/yoga/category/teacher/jason-crandell" target="_blank">here.</a></p></td></tr> <tr><td class="style4" colspan="3"><img src="/images/LBO/09_1201_spacer.jpg" alt="" width="700" height="20" /></td></tr> <tr><td colspan="3"><p class="style4"><strong class="style10">About Lulu Bandha's Yoga School</strong></p></td></tr> <tr valign="top"><td><a href="http://www.lulubandhas.com/yogaschool.htm" target="_blank"><img src="/images/LBO/09_1204_yogaschool_logo" border="0" alt="Yoga_School" width="150" height="150" /></a></td> <td>&nbsp;</td> <td><p class="style4">Established in March 2002, Lulu Bandha's is a community of blossoming yogis tuning into their own alignment. We are not committed to one particular style, rather we are interested in offering techniques that might help you relax into yourself.</p> <p class="style4">We believe that the best way to learn how to teach yoga is to <em>start</em> teaching yoga. The authentic inspiration to share and connect with others will naturally guide you. However, we also know that deepening the understanding through trainings and workshops with yogis already on the path can help grow the heart, strengthen the confidence and connect you with a supportive community. We are a Registered Yoga School with the <a href="http://www.yogaalliance.org/" target="_blank">Yoga Alliance.</a> Read more about <a href="http://www.lulubandhas.com/yogaschool.htm" target="_blank">Lulu Bandha's Yoga School.</a></p></td></tr> <tr><td class="style4" colspan="3">&nbsp;</td></tr></tbody></table> <p>&nbsp;</p>

Just added the contents of apachesolr_clean_text() to node.tpl.php to see what it does:

  $text = $node->body;
  $text = filter_xss(str_replace(array('<', '>'), array(' <', '> '), $text), array());
  $output = htmlspecialchars(html_entity_decode($text, ENT_QUOTES, 'UTF-8'), ENT_QUOTES, 'UTF-8');
  dpm($output);

It outputs this:

$0.00        Untitled Document  
  <!-- .style4 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 12px; } .style10 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 14px; 	font-weight: bold; } -->   
...

Looks like the tags just get converted to html entities.

And here's the output of the $document as it's being indexed (via hook_apachesolr_update_index()):

Apache_Solr_Document::__set_state(array(
   '_documentBoost' => false,
   '_fields' => 
  array (
 ...
    'body' => '                      $0.00      
      
              
                    2          
          
  
  
      
              
                      Saturday, April 14, 2007 (All day)            
          
  
  
      
              
                    Jason Crandell demonstrates a method of helping each other play towards Viparita Chakrasana          
          
  
    Untitled Document  
  <!-- .style4 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 12px; } .style10 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 14px; 	font-weight: bold; } -->   
...

Replacing

  $text = $node->body;
  $text = filter_xss(str_replace(array('<', '>'), array(' <', '> '), $text), array());
  $text = htmlspecialchars(html_entity_decode($text, ENT_QUOTES, 'UTF-8'), ENT_QUOTES, 'UTF-8');
  dpm($text);

With

  $text = $node->body;
  $text = filter_xss(str_replace(array('<', '>'), array(' <', '> '), $text), array());
  $text = htmlspecialchars(html_entity_decode($text, ENT_QUOTES, 'UTF-8'), ENT_QUOTES, 'UTF-8');
  $text = check_markup(htmlspecialchars_decode($text), 1, true);
  dpm($text);

Returns

<p>$0.00        Untitled Document  </p>
<p>    Jason Crandell Yoga Workshop at Lulu Bandha&#039;s    </p>
<p>  This footage is from a 3 day Workshop in April 2007 at Lulu Bandha&#039;s with   Jason Crandell.    </p>

It's checked against the Filtered HTML filter format. This may not be an ideal way of doing it, but it does seem to work in this case.

OK, here's a patch that alters apachesolr_clean_text() and removes:

<style>
<script>
<object>
<embed>
<applet>
<noframes>
<noscript>
<noembed>

Including the content these tags contain.

updated patch to fix the:

<!--
*******************
-->

issue as well

updated patch. changed the syntax a bit. added iframe to the list of excluded tags -- youtube now uses iframe for embeds.

weird, last patch didn't come through fully... here it is attached again.

--- apachesolr.index.inc.orig	2011-05-11 09:22:41.000000000 -0700
+++ apachesolr.index.inc	2011-05-11 17:35:27.000000000 -0700
@@ -21,7 +21,27 @@ function apachesolr_clean_text($text) {
   // Add spaces before stripping tags to avoid running words together.
   $text = filter_xss(str_replace(array('<', '>'), array(' <', '> '), $text), array());
   // Decode entities and then make safe any < or > characters.
-  return htmlspecialchars(html_entity_decode($text, ENT_QUOTES, 'UTF-8'), ENT_QUOTES, 'UTF-8');
+  return htmlspecialchars(apachesolr_strip_tags(html_entity_decode($text, ENT_QUOTES, 'UTF-8')), ENT_QUOTES, 'UTF-8');
+}
+
+function apachesolr_strip_tags($text) {
+  $text = preg_replace(
+    array(
+        '/<style[^>]*>.*<\/style>/si',
+        '/<script[^>]*>.*<\/script>/si',
+        '/<object[^>]*.*<\/object>/si',
+        '/<embed[^>]*.*<\/embed>/si',
+        '/<iframe[^>]*.*<\/iframe>/si',
+        '/<applet[^>]*.*<\/applet>/si',
+        '/<noframes[^>]*.*<\/noframes>/si',
+        '/<noscript[^>]*.*<\/noscript>/si',
+        '/<noembed[^>]*.*<\/noembed>/si',
+        '/<!--[^>]*-->/si',
+    ),
+    array(' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' '),
+    $text
+  );
+  return $text;
 }
 /**

You won't believe it. Another patch :-)

I realized that filter_xss() strips the script tags leaving the content between them, so the content needs to be intercepted before that happens.

Tested this with the following bits of code:

<script type="text/javascript" >
$(document).ready(function(){
  $('.accordion_teachers').accordion({ collapsible:true, autoHeight:false });
  $('.accordion_teachers').accordion('activate', false);

  $('.accordion0').click(function(){
    $('.accordion_teachers').accordion('activate', 0);
  });
  $('.accordion1').click(function(){
    $('.accordion_teachers').accordion('activate', 1);
  });
  $('.accordion2').click(function(){
    $('.accordion_teachers').accordion('activate', 2);
  });
  $('.accordion3').click(function(){
    $('.accordion_teachers').accordion('activate', 3);
  });
});
</script>

and

<p>Untitled Document</p><!-- .style4 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 12px; } .style10 {font-family: Verdana, Arial, Helvetica, sans-serif; 	font-size: 14px; 	font-weight: bold; } --> 

It strips the tags and everything between them.