Problems with double-encoded ampersands

i did a diff between the b10 and rc2 versions of apachesolr.index.inc

[root@bfc33 tmp]# diff apachesolr_b10/apachesolr.index.inc apachesolr_rc2/apachesolr.index.inc 
2c2
< // $Id: apachesolr.index.inc,v 1.1.2.4 2009/04/30 17:38:40 pwolanin Exp $
---
> // $Id: apachesolr.index.inc,v 1.1.2.6 2009/07/01 22:43:50 pwolanin Exp $
31,34d30
<   return _apachesolr_strip_decode(preg_replace('@[\x00-\x08\x0B\x0C\x0E-\x1F]@', ' ', $text));
< }
< 
< function _apachesolr_strip_decode($text) {
38c34,37
<   return htmlspecialchars(html_entity_decode($text, ENT_NOQUOTES, 'UTF-8'), ENT_NOQUOTES, 'UTF-8', FALSE);
---
>   $text = htmlspecialchars(html_entity_decode($text, ENT_NOQUOTES, 'UTF-8'), ENT_NOQUOTES, 'UTF-8');
>   // We must strip low bytes second in case there was an encoded
>   // low-byte character.
>   return apachesolr_strip_ctl_chars($text);
91c90
<     $document->body = _apachesolr_strip_decode($text);
---
>     $document->body = apachesolr_clean_text($text);
[root@bfc33 tmp]# 

and noticed the htmlspecialchars in b10 is using double_encode=FALSE, but that is not the case in rc2. would this be causing the double encoding attempt for & to & ?

php version is 5.2.9

thx.

yashesh bhatia

hi, i changed

$text = htmlspecialchars(html_entity_decode($text, ENT_NOQUOTES, 'UTF-8'), ENT_NOQUOTES, 'UTF-8')

to

$text = htmlspecialchars(html_entity_decode($text, ENT_NOQUOTES, 'UTF-8'), ENT_NOQUOTES, 'UTF-8', FALSE)

deleted the index and rebuilt it but am getting the same problem. back to debugging.
thx
yashesh bhatia

u can try it at

http://publicart.callforentry.org/

type 'lipski' in the search box

it gives u - http://publicart.callforentry.org/search/apachesolr_search/lipski
click on the 3rd link in the 3rd block.

"Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin & Martin, Engineers, Lakewood, CO (1)"

and it goes to
http://publicart.callforentry.org/search/apachesolr_search/lipski?filter...

works fine for 6.x-1.0-beta10

digging a bit more into the file apachesolr.module and specifically the functions apachesolr_facet_block and apachesolr_l it seems the problem is in the apachesolr_l. here's the breakpoints output from the function

----------------------------------------------------------------------------------------------------------------------------------------------------------------

WORKING CODE - 6.x-1.0-beta10

Sep 15 11:39:04  [info] entering function apachesolr_facet_block() in apachesolr.module
Sep 15 11:39:04  [info] $facet_field = sm_cck_field_work_agent_name
Sep 15 11:39:04  [info] Breakpoint 10
Sep 15 11:39:04  [info] entering function apachesolr_l() in apachesolr.module
Sep 15 11:39:04  [info] $text = Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin & Martin, Engineers, Lakewood, CO (1)
Sep 15 11:39:04  [info] $path = search/apachesolr_search/lipski
Sep 15 11:39:04  [info] $options = Array
(
    [query] => filters=sm_cck_field_work_agent_name%3A%22Brooke%20Michl-Smith%2C%20Senior%20Architectural%20Designer%2C%20Karlsberger%2C%20Architects%2C%20Columbus%2C%20Ohio%20Martin%20%26amp%3B%20Martin%2C%20Engineers%2C%20Lakewood%2C%20CO%22
)

Sep 15 11:39:04  [info] $retval = <a href="/search/apachesolr_search/lipski?filters=sm_cck_field_work_agent_name%3A%22Brooke%20Michl-Smith%2C%20Senior%20Architectural%20Designer%2C%20Karlsberger%2C%20Architects%2C%20Columbus%2C%20Ohio%20Martin%20%26amp%3B%20Martin%2C%20Engineers%2C%20Lakewood%2C%20CO%22">Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin &amp;amp; Martin, Engineers, Lakewood, CO (1)</a>

----------------------------------------------------------------------------------------------------------------------------------------------------------------

NOT WORKING CODE - 6.x-1.0-rc2

Sep 15 17:11:57  [info] $facet_field = sm_cck_field_work_agent_name
Sep 15 17:11:57  [info] Breakpoint 10
Sep 15 17:11:57  [info] entering function apachesolr_l() in apachesolr.module
Sep 15 17:11:57  [info] $text = Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin &amp; Martin, Engineers, Lakewood, CO (1)
Sep 15 17:11:57  [info] $path = search/apachesolr_search/lipski
Sep 15 17:11:57  [info] $options = Array
(
    [query] => Array
        (
            [filters] => sm_cck_field_work_agent_name:"Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin &amp; Martin, Engineers, Lakewood, CO"
        )

    [attributes] => Array
        (
            [class] => apachesolr-facet
        )

)

Sep 15 17:11:57  [info] $retval = <a href="/search/apachesolr_search/lipski?filters=sm_cck_field_work_agent_name%3A%22Brooke%20Michl-Smith%2C%20Senior%20Architectural%20Designer%2C%20Karlsberger%2C%20Architects%2C%20Columbus%2C%20Ohio%20Martin%20%2526amp%3B%20Martin%2C%20Engineers%2C%20Lakewood%2C%20CO%22" class="apachesolr-facet">Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin &amp;amp; Martin, Engineers, Lakewood, CO (1)</a>

----------------------------------------------------------------------------------------------------------------------------------------------------------------

The above debugging statements are the parameters and the return value from the function apachesolr_l in apachesolr.module
i'll dig more deeper, it's not solved yet but wanted to share the debugging information.

yashesh bhatia

sorry i forgot to mention the drupal versions clearly.

Working code - Drupal 6.10, apachesolr 6.x-1.0-beta10
Broken code - Drupal 6.13, apachesolr 6.x-1.0-rc2

i'll debug the common.inc for the function url() for drupal 6.10 and 6.13

yashesh

some more debugging. it's breaking down in includes/common.inc in function url().
at the place

  if (is_array($options['query'])) {
    $options['query'] = drupal_query_string_encode($options['query']);
  }

------------------------------------------------
working code 
Sep 15 13:24:51  [info] entering function url() in common.inc
Sep 15 13:24:51  [info] $path = search/apachesolr_search/lipski
Sep 15 13:24:51  [info] $options = Array
(
    [query] => filters=sm_cck_field_work_agent_name%3A%22Brooke%20Michl-Smith%2C%20Senior%20Architectural%20Designer%2C%20Karlsberger%2C%20Architects%2C%20Columbus%2C%20Ohio%20\
Martin%20%26amp%3B%20Martin%2C%20Engineers%2C%20Lakewood%2C%20CO%22
    [attributes] => Array
	(
        )

    [html] =>
)
------------------------------------------------
not working
Sep 15 18:29:53  [info] entering function url() in common.inc
Sep 15 18:29:53  [info] $path = search/apachesolr_search/lipski
Sep 15 18:29:53  [info] $options = Array
(
    [query] => Array
        (
            [filters] => sm_cck_field_work_agent_name:"Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin & Martin, Engineers, Lakewood, CO"
        )
    [attributes] => Array
        (
            [class] => apachesolr-facet
        )
    [html] => 
)
------------------------------------------------

now need to find out why the two $options are different or is it a bug with the function drupal_query_string_encode

some more debugging

it seems the way the links are generated in the function apachesolr_facet_block is causing the problem

------------------------------------------------
working code - Beta 10 - 6.x-1.0-beta10
line 658
      else {
        $new_query->add_filter($facet_field, $facet, $exclude);
        $path = $new_query->get_path();
        $querystring = $new_query->get_url_querystring();
      }
      if ($count || $active) {
        $items[$sortpre . '*' . $facet_text] = theme('apachesolr_facet_item', $facet_text, $count, $path, $querystring, $active, $unclick_link, $response->response->numFound, $options);
      }

------------------------------------------------
broken code  - RC2 - 6.x-1.0-rc2
line 694
      else {
        $new_query->add_filter($facet_field, $facet, $exclude);
        $options['query'] = $new_query->get_url_queryvalues();
        $link = theme('apachesolr_facet_link', $facet_text, $new_query->get_path(), $options, $count, $active, $response->response->numFound);
      }
      if ($count || $active) {
        $items[$sortpre . '*' . $facet_text] = $link;
      }
------------------------------------------------

the way the links are generated is different for the theme functions apachesolr_facet_link and apachesolr_facet_item

hi. i applied the patch. but it's not fixing the url yet. here's the debugging information in the function apachesolr_l

Sep 16 16:56:44  [info] entering file /work/projects/panmvng/devel/panmvng/trunk/index.php
Sep 16 16:56:44  [info] entering function apachesolr_l() in apachesolr.module
Sep 16 16:56:44  [info] $text = Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin & Martin, Engineers, Lakewood, CO (1)
Sep 16 16:56:44  [info] $path = search/apachesolr_search/lipski
Sep 16 16:56:44  [info] $options = Array
(
    [query] => Array
        (
            [filters] => sm_cck_field_work_agent_name:"Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin & Martin, Engineers, Lakewood, CO"
        )

    [attributes] => Array
        (
            [class] => apachesolr-facet
        )

    [html] => 1
)

// $options after the array merge.

Sep 16 16:56:44  [info] $options = Array
(
    [query] => Array
        (
            [filters] => sm_cck_field_work_agent_name:"Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin & Martin, Engineers, Lakewood, CO"
        )

    [attributes] => Array
        (
            [class] => apachesolr-facet
        )

    [html] => 1
)

//   $genurl = check_url(url($path, $options));

Sep 16 16:56:44  [info] $genurl = /search/apachesolr_search/lipski?filters=sm_cck_field_work_agent_name%3A%22Brooke%20Michl-Smith%2C%20Senior%20Architectural%20Designer%2C%20Karlsberger%2C%20Architects%2C%20Columbus%2C%20Ohio%20Martin%20%2526amp%3B%20Martin%2C%20Engineers%2C%20Lakewood%2C%20CO%22

// $retval =  '<a href="'. check_url(url($path, $options)) .'"'. drupal_attributes($options['attributes']) .'>'. ($options['html'] ? $text : check_plain($text)) .'</a>';


Sep 16 16:56:44  [info] $retval = <a href="/search/apachesolr_search/lipski?filters=sm_cck_field_work_agent_name%3A%22Brooke%20Michl-Smith%2C%20Senior%20Architectural%20Designer%2C%20Karlsberger%2C%20Architects%2C%20Columbus%2C%20Ohio%20Martin%20%2526amp%3B%20Martin%2C%20Engineers%2C%20Lakewood%2C%20CO%22" class="apachesolr-facet">Brooke Michl-Smith, Senior Architectural Designer, Karlsberger, Architects, Columbus, Ohio Martin &amp; Martin, Engineers, Lakewood, CO (1)</a>

Sep 16 16:56:44  [info] leaving  function apachesolr_l() in apachesolr.module

here's the theme function theme_apachesolr_facet_link after the patch was applied

function theme_apachesolr_facet_link($facet_text, $path, $options = array(), $count, $active = FALSE, $num_found = NULL) {
  $options['attributes']['class'][] = 'apachesolr-facet';
  if ($active) {
    $options['attributes']['class'][] = 'active';
  }
  $options['attributes']['class'] = implode(' ', $options['attributes']['class']);
  $options['html'] = TRUE;  // added by patch
  return apachesolr_l($facet_text ." ($count)",  $path, $options);
}

thx

yashesh bhatia