Special (nordic) characters in title breaks URL alias

Yep - as with the other issue, I'm open to suggestions from those with more experience with character set issues. I'm hoping there's a simple function call somewhere, I don't want to resort to my own table of character mappings...

Don't map them at all. Browsers with IRI (internationalized resource identifiers) support can handle Unicode characters in URLs. The UTF-8 bytes are escaped with %XX syntax. Search engines understand this.

For example:

http://localhost/日本語
http://localhost/Français
http://localhost/Fran%C3%A7ais

Check the status bar when hovering over these links. See where you are directed when you click them. The only place where it looks ugly is in the browser addressbar, and even then there is work to fix that. Mozilla has IRI support already, but I believe it is off by default.

Yes, but...

1. Currently pathauto's use of
   

     $output = preg_replace('/[\W]+/ ', $separator, $output);
   

   removes those characters entirely. How can I preserve them?

2. Mozilla has IRI support already, but I believe it is off by default.
   Does IE have IRI support? If IE doesn't support it, and Mozilla has it off by default, then 95% of people browsing the web can't use it today...

I'm open to either a solution that preserves these characters within the URL, or replaces them with ASCII equivalents, as an improvement on the current state where they simply get removed - I'm still looking for a suggestion on how to do that. It's probably obvious to those who deal with multi-lingual issues on the Internet regularly, but I'm a red-blookded true-blue American.

[Obligatory joke:

What do you call someone who speaks two languages?

Bilingual.

What do you call someone who speaks three languages?

Trilingual.

What do you call someone who speaks one language?

American.
]

The following code replaces accented characters with their ASCII counterparts. This is from a patch submitted earlier by elonen at http://drupal.org/node/18691. The patch adds the ability to add php scripts for path aliases which may be a little much for some drupal administrators, but it has several other good enhancements including this one.

Here is the code from that patch:

function _pathauto_cleanstring($string, $separator) {
  // Drop apostrophes
  $output = str_replace("'", "", $string);
  // Convert accented characters to their ASCII counterparts
  $output = utf8_encode(strtr(utf8_decode($output),
  "\xe1\xc1\xe0\xc0\xe2\xc2\xe4\xc4\xe3\xc3\xe5\xc5".
  "\xaa\xe7\xc7\xe9\xc9\xe8\xc8\xea\xca\xeb\xcb\xed".
  "\xcd\xec\xcc\xee\xce\xef\xcf\xf1\xd1\xf3\xd3\xf2".
  "\xd2\xf4\xd4\xf6\xd6\xf5\xd5\x8\xd8\xba\xf0\xfa".
  "\xda\xf9\xd9\xfb\xdb\xfc\xdc\xfd\xdd\xff\xe6\xc6\xdf",
   'aAaAaAaAaAaAacCeEeEeEeEiIiIiIiInNoOoOoOoOoOoOoouUuUuUuUyYyaAs'));
  // Any sequence of one or more non-alphanumerics
  // should become a separator
  $output = preg_replace('/[\W]+/ ', $separator, $output);
  // Trim any leading or trailing separators (note the need to
  // escape the separator if and only if it is not numeric)
  if (ctype_alnum($separator)) {
    $seppattern = $separator;
  } else {
    $seppattern = '\\'.$separator;
  }
  $output = preg_replace("/^$seppattern+|$seppattern+$/", "", $output);
  return $output;
}

I would still prefer to leave the accents but this is a good workaround till someone figures it out.

Thanks

JJ

I've committed a fix using an example found in the PHP manual's strtr comments, which handles the ligatures hrkaspar was concerned with as well as stripping accents.

Whether leaving accents in the URL is desirable or not, even if pathauto passes them to path_set_alias() they will fail, Drupal's valid_url() function will not permit them.