remove transliteration and/or provide it as an option

I think it would be cool to have the choice: For example, I'd like to replace spaces with dashes, but leave accented characters like they are.

Having just written http://drupal.org/node/106817 , I would think that transliteration still has a future. As I understood, what the issue here is suggesting is to leave the title in the URL un-transliterated, presumably because we now have IDN. But from what I see here in the Czech Republic, IDN will not be universal soon: it has been rejected twice by the local domain name regulator (nic.cz) and polls always indicate a general dislike of the IDN idea among the domain owners (duplication of costs, confusion, speculation).

Perhaps a truly Drupal solution would be to allow pathauto to create both: a transliterated and non-transliterated alias URL simultaneously.

IDN seems very nice until you actually have to type a URL using characters not on your keyboard.

Actually, transliteration as we have it, if I'm not mistaken, is not to western latin (ISO-8859-1[5]) but just to a subset of ASCII-96. I think this is what you are referring to when saying western latin ?

I'd be in favor of a setting, just like the one for apostrophes merging/converting.

I can't confirm whether the transliteration table is correct: after all many letters in some languages don't even have an exact sound-alike equivalent in ASCII-96, so it may well be imperfect.

But FWIW, the right-hand letters are ASCII-96, and for most Western Latin languages, the left-to-right conversion seems correct.

One thing to keep in mind, although I don't think it has an impact on pathauto, is that this transformation is obviously surjective, meaning there's no way back from the transliterated version.

looks good.

Maybe I didn't look hard enough, but this patch is looking for the file i18n-ascii.txt but the included file is i18n-ascii.example.txt with no mention of renaming this file.

However, I think a far better option is to follow the translit module: http://drupal.org/project/file_translit -- this does the same thing for file names and is super easy to use. Including this PHP package with pathauto would offer the most robust support IMO and would be easy to implement.

Maybe I should have been more clear. That module cleans filenames like this issue cleans URLs.

It uses this open source package: http://sourceforge.net/projects/phputf8 -- this is far more robust than the i18n-ascii file in pathauto.

All pathauto would need to do is include that library (it's essentially the same as the on pathauto includes but has support for all languages) and then simply add this code:

$url = file_translit_clean($url); 

What I'm suggesting is remove the i18n-ascii, add in this UTF-8 package, and then simply that one line of code. No dependencies, and it's just as fast, if not faster than what is there now.

Ooops sorry, it would need to use the function in the project, but it's only 20+ lines long or so, file translit does this:

/**
 * Transliterate UTF-8 text to ASCII.
 * Based on the PHP UTF-8 project http://sourceforge.net/projects/phputf8/
 *
 * @param $string
 *   A UTF-8 encoded string.
 * @param $unknown
 *   Replacement for unknown (erroneous) characters.
 * @return
 *   String transliterated to ASCII.
 */
function file_translit($string, $unknown = '?') {
  // Quickly return if it's pure ASCII
  if(!preg_match('/[\x80-\xff]/', $string)) {
    return $string;
  }

  static $banks = array();
  $result = '';

  for ($i = 0, $len = strlen($string); $i < $len; $i++) {
    $c = $string{$i};
    if (ord($c) >=   0 && ord($c) <= 127) { $result .= $c; continue; } // ASCII - next please
    if (ord($c) >= 192 && ord($c) <= 223) { $ord = (ord($c)-192)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 224 && ord($c) <= 239) { $ord = (ord($c)-224)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 240 && ord($c) <= 247) { $ord = (ord($c)-240)*262144 + (ord($string{++$i})-128)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 248 && ord($c) <= 251) { $ord = (ord($c)-248)*16777216 + (ord($string{++$i})-128)*262144 + (ord($string{++$i})-128)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 252 && ord($c) <= 253) { $ord = (ord($c)-252)*1073741824 + (ord($string{++$i})-128)*16777216 + (ord($string{++$i})-128)*262144 + (ord($string{++$i})-128)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 254 && ord($c) <= 255) { $result .= $unknown; continue; } // Error

    $bank = $ord >> 8;
    if (!isset($banks[$bank])) {
      $file = drupal_get_path('module', 'file_translit') .'/translit/'. sprintf('x%02x', $bank) .'.php';
      if (file_exists($file)) {
        $banks[$bank] = include($file);
      } else {
        $banks[$bank] = array();
      }
    }
    
    $char = $ord & 255;
    if (isset($banks[$bank][$char])) {
      $result .= $banks[$bank][$char];
    } else {
      $result .= $unknown;
    }
  }
  
  return $result;
}      

Ooops sorry, it would need to use the function in the project, but it's only 20+ lines long or so, file translit does this:

/**
 * Transliterate UTF-8 text to ASCII.
 * Based on the PHP UTF-8 project http://sourceforge.net/projects/phputf8/
 *
 * @param $string
 *   A UTF-8 encoded string.
 * @param $unknown
 *   Replacement for unknown (erroneous) characters.
 * @return
 *   String transliterated to ASCII.
 */
function file_translit($string, $unknown = '?') {
  // Quickly return if it's pure ASCII
  if(!preg_match('/[\x80-\xff]/', $string)) {
    return $string;
  }

  static $banks = array();
  $result = '';

  for ($i = 0, $len = strlen($string); $i < $len; $i++) {
    $c = $string{$i};
    if (ord($c) >=   0 && ord($c) <= 127) { $result .= $c; continue; } // ASCII - next please
    if (ord($c) >= 192 && ord($c) <= 223) { $ord = (ord($c)-192)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 224 && ord($c) <= 239) { $ord = (ord($c)-224)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 240 && ord($c) <= 247) { $ord = (ord($c)-240)*262144 + (ord($string{++$i})-128)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 248 && ord($c) <= 251) { $ord = (ord($c)-248)*16777216 + (ord($string{++$i})-128)*262144 + (ord($string{++$i})-128)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 252 && ord($c) <= 253) { $ord = (ord($c)-252)*1073741824 + (ord($string{++$i})-128)*16777216 + (ord($string{++$i})-128)*262144 + (ord($string{++$i})-128)*4096 + (ord($string{++$i})-128)*64 + (ord($string{++$i})-128); }
    if (ord($c) >= 254 && ord($c) <= 255) { $result .= $unknown; continue; } // Error

    $bank = $ord >> 8;
    if (!isset($banks[$bank])) {
      $file = drupal_get_path('module', 'file_translit') .'/translit/'. sprintf('x%02x', $bank) .'.php';
      if (file_exists($file)) {
        $banks[$bank] = include($file);
      } else {
        $banks[$bank] = array();
      }
    }
    
    $char = $ord & 255;
    if (isset($banks[$bank][$char])) {
      $result .= $banks[$bank][$char];
    } else {
      $result .= $unknown;
    }
  }
  
  return $result;
}