Wrong rewrite on requesting url with special chars

Drupal uses UTF-8 internally everywhere. If you enter a path in UTF-8 encoding (as should be the default behaviour), it should come out as UTF-8 escaped bytes in the path.

For example, using Drupal 5:

http://acko.net/Testing-%C3%9B%C3%B1ic%C3%B6d%C3%A9-%E2%80%B9%E2%82%ACha...
http://acko.net/Testing-Ûñicödé-‹€harachters›-∞

Clicking the bottom link works fine for me and pasting it into the address bar works as expected. Any browser that supports some form of internationalization should default to using UTF-8 for URLs. However, given that Drupal always outputs the former escaped variant and not the latter, there should be no problems.

The problem is caused by some browsers requesting URI's in local encoding (ISO-8859-1 in my case) rather than UTF-8. Since Drupal does nothing to handle the encoding of $_GET['q'], things are messed up good.

(I noticed that Firefox 1.4 and 2.0 has this problem, while IE6 (surprisingly) does send URL's in UTF-8.)

I've made a small patch (attached) to path.inc that adds a check/conversion for non-UTF-8 sequences in $_GET['q'] before the rest of the code in drupal_init_path is executed. A limitation is that non-UTF-8 encoded strings are assumed to be in ISO-8859-1.

This will work on most installations. There might, however, be problems on some servers due to a bug in the Apache rewrite engine. So far, it has only appeared on my windows installation of Apache 1.3.34. My Linux servers are fine. The problem is that the rewrite engine screws up with UTF-8 strings and manages to alter them in a way that they are no longer legal UTF-8 sequences.

I've installed the following script on several servers (index.php):

<?php
print($_SERVER['REQUEST_URI']);
print "<br />";
print(urlencode($_GET['q']));

together with the Drupal .htaccess file.

On my windows machine the following requests and results are produced:

Request (UTF-8 in first character):
http://localhost/%C3%B6
Response:

/%C3%B6
%E3%B6

Request (UTF-8):
http://localhost/abc%C3%B6%C3%B6abc/def%C3%B6def
Response:

/abc%C3%B6%C3%B6abc/def%C3%B6def
abc%E3%B6%E3%B6abc%2Fdef%C3%B6def

Request (ISO-8859-1):
http://localhost/%F6
Response:

/%F6
%F6

As you see, the windows Apache server destroys the UTF-8 characters that appear in the part before the first slash (after rewrite base). This never causes a problem with Liquid since all ids are of the type wiki/*, but for general paths this might be a problem.

Hope this helps.

Requesting http://rilacademy.org/drupal52dev/Länk in my browser (Mozilla 1.5.0.9, Sv) still gives http://rilacademy.org/drupal52dev/L%E4nk though, while after editing the node I see http://rilacademy.org/drupal52dev/L%C3%A4nk

For some reason Mozilla/Firefox never displays URL's in local formating. The reason for the URL's being different is quite simple; When typing the character on your keyboard, Mozilla is given the character in local formating (ISO-8859-1) and converts it to %E4. When the link is returned by Drupal, it has been converted to UTF-8 (by my patch) and thus Mozilla converts it to %C3%A4 which is the UTF-8 code for the character.

If it's important to have one unique URL for each node, there is a possibility to place a redirect when the requested URL is in ISO-8859-1 (this is what Liquid does). I'm, however, not sure it's worth it.

// Soren

I cannot reproduce this problem myself, and if it is indeed Apache doing it, then you need to file an Apache bug report, upgrade to a newer version or find the right setting.

Drupal can only work with what it is given, and we expect that to be UTF-8, as per the web standards that all the browsers implement. Apache should not alter such strings.