Non unicode in URL changes to ___.html

another option may be to make pathauto transliterate (with the help of the transliteration module ) urls to ascii (not always wanted though) (google knows about transliteration and will list your cyrillic pages no matter how do you search them or what their url (latin or cyr) )

Cheers from a bulgarian.

I tried the latest boost 6.x.dev on Drupal 6.10 with the german title of a node "Österreich park test of the unlaut" with the path alias of "http://freizeitpark.co.at/%C3%B6sterreich-park-test-unlaut" (attached a screenshot - cause the link may dissapear soon)

the analoguous line to comment in 6.x is the 583 line of .module file. (search for a "a-z" string on line with preg_replace - it is the only one)

After commenting it the boost static cache of the special page works - both the boost comment and the loading times improved...

in bash when listing the symlink it has a ?? in the place of the "Ö" char but it works in the browser - I guess I should germanize my bash...

issue changed to 6.x version. and issue title changed

hey, guys tested this line commenting on several of my sites - it works nice. I wonder what could go wrong if we patch boost that way? Which chars are meant to be removed by this preg_replace (except for non-UNICODE)?

Most sites use pathauto which by default is set to remove odd chars (!@$%$^&) from the alias.... so?

Need review - anybody?

The proposed patch is :

// Convert any other undesirable characters in the path to underscores:
- $path = preg_replace('@[^/a-z0-9_\-&=,\.:]@i', '_', $path);

...

Remove/comment the 2nd line.

Having this as a setting that you can click on or off would be a better idea IMHO. It was originally put in there for a reason, so it has to be useful to someone.

I want to know the reason. Then comes the setting-patch.

I think this is the reason
http://tools.ietf.org/html/rfc3986#section-2

But as noted, this is for the filename not the url. Apache should change the Percent-Encoding back to its original UTF-8 character string. Can more people please test this out, as I'm starting to think that boost was over compensated when dealing with non ASCII characters.

I do not fully understand the theory behind this and the exact process but here is what I observe:

- name of page:
Elvia Mondial Reiseversicherung Österreich

- url of the page:

/elvia-mondial-reiseversicherung-%C3%B6sterreich - percent encoding for non-ascii unicode chars - the encoded Char is the capital german letter Ö .

- url as copied from ff addressbar

http://versichern24.org/elvia-mondial-reiseversicherung-%C3%B6sterreich

- url visually in the ff address bar

http://versichern24.org/elvia-mondial-reiseversicherung-österreich

- path in the file system - seen via ls - 'locale' command output below :

elvia-mondial-reiseversicherung-??sterreich.html

LANG=C
LANGUAGE=de_DE:en_US:de_LU:de_CH:de_BE:de_AT
LC_CTYPE="C"
LC_NUMERIC="C"
LC_TIME="C"
LC_COLLATE="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_PAPER="C"
LC_NAME="C"
LC_ADDRESS="C"
LC_TELEPHONE="C"
LC_MEASUREMENT="C"
LC_IDENTIFICATION="C"
LC_ALL=

but I won't bother setting my locale right to just see the umlaut characters right - if they work in apache as they do it is ok then ...

I am begining to think that this preg_replace was done just in case cause there was no time to test how would apache/file system manage non-ascii chars ...

If several people from diff. countries and with standart common linux based servers report that it works - then we commit. Some windows server feedback would be good as well though not compulsory to the commit .

The idea about a setting fades away if it works everywhere, right?

No need for a boost setting if this works everywhere. Not sure if we will get any windows testing in the near future, since boost hasn't supported windows for very long :). I would like at least 3 other people to test the proposed changes before moving forward. An alt to killing it is detecting if pathauto is installed, if not installed then default to ___

Related Issues
#180689: Does not work for URL aliases ending in .html
#364327: boost_file_path generates file names that are never processed by htaccess
#210671: Boost creating wrong cache file names

This is going into the latest dev by mid week, so I'm assuming no news is good news in this case.

Found something...
URL variables don't get processed, thus they contain spaces. Spaces in the url mess up boost, with that line removed.

Type	page not found
Date	Saturday, April 25, 2009 - 7:05pm
User	Visitor
Location	http://www.example.com/info/contact-us?title=Chariot%2520Eagle%2520West%2520-%2520CH1-2852A%2520
Referrer	http://www.example.com/manufactured-homes/new/chariot-eagle-west-ch1-2852a
Message	cache/www.example.com/info/contact-us_title=Chariot Eagle West - CH1-2852A .html.gz
Severity	warning
Hostname	76.255.139.87
Operations	

Type	page not found
Date	Saturday, April 25, 2009 - 6:14pm
User	Visitor
Location	http://www.example.com/info/contact-us?title=Pre-Owned%2520-%2520Laurel%2520Creek%2520park%2520model%25202006
Referrer	http://www.example.com/park-models/pre-owned/pre-owned-laurel-creek-park-model-2006
Message	cache/www.example.com/info/contact-us_title=Pre-Owned - Laurel Creek park model 2006.html.gz
Severity	warning
Hostname	207.200.116.133
Operations	

info/* are http://drupal.org/project/webform pages.

Could use http://www.php.net/str_replace to replace all " " with "+" which incorporates #364327: boost_file_path generates file names that are never processed by htaccess. But what else does this patch mess up?
The browsers for those errors according to my stats is IE6 & Safari 1.2.

The generated file looks like
contact-us_title=Chariot%20Eagle%20West%20-%20F-22%20.html.gz
On the web server.

Came up with a better idea.

eh that doesn't work 100% of the time... leaning more towards a setting that can be turned on/off. Using mb_ereg_replace() instead of preg_replace() with the correct expression would allow the passage of non ASCII characters. Another way of looking at this is figuring out what characters need to be escaped because of apache, allow all others. Adding the output of boost_file_path() to the boost block would be helpful. For now, thinking about using preg_match() and returning false if it doesn't pass. Example from #364327: boost_file_path generates file names that are never processed by htaccess

//  $path = preg_replace('@[^/a-z0-9_\-&=,\.:]@i', '_', $path);
  if (preg_match('@[^/a-z0-9_\-&=,\.:]@i', $path)) {
    return FALSE;
  }

  // Don't cache path if it can't be served by apache.
  if (BOOST_ONLY_ASCII_PATH) {
    if (preg_match('@[^/a-z0-9_\-&=,\.:]@i', $path)) {
      return FALSE;
    }
  }

BOOST_ONLY_ASCII_PATH is set on the performance page under advanced

  $form['advanced']['boost_only_ascii_path'] = array(
    '#type'          => 'checkbox',
    '#title'         => t('Only allow ASCII characters in path'),
    '#default_value' => BOOST_ONLY_ASCII_PATH,
    '#description'   => t('Only allowing ACSII characters is a safe way to cache pages. It severely limits i18n support so this can be turned off. Fair warning, disabling this may cause "page not found" errors depending on your url structure (spaces are bad, ect...). If you follow RFC 3986 you should be ok.'),
  );

committed

just for completeness - checked the php manual:
1. the preg_match above is case insensitive - nice
2. between [] the \ is fine to escape characters such as "." and "-"

All ok.