utf-8 urls are not cached: filenames encoded wrongly

related but not the solution
#782274: Allow % when using the "Only allow ASCII characters in path" setting - this would allow % to pass the ascii check

Are the files for the hebrew pages being created?

Here's how I get the filename and write it

function boost_init() {
...
  //set variables
  if (isset($_REQUEST['q'])) {
    $GLOBALS['_boost_path'] = $_REQUEST['q'];
  }
  else {
    //front page
    $GLOBALS['_boost_path'] = '';
  }
...
}

function _boost_ob_handler() {
...
  boost_cache_set($GLOBALS['_boost_path'], $decompressed_buffer, BOOST_FILE_EXTENSION);
...
}

function boost_cache_set($path, $data, $extension = BOOST_FILE_EXTENSION) {
...
  $filename = boost_file_path($path, TRUE, $extension);
  boost_cache_write($filename, $data);
...
}

/**
 * Returns the static file path for a Drupal page.
 *
 * @param $path
 *   path to convert to boost's file naming convention
 * @param $query
 *   add query to path
 * @param $extension
 *   add extension to end of filename
 * @param $file_path
 *   Optional: BOOST_FILE_PATH
 *
 * $path = $GLOBALS['_boost_path'] most of the time
 */
function boost_file_path($path, $query = TRUE, $extension = BOOST_FILE_EXTENSION, $file_path = NULL) {
  //handling of url variables
  if ($GLOBALS['_boost_query'] != BOOST_CHAR) {
    if ($query) {
      $path .= $GLOBALS['_boost_query'];
    }
    else {
      $path .= BOOST_CHAR;
    }
  }
  else {
    $path .= $GLOBALS['_boost_query'];
  }

  // Under no circumstances should the incoming path contain '..' or null
  // bytes; we also limit the maximum directory nesting depth of the path
  if (   strpos($path, '..') !== FALSE
      || strpos($path, "\0") !== FALSE
      || count(explode('/', $path)) > BOOST_MAX_PATH_DEPTH
      ) {
    return FALSE;
  }
  $file_path = is_null($file_path) ? BOOST_FILE_PATH : $file_path;

  return implode('/', array($file_path, $path . (is_null($extension) ? '' : $extension)));
}


/**
 * Writes data to filename in an atomic operation thats compatible with older
 * versions of php (php < 5.2.4 file_put_contents() doesn't lock correctly).
 *
 * @param $filename
 *   Name of file to be written
 * @param $buffer
 *   Contents of file
 */
function boost_cache_write($filename, $buffer) {
  $filenames = boost_get_all_filenames($filename);
  foreach ($filenames as $key => $values) {
    if ($key == 'gzip') {
      $data = gzencode($buffer, 9);
    }
    else {
      $data = $buffer;
    }
    foreach ($values as $filename) {
      if (!_boost_mkdir_p(dirname($filename))) {
        if (BOOST_VERBOSE >= 3) {
          watchdog('boost', 'Unable to create directory: %dir<br /> Group ID: %gid<br /> User ID: %uid<br /> Current script owner: %user<br />', array('%dir' => dirname($filename), '%gid' => getmygid(), '%uid' => getmyuid(), '%user' => get_current_user()), WATCHDOG_WARNING);
        }
      }
      $tempfile = $filename . getmypid();
      if (@file_put_contents($tempfile, $data) === FALSE) {
        if (BOOST_VERBOSE >= 3) {
          watchdog('boost', 'Unable to write temp file: %file<br /> Group ID: %gid<br /> User ID: %uid<br /> Current script owner: %user<br />', array('%file' => $tempfile, '%gid' => getmygid(), '%uid' => getmyuid(), '%user' => get_current_user()), WATCHDOG_WARNING);
        }
        return FALSE;
      }
      else {
        if (is_numeric(BOOST_PERMISSIONS_FILE)) {
          @chmod($tempfile, octdec(BOOST_PERMISSIONS_FILE));
        }
        // Erase old file
        if (BOOST_OVERWRITE_FILE) {
          @unlink($filename);
        }
        // Put temp file in its final location
        if (@rename($tempfile, $filename) === FALSE) {
          if (BOOST_VERBOSE >= 5) {
            watchdog('boost', 'Unable to rename file: %temp  to  %file<br /> Group ID: %gid<br /> User ID: %uid<br /> Current script owner: %user<br />', array('%temp' => $tempfile, '%file' => $filename, '%gid' => getmygid(), '%uid' => getmyuid(), '%user' => get_current_user()), WATCHDOG_WARNING);
          }
          @unlink($tempfile);
          return FALSE;
        }
      }
    }
  }
  return TRUE;
}

/**
 * Returns all possible filenames given the input and current settings
 *
 * @param $filename
 *   Name of file
 * @param $base_dir
 *   Value from base_dir column in database
 * @return
 *   returns a 2 dimensional array
 *    1st dimension key is either gzip or normal
 *    2nd dimension contains all the filenames
 */
function boost_get_all_filenames($filename, $base_dir = NULL) {
  $namesA = array();
  $namesB = array();
  $filenames = array();
  $base_dir = is_null($base_dir) ? BOOST_FILE_PATH : $base_dir;

  if (stristr($filename, '[')) {
    $paths = explode('/', $filename);
    $end = array_pop($paths);
    $end = preg_replace("(\x5B[0-9]\x5D)", ']', $end);
    $paths[] = $end;
    $namesA[] = implode('/', $paths);
  }
  $namesA[] = $filename;

  foreach ($namesA as $filename) {
    $namesB[] = $filename;
    // Generate urlencoded filename, if name contains decoded characters
    $paths = explode('/', $filename);
    $end = array_pop($paths);
    //$end = str_replace('[', '%5B', $end);
    //$end = str_replace(']', '%5D', $end);
    $end = str_replace(',', '%2C', $end);
    $end = str_replace(' ', '%20', $end);
    $paths[] = $end;
    $namesB[] = implode('/', $paths);
  }
  $namesB = array_unique($namesB);

  // Generate gzip filenames
  foreach ($namesB as $name) {
    $filenames['normal'][] = $name;
    if (BOOST_GZIP) {
      // Replace the correct dir with the gzip version for the given base dir.
      $gzip_base_path = implode('/', array_filter(explode('/', str_replace(BOOST_ROOT_CACHE_DIR . '/' . BOOST_NORMAL_DIR, BOOST_ROOT_CACHE_DIR . '/' . BOOST_GZIP_DIR . '/', $base_dir))));
      $filenames['gzip'][] = str_replace($base_dir, $gzip_base_path, $name) . BOOST_GZIP_EXTENSION;
    }
  }
  return $filenames;
}

If you know of someway to write files in php with multi byte names, let me know http://php.net/mbstring.
A quick test to see if its possible is run this code and see what filename it generates.

  file_put_contents('cache/' . $GLOBALS['_boost_path'], $_REQUEST['q']);

output should be in the cache directory. This will let me know if I'm converting the input string or something else.

http://php.net/iconv appears to be the function I want if the filename doesn't come out right.

The Transliteration module goes from utf-8 to acsii. You want to from utf-8 to your specific language CP1255 (Hebrew). Let me know if the Transliteration module does this but I don't think it does. I believe this will have to be a semi-complex patch to get this working. Do you know if PHP can auto detect the language used?

I'm pretty sure apache won't serve those the transliterated filenames; or did you test it and it indeed does?

look at the link in the html, it points to kjd_jif_s.pdf

If the link points to שלום.pdf and apache looks for, and gives you kjd_jif_s.pdf then the Transliteration module idea would work for boost. I'm pretty certain it won't work this way, but I do not do much with i18n so I could be wrong.

If you use the Transliteration module with path auto then you will get the desired effect; no need to do anything with boost.
http://drupal.org/project/pathauto

Boost saves the filename as the pathname; so if you use pathauto to rename all your paths the ascii then apache will be able to serve them

For this change its used in 1 place. Here's a patch just for Hebrew; Let me know it works & I will be making a setting for this.

Should take hints from http://api.drupal.org/api/function/drupal_convert_to_utf8/6

subscribing...

...subscribing...

@ukrdrupal
try the patch

committed

put it in the wrong location; try this patch

committed

please add fix in 7.x version

I updated the module to latest dev to overcome the hebrew problem.
It is now showing me the cache status block, which wasn't there before.
it even says that it will expire in 58 minutes and 19 seconds, like it should.
but it also says that: cache/normal/www.mysite.co.il/חדשות_.html Does not exist.
there is no comment at the bottom of the source that says that the page is cached by boost.

any workaround for that?

Since Boost 6.x and 7.x are pretty different, let's keep the issues separate.

Here is a 7.x-1.x issue for this: #1386166: PDOException with UTF-8 aliases / boost does not cache pages with UTF-8 aliases. Please review the patch.

Thanks