Closed (outdated)
Project:
Glossary
Version:
6.x-1.8
Component:
Code
Priority:
Normal
Category:
Bug report
Assigned:
Unassigned
Reporter:
Created:
11 May 2012 at 12:36 UTC
Updated:
1 May 2017 at 17:19 UTC
Jump to comment: Most recent
The latest glossary.module contains this:
1252 function _glossary_is_boundary($char) {
1253 if (extension_loaded('mbstring')) {
1254 return (mb_strpos("!\"#\$%&'()*+,-./:;<=>?@[\]^_`{|}~<92> <93><94><B4><B0><AE><A9><BC><BD><BE> \t\n\r", $char) !== FALSE);
1255 }
1256 else {
1257 return (strpos("!\"#\$%&'()*+,-./:;<=>?@[\]^_`{|}~<92> <93><94><B4><B0><AE><A9><BC><BD><BE> \t\n\r", $char) !== FALSE);
1258 }
1259 }
The .module file contains invalid UTF-8. Both `file` and `enca` give the .module file as "unrecognized encoding".
I suspected that my system had somehow munged the file on extracting the tarball, so I checked against a git clone, and it is the same, there are no differences between a git checkout and the extracted tarball.
I have a reasonable idea what this function should be doing and what those characters need to be in UTF-8, but I'd like to get confirmation from the module maintainer what the encoding of this file might be currently and what those characters should be.
Oddly enough, git blame gives the last change on this line as back in 2008, so looks like these may have "always" been there.
There's actually a good chance that no-one has ever observed symptoms from this, given the characters involved there, so leaving this as "normal" until there are any reports otherwise.
Comments
Comment #1
nancydruWell, I don't know about "always" but that's been there a long time (before I took on the module). I think it's fairly standard filter code for boundary finding.
The module was just updated for security reasons, so you are not the first to test it. Nothing in Git or Drupal.org complained. I would think a module file should be just normal ASCII.
Comment #2
alan evans commentedI think that's actually the problem though: hex 92 and above can't be standard ASCII, which stops at 7F. How those characters render depends on what character encoding you choose, and there's no good way to specify the encoding for a PHP file, it will typically depend on the locale of the system running it. It's probably safe to assume most systems will use UTF-8 these days (where these isolated bytes are invalid), but that might not be true of all hosts.
I couldn't find any drupal standard for the character encoding of PHP code itself. Maybe we need one - either ASCII: 00-7F (restrictive but safe) or UTF-8 (risks excluding some hosts that are not UTF-8 and don't allow users to change), in which case those characters could be changed to the UTF-8 encoding. At the moment, I'd lean toward ASCII, as the lowest common denominatore for PHP scripts is 8-bit (single-byte) encoding, and we've just seen what issues can happen when using characters above 7F (128).
Probably in order to be really correct and usable on any system, these need to be switched for something like a regular expression using the unicode codepoints for these characters instead of literal characters (and if you're going with regular expressions, then the whole thing could probably be refactored using native regexp techniques rather than specifying all of the individual characters that might be non-word chars.) It might need a separate string for sites that have PCRE without utf-8 mode if going with a regex that specifies all the character literals as hex values.
Comment #2.0
alan evans commentedChange description to make it more general: describe the current state rather than diff between current and patched version.
Comment #3
nancydru