Posted by kotnik on February 17, 2012 at 1:47pm
34 followers
| Project: | Chaos tool suite (ctools) |
| Version: | 6.x-1.10 |
| Component: | Code |
| Category: | bug report |
| Priority: | critical |
| Assigned: | Unassigned |
| Status: | closed (fixed) |
Issue Summary
Constant CTOOLS_PREG_CLASS_ALNUM in includes/cleanstring.inc contains \x{d800}- which is ill-formed code point. PHP issues this warning:
Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in ctools_cleanstring() (line 157 of /srv/http/XXXX/www/sites/all/modules/contrib/ctools/includes/cleanstring.inc).
Easily reproducable by running this:
<?php
define('CTOOLS_PREG_CLASS_ALNUM',
'\x{0}-\x{2f}\x{3a}-\x{40}\x{5b}-\x{60}\x{7b}-\x{bf}\x{d7}\x{f7}\x{2b0}-' .
'\x{385}\x{387}\x{3f6}\x{482}-\x{489}\x{559}-\x{55f}\x{589}-\x{5c7}\x{5f3}-' .
'\x{61f}\x{640}\x{64b}-\x{65e}\x{66a}-\x{66d}\x{670}\x{6d4}\x{6d6}-\x{6ed}' .
'\x{6fd}\x{6fe}\x{700}-\x{70f}\x{711}\x{730}-\x{74a}\x{7a6}-\x{7b0}\x{901}-' .
'\x{903}\x{93c}\x{93e}-\x{94d}\x{951}-\x{954}\x{962}-\x{965}\x{970}\x{981}-' .
'\x{983}\x{9bc}\x{9be}-\x{9cd}\x{9d7}\x{9e2}\x{9e3}\x{9f2}-\x{a03}\x{a3c}-' .
'\x{a4d}\x{a70}\x{a71}\x{a81}-\x{a83}\x{abc}\x{abe}-\x{acd}\x{ae2}\x{ae3}' .
'\x{af1}-\x{b03}\x{b3c}\x{b3e}-\x{b57}\x{b70}\x{b82}\x{bbe}-\x{bd7}\x{bf0}-' .
'\x{c03}\x{c3e}-\x{c56}\x{c82}\x{c83}\x{cbc}\x{cbe}-\x{cd6}\x{d02}\x{d03}' .
'\x{d3e}-\x{d57}\x{d82}\x{d83}\x{dca}-\x{df4}\x{e31}\x{e34}-\x{e3f}\x{e46}-' .
'\x{e4f}\x{e5a}\x{e5b}\x{eb1}\x{eb4}-\x{ebc}\x{ec6}-\x{ecd}\x{f01}-\x{f1f}' .
'\x{f2a}-\x{f3f}\x{f71}-\x{f87}\x{f90}-\x{fd1}\x{102c}-\x{1039}\x{104a}-' .
'\x{104f}\x{1056}-\x{1059}\x{10fb}\x{10fc}\x{135f}-\x{137c}\x{1390}-\x{1399}' .
'\x{166d}\x{166e}\x{1680}\x{169b}\x{169c}\x{16eb}-\x{16f0}\x{1712}-\x{1714}' .
'\x{1732}-\x{1736}\x{1752}\x{1753}\x{1772}\x{1773}\x{17b4}-\x{17db}\x{17dd}' .
'\x{17f0}-\x{180e}\x{1843}\x{18a9}\x{1920}-\x{1945}\x{19b0}-\x{19c0}\x{19c8}' .
'\x{19c9}\x{19de}-\x{19ff}\x{1a17}-\x{1a1f}\x{1d2c}-\x{1d61}\x{1d78}\x{1d9b}-' .
'\x{1dc3}\x{1fbd}\x{1fbf}-\x{1fc1}\x{1fcd}-\x{1fcf}\x{1fdd}-\x{1fdf}\x{1fed}-' .
'\x{1fef}\x{1ffd}-\x{2070}\x{2074}-\x{207e}\x{2080}-\x{2101}\x{2103}-\x{2106}' .
'\x{2108}\x{2109}\x{2114}\x{2116}-\x{2118}\x{211e}-\x{2123}\x{2125}\x{2127}' .
'\x{2129}\x{212e}\x{2132}\x{213a}\x{213b}\x{2140}-\x{2144}\x{214a}-\x{2b13}' .
'\x{2ce5}-\x{2cff}\x{2d6f}\x{2e00}-\x{3005}\x{3007}-\x{303b}\x{303d}-\x{303f}' .
'\x{3099}-\x{309e}\x{30a0}\x{30fb}-\x{30fe}\x{3190}-\x{319f}\x{31c0}-\x{31cf}' .
'\x{3200}-\x{33ff}\x{4dc0}-\x{4dff}\x{a015}\x{a490}-\x{a716}\x{a802}\x{a806}' .
'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}' .
'\x{fdfc}-\x{fe6b}\x{feff}-\x{ff0f}\x{ff1a}-\x{ff20}\x{ff3b}-\x{ff40}\x{ff5b}-' .
'\x{ff65}\x{ff70}\x{ff9e}\x{ff9f}\x{ffe0}-\x{fffd}');
if (preg_match('/^[^' . CTOOLS_PREG_CLASS_ALNUM . ']+$/uD', '-')) {
echo 'Warning issued';
}
?>Patch to be posted in comment.
Comments
#1
Patch with fix attached.
#2
Nice! And a properly formatted commit message which is rare! Thank you.
#3
This bug is likely caused by pcre 8.30+, most Linuxes still use older versions, but FreeBSD has 8.30_1 currently. But this version of pcre will be on Linux soon - it's developed on FreeBSD.
It breaks core search too in D6/7.
Ctools has it because it has code in cleanstring.inc that is copied from includes/unicode.inc (D7 core) or modules/search/search.module in D6.
It's examined in some depth here:
Invalid Unicode code range in PREG_CLASS_UNICODE_WORD_BOUNDARY fails with PCRE 8.30
There are patches for D6 and D7, and these work also if adapted for ctools' unicode.inc
These patches are slightly different to the one above. They replace the D800 parameter with E800 and this works. The patch above removes it entirely. I don't understand this code or what it does, but it seems like the patches that I'm referencing are more evolved than the one given on this page (which removes this value rather than replacing it). Maybe someone who understands its function might want to look at it?
#4
Yes, this is just the same issue as one described in #1446372: Invalid Unicode code range in PREG_CLASS_UNICODE_WORD_BOUNDARY fails with PCRE 8.30 (pcre > 8.30, observable in FreeBSD and Arch, probably Gentoo too).
As a matter of fact, CTOOLS_PREG_CLASS_ALNUM (changed in patch from #1) is copied from search.module's PREG_CLASS_SEARCH_EXCLUDE.
Removing d800 is quite safe, since CTOOLS_PREG_CLASS_ALNUM is used to escape separator if it's not alphanumeric.
#5
Automatically closed -- issue fixed for 2 weeks with no activity.
#6
Hi, I really don't know if this patch is right, but for the moment there is no more messages on my websites (This is D6.25)
#7
Updated to be in sync with #1446372: Invalid Unicode code range in PREG_CLASS_UNICODE_WORD_BOUNDARY fails with PCRE 8.30.
As noted in 1446372's comment #16 it's better to scan Unicode private use instead of surrogate area, then to ignore it.
This code has been merged in Drupal core 8.x and 7.x (patch waiting to be ported to 6.x), and I think ctools should do the same.
PS. @Marc Angles: yes, that's the same fix that'll be included in next 6.x core.
#8
I've just manually replaced
'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.with
'\x{a80b}\x{a823}-\x{a82b}\x{e000}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.as in #7 and it fixed the error messages in 6.x-1.8+50-dev
Since the first half of the patch seems to be already committed, that last change is all that is needed to remove the message (at least using my configuration).
RTBC, anyone?
#9
The exact same fix as in #7 has already been commited in D6, D7, and D8. Marking it as RTBC.
#10
are they committed to dev? awesome! also, maybe this should be marked as fixed?
thanks!
#11
@lucascaro:
No, patch from #7 needs to be commited to ctools, since the exact same fix was applied to Drupal 6, 7 and 8.
#12
ooh, nice. This is starting to trigger error messages in all of my projects, so I'd love it to be committed. How can I help making it happen?
#13
For tonight.
#14
awesome!
#15
Oh wow the format-patch threw me off there. This is much more straightforward then I thought.
Also, why'd you unassign me? I almost couldn't find the issue.
http://drupalcode.org/project/ctools.git/commit/a1c5c9b
#16
og god, sorry about that. I don't know why that happened, I didn't touch the assigned box. :S
#17
Automatically closed -- issue fixed for 2 weeks with no activity.
#18
Sorry to reopen, but it looks to me like this didn't make it into 6.x-1.x-dev.
Attached is a patch for 6.x-1.x-dev made by copying PREG_CLASS_SEARCH_EXCLUDE from search.module from Drupal 6.26. Works for me, but not thoroughly tested.
Best,
-G
#19
I agree that this seems to be unfixed still.
@gdl: Why the difference between your patch and the one in #1697538: Unicode compilation fails because of invalid code range in CTOOLS_PREG_CLASS_ALNUM?
#20
It looks like the patch in #1697538 changes just a single code point, while mine changes the list of code points in CTools to match the code points from the core search module. I guess it's up the the developers to determine which approach to take. I don't have an informed opinion about which is better - I just want to see the issue resolved!
Cheers,
-G
#21
Fix in #18 gets rid of the warnings in 6.x-1.x.
#22
The #18 patch works here, with pcre package version 7.8 and ctools 6.x-1.10+0-dev.
#23
confirmin, patch from #18 fixes the issues you get when using PHP > 5.3.18 . Applying did not succeed though. Thanks!
*removed since it up there already*
#24
Just upgraded to 6.x-1.10, and found the error from the OP was again occurring, #8 again solves the problem.
Why is this not changed in the module core?
#25
Thanks guys. Tested and applied #1697538-2: Unicode compilation fails because of invalid code range in CTOOLS_PREG_CLASS_ALNUM since it brings the regex into a state matching the 7.x-1.x branch it seemed more reasonable. If we need the extra change here can you reroll with an explanation?
#26
hello
i am not professional in Drupal core settings
and this error appear for me too
warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in C:\inetpub\vhosts\xxxxx\httpdocs\sites\all\modules\ctools\includes\cleanstring.inc on line 157.
how can i fix this problem pleas ????
#27
#8 Work for me :D
I've just manually replaced
'\x{a80b}\x{a823}-\x{a82b}\x{d800}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.
with
'\x{a80b}\x{a823}-\x{a82b}\x{e000}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.
#28
this is happening more often after I search and search, then post, I find the issue when searching google on drupal. I could not find this issue in Drupal search.
possible duplicate: http://drupal.org/node/1878284
This seems still unfixed in the module.
#29
I solved it doing this. I TRUNCATE the following tables: cache_page, acceslog, cache_menu and cache_filter.
I think that commen #27 don't work for me, but i set the line like beto_beto say:
'\x{a80b}\x{a823}-\x{a82b}\x{e000}-\x{f8ff}\x{fb1e}\x{fb29}\x{fd3e}\x{fd3f}'.
#30
I've the same bug and also the patch from #18 fixed it.
#31
I just ran into this problem for this first time today (6.x-1.10).
#8 solved the problem.
#32
I'm on 6.x-1.10, running PHP 5.3.21, and running Drupal 6.28. Anyone know if #18 or #8 is the better route to take?
#33
I've used the approach from #8 and that's solved the problem for me.
#34
I was getting
warning: preg_match() [function.preg-match]: Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1811 in /var/www/mysite/public_html/sites/all/modules/ctools/includes/cleanstring.inc on line 157
a number of times after upgrading my server from
PHP 5.2.17-0.dotdeb.0
MySQL 5.1.56
to
PHP 5.3.21-1~dotdeb.0
MySQL 5.1.66
on a Drupal 6.28 site running cTools 6.x-1.10, but #8 solved the problem. Not sure why this hasn’t been committed!
#35
Same as Grabby for me, #8 fixed the problem on my drupal6 installations. I really believe it should be commited.
#36
Thanks! I had the same issue as #34: the error appeared after our servers were upgraded to PHP5.3.x. The simple fix in #8 resolved the problem.
#37
The fix from #8 works but doesn't affect search.module. This patch incorporates #8 with a fix for the search module using the same code.
Works for me on:
FREEBSD
Drupal 6.16
PHP 5.3.19
APACHE 2.2.23
#38
Another vote for solution in #8
#39
Confirm that patch in #18 worked - CTools 6.x-1.10, Drupal 6.28, PHP 5.3.19, Debian 6.0.6
#40
#18 worked very well for my installation :
CTools 6.x-1.10
PHP 5.3.23
Drupal 6.28
I updated to "tested by the community" because it's enough trivial to include it into git.
#41
The last submitted patch, ctools-1444006-disallowed-unicode-D6-37.patch, failed testing.
#42
As per #25, this was marked fixed; neclimdul committed the patch. Why are there 17 comments after that asking for this to be committed?
The reason it failed testing is the patch, which is already committed, doesn't apply.
Do people actually read issues before they start in? Because it seems like a giant waste of time to have so many people asking why a fix, which is committed, hasn't been committed. Because it's been committed.
#43
There are 17 comments because, as I said already in comment 19, the fix was only committed to the 7.x-1.x branch, not the 6.x-1.x branch. Hence, the bug is still present on 6.x-1.10... This is clearly frustrating for people using Drupal 6. (The fix is almost one year old.)
OK, merlin, you are right. The fix was applied to the 6.x-1.x branch - ref this commit. So the only problem is that there hasn't been a new release of Ctools since 6.x-1.10.
#44
Updating version info since this isn't fixed for 6.x-1.10, only in 6.x-1.x-dev.
#45
Sorry about the noise. The bug, which is fixed, was in version 6.x-1.10. Some times I'm just that stupid.