| Project: | Drupal core |
| Version: | 6.x-dev |
| Component: | install system |
| Category: | bug report |
| Priority: | critical |
| Assigned: | Unassigned |
| Status: | closed (fixed) |
| Issue tags: | 7.13 release notes, D7 stable release blocker, needs backport to D6, needs backport to D7 |
Issue Summary
The latest update of PCRE to 8.30 produces the following warning in drupals installation:Warning: preg_match(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 2112 in truncate_utf8() (line 339 of /home/sites/drupal7/includes/unicode.inc).
This is discussed in full here: http://drupal.org/node/1442876
Systems affected so far are Archlinux and FreeBSD. It is possible that more systems will display such warnings once they update PCRE to 8.30
A user has mentioned that this also affects drupal-searches (http://drupal.org/node/1442876#comment-5622720). I haven't tried it yet myself so I cannot tell.
I created a bug-report on Archlinux (https://bugs.archlinux.org/task/28533) but they suggest that this get fixed on the drupal-side.
Comments
#1
The attached solves the problem. I simply removed the
\x{D800}-\x{F8FF}part as Pierre Schmitz suggests here: https://bugs.archlinux.org/task/28533#2
Silly me... The above patch is against drupal-7.x-dev as it should be. Forgot to update the issue status
#3
The last submitted patch, 1446372-modified-unicodeinc-to-play-nice-with-PCREv830.patch, failed testing.
#4
I got the same error on FreeBSD8.2, pcre 8.30, apache22, php5.3, mysql55. On my another FreeBSD system, php5.2, php52-pcre, drupal7.12 works well.
#5
Looks like the patch is breaking existing test coverage, didn't look to see if it's a problem with the test or a problem with the patch. Note all bugfixes are applied to 8.x first then backported.
Assuming this will need to be backported to D6 as well.
#7
The patch has 1 fail and 38700 passes. Probably something very specific fails that needs the characters I patched.
@alan mccoll, there is a subscribe button now in drupal.org, you do not need to answer in order to subscribe
#8
Yes it's the search test that breaks, so it's directly related to the patch.
#9
#1: 1446372-modified-unicodeinc-to-play-nice-with-PCREv830.patch queued for re-testing.
#10
The last submitted patch, 1446372-modified-unicodeinc-to-play-nice-with-PCREv830.patch, failed testing.
#11
It was said that PCRE 8.30 uses libpcre.so.0 instead of libpcre.so.1, which causes the problem. I will test on my system tomorrow.
#12
Help me out here, we should be testing against D8??
I haven't had any experience with D8, I'll look at it this week
#13
Recently I have updated PCRE to version 8.30 and now I constantly get this error when do any search:
•warning: preg_replace(): Compilation failed: disallowed Unicode code point (>= 0xd800 && <= 0xdfff) at offset 1816 in /www/istmat/soft/drupal/modules/search/search.module on line 333.
I think that error must be eliminated ASAP as it really nasty bug and from now more and more sites will have updated PCRE!
#14
bserem, invalid codepoints in modules/search/test/UnicodeTest.txt are most likely the culprit.
#15
Thanks Heine.
Anybody having any ideas on the patch? It failed (as it says) one time in the search module (it had 650 passes of course).
That is against D7 of course, not D8.
I'm out of ideas
#16
The test fails on line 311 of UnicodeTest.txt which contains the byte sequences:
ee 80 80 ef a3 bfef a3 bf is UTF-8 for U+F8FF, which has been removed with the patch.
It appears that a sequence is not possible because it encompasses the following Unicode areas:
Surrogate Area: U+D800 - U+D8FF
Private Use Area: U+E000 - U+F8FF
The surrogate area is not allowed in PCRE 7.3+. Attached patch removes it.
(I also believe we shouldn't break on U+FEFF which is a Word Joiner, but that's best left for another issue).
#17
This is the Drupal 8 patch. Please test with PCRE 8.30 as I do not have a test install with it atm.
#18
The last submitted patch, do-1446372-remove-surrogate.patch, failed testing.
#19
*sigh*
Patch in the Dark Ages format :(
#20
#21
Just did a minimal-D8 install, all went fine.
It should however be backported to 7 and 6 in order to test it with the search function.
#22
Drupal 7 patch in updated format to aid testers. D8 contains the same test BTW.
#23
A similar patch is needed for D6 where the code is located in PREG_CLASS_SEARCH_EXCLUDE (search.module).
Thanks. -virgil
#24
I can confirm that on D6 after applying patch from #23 error gone and search works again.
#25
I confirm D8, though I didn't have a site with a search-index to test search.
All my D7 sites are in PCRE8.2 still, so we need somebody for D7 to confirm it.
#26
Go. Before catch pointed out this issue (I was only looking at criticals) I had the same patch prepared.
#27
I checked and the next Ubuntu release due Apr seems to be using PCRE 8.12 http://packages.ubuntu.com/search?keywords=pcre&searchon=names&suite=pre... . CentOS/RedHat is using PCRE 7, we can expect this most excellent and modern OS to break some time around the Sun grows cold. I suspect Fedora will be the next to break on May 8 when Fedora 17 releases because their trunk ("Rawhide") contains 8.30.
#28
I'm going to tag this as Drupal 7 stable release blocker and critical. The next D7 release is likely to be end of March, that only gives people 5 weeks to upgrade before the Fedora upgrade, after which point I'd expect us to start seeing duplicate tickets getting opened etc.
#29
The last submitted patch, do-1446372-remove-surrogate-D6.patch, failed testing.
#30
Not sure why the suffixed patches are being tested, back to RTBC.
#31
Not sure why they got tested suddenly now when they were submitted previously, but:
#1092232: Bot needs to handle patches named for all core versions -D[678]
#32
Committed/pushed this to 8.x.
There was some discussion on irc about trying to get this into 6.25, I don't think we should try to do that, but it'd be good to get it into 6/7 so there's time for it to bed in before and end of march release (i.e. 6.26).
I'm leaving this RTBC for 7.x since #22 is the same patch re-rolled.
#33
Committed and pushed to 7.x. Thanks!
Moving to 6.x for backport.
#34
#23 has the patch.
#35
#23: do-1446372-remove-surrogate-D6.patch queued for re-testing.
#36
#22: do-1446372-remove-surrogate-D7.patch queued for re-testing.
#37
thanks, went to dev 7, issue seems to be gone for now.
#38
Hrm, we wanted to get this in this month, right?
#39
#23: do-1446372-remove-surrogate-D6.patch queued for re-testing.
#40
The problem has also turned up on Debian wheezy now. The patch in comment #23 resolves it.
#41
Thanks all, committed and pushed #23 to Drupal 6. Should be in the next bugfix release.
#42
The D7 patch from comment #22 fixed the issue of 'You must include at least one positive keyword with 3 characters or more.' on any search under Debian Testing. Actually I couldn't get the patch to apply but I manually replaced the one line from the patch file and it fixed search.
# dpkg -l | grep pcre
ii libpcre3:amd64 1:8.30-4 Perl 5 Compatible Regular Expression Library - runtime files
...
#43
#22: do-1446372-remove-surrogate-D7.patch queued for re-testing.
This is a Drupal 7 patch but the test system thinks it's a Drupal 6 patch. :(
#44
Please do not re-test the patch. The issue is marked fixed. This means the patches have been committed to all applicable development branches for D6, D7, and D8.
#45
#22: do-1446372-remove-surrogate-D7.patch queued for re-testing.
#46
#47
Path #19 worked for me in a D7 with FreeBSD 9 (pcre 8-30_1)
#48
#49
Calling this out as something to mention in the release notes.
#50
Automatically closed -- issue fixed for 2 weeks with no activity.