drupal_substr() is slow on long strings with large start indices. To tokenize a string, it's much faster to repeatedly call:
$token = drupal_substr($string, 0, $tok_len);
$string = drupal_substr($string, $tok_len);
than it is to iterate $i through the string and call:
$token = drupal_substr($string, $i, $tok_len);
It seems like it might be possible to use preg_match or preg_replace along with the {} quantifier and the /u modifier to good effect here. Is there some reason that this is a bad idea or not possible?
Alternatively I just tried what I thought was a fairly naive iconv() implementation (convert the whole string to UTF32, call substr(), convert back to UTF8), and it's way faster than the current drupal_substr implementation.
| Comment | File | Size | Author |
|---|---|---|---|
| #4 | speed.php__0.txt | 9.04 KB | Wesley Tanaka |
| #1 | speed.php_.txt | 3.07 KB | Wesley Tanaka |
Comments
Comment #1
Wesley Tanaka commentedscript used for testing
put in drupal directory and run with "php speed.php"
Comment #2
Wesley Tanaka commentedThat iconv implementation might be returning the wrong substring if the UTF32 output contains a byte order mark. But the performance characteristic would be roughly the same after it was fixed.
Comment #3
Wesley Tanaka commentedThere's some promising-looking LGPL code here.
This may also be a good thing to incorporate if it turns out to be true:
Comment #4
Wesley Tanaka commentedUploading a new test script which:
1. fixes the bug where outer-loop $i was getting clobbered by inner-loop $i. Oops.
2. tests the splitbrain.org utf8_substr
3. uses a longer test string. this causes the current drupal_substr to really croak.
Comment #5
gábor hojtsyThis issue is better suited as being part of the base system, not the language system.
Comment #6
nancydruAre there some performance suggestions here? Since there is no chance of this getting into either 5.x or 6.x, how about changing it to 7.x?
See also: http://drupal.org/node/236924
Comment #7
kbahey commentedWesley
Is this noticeable on an average Drupal page view? Can a change there show measurable improvement, or it is all under the 1% or 2-3 ms range?
I am not saying that what you propose is wrong, but it may not be worth the effort or added code complexity vs. something like tuning one query for example ...
Try to create a prototype and benchmark it using Drupal, so its net effect is assessed, not theoretical effect.
Comment #8
nancydruThis may very well have a significant impact on filters and translation efforts (e.g. i18n). The issue I pointed to is reporting 47 second page loads using the Glossary module, which is set up to handle non-English, non-Latin, and multi-byte character sets.
Comment #9
Wesley Tanaka commentedKhalid,
I don't know. I believe I noticed the problem while calling drupal_substr() from my own code.
Comment #10
Wesley Tanaka commented> Are there some performance suggestions here?
Nancy, I went back and took a look at what I did -- I ended up not using drupal_substr() but instead doing an iconv(substr(iconv())) like the one in the test file attached here: http://drupal.org/files/issues/speed.php__0.txt
Or you could try and replace drupal_substr() wholesale, with either the iconv() version or the utf8_substr() that's also in the above file.
Khalid, the question is not with a particular Drupal core menu callback being slow, it's whether or not drupal_substr() is fast enough to be a generally useful public Drupal API function for contrib modules to use.
Comment #11
nancydruGlossary benchmarks (http://drupal.org/node/236924#comment-781505) show utf8-substr to be slightly SLOWER that drupal_substr.
Comment #12
lilou commentedAdd tag.
Comment #13
killes@www.drop.org commentedPresented eveidence is inconclusive. I mark this as "won't fix".