drupal_substr() is slow on long strings with large start indices. To tokenize a string, it's much faster to repeatedly call:

$token = drupal_substr($string, 0, $tok_len);
$string = drupal_substr($string, $tok_len);

than it is to iterate $i through the string and call:

$token = drupal_substr($string, $i, $tok_len);

It seems like it might be possible to use preg_match or preg_replace along with the {} quantifier and the /u modifier to good effect here. Is there some reason that this is a bad idea or not possible?

Alternatively I just tried what I thought was a fairly naive iconv() implementation (convert the whole string to UTF32, call substr(), convert back to UTF8), and it's way faster than the current drupal_substr implementation.

CommentFileSizeAuthor
#4 speed.php__0.txt9.04 KBWesley Tanaka
#1 speed.php_.txt3.07 KBWesley Tanaka

Comments

Wesley Tanaka’s picture

StatusFileSize
new3.07 KB

script used for testing

put in drupal directory and run with "php speed.php"

Wesley Tanaka’s picture

That iconv implementation might be returning the wrong substring if the UTF32 output contains a byte order mark. But the performance characteristic would be roughly the same after it was fixed.

Wesley Tanaka’s picture

There's some promising-looking LGPL code here.

This may also be a good thing to incorporate if it turns out to be true:

 111  /**
 112   * Unicode aware replacement for strlen()
 113   *
 114   * utf8_decode() converts characters that are not in ISO-8859-1
 115   * to '?', which, for the purpose of counting, is alright - It's
 116   * even faster than mb_strlen.
 117   *
 118   * @author <chernyshevsky at hotmail dot com>
 119   * @see    strlen()
 120   * @see    utf8_decode()
 121   */
 122  function utf8_strlen($string){
 123    return strlen(utf8_decode($string));
 124  }
Wesley Tanaka’s picture

StatusFileSize
new9.04 KB

Uploading a new test script which:

1. fixes the bug where outer-loop $i was getting clobbered by inner-loop $i. Oops.
2. tests the splitbrain.org utf8_substr
3. uses a longer test string. this causes the current drupal_substr to really croak.

gábor hojtsy’s picture

Component: language system » base system

This issue is better suited as being part of the base system, not the language system.

nancydru’s picture

Are there some performance suggestions here? Since there is no chance of this getting into either 5.x or 6.x, how about changing it to 7.x?

See also: http://drupal.org/node/236924

kbahey’s picture

Wesley

Is this noticeable on an average Drupal page view? Can a change there show measurable improvement, or it is all under the 1% or 2-3 ms range?

I am not saying that what you propose is wrong, but it may not be worth the effort or added code complexity vs. something like tuning one query for example ...

Try to create a prototype and benchmark it using Drupal, so its net effect is assessed, not theoretical effect.

nancydru’s picture

Version: 5.1 » 7.x-dev

This may very well have a significant impact on filters and translation efforts (e.g. i18n). The issue I pointed to is reporting 47 second page loads using the Glossary module, which is set up to handle non-English, non-Latin, and multi-byte character sets.

Wesley Tanaka’s picture

Khalid,

I don't know. I believe I noticed the problem while calling drupal_substr() from my own code.

Wesley Tanaka’s picture

> Are there some performance suggestions here?
Nancy, I went back and took a look at what I did -- I ended up not using drupal_substr() but instead doing an iconv(substr(iconv())) like the one in the test file attached here: http://drupal.org/files/issues/speed.php__0.txt

Or you could try and replace drupal_substr() wholesale, with either the iconv() version or the utf8_substr() that's also in the above file.

Khalid, the question is not with a particular Drupal core menu callback being slow, it's whether or not drupal_substr() is fast enough to be a generally useful public Drupal API function for contrib modules to use.

nancydru’s picture

Glossary benchmarks (http://drupal.org/node/236924#comment-781505) show utf8-substr to be slightly SLOWER that drupal_substr.

lilou’s picture

Issue tags: +Performance

Add tag.

killes@www.drop.org’s picture

Status: Active » Closed (won't fix)

Presented eveidence is inconclusive. I mark this as "won't fix".