XML-RPC request (string) values are not safe for UTF-8

This looks like the right thing to do do me; we shouldn't work around broken clients because it breaks valid clients.

UTF-8 is the new ISO-8859-1, and has been for a while (at least in Drupal).

Oh my ...

We've been discussing this some more and found that not all valid UTF-8 characters are valid XML characters. See http://www.w3.org/TR/2000/REC-xml-20001006#NT-Char.

Given that we're returning XML, additional filtering is required. This probably affects all of our XML-generating code; including format_rss_item().

That's not true. All you have found is that not all valid UTF-8 byte streams are valid XML characters. Guess what? They are not Unicode characters either.

OK so now I understand. What you can not do is to take an arbitrary bytestream, encode it with the algorithm called UTF-8 and expect the results to be valid characters by the Unicode standard. UTF-8 is just that, an encoding of a bunch bytes into another bunch of bytes. What we need here, I think is preg_replace('/.+/u', '\0', htmlspecialchars($xmlrpc_value->data, ENT_COMPAT, 'UTF-8')).

Also, don't we need tests?

Edit: or we actually need to take #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF] and turn it into a preg to be sure.

Everyone is tired? check_plain() is just perfectly what we need here.

If the input stream is not valid UTF-8, check_plain() will kill it. Which is what we want anyway.

If PHP >= 5.2.5, we don't perform that filtering,...

Because PHP does that for us since 5.2.5.

and even for lower versions, check_plain() actually checks the beginning of the string only, not caring for invalid characters elsewhere. Or at the very least, if check_plain() is supposed to filter out invalid characters, then it is not doing what we expect it should be doing on my local test site on PHP 5.2.6/Win32.

preg_match() will check the validity of the whole input string before doing the matching.

So indeed, we are letting characters pass-thru that might not be valid in XML, ie. which are not in:

#x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

Most characters in [#x1-#x1F] are not valid, and we should strip them.

Question: should we also remove those from HTML output? It might be a costly operation.

+    // @todo chr() #fail
+    $invalid = array_merge(
+      $invalid,
+      array_map('chr', range(hexdec('D800'), hexdec('DFFF')))
+    );

You have to encode those in UTF-8. chr($code) is the UTF-8 representation of $code only for the lowest 7 bits.

It seems that the XML extension has a nice utility function to do that:

http://php.net/manual/en/function.utf8-encode.php

I've tried to look up what the HTML5 spec defines, but it does not seem to define precise character ranges, just Text must consist of Unicode characters, and must not contain U+0000 characters, noncharacters, or control characters (except spaces).

I'd say the specs are actually pretty precise: any noncharacter or control characters need to be encoded as numeric entities (see 8.1.4 Character references). So we're not talking about removing any characters, but properly encoding them.
Scratch that, it clearly says they're not even allowed as entities.

Probably related: folks over at Views Bonus Pack (XML export) are struggling with a similar issue: #603420: XML export mangles <, >, and &

After #1285726-48: Remove XML-RPC moving this to Drupal 7.

But it could also be appropriate to move it to the contrib xmlrpc module.