Form API accepts values containing invalid UTF-8

I think it makes sense to reject invalid UTF-8 values when submitting a form. But I would simply do that in drupal_build_form() when we map $_GET or $_POST to $form_state['input'].

Or even earlier in the process?

A (maybe faster) alternative.

I agree that this is needed. We often get invalid UTF-8 submitted through comment forms.

Always reject form submissions that contain invalid UTF-8.

No, please don't do that, as it's very bad for UX. Invalid UTF-8 is always generated by machines, and you leave the user with a bunch of text (usually copy-pasted from somewhere) that they cannot use, and present them an error message. How is the user supposed to proceed? Instead, invalid UTF-8 can be easily filtered out, at least if you know a value is supposed to be plain text.

I ... do not know. There is sense in this for sure. But do we really want this to be done in bootstrap when form API has recursion handling already? Also, our philosophy was no to mangle user data but to change on output only. And then there is the question -- did the whole world migrate to UTF8 solidly? Is there no need for a $conf override to opt out?

For example, in the case of the Mollom module, this is what happens when a node or comment with invalid UTF-8 is submitted through Drupal:

1. The Mollom module converts it into XML so it can be send over XML-RPC. Because of the invalid UTF-8, the resulting XML is invalid too. The backend parser throws up and returns an XML-RPC error. If the Mollom fallback mode is set to 'accept comments' when an error occurs, the spam is accepted as ham. This provides spammers an easy way to by-pass Mollom. On the flip side, it can also cause valid ham with broken UTF-8 to be rejected.
2. If the node or comment is shown on a page with doctype XHTML, the page won't validate.
3. If the node or comment is shown in an RSS feed, the RSS feed won't render in many feed readers.

Our options:

1. Reject invalid UTF-8 on input. This is implemented by this patch.
2. Turn invalid UTF-8 into valid UTF-8 on input.
3. Turn invalid UTF-8 into valid UTF-8 on output.

Note that check_plain already turns invalid UTF-8 into the empty string. I would be OK with a form based approach if there is an opt out.

Presumably option #10.3 either already is or can be supported via an input format filter. Regradless of that, there seems to be use-cases for #10.1 and #10.2 and for nothing at all. And potentially someone might want to choose a different option based on the form/URL or other context information (for example, #10.1 for anonymous and #10.2 for admins).

Do we already have an early bootstrap handler we can piggy back on to? If not, can we make a new one? And let this problem be solved in contrib?

• Do we want to throw out user input just because of a single non-UTF-8 character, who knows copied from where? I will presume no.
• Do we want to burden bootstrap with a tricky regex? No.

That's not quite enough. But this is. I might come up with a test, even.

I fail to see the intend of #15 and #16, but I'm very reluctant to just remove characters from the input stream like this. It might have some nasty consequences.

The goal here is to remove invalid UTF-8 sequences. The regexp captures valid UTF-8 sequences and also matches anything else but those are not captured and so they get removed.

Rerolled on a current 7.x.

#21 yes, yes and yes. The rest, no, you can't possibly suggest to just throw away some user input without notifying the user.

A couple of reasons to not submit a form with the portion of text that fails the UTF8 regex test excluded are:
1) If there is an error in a UTF8 string, you can't be positive of the error boundaries, as part of the erroneous sequence could by coincidence be part of a regex match.
2) A completely random byte sequence would likely contain some matches, as the categories are pretty broad.

It seems to me that this could potentially be a security issue. It's been dormant for so long, it may not be an issue any more. Just trying to clean up assigned & inactive issues.

Reviewed this as part of the bug smash initiative. Judging by the discussion and debate on this issue it seems like this is probably a feature request. Happy for it to be changed back if I've misunderstood.

Needless to say this needs a reroll, looks like \Drupal\Core\Form\FormBuilder::handleInputElement has replaced _form_builder_handle_input_element. We also need steps to reproduce and test coverage.

Rerolled the patch #25 in Drupal 10.1.x.

The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.