Mailhandler does not handler text encoding properly. For example, the ISO-8889-1 text part of an email is not converted to UTF-8 before beeing saved as a story node.

Comments

moshe weitzman’s picture

Any tips on how to resolve this?

Alexandre Gomes@ensaio.poderdapalavra.com.br’s picture

Assigned: Unassigned » Alexandre Gomes@ensaio.poderdapalavra.com.br
Category: bug » feature

I have the same problem. I need that all messages will be converted to utf-8, since most of my users will use email clients with the iso-8859-1 encoding.
A workaround of this will be using Iso enconding all the site, but I prefer to convert the mailhandler output.
It is possible?
(using mailhandle last CVS and drupal 4.4.1)

isis-1’s picture

I guess we can use "imap_mime_header_decode" and "iconv" to solve this problem. use imap_mime_header_decode (or iconv_mime_header_decode?) to get the charset of that email and then use iconv convert to UTF-8.

but.....I'm a newbie to PHP and now I can only write simple stuff, I have no idea how to add correct code to right place...

mobile device in my country always use BIG5 as default, so this problem bothers me a lot.

isis-1’s picture

i added something below line 217 and it now works for me (BIG5). I just start to learn PHP for 1 week so I have no idea if it's the best way to solve this problem or proper to other languages.

  // decode encoded subject line
  $subjectarr = imap_mime_header_decode($header->subject);
  for($i=0;$i<count($subjectarr);$i++) {
    $charset = $subjectarr[$i]->charset;
    $node->title .=  $subjectarr[$i]->text;
    if(!strcasecmp($charset, "utf-8") ==0){
         $node->title = iconv($charset, "utf-8", $subjectarr[$i]->text);
         $node->body = iconv($charset, "utf-8", implode("\n", $tmp ));
  }
  }

isis

isis-1’s picture

Title: Doesn't handle text encoding properly » is this better?

fix title problem.


  // decode encoded subject line
  $subjectarr = imap_mime_header_decode($header->subject);
  for($i=0;$i<count($subjectarr);$i++) {
    $charset = $subjectarr[$i]->charset;
    $text=  $subjectarr[$i]->text;
    if(!strcasecmp($charset, "utf-8") ==0){
         $node->title .= iconv($charset, "utf-8", $text);
         $node->body = iconv($charset, "utf-8", implode("\n", $tmp ));
    }else {
        $node->title .= $text;
        $node->body = implode("\n", $tmp);
    }
}

Steven’s picture

drupal_xml_parser_create() contains similar conversion code (for xml documents), but it tries iconv, gnu recode and mbstring in the hope that one of them is available.

Maybe a general drupal_encoding_convert() wrapper could be useful?

The snippet in question makes the assumption that the body is encoded in the same character set as the subject. This is of course mostly the case, but it's probably still best to extract the character set from the Content-Type header instead.

The Content-Type header looks like:
Content-Type: text/plain; charset=us-ascii

Also, us-ascii encoded data does not need to be converted to UTF-8 because U+0 to U+7F are encoded in UTF-8 as the literal bytes 0x00 to 0x7F.

moshe weitzman’s picture

Title: is this better? » recognize charset of incoming email and convert as needed
moshe weitzman’s picture

Anonymous’s picture