Hotmail and UTF-8, what a headache!

There was, in an earlier version of simplenews. Then, people were crying for UTF-8, and because that's the standard in Drupal, I removed the character encoding dropdown list. If I find some time, I'll reintroduce it.

I just replaced sn_mail_send with the following. It has been tested and seems to work fine for me but I have no idea whether there might be bad side effects or other problems...

Of course, with this "fix" you can't send in UTF-8 anymore.

function sn_mail_send($mail) {
  $mail->to = trim($mail->to);
  require_once('activeMailLib.php');
  $email = new activeMailLib($mail->s_format);
  $email->From(utf8_decode($mail->from_address), utf8_decode($mail->from_name));
  $email->To(utf8_decode($mail->to));
  $email->Subject(utf8_decode($mail->title));
  $email->Message(utf8_decode($mail->message), 'iso-8859-1', '8Bit');
  $email->priority($mail->priority);
  if ($mail->receipt) {
    $email->Receipt(utf8_decode($mail->from_address));
  }
  $email->Send();
  return $email->isSent($mail->to);
}

I have this problem also - but with a multitude of character sets, including notably Korean and Russian. What about using the multibyte_encoding functions? This would provide a wider choice than the standard PHP. A drop-down list for the site to configure in what coding the message should be sent in could be very useful.

We have now request of utf-8 and iso-8859-1. What encoding do you propose?

Please explain your "What about using the multibyte_encoding functions?"

Actually what I'm thinking of is something like this: use the multi-byte encoding functions available in PHP (though not ALWAYS available I think, so a call to function_exists would be in order no doubt) to allow the administrator to choose which encoding he wanted to use. Therefore, in the admin/settings part of Simplenews you would have a drop-down list to choose your mail encoding, which could simply be the output from mb_list_encodings.

Then the send mail function would look something like this:

function sn_mail_send($mail) {
  $encoding = variable_get('simplenews-encoding', 'UTF8');
  $mail->to = trim($mail->to);
  require_once('activeMailLib.php');
  $email = new activeMailLib($mail->s_format);
  $email->From(mb_convert_encoding($mail->from_address, $encoding), mb_convert_encoding($mail->from_name, $encoding));
  $email->To(mb_convert_encoding($mail->to, $encoding));
  $email->Subject(mb_convert_encoding($mail->title, $encoding));
  $email->Message(mb_convert_encoding($mail->message, $encoding), $encoding, '8Bit');
  $email->priority($mail->priority);
  if ($mail->receipt) {
    $email->Receipt(mb_convert_encoding($mail->from_address));
  }
  $email->Send();
  return $email->isSent($mail->to);
}

Note I'm not sure this is exactly right - have not had a chance to test it yet. This would truly internationalise the simplenews function - and I could send newsletters out in Russian and Korean, not to mention not have funny characters appear in French, Portuguese, or indeed Swedish (which is not iso-8859-1 either in fact). In other words, you would be handling internationalisation correctly and as completely as possible in PHP (I think - I have to confess I'm not sure I have understood everything here...)

Except I just noticed that mb_list_encodings is PHP5 only, so you would have to do that by hand to accomodate PHP4

Found some alternative conversion functions in the function. I post it here for future reference.

Sorry, you found it where? And does future reference mean it will get into Simplenews? He asked, hopefully?

Hmm, I wasn't awake this morning. The drupal_convert_to_utf8 function has two alternative conversion functions besides mb_list_encodings. Perhaps this can help in some way.

I recognize your problem because I also use those funny characters in my mails (Dutch) but the feature list is _long_ and not all can be implemented. Patches are very welcome and will help getting this functionality implemented.

Looks interesting - clearly this is a way to go for a complete solution. What a shame that there is not a drupal_convert_FROM_utf8 function!

I adapted the "solution" to send iso-8859-1 formated messages to hotmail email adresses only :

- download "activemaillib.php" and put it inside simplenews module folder (you have something like : drupal/modules/simplenews/activemaillib.php). I found it at http://www.phpclasses.org/browse/file/8618.html

- edit simplenews.module, around line 1188 (just before if (module_exists('mimemail')) inside function simplenews_mail_send($mail)), and add the following :

  // if it is an hotmail address provide an other system for mail sending ...
  if (strstr($mail->to, 'hotmail.com')) {
	  $mail->to = trim($mail->to);
	  require_once('activemaillib.php');
	  $email = new activeMailLib($mail->s_format);
	  $email->From(utf8_decode($mail->from_address), utf8_decode($mail->from_name));
	  $email->To(utf8_decode($mail->to));
	  $email->Subject(utf8_decode($mail->title));
	  $email->Message(utf8_decode($mail->body), 'iso-8859-1', '8Bit');
	  $email->priority($mail->priority);
	  if ($mail->receipt) {
		  $email->Receipt(utf8_decode($mail->from_address));
	  }
	  $email->Send();
	  return $email->isSent($mail->to);
  }
 

- save and "enjoy"

NOTE :
This is an ugly hack, and it uses an exernal mail lib. It is not recomended to modify a module file. But since it only affects emails ending with "hotmail.com" it should not have side effects to other emails.

I've been working of this issue and it is not final. In fact it is waiting for further work on Mimemail.
Since there is frequent response on this issue, I decided to post it so it may help some.

This patch:
* Has one encoding selection for all e-mails send with Simplenews.
* Only works with plain text and without Mimemail module enabled.
* Is hardly tested, so at you own risk!

Actually, I'm having the same problem in Gmail, so it's not a Hotmail-related issue.

Is there some permanent fix available, either in Mime Mail or in Simplenews?

Same thing here !

I don't get it. Hotmail makes a mess out of UTF-8??

thanks for the patch, but i see it is for version 1.56. i am using version 1.1 and cannot find 1.56 for download. i tried patching 1.1 but got an error "invalid foreach on line 372". what am i doing wrong?

i ended up adding philippejadin's code instead of applying the patch. it seems to have worked fine (also using french characters), but i had to tweak two things, so i'm posting here in case it helps anyone.

in order to fix the problem for all hotmail _and_ yahoo addresses, i changed the first line:

  if (strstr($mail->to, 'hotmail.com')) {

to:

  if ((strstr($mail->to, 'hotmail')) || (strstr($mail->to, 'yahoo'))) {

in order to display special characters correctly in the titles of my emails, i changed this:

      $email->Subject(utf8_decode($mail->title));

to:

      $email->Subject(utf8_decode($mail->subject));

Using iso-8859-1 in an all-UTF-8 environment like Drupal is like continuing to drive with one flat tire because in a straight line that still works reasonably well. In my experience, the problem is usually Content-Transfer-Encoding, and not the Content-Type.
I had problems with UTF-8 myself, and it appeared to have been a combination of Content-Transfer-Encoding: 8bit and an outdated version of amavisd-new (an interface between an MTA and scanners such as clamav and SpamAssassin).
Drupal uses Content-Transfer-Encoding: 8bit by default. In my opinion, 8bit is the best setting, but it can cause problems with old and buggy software. SMTP originally only supported 7 bit characters, which gives a problem with UTF-8. 8 Bit has been standardized a long time ago, but as usual with these kinds of things, incompatibilities still arise now and then.
To cope with this, an additional layer of encoding was invented between the contents of the email and the MTA: the Content-Transfer-Encoding. Setting this to 8bit is the most simple way, and it expects clients and MTA's to simply interpret the text as 8 bit text. To prevent non-confirming software from messing up the mail two options are available for Content-Transfer-Encoding:
- quoted-printable. This encodes high characters using some kind of trick with = and ? signs. The resulting text is relatively readable when there are not many special characters used, hence the name. The standard PHP function is imap_8bit(), but this requires the imap extension to be available. There are native PHP replacements, of course.
- base64. Using base64 the complete text is encoded using the base64 encoding function. The downside is that the encoded text is completely unreadable. The upside is that the related PHP function (base64_encode) is part of the core and available in every PHP installation.

The point is: UTF-8 using 8bit should work, but if you want to be on the safe side, use quoted-printable. This bug is very old, and I think it was posted even before Microsoft moved to the Live platform. On the current Hotmail/Live Mail version, UTF-8/8bit is no problem anymore. Also on Outlook 2003 there is no problem.

An interesting observation is that when you send special characters using Outlook 2003, it chooses to send it using UTF-8 and quoted-printable. Perhaps a wise thing to do is just choose the way of the biggest player in the market.

The last thing I would like to say is that this problem should not be fixed in simplenews, but in Drupal itself, if it can be considered a problem at all.

Just added this patch:
http://drupal.org/node/225731

And now HTML comes into hotmail, and exchange server / outlook 2007.

Some other issues have cropped up though, which we're still looking into.

ok, other issues were our own fault.. set to filtered html somehow. This patch looks good.

patch in #225731: Outlook 2203/2007 + filename solves this problem. the patch that worked for me was the one against 5.x-1.0 not dev, comment #2

I've just had this problem reported to me by a client -- or rather #175781: HTML mail sent as attachment to hotmail but that says it's caused by hotmail not handling UTF8.

What's the status of this issue?
Is the most recent thing the patch in #13? Which version was that against?

The patch is quite old (april 2007). Feel free to re-roll for the current head.

subscribing

I saw lots of solutions...but which one is currently more up to date? which one works fine for 6 version??

#12 didnt work for me...

the emails are been received in attatched files for hotmail.

[editing]

it works if I test only with a single email from hotmail...

i got the solution here

http://drupal.org/node/741984

At this point, this issue is going to be marked as won't fix.
Both Simplenews & Mimemail have had new versions since then, so please create a new issue if you still have a problem and the solution provided by #741984: Problem with HTML Newsletter is not sufficient.

function sn_mail_send($mail) {

Could someone please tell me where i could find this function? and if i use the patch at #6, will iso 8859 been corretly displayed in html mails?

You can't find it, that function doesn't exist. This issue has been closed more than a year ago. If you have issues, open a new one.