When sending a newsletter with a from name that includes German umlauts (ö, ä, ü, ß, ...), the sender name is MIME-encoded twice.

The problem is this line in simplenews.mail.inc:

// Windows based PHP systems don't accept formatted emails.
$formatted_address = drupal_substr(PHP_OS, 0, 3) == 'WIN' ? $address : '"' . mime_header_encode($name) . '" <' . $address . '>';

Sender name is MIME-encoded here once and then passed to the Mime Mail module later on, if you send HTML emails with Simplenews. Mime Mail module, however, also MIME-encodes the from name, so it gets encoded twice. The result is a broken from name in newsletters.

For me, removing the call of mime_header_encode() solved the problem. But I don't know if this still works, if you use Simplenews without Mime Mail module.

Also see http://drupal.org/node/1209018 for this problem.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Berdir’s picture

Version: 6.x-2.0-alpha2 » 7.x-1.x-dev

#1209018: From: in headers not properly encoded confirms that core does encode headers, so we are doing it wrong.

Moving to 7.x-1.x to get fixed there first.

Berdir’s picture

Status: Active » Needs review
FileSize
2.58 KB

Ok, attached is a patch that removes the mime encoding and also adds a simple test for this that simply makes sure that the from address isn't encoded.

Berdir’s picture

Version: 7.x-1.x-dev » 6.x-2.x-dev
Status: Needs review » Patch (to be ported)

Commited, needs to be backported to 6.x-2.x

Simon Georges’s picture

Status: Patch (to be ported) » Fixed

Fixed on 6.x-2.x too. Thanks!

caktux’s picture

Still seeing this issue using Mime Mail, which seems to encode the whole sender instead of just the name...

Simon Georges’s picture

Are you using the last 6.x-2.x version?

Berdir’s picture

Yes mimemail is encoding the whole header string.

There is nothing we can do, we actually don't do any encoding anymore at all.

You need to report that to mimemail.

vitovt’s picture

Version: 6.x-2.x-dev » 6.x-2.0-alpha3
Status: Fixed » Active

I have the same problems with simplenews -alpha3
Sender: =?UTF-8?B?ItCd0LDRgNC+0LTQvdC40Lkg0J7Qs9C70Y/QtNCw0YciIDxuZXdzQHNkLm9yZy4=?=
=?UTF-8?B?dWE+?=
From: =?UTF-8?B?ItCd0LDRgNC+0LTQvdC40Lkg0J7Qs9C70Y/QtNCw0YciIDxuZXdzQHNkLm9yZy4=?=
=?UTF-8?B?dWE+?=

It happens only when I use non-ascii symbols in
/admin/content/simplenews/types/edit/2 --> Sender information --> From name:
When I write there only ASCII - it is ok.

I've tried disable mimamail - it didn't help. Only Sender field became OK, but From still has problem.

lucuhb’s picture

I got this problem too when a non ASCII character was in the sender of a newsletter.

The problem seems to be solved when I put, in the from name of the sender information, the name directly encoded in base 64 , added with the part =?UTF-8?B? before and ?= after this encoded words.

For exemple, if the sender of the email that should appear is "Université", that is =?UTF-8?B?VW5pdmVyc2l0w6k=?= what must indicate in the settings of the newsletter.

Berdir’s picture

What module are you using the send mails?

Based on what we have seen do Drupal core and Mime Mail automatically mime encode all headers. You are *not* supposed to have to do that yourself.

We need to follow the model that is given by core to prevent double encoding, if another module is not doing the encoding then that is IMHO a bug in that module and you might want to report that.

lucuhb’s picture

I use mimemail with simplenews, and of course it is not a real solution I give in my post.

In fact the problem I try to solve is that our postfix server don't recognize the email address and adds @localdomain.local after the sender informations.

It seems that it is because in Sender and From of the headers, not only the name of the sender is encoded, but also the email address is encoded, and so not understandable.

Perhaps it was possible to encode only the name of the sender but not the email address ?

I made a test to compare with webform module, with a same sender (name = Université and email address toto@titi.fr".
When a email was send when I use the webform module, I have :
[Sender] => "=?UTF-8?B?VW5pdmVyc2l0w6k=?=" <toto@titi.fr>\n [From] => "=?UTF-8?B?VW5pdmVyc2l0w6k=?=" <toto@titi.fr>\n
in headers, and no problem with postfix.

But with simplenews, I have something like :
[Sender] => =?UTF-8?B?IlVuaXZlcnNpdMOpIiA8dG90b0B0aXRpLmZyPg==?=\n [From] => =?UTF-8?B?IlVuaXZlcnNpdMOpIiA8dG90b0B0aXRpLmZyPg==?=\n
Here you see that the words Université <toto@titi.fr> are all encoded.

Berdir’s picture

Yes, that is in fact what we did before. The problem is that both core and mime mail unconditionally encode all headers so we had to remove that to prevent the mentioned double encoding. See http://api.drupal.org/api/drupal/modules!system!system.mail.inc/function...

I am not sure what the best solution is but it's not something that we can fix in this module. I think we need to fix core and mimemail to treat the From and similar headers like Return-Path differently and don't encode everything. Or at least prevent double-encoding so we can encode it again ourself.

Maybe you can check if there are existing related issues in Mimemail/Core and link them here so that we can further discuss this in these issues.

Berdir’s picture

Looking at http://api.drupalhelp.net/api/mimemail/mimemail.inc/function/mimemail_ad..., that's maybe not actually the case for mime mail, it seems to support partial encoding but we need to pass in an array. I will need to discuss this with the maintainer of that module.

lucuhb’s picture

Here is a patch that solves my problem.
This patch seems correct?

Berdir’s picture

The problem is that Drupal core and probably all other mail send modules expect a string and not an array. So while that works for Mime Mail, it will fail pretty hard without it :)

So one way would be to add some module_exists() checks and pass different data in but especially in Drupal 7, you don't know which mail system will actually be used, you might also have something like mail log installed to prevent mails from being sent on development sites, which would then fail again.

Something that might work is if we could set the name/address array keys within $params and mime_mail would check there first. But that's something that I need to discuss with the mime mail maintainer.

lucuhb’s picture

OK.
So it is perhaps better that I add, as this was before, the call to mime_header_encode in the _simplenews_set_from function (see #1209018: From: in headers not properly encoded ).
It is strange that the call to mime_header_encode function had to be removed since this function only encode strings that contain non-ASCII characters, so a from already encoded should not be re-encoded, no?

mducharme’s picture

Status: Active » Needs review

#2: no_mime_encoding.patch queued for re-testing.

Status: Needs review » Needs work

The last submitted patch, simplenews.mail_.inc_from_sender.patch, failed testing.

anrikun’s picture

Version: 6.x-2.0-alpha3 » 7.x-1.x-dev

Let's fix this in 7.x first.

anrikun’s picture

Status: Needs work » Needs review
FileSize
912 bytes

This patch fixed the issue for me.
And it seems to work whether Mime Mail is used or not.

The last submitted patch, fix_formatted_fromaddress-1387648-20.patch, failed testing.

anrikun’s picture

Status: Needs review » Needs work

This test fails:

<?php
      // The mail system is responsible for mime encoding, make sure the from
      // address is not yet encoded.
      $this->assertEqual('"' . $edit_category['from_name'] . '" <' . $edit_category['from_address'] . '>', $mail['from']);
?>

But is this itseft correct?
System encodes headers but here, we need to encode the name itself, not the '" <' . $edit_category['from_address'] . '>' part.
After this, any further call to mime_header_encode will ignore this header because there is no non-ASCII char left.
I guess the test itself is wrong here.

anrikun’s picture

Here is an updated patch that fixes test too.

anrikun’s picture

Status: Needs work » Needs review
Berdir’s picture

Not necessarly. The test was added as part of the removal of mime_encode_header() and might be wrong.

We had issues with double encoded from names, are you sure that this doesn't happen with your patch?

It's not trivial to test, not sure what the best way is. What we could try is to call mime_encode_header() again in the test and make sure it's not encoded twice or something like that.

anrikun’s picture

Ok let's try this (added an extra test to ensure there won't be twice encoding, as you suggested).

Berdir’s picture

Status: Needs review » Needs work
+++ b/tests/simplenews.testundefined
@@ -2613,9 +2613,11 @@ class SimplenewsSourceTestCase extends SimplenewsTestCase {
+      // And make sure it won't get encoded twice.
+      $this->assertEqual(mime_header_encode($from), $mail['from']);

The result is the same, but logically, it would make more sense to me to the encoding on $mail['from'] and then compare it with what we expect it to be instead of the other way round.

Have you tested this with Mimemail? If it works correctly there as well, then I guess I messed up :)

anrikun’s picture

Status: Needs work » Needs review
FileSize
1.97 KB

@Berdir: thank you for your feedback.
Yes you're right, even if the result is the same, it's more logical to do it the second way.
Here is an updated patch.

Yes, I confirm it works as expected with Mime Mail: I use it on my site!

Berdir’s picture

Version: 7.x-1.x-dev » 6.x-2.x-dev
Status: Needs review » Patch (to be ported)

Ok, great, thanks, commited and pushed. Back to 6.x-2.x

anrikun’s picture

Version: 6.x-2.x-dev » 7.x-1.x-dev
Status: Patch (to be ported) » Needs review
FileSize
802 bytes

I have just found another place where From was not properly encoded.
Here is a patch to fix it too (tested live).

andrenoronha’s picture

I dont know if here is the right place for this, but the email sent by simplenews shows the sender name as

"\"Name of my site\"

How to get rid of those "\" ?

I'm using Simplenews 7.x-1.0-beta2 after update from drupal 6.
Is this corrected in the last dev version from may 5?

Thanks

Berdir’s picture

Version: 7.x-1.x-dev » 6.x-2.x-dev
Status: Needs review » Patch (to be ported)

Thanks, commited.

andrenoronha’s picture

I noticed that my problem in #31 was due to the 'ú' caracter of my site's name.
I tried '&uacute' but it didn't work.
What code could I use to print an 'ú' in the 'From' field of the email?

anrikun’s picture

Status: Patch (to be ported) » Needs review
Issue tags: +Needs tests
FileSize
746 bytes

Here is the patch for 6.x-2.x

perk11’s picture

This patch worked for me, please commit it to the release

Berdir’s picture

Status: Needs review » Fixed

Commited to 6.x-2.x and 6.x-1.x

Automatically closed -- issue fixed for 2 weeks with no activity.

GaëlG’s picture

Issue summary: View changes

If these patches are not enough to solve the bug, try using this one too (both at the same time): https://www.drupal.org/node/300387#comment-985929