8bit encoding for base64 encoded message body

What you see is the Base64 encoded message body and it should be fine. You should check with other mail clients.

As I mentioned the Base64 encoding was added by #321026: HTML messages are broken by line breaks.

I just tested it again and it works fine. I guess the cause of the error is something else. You can see a list of changes in the release note.

Already did.

If you revert some changes, you may get rid of the error but that won't fix the issue. Base64 is a common standard and should not be a problem. I am guessing that bug can be related to the base64 encoding but something else breaks your messages. Are you using any other contributed module related to sending messages?

The problem is that SMTP uses 8bit encoding in the Content-Transfer-Encoding header for the base64 encoded body.

With default Mime Mail engine:

--bffdb4b41f7ec8ec05b089d32a12ae01
Content-Type: text/html; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: base64

With SMTP module:

--b1_9e0dd0fa3f4c5693e7c3c6ac727b14ec
Content-Type: text/html; charset = "utf-8"
Content-Transfer-Encoding: 8Bit

I am moving the issue to the right place. In the meantime you can try to revert the patch in #321026: HTML messages are broken by line breaks or try the PHPMailer module.

Hi,

I had the same problem (using Gmail's SMTP), and tracked it down to adding:
$mail->Body = base64_decode($mail->Body);
.. just before "Set the authentication settings" (line ~768 in my version), in smtp.module.

Works like a charm on my test case, don't know if it breaks other stuff ...

edit /
Better inserting the following, to detect base64 (because in some cases it's not 64-encoded ..):

$base64 = false;
$body_parts = _smtp_boundary_split($mail->Body, $boundary);
foreach ($body_parts as $body_part)
$base64 = $base64 || drupal_strtolower(_smtp_get_substring($body_part, 'Content-Transfer-Encoding', ' ', "\n")) == 'base64';
if ($base64)
$mail->Body = base64_decode($mail->Body);

I encountered the same problem, and it started as soon as I upgraded Mimemail which included the patch from #321026: HTML messages are broken by line breaks. Rather than rolling back the patch, I stepped through the process and came up with a (possible) fix for SMTP module.

It looks as though under the right circumstances, a base64 encoded text/html body makes it to $body_part2 at line 681. I grabbed the decode_base64() attachment logic from lines 740-742 and applied it to $body_part2. It does appear to fix my specific issue, although I can't vouch for whether this breaks anything else.

I'm attaching the patch file to start the discussion, and I'm open to any/all feedback. Thanks!

I don't have a test environment for this, so if anyone could review it I'd really appreciate.

I have the same problem on the 7.x-1.x branch.

Attached you can find an almost straight forward port of #14 that fixed the issue for me. The only addition I did is forcing base64 encoding for the whole message (since it seems PHP Mailer does not support specifying a different format for alt and body), to ensure that base64 encoding is preserved, which is needed for #321026: HTML messages are broken by line breaks to work.

I won't RTBC the patch since I wrote it myself but I'm pretty confident it's ok. I didn't test the 6.x version, but since the port is good I'd say the original is too, aside from the encoding issue described above.

To reproduce the issue:

1. Install Drupal 7.20, Mail System 7.x-2.34, Mime Mail 7.x-1.0-alpha2, Simplenews 7.x-1.0.
2. Configure a new mail system class using Mime Mail as formatter and Smtp as mailer and assign it to the Simplenews module.
3. Configure a newsletter category to use HTML format.
4. Create a test newsletter with a single-lined body exceeding 1000 characters (Dummy Lipsum is very good at that ;).
5. Send a test newsletter.

Expected result: a nice HTML mail is showed.
Actual result: an ugly base64-encoded text is showed.

#16 patch worked great for me. Solved the problem perfectly.

#1964218: Mail body corrupted in HTML format is marked as a duplicate of this.

Experienced the same issue with a combination of MIME Mail and SMTP modules. Patch from #16 should be committed, because it fixes the problem.

I can confirm that the patch in #1741082-16: 8bit encoding for base64 encoded message body fixes this issue. Tens of thousands of successful emails later I believe it is ready to be committed as well.

I can do additional testing if needed.

In our case it appeared broken again after patching. However, I'm not sure what was the actual reason.

We've re-wrote sending class from scratch without PHPMailer at all by just using the message + headers composed by MIME Mail module and low-level Smtp class from this module. Because with MIME Mail all the parsing of the multi-part messages to build it again is not the way it should be. Especially because the result is not really matching the original message and parsing is done in a non-safe ways like matching substrings in a whole text (with both headers & body). We could probably share the code if someone is interested. It could be a patch to MIME Mail or SMTP, or even as a separate module.

#16 Patch worked for us too! Thanks!

Committed b9d1d33 to 7.x
Can someone confirm the 6.x works well too?

The patch in comment #16 is for 7.x
Here is the patch I backported for 6.x

16: smtp-base64-1741082-16.patch queued for re-testing.

I can confirm that the described problem still exists with the latest versions of this module.
I strongly recommend to use "phpmailer.module" for Drupal 7. After lots of WTF's I decided to change from SMTP to PHPMailer and everything worked fine ad hoc.

I added base64_decode checking if the body is in base64.

Patch from #16 worked for me after some changes, replaced

$body_part2_encoding = $this->_get_substring($body_part2, 'Content-Transfer-Encoding', ' ', "\n");
with
$body_part2_encoding = $this->_get_substring($body_part2, 'Content-Transfer-Encoding', ':', "\n");

smtp 7.x-1.0, mimemail 7.x-1.0-alpha3, D 7.28

another solution is to revert patch #42 from mimemail issue https://drupal.org/node/321026 since it's safe to assume phpmailer will deal with line breaks and as a bonus your server will not have to base64_encode in mimemail, base64_decode in smtp, then again base64_encode in phpmailer

#30 worked for me (#16 did not).

Emails were sent as base64 if it contained a really long line, but this patch + tweak fixed the problem.

Rolled comment #30 as a patch. I don't know why changing that space to a colon fixes the issue, but it worked for us. We were able to still reproduce the problem following roughly the same steps as comment #17, except with 7.x-1.0-beta3 of mimemail and latest git of smtp. Simplenews is not needed to reproduce the issue, just a very long e-mail body.

#32 worked like a charm for me! Thanks a lot to everybody for your work in this issue!

Based on #33 and #31

Adding my support to these changes, fixed my issue. Just added a unified patch.

While this has been tested by the community a maintainer hasn't actually reviewed this as yet and committed the changes. Due to this I am changing the status to Needs Review as they still need to commit these changes to the repo.

I've tested and am using the patch, hence the RTBC of #32. The maintainer can knock it back if there is something wrong or anybody for that matter.
This may help? https://www.drupal.org/node/156119

What do you mean "unified patch"? #32 applies. Are you adding more things that are need into the patch? My only guess why that fails is it may be made off of 7.x-1.0 instead of 7.x-1.x-dev which I believe already has what you are trying to add hence the patch not applying. Just a guess from a quick look though, let us know:)

Thanks guys. This fixed the problem for me as well. Like @joelpittet says, #32 applies to 1.x dev and is the one that should be applied. If you're running 1.0, the patch in #36 includes #25 and #32 together.

Could someone please re-roll the patch against latest dev?

It still applies it seems.

[ develop ] $ patch -p1 < smtp-base64-1741082-30.patch
patching file smtp.mail.inc
Hunk #1 succeeded at 387 (offset 17 lines).

git pull? :)

FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch smtp_0.patch. Unable to apply patch. See the log in the details link for more information.

(Also, I tried in my local and it didn't work)

Here's a clean roll of the patch in #32 against dev if that helps any.

Ohh, my bad, I was rolling #36 not #32 ;(

Thanks again @wundo

7.x-1.2 was released, which includes this fix...

Not working !
The issue is still here in 7.x-1.2 as in 7.x-1.x-dev
(see https://www.drupal.org/node/2550151)

Hey there,

I just wanted to reach out and let you all know that on upgrading to the beta4 version, I am getting the same encoding errors as has been mentioned, whereas in the beta3 version, this is working as intended.

I will try to take some time and dig into it, but for now, we will be reverting to the beta3.

Just wanted to give you guys a heads up as I'm sure I won't be the only person experiencing this issue.

Thanks

Hey all,

Updating the status and adding more information. This issue with beta4 is involved around the following contrib modules. We're using:

Mimemailer
SMTP
Simplenews
Workflows

After some sleuthing, interestingly enough, using the Mimemail and SMTP together does not result in this issue when sending a test email from the SMTP module, or when sending out notifications via Workflows email system. But when I try to send a Simplenews Newsletter, OR use the Contact form (default module), I receive the garbled encoding.

Like I said, I'll try and poke at it later, but I'm just trying to bring this to the attention of the group. Sorry for any inconvenience.

Thanks

PS - We are not using Mandril

@kungfu4 please try the dev version and report.

Same here. After upgrading to beta4 I sometimes (my guess is, when I have long lines in html body of email)
receive garbled base64 encoded emails. Everything was fine with beta3.

Same here only sometimes.

I have been having this same problem. After some debugging I have found that the header value had a leading space that I had to trim out. I don't have time to debug futher to see where it's coming from but I simply changed:
if (drupal_strtolower($body_part2_encoding) == 'base64') {
to
if (trim(drupal_strtolower($body_part2_encoding)) == 'base64') {
in smtp.mail.inc and they are sending good for me now.

#62 has solved my problem for now thanks mrconnerton

#62 solved the issue for me as well

That's cool that you have a solution to the problem @mrconnerton. Though it would be really nice to know how trimming the encoding would fix this. Why would the encoding have whitespace?

Like I said I haven't had time to debug the "why". I'm sure a simple debug_backtrace() would do the trick.

ok @mrconnerton, here's a backtrace patch that everybody can use to trackdown what's up in their logs.

I'm getting this sporadically as well, so hopefully collectively we can resolve this.

same problem here, randomly mails are base64 encoded

applied patch #67, will report the results if any

#62 has solved my problem. Kudos mrconnerton

The white space can easily come from an earlier line depending on what is setting the encoding. This line actually needs a trim() around it as shown below (line 388 in the version I am using). I can submit a patch if you need it:

388: $body_part2_encoding = trim($this->_get_substring($body_part2, 'Content-Transfer-Encoding', ':', "\n"));

Patched.

The patch in #71 worked like a charm. Thank you very much @Dave the Brave!

Dan

Patch #71 works for me. Saved me tons of debugging… Please commit

Committed!