The encoding of the subject includes a call to drupal_html_to_text. This function is probably not necessary for the subject line since any HTML in the subject line would not be rendered (that I know of). Although it shouldn't cause a problem with the encoding, drupal_html_to_text occasionally inserts a linefeed. This corrupts the subject line and causes the call to the PHP mail function to fail.

Attached is a patch to eliminate the call to drupal_html_to_text for the subject only.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mhdg’s picture

Using Notifications module with mime mail, SpamAssasin was telling me that : "1.5 SUBJECT_ENCODED_TWICE Subject: MIME encoded twice"

Subject: =?UTF-8?B?dGVzdC5pbnRlcmZpbmFuY2Uub3JnIHN1YnNjcmlwdGlvbiB1cGRhdGUgZm9yIG0=?= =?UTF-8?B?aWNoZWxAaGRnLmJlCg==?=

After patching:

function mime_header_encode($string) {
  return $string; // added for test
  if (preg_match('/[^\x20-\x7E]/', $string)) {
    $chunk_size = 47; // floor((75 - strlen("=?UTF-8?B??=")) * 0.75);
    $len = strlen($string);
    $output = '';
    while ($len > 0) {
      $chunk = drupal_truncate_bytes($string, $chunk_size);
      $output .= ' =?UTF-8?B?'. base64_encode($chunk) ."?=\n";
      $c = strlen($chunk);
      $string = substr($string, $c);
      $len -= $c;
    }
    return trim($output);
  }
  return $string;
}

I got:
Subject: =?UTF-8?B?Tm90aWZpY2F0aW9ucyBkaWdlc3QK?=

Then I revert the change in mime_header_encode and applying the patch (eliminate the call to drupal_html_to_text for the subject only)

I got a correct result:
Subject: Notifications digest

Conclusion: correct solution, no false spam anymore.

Michel

a_c_m’s picture

We've noticed this too. We are using phpmailer as well, and outlook users are seeing a square char at the end of each subject line.

Will test this and report back.

derjochenmeyer’s picture

FileSize
3.45 KB

Patch works. After patching the subject looks good in outlook.

Peter Bex’s picture

+1 on this bugfix. It removes the weird little square.

Actually, some versions of Outlook behave even worse; they think that the email body starts after that linefeed. So when the user opens his mail, all he sees is the raw message contents, starting with the header following "Subject:".
I see similar behaviour in mutt, only less extreme; it shows one or two headers (To and Date) as text contents of the mail, but it still shows the different mime parts as different attachments.

I didn't get the error described by the OP, though. Perhaps that depends on the particular MTA used?

It would be great if this patch could be integrated in the next release of MimeMail.

muhleder’s picture

+1 on this, we're seeing this issue too.

mrfelton’s picture

+ 1 from me too. Thanks

Damien Tournoud’s picture

Status: Active » Fixed

This has been fixed by http://drupal.org/cvs?commit=214382 a while ago.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

new_B’s picture

Version: 6.x-1.x-dev » 6.x-1.0-alpha1

Actually adding trim() did not fix the problem. The original solution above, removing html_to_text appears to work.

new_B’s picture

Status: Closed (fixed) » Active
new_B’s picture

In my case, the weird little square shows up near the end (but not at the end) of long subject lines in Outlook 2007. No square appears for shorter ones.

bjcool’s picture

Subscribe

christefano’s picture

Did commit #214382 get into 6.x-1.0-alpha2? I don't see any use of trim() in mimemail.module anymore and just saw this problem reoccur.

Whoops, I was looking at the wrong version. Never mind.

sgabe’s picture

Version: 6.x-1.0-alpha1 » 7.x-1.x-dev
DanChadwick’s picture

I am still having trouble with 1.0-alpha5. If the subject line is long enough, the call to drupal_html_to_text insert a linefeed ("\n"). This triggers the mime encoding. GMail receives this message fine, but Windows Live Mail (aka Outlook Express) doesn't even show the message anywhere.

It isn't clear to me what filtering the subject needs, and certainly drupal_html_to_text would do horrible things to the subject if it contained, say, h1 tags. I wonder if it wouldn't be best to skip the call altogether. Absent that, perhaps removing the linefeeds from the result, before passing to trim and on to mine_header_encode would be appropriate.

I am referring to mime_mail_prepare_message, line 127,
$subject = mime_header_encode(trim(drupal_html_to_text($subject)));

It would be super if this could get committed. It would seem to be a fairly serious problem for anyone using SimpleNews to send newsletters (which tend to have long subjects, since they are in the format "[Newsletter Name] Newsletter Node Title". It is hard to make a meaningful title if your newsletter name is long-ish.

sgabe’s picture

I couldn't reproduce this so far with 255 character, which is the maximum length of the title field of nodes BTW. I tried Thunderbird 3.1.3, Outlook 2007, Windows Mail. Please, provide more details how to reproduce.

DanChadwick’s picture

sgabe - Are you saying that you didn't get a linefeed in the subject after it came back from drupal_html_to_text, or that the linefeed (after mime encoding) didn't cause a problem.

I am using Windows 7, Windows Live Mail version 2009 build 14.0.8089.0726 fetching the mail from gmail via IMAP. I also reproduced it on Vista with Windows Mail version 6.0.6000.16386. I tried disabling all the anti-spam, anti-phishing, and other security features, and I added the domain to the approved white list. No luck. The mail does not end up in the "junk" folder -- it just appears nowhere. I tested with Macintosh Mail and it worked fine.

Please e-mail your e-mail address to me at dan a t kindredcocktails d o t com. I will then send you a test message directly from the site.

I'm not sure it's even necessary for you to reproduce it. Surely a linefeed in the subject is a bad thing. Even if your particular mail client doesn't complain, others might (and apparently do). And the current Microsoft mail clients are obviously extremely popular.

Thanks for your help.

sgabe’s picture

@DanChadwick: I still couldn't reproduce it. Your test message arrives fine and looks correct. I even installed Windows Mail 14.0.8089.0726 to check it out.

BTW there is a similar issue #869652: Wordwrap in From header value causing not sending email about the line feeds. As I wrote there according to RFC 2822 folding should be okay. See the section about the long header fields.

Subjects in multiple lines like yours should not be a problem:

Subject: =?UTF-8?B?W0tpbmRyZWQgQ29ja3RhaWxzIG5ld3NsZXR0ZXJdIE5yIDEsIFNlcHQgMjAxMC4=?=
    =?UTF-8?B?IFN1bW1lciBjb2NrdGFpbHMgYW5kIHdlYnNpdGUgIApnZWVrZXJ5?=

The subject of my test message looks like this:

Subject: =?UTF-8?B?KlBoYXNlbGx1cyBpbiBsZWN0dXMgc2l0IGFtZXQgcmlzdXMgdml2ZXJyYSB2YXI=?=
    =?UTF-8?B?aXVzIGEgdmVsIG1pLiogRG9uZWMgbHVjdHVzICAKZWxpdCBlZ2V0IC9sb3JlbSA=?=
    =?UTF-8?B?cGhhcmV0cmEvIHBvcnRhLiBOYW0gc3VzY2lwaXQgcmlzdXMgc2VkIGxhY3VzIHA=?=
    =?UTF-8?B?aGFyZXRyYSBpZCAgCnVsbGFtY29ycGVyIHZlbGl0IGNvbnNlcXVhdC4gTnVsbGE=?=
    =?UTF-8?B?bSBsb2JvcnRpcyB0aW5jaWR1bnQgdml2ZXJyYS4gQ3JhcyBsZW8gIAp0b3J0b3I=?=
    =?UTF-8?B?LCBpbnRlcmR1bSB2aXRhZSBhZGlwaXNjaW5nIGVnZXQsIHVsdHJpY2llcyBhdCA=?=
    =?UTF-8?B?aXBzdW0uIFNlZCBhYyBqdXN0byAgCnBvc3VlcmUu?=

and it shows up just fine.

However I tested with SMTP and PhpMailer modules on Outlook 2007, Windows Mail, Thunderbird. The test message with ultra long subject shows up fine in every case. The "weird little square" has been fixed with a trim(). I can't see any other issue here. Are you sure that the drupal_html_to_text() call is responsible for your issues?

DanChadwick’s picture

I am using PHPMailer. It may be related to the IMAP from gmail.

I commented out the offending drupal_html_to_text and it goes through fine, so it's definitely causing the problem. It also goes through fine with short subjects that aren't long enough to make drupal_html_to_text wordwrap. The linefeeds in your quoted example are just wordwrapping of the encoded MIME, no? I believe buried within that mime is a linefeed character, which is probably causing the problem.

If the problem is IMAP from gmail, then that too is probably a big enough problem that it warrants a workaround. I don't see the advantage of running it through drupal_html_to_text anyhow. mime_header_encode will break it into lines that are short enough.

I'm not sure what to say about the spec. It says you can break subject lines when they are in plain text. I'm not sure what it says about encoding linefeeds. It could also be that it doesn't like (decoded) linefeeds, rather than CRLF's. Since the mime encoding already breaks it into lines, perhaps some part of the IMAP delivery isn't expected to decode it and still have linefeeds.

Out, out, damn linefeed.

sgabe’s picture

FileSize
770 bytes

The drupal_html_to_text() call was added by #369983: Plain text Email subject shows & as _amp; and " as _quot; , this prevents characters like & showing up in amp; format, so we can't just remove it. Another approach suggested by AlexisWilke in #850694: Subject line includes a newline character (or more) is to remove the line feeds with an str_replace() call. See the attached patch.

sgabe’s picture

Status: Active » Needs review

Changing status.

DanChadwick’s picture

Downloaded HEAD and applied patch. Works correctly for me. Thank you.

sgabe’s picture

If some people can confirm that patch as a fix, it could get into the next release.

DanChadwick’s picture

Any chance of getting this committed, since I tested it?

haroldb’s picture

sgabe’s picture

@haroldb: Previously you posted that the patch causes WSOD, later you deleted your post. Does this mean that the patch is good after all?

haroldb’s picture

@sgabe: Yes, the patch is good.

DanChadwick’s picture

Hoping for a commit....

sgabe’s picture

Title: drupal_html_to_text inserts LF in subject causing outgoing e-mail to fail » Remove line feeds in subject
Status: Needs review » Reviewed & tested by the community
sgabe’s picture

Status: Reviewed & tested by the community » Fixed

Committed to HEAD.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.