drupal_mail() mismatches the 'Sender' header and the 'From' header

Contact module doesn't set any From value to the site's e-mail address. That's done by core's drupal_mail() function. Not sure we can blame Drupal for also doing what it's supposed to do and mail clients not handling the headers properly.

Agreed. We are doing the right thing here. Masquerading the user in From: is not an option.

Sorry if I'm confused, but why is this a "won't fix" issue? (sorry if i've overstepped my bounds by refiling this as "needs review" and renaming the issue, not sure if that's something i should be doing.. )

I renamed this from "Some mail clients improperly use the 'Sender' header instead of the 'From' header" because the root of the problem is more specific.

I had this same problem with every drupal-sent mail that was supposed to be from an address different than the primary address of the site set up on the "site configuration" menu.

I understand if the contact module properly uses the drupal_mail function, but who would be able to fix the drupal_mail function if it's not putting in proper email headers?

I think the problem is..

that the Sender and From headers don't match, so some email clients read one, others another, others say "on behalf of", and it seems that some servers just don't even send it.

It seems to me that the issue is here in the drupal_mail function:
(referencing http://api.drupal.org/api/function/drupal_mail )

<?php
  if ($default_from) {
    // To prevent e-mail from looking like spam, the addresses in the Sender and
    // Return-Path headers should have a domain authorized to use the originating
    // SMTP server. Errors-To is redundant, but shouldn't hurt.
    $headers['From'] = $headers['Sender'] = $headers['Return-Path'] = $headers['Errors-To'] = $default_from;
  }
  if ($from) {
    $headers['From'] = $from;
  }
?>

So, if you set a site email, then there will be a mismatch in the email header if you designate a different $from than what is the site email. This is because only the $headers['From'] value gets reset if the $from parameter is detected, and the other header parameters remain the default.

The result email header then is then rendered as such:

Errors-To: [primary site email]@[your site]
Sender: [primary site email]@[your site]
From: [designated email]@[your site]

I'm not an email expert, but I think this is the root of the issue. If the end user's email contains more than the designated From address, it could get confusing if that user only needs to see [designated email]@[your site]

possible solution?

I don't know how to write a patch ( but i shouldn't if this is the core anyway, right? ), but a custom module that uses hook_mail_alter() could resolve the problem with this:

<?php
function hook_mail_alter(&$mailkey, &$to, &$subject, &$body, &$from, &$headers) {
  $headers['Sender'] = $headers['Return-Path'] = $headers['Errors-To'] = $headers['From'];
}
?>

(not tested, yet... )

this would ensure that the 4 header designations match, no matter what.

I suppose leaving it alone might be desirable, but as aaront reports above, the mismatch creates inconsistent results in varying email recipients.

"From" should be the user's email
"Sender" should be the site email

Nothing wrong here that I can see. This is still won't fix as far as I'm concerned.

I see why you have them different....
So a user can use his own user email on the form, and send it ( as outlook says ) "on behalf of" the site name.

But what if you want to use drupal_mail() with php and send a site default email with a specific from address that does not match the site default email, and should not have a mismatch of email headers?

In my case I'm using this:

<?php 
$email_from = "specifcname@mysite.com";
$mail = drupal_mail('user', 'register_admin_created', $account->mail, $language, $params, $email_from, TRUE);
?>

This is called in special user creation cases, and the site default email is used in other areas of the site via the contact module.

When I use the above php, i want the Sender to match the From address.

Is there a different function I should be using to accomplish that ( a designated from address with matching Sender, Errors-To, and Return-Path )?

I've attached a simple module that will force these headers to match on all site emails for anybody who might stumble on this...

Damien, your point is valid, but then again we are not the ones making the rules. When you have different sender and from field you will usally get this in gmail, and sometimes also just be blocked or go to spam. Matching the 2 could help this not happening.

This message may not have been sent by: mymail@gmail.com Learn more Report phishing

Also some email clients confuse the 2 and then I get replies (my site email) instead of sender of email.

The rules (which Damien's comment #5 refers to) are located at https://tools.ietf.org/html/rfc6854#section-2.1 (subject to replacement... as of this comment it's the latest RFC):

The "From:" field specifies the author(s) of the message,
that is, the mailbox(es) of the person(s) or system(s) responsible
for the writing of the message. The "Sender:" field specifies the
mailbox of the agent responsible for the actual transmission of the
message. For example, if a secretary were to send a message for
another person, the mailbox of the secretary would appear in the
"Sender:" field and the mailbox of the actual author would appear in
the "From:" field. If the originator of the message can be indicated
by a single mailbox and the author and transmitter are identical, the
"Sender:" field SHOULD NOT be used. Otherwise, both fields SHOULD
appear.

Note: The transmitter information is always present. The absence
of the "Sender:" field is sometimes mistakenly taken to mean that
the agent responsible for transmission of the message has not been
specified. This absence merely means that the transmitter is
identical to the author and is therefore not redundantly placed
into the "Sender:" field.

The originator fields also provide the information required when
replying to a message. When the "Reply-To:" field is present, it
indicates the address(es) to which the author of the message suggests
that replies be sent. In the absence of the "Reply-To:" field,
replies SHOULD by default be sent to the mailbox(es) specified in the
"From:" field unless otherwise specified by the person composing the
reply.

I guess Damien is saying that the Drupal Site Admin is analogous to the "secretary"/"transmitter" in the RFC's language. I disagree.

In the case of, say, a Drupal site that lets the interactive user type into a field and then that submission immediately fires off an email containing the field value, the interactive user can very reasonably be considered the "secretary"/"transmitter" since they are punching the keyboard and triggering the send. The Drupal Site Admin most certainly is not triggering the send in this case. The human being who triggers the send (in the case of a human, anyway, such as in a webform or contact form) is, per the RFC, the "secretary"/"transmitter".

Yes there are other ways to interpret the RFC, my opinions above are not the only way to interpret it. If the interactive user is submitting to a batch queue and a cronjob grabs it later and transmits it, then yeah, the transmitter is the machine and not the human, and you might say that the Drupal Site Admin is the best way to represent said machine. But that is not necessarily the case, and thus drupal_mail() should not be so opinionated.

By the way, it seems that Outlook has switched from rendering "[sender] on behalf of [from]" to now rendering "[from] sent by [sender]" so at least sender is taking a back seat. Still not ideal.

D6 is no longer supported.

Sorry about that, I didn't even notice when editing before. This behavior continues in D7 exactly as it did in D6, so moving this to the D7 queue makes sense.

In line 150 at http://cgit.drupalcode.org/drupal/tree/includes/mail.inc?h=7.x#n150 From and Sender are both set to Site Admin (which itself violates the RFC quoted in my previous comment; it says if they are the same address, do not set a Sender) and then in line 153 From is replaced if the optional argument is set.

It seems like the only way to override the value of Sender -- regardless of whether you want to unset it (to avoid seeing "sent by" / "on behalf of" in mail clients, which is the likely use case), or set it to anything else -- is to invoke hook_mail_alter() and then conditionally unset($message['headers']['Sender']) when instead drupal_mail() could get a new optional argument like $site_admin_sender = TRUE that, when explicitly FALSE, omits Sender.

To summarize: if both the content of the email and the act of "hitting send" are both explicitly coming from a human user who is looking at some kind of email composition screen such as a contact form, then that user is both the From and the Sender, and thus a Sender header should not be sent. If, on the other hand, it's not the human user's intention to literally to write an email and instead the email is transactional or similar, then it does make sense to list the Site Admin as the Sender. Why should drupal_mail() not handle both use cases?

I recently noticed that while some of my sites are afflicted with mismatched From and Sender headers, others are not (even without a custom hook_mail_alter() implementation). After reviewing the differences between the sites, I concluded that the ones that are not mismatched (i.e., the more desirable behavior for the particular use case this issue is solving for) all use the mimemail module. It's actually the default behavior of that module to set Sender to be equal to From in conditional but highly typical circumstances (see http://cgit.drupalcode.org/mimemail/tree/mimemail.inc#n65 where it has $headers['Sender'] = $from;).

I mention this for 2 reasons: the first being that if this isn't solved in core, anyone who doesn't feel like doing a super quick hook_mail_alter() could instead use that (albeit heavyweight) module; the second being that that module's default behavior establishes a little bit of a precedent that this behavior is desirable often enough to be the default.

At least if this does again go to "wontfix" we have somewhere to point people who need this. I have also contributed a simple module with the hook_mail_alter() one-liner for anyone who doesn't want all the bells and whistles of the mimemail module: No Sender.

#8

I guess Damien is saying that the Drupal Site Admin is analogous to the "secretary"/"transmitter" in the RFC's language. I disagree.

...a Drupal site that lets the interactive user type into a field and then that submission immediately fires off an email containing the field value, the interactive user can very reasonably be considered the "secretary"/"transmitter"...

That's a literal interpretation of the analogy but the specs outlined in the RFCs aren't referring to human interaction or human mailboxes.

The Sender header in this case is referring to the Drupal website, which is in fact the "transmitter" of the message.

However, I do believe Drupal's mail system could handle some improvement. First, some background. Apologies, this got a bit wordy.

Sending Email from a Website

There are generally 2 main ways email can be sent by a web site, with variations in implementations.

1. Email generated on the website is sent from the web server directly to the recipient's mail server.

This is the traditional method and can be implemented using a PHP library, local binary such as Sendmail or directly via socket communication.

2. Email from the website is sent via an external delivery service such as Sengrid or Mailgun.

Sending email via a reputable delivery service enables advanced tracking and it is less likely for emails to be flagged as spam. This method also tends to be more reliable.

This is the way I think all website email delivery is, or at least should be, heading. There are free tiers with most services which are usually generous enough for small sites.

These services often have a REST API for sending emails or they can be used via normal authenticated SMTP.

The problem of SPAM

Recipient mail servers like to know where emails have originated, on behalf of who and whether they are reputable or not.

Take a Webform which can receive submissions from the general public. When it is filled in and submitted, including General Public's email address, a copy is sent to an appropriate email address relating to the website.

Under option 1:

The "From" email header might be set to that of the person filling out the form, myemail@mycompany.com. If no "Sender" header is set or, worse, it is set to the same email address as the "From" header, the recipient's mail server wouldn't be able to verify the sending server as valid for the email address myemail@mycompany.com and flag the email as SPAM, as well as potentially blacklisting the website's server for sending SPAM.

In this case, the "Sender" header must much a domain for which the website's server is authorised somehow. Plain DNS, DKIM, SPF records etc. are all possible. This means the "Sender" header must contain an email with a domain authorised as relating to the website, for example the Drupal admin address. This is not the email address of the person filling out the form.

Under option 2:

The delivery service itself would be authorised to deliver email from the website's domain. The "Sender" header would still need to be set correctly as above.

Email headers, terminology

Setting some of the email headers the same I suspect was a quick way to get things up and running but not always appropriate.

While From, Sender and Reply-To headers are fairly well explained, one that still confuses many people is Return-Path.

In a nutshell, the Return-Path header is set by the recipient's mail server based on the envelope sender. This is the email address supplied to the remote server using the SMTP command MAIL FROM. For example, MAIL FROM: myemail@mycompany.com.

The Return-Path header should never be set directly by Drupal or any script which sends email. It's sometimes explained that the header can be set by specifying the "-f" option with Sendmail, but this is setting the envelope sender rather than Return-Path directly. Mentioning "Return-Path" at this stage leads to further confusion about what is actually happening.

The Return-Path address is where delivery error messages are sent.

Suggested changes

1. Remove all reference to "Return-Path" as a header and configuration setting
2. Better explain headers such as "Sender" and allow them to be directly configured where appropriate
3. Allow "envelope sender" to be set directly as an advanced option for those systems which allow it

RFCs

• Internet Message Format - Trace fields: https://tools.ietf.org/html/rfc5322#section-3.6.7
• Simple Mail Transfer Protocol - Trace information: https://tools.ietf.org/html/rfc5321#section-4.4

There seems to be some good discussion happening in another issue.

@imclean: wordy but good! Given all the confusion around this stuff (after all, most of us deal far more in HTTP land than SMTP land), the more correct information to overshadow misguided information, the better.

For sites that allow sending mail with arbitrary domain names in the From header, it's critical that such sites take measures to ensure anti-spam compatibility by way of something like a fixed-domain Sender header exactly as you say. In my particular case, I've got a site where every user's email address shares the same domain name (being a university), and when mail is generated by our site the From header is set to $user['mail'] (not arbitrary form input), so spam filter false-positives are not a concern. I realize this may very well be a minority and defaults should be appropriate for the majority, which is why I built No Sender.

That said, I do believe the RFC is referring to humans when the mail is composed and triggered in realtime by a human, which is how my site works (my site is essentially an alternative to composing in Outlook, offering templates and things so that our university memos look nice, and why would the RFC care about whether the composition occurs in a desktop client or a web app?). In *other* cases, like a new user welcome email, a webform thank-you email, etc. then I agree 100% that the Sender value should represent the site. But the point is, a core function should not be so opinionated as to make the former difficult.

Yes, I'd say yours is a special case as you're hosting a webmail service for a specific domain.

That said, I do believe the RFC is referring to humans when the mail is composed and triggered in realtime by a human...

If the mail server which you're sending through is authorised to deliver email from the domain in the From header then you don't need the Sender header at all. This is no different to using an email client to send via an authorised SMTP server.

In your case, the envelope sender (MAIL FROM command) would be the same as the From header.

This is why I believe everything should be configurable. There should also be accurate descriptions of each header and command with sensible defaults.

@mforbes, existing modules weren't suiting our needs so I created PHPMailer SMTP. It allows some configuration of how the Envelope Sender is set, which may be useful in your specific case. One option is to use the "From" address.