Set fixed "from:" and add "Reply-to:" to comply with DMARC

Drupal 5 has been released, so new features go to Drupal 6. That said, it sounds like a reasonable feature.

It doesn't matter how many lines of code it is. A stable release is feature-frozen by definition. Drupal 5 will only be accepting bug fixes from now on.

This is a sensible feature request.

Were duplicates:

#195298: From addresses showing Recipients addresses in e-mails sent to contact requesters.
#350062: Site wide contact wrong from address

Also please keep in mind this recent change in the D7.x branch: #330084: Setting Reply-To in mails to the same as From is redundant

Feature requests are closed in D7 now.

#611176: Big spam block issue. From address is user's input address. appears to be a duplicate of this as well.

I am new to drupal and am having the problems with gmail blocking contact form emails due to the from address spoofing, etc. Is there any update on this? Is there a patch I can apply to fix the from address and make the reply-to address be the users address? I thought this was a patch, but I don't see a patch file. Does the comment related to D7 above mean that this will be fixed in Drupal 7? Any suggestions for a newb would be great.

Feature requests go only against the development version, and Drupal 7 is in feature freeze. This may be fixed in Drupal 8. Please do not change the version tag on an issue back to a stable release.

This could be justified for Drupal 7 if it could be argued that it's security related, but otherwise it's like so many issues that just never made it onto anyone's radar for this release.

Oceria: "none of the developers" implies that there's some small cadre of people who are the only ones that touch code. :-) If it's a problem you're having, patches are welcome from anyone. If you don't know how to write the patch, find a friend who does.

None of the shared hosts I've been on have had an issue here, so I couldn't really test it either way.

[Edit: Wow that was bad grammar. Fixed.]

Moving to the mail system component, as this can only be addressed in drupal_mail() itself, not contact.module.

It actually wouldn't be that hard to write a small module that implements hook_mail_alter() and adjusts the mail headers itself.

Thank you for posting about this issue.

I host my site at Bluehost.com. They have a very strict mailing policy which effectively breaks Drupal's e-mail functionality. Bluehost's policy dictates the following:

1. You MUST use bluehost's servers to send e-mail. SMTP modules for Drupal which attempt to use an external SMTP server (like Gmail) are blocked by Bluehost.

2. E-mail addresses in the "FROM" field MUST be populated with an address that is configured as a valid e-mail address in your Bluehost account. You can only add e-mail addresses to your Bluehost account with domain names that you have legitimate control of. (For instance, I could not configure a Yahoo e-mail address as an address on my Bluehost account.)

Here's how item #2 breaks Drupal: If you have a user attempting to send another user a message using the contact form, Drupal will populate the "FROM" field with the originating user's e-mail address. When this happens, Bluehost will send the message, but it will overwrite the e-mail address in the "FROM" field with "your-bluehost-login-name@yourbluehost-box-number.bluehost.com." (i.e. jsmith@box341.bluehost.com) This address will make no sense to the recipient and if replied to, will not reach the sender of the message!

What is desperately needed, is the ability to populate the "FROM" field with an "admin" or "no-reply" account, which the Bluehost user can configure in their account. THEN, add a "REPLY-TO" field which is populated with the sender's e-mail address. (Bluehost doesn't care what you put in the reply-to field.) That way, the "FROM" field is representative of the domain running the Drupal site, and the recipient can also reply to the sender.

I would love to write patch for this, but I lack the necessary knowledge of how to do this at the moment. While far from being an expert, I have done a fair amount of PHP programming, so perhaps I will be able to figure it out. I don't really have a choice. Either I figure it out, or I don't launch the site. I'll report back here if I am successful.

I finally figured it out. The drupal module called "PHPMailer" effectively solves this issue. If you're hosted on Bluehost, here's what you need to do. Some of the steps are applicable to other hosting providers.

1. Host your mail somewhere other than bluehost. (i.e. Google apps.)

2. Go to mx_entry button in cpanel. Make sure that you remove the local mx record already there. Set the radio button to "Remote" rather than local.

3. Go to E-mail Accounts in cpanel. Create an e-mail address that you would like to display in the FROM field of drupal e-mails.

4. Ask Bluehost to put your domain on the remote domains list.

5. Install the "PHPMailer" module on your drupal site.
http://drupal.org/project/phpmailer

6. (Read the instructions and don't forget to install the PHPMailer Library, a separate download.)
http://sourceforge.net/projects/phpmailer/files/phpmailer%20for%20php5_6/

7. In the PHPMail module settings, do this:
- Check "Use PHPMailer to send e-mails"
- Primary SMTP server: localhost
- SMTP Port 25
- Use secure protocol: NO
- Leave SMTP Authentication blank
- Under Advanced SMTP settings, Enter a FROM name
- Under Advanced SMTP settings, CHECK "Always set "Reply-To" address"
- Under Advanced SMTP settings, Don't check "Keep connection alive"

This makes drupal set the FROM field to be an address that Bluehost sees as legitimate (the one you created in step 3.) The reply-to is set to the address of the drupal user who sent the message using the contact form. Even thought the FROM field won't be accurate, when replying to the message, it will go to the right place.

I completely agree with Oceria's remarks and I think we must not give the chance to modify the "from" field whereas the e-mails are technically *always* sent by the application.
In the patch provided as attached, I've changed the signature of drupal_mail() to replace $from by $reply.
Then, the e-mails are *always* sent by the e-mail defined at the site level and if a module provides the parameter $reply, this value will be set in the section 'Reply to' of the e-mail.

Patch in #17 works for me. Thank you montesq
Just one question... Would it be possible to set the site name as well in the emails?

I just came across this problem with the Contact module sending email from the user's email, which of course doesn't work through the sites SMTP server.

The patch in #17 resolves the issue perfectly for me, where I now see messages from the site, but have a reply address to the user's email.

I can't believe this was spotted way back in 2007 and is still there!

I'd like to see this fix backported to drupal 6 and 7. In the meantime, I'll keep this patch handy for sure!

COntact form should never use user-provided From: address directly.

That will make mail validation fail, for whatever techniques used (SPF, DKIM etc).

This will also raise chances to mark the Drupal-running site to be marked as spam origin.

I had to use simple 'hack' that moves the From: address to the message body, and uses site-wide address instead. It was the only simple way to make SMTP services such as Amazon SES process Drupal system messages.

A related problem is if you have multiple recipients in the sitewide contact form, they all get used as the confirmation message's From address. This technically violates RFC2822 since you MUST add a Sender field in that case, which Drupal doesn't do.

http://www.ietf.org/rfc/rfc2822.txt

3.6.2. Originator fields

The originator fields of a message consist of the from field, the
sender field (when applicable), and optionally the reply-to field.
The from field consists of the field name "From" and a
comma-separated list of one or more mailbox specifications. If the
from field contains more than one mailbox specification in the
mailbox-list, then the sender field, containing the field name
"Sender" and a single mailbox specification, MUST appear in the
message. In either case, an optional reply-to field MAY also be
included, which contains the field name "Reply-To" and a
comma-separated list of one or more addresses.

Just for future searchers that don't want to have to apply a patch, we've whipped up a small D6 module that sends all contact form email from the website's email address with Reply-To set to the from address. It's still in sandbox at the moment but we are using in production sites with Amazon SES. Our D7 sites aren't running on AWS at the moment but I'm sure a D7 version will follow at some point.

I had the same problem as @markabur (#23) and am also using Bluehost. I modified the contact_mail_page_submit() function as shown below (sorry I wasn't able to create a patch file in the proper format). My reasoning is that the "site_mail" system variable should be used (if present) as stated on the Site Configuration page for Email Address...

The From address in automated e-mails sent during registration and new password requests, and other notifications.

In the unlikely event that the "site_mail" system variable doesn't exist, it falls back to the first email address in the list of "recipients"...

  // Send an auto-reply if necessary using the current language.
  if ($contact['reply']) {
    $recipients = explode(',', $contact['recipients']);
    $site_mail = variable_get('site_mail', $recipients[0]);
    drupal_mail('contact', 'page_autoreply', $from, $language, $values, $site_mail);
  }

Many thanks to Rj for making a solution available. I've got to say that not having this issue sorted out in core is totally crazy. I'm sure I'm not alone in struggling to find a solution to this issue, which frankly should have been addressed years ago, and looks to have been largely ignored, thereby limiting the usefulness of Drupal to many people.

I see this has been bumped (yet again) to 8.x by Dave Reid in comment #7 with the explanation being "feature requests are closed in D7 now".

I don't see this as a "feature request". I see it as a bug. And if security fixes and other bug fixes can be applied to core (even after it has been "officially released"), then this should be able to be fixed in D7.

What do you say, community?... agree?... disagree?

I agree, it's a bug.

All the emails that are generated by Drupal core's contact module on various Drupal sites of mine are now landing in my Gmail inbox with a giant warning on them stating:

This message may not have been sent by: someone@notmydomain.com

It's bad behavior for Drupal to cause these kinds of warnings, and since the solution appears to be really straightforward, we should just do it :) That siad, it probably still needs to be fixed in D8 first, then backported to D7.

So... attached is a patch against D8, and a re-roll of the D7 patch ready to be backported to D7 (named do-not-test)

A test had been written for From-headers in the meantime, so the tests were failing. Here is a D8 patch that fixes the tests.

I am not entirely sure wether the From: header should still include the site name like "Drupal site <someemail@example.com>"? I removed it for now in the test.

Nice work, thanks :)

Looks like the D7 patch is still good, unless we want to backport the new tests, too?

This looks good. Do we need to update any code in Drupal (other than the tests) to reflect this API change?

A quick grep seems to suggest there may be a couple of code paths to look at:

vortex:drupal dries$ grep -r drupal_mail * | grep -v test | grep -v api | grep modules
core/modules/action/action.module:  if (drupal_mail('system', 'action_send_email', $recipient, $langcode, $params)) {
core/modules/contact/lib/Drupal/contact/MessageFormController.php:    drupal_mail('contact', 'page_mail', $to, $recipient_langcode, $params, $sender->mail);
core/modules/contact/lib/Drupal/contact/MessageFormController.php:      drupal_mail('contact', 'page_copy', $sender->mail, $language_interface->langcode, $params, $sender->mail);
core/modules/contact/lib/Drupal/contact/MessageFormController.php:      drupal_mail('contact', 'page_autoreply', $sender->mail, $language_interface->langcode, $params);
core/modules/update/update.fetch.inc:        $message = drupal_mail('update', 'status_notify', $target, $target_langcode, $params);
core/modules/user/user.module:    $mail = drupal_mail('user', $op, $account->mail, $langcode, $params, $site_mail);
core/modules/user/user.module:      drupal_mail('user', 'register_pending_approval_admin', $site_mail, language_default()->langcode, $params);

Changing status because of #32. Feel free to set back to RTBC if this is a non-issue. Thanks.

I took a look at the instances where an explicit sender is used. They all (I found two) using sender correctly I would say.

@nanox: thanks for checking. Good to know we checked all callers. Unfortunately, it seems like the patch no longer applies and that a re-roll may be required. Asking for a re-test.

#30: 111702-30.patch queued for re-testing.

Here is a reroll.

Fixed the failing test.

Hi, could someone tell me how can I get this issue solved for D7?

Thanks!

@javier.drupal.2012 the patch needs to be committed to D8 first and then backported to D7 so it will take a little while.

But you can reroll the patch and apply it to your D7 installation while you wait :)

Hi @naxoc, Sorry, but I don't know how to get the patch for 7.x version. I'm waiting for the patch for D7 but I don't know how long it would take....Can you let me know whether there is any way to know when is going to be the patch backported?

I really don't know how to reroll the patch to apply it in my d7 version

Thanks heaps
Javier

#40: 111702-40.patch queued for re-testing.

for anyone on D6 or D7 who has this problem and doesn't want to use this core patch on their site, the module contact reply-to offers the same functionality.

Crazy that this isn't solved after this many years :-)

setting back to RTBC because a) i tested the patch and b) the only thing wrong earlier was a failing test.

#40: 111702-40.patch queued for re-testing.

trying to fix the test case

Back to RTBC, tests are green and look right.

This issue was RTBC and passing tests on July 1, the beginning of API freeze.

+++ b/core/includes/mail.incundefined
@@ -118,13 +118,13 @@
+  $from = $site_mail;

I don't think we need this assignment anymore... can just use $site_mail instead of $from

+++ b/core/includes/mail.incundefined
@@ -147,15 +148,14 @@ function drupal_mail($module, $key, $to, $langcode, $params = array(), $from = N
-  if ($default_from) {
+  if ($from) {

I think this if is unnecessary now as we should always have a $site_mail

Also I think we need to update the watchdog message in drupal_mail too. The current code is:

      if (!$message['result']) {
        watchdog('mail', 'Error sending e-mail (from %from to %to).', array('%from' => $message['from'], '%to' => $message['to']), WATCHDOG_ERROR);
        drupal_set_message(t('Unable to send e-mail. Contact the site administrator if the problem persists.'), 'error');
      }

Maybe to something like what is below since the from is going to be unchanging for 99 use-cases out of 100...

  watchdog('mail', 'Error sending e-mail (to %to with reply-to %reply).', array('%to' => $message['to'], '%reply' => $message['reply-to']), WATCHDOG_ERROR);

Although in the case of multiple domains...

  watchdog('mail', 'Error sending e-mail (from %from to %to with reply-to %reply).', array('%from' => $message['from'], '%to' => $message['to'], '%reply' => $message['reply-to']), WATCHDOG_ERROR);

Might be more helpful... but then again you should get the uri on the watchdog message anyway...

thanks for the thorough review, alexpott.

I don't think we need this assignment anymore... can just use $site_mail instead of $from

I think this if is unnecessary now as we should always have a $site_mail

good points. i've fixed those now.

Also I think we need to update the watchdog message in drupal_mail too.

on the grounds that more debugging information is generally better than less, i've used your second suggestion.

actually, let me try that again: i wasn't printing a useful watchdog message in the case where no reply-to value had been set. now i'm printing the words "not set".

55: drupal-mail-fixed-from-111702-55.patch queued for re-testing.

Testbot response didn't arrive, but the patch at #55 does not apply anymore.

re-roll of #55.

@penyaskito: thanks for your interest; i was beginning to feel like this issue has been abandoned :)

oops, forgot to change the status...

+++ b/core/includes/mail.inc
@@ -147,15 +147,12 @@ function drupal_mail($module, $key, $to, $langcode, $params = array(), $from = N
-    $headers['From'] = $site_config->get('name') . ' <' . $default_from . '>';
...
+  $headers['From'] = $headers['Sender'] = $headers['Return-Path'] = $headers['Errors-To'] = $site_mail;

If I'm right, we are losing here the ability of putting a beautiful human-from, only allowing the address.
I will like that emails sender are shown in clients like "Example Website", not info@example.org.

This would cause a regression of #209672: Use site name in From: header for system e-mails. As far as I know using that won't affect deliverability.

@penyaskito: oops, you're right. here's a new patch that doesn't regress #209672: Use site name in From: header for system e-mails. i had missed that, but in my defense, this issue is a year older :)

let me know if you see any other reason not to set this to RTBC.

gah, forgot to hide old files...

+++ b/core/modules/system/lib/Drupal/system/Tests/Mail/MailTest.php
@@ -90,12 +90,12 @@ function testFromHeader() {
-    $this->assertEqual('Drupal <simpletest@example.com>', self::$sent_message['headers']['From']);
+    $this->assertEqual('simpletest@example.com', self::$sent_message['headers']['From']);

This line should not change now.

+++ b/core/modules/contact/lib/Drupal/contact/Tests/ContactPersonalTest.php
@@ -73,7 +73,7 @@ function testSendPersonalContactMessage() {
-    $this->assertEqual($mail['from'], $this->web_user->getEmail());
+    $this->assertEqual($mail['reply-to'], $this->web_user->getEmail());

I would maintain an assert for $mail['from'].
Adding one to reply-to is super!

Thanks to your insistence!

Fixed #66 comments and also changed a comment that wasn't applicable anymore.
Renamed variable for readibility.

Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate 268435456 bytes) in /var/lib/drupaltestbot/sites/default/files/checkout/core/lib/Drupal/Core/Database/Connection.php on line 336
FATAL Drupal\system\Tests\Form\FormTest: test runner returned a non-zero error code (255).

Looks like a testbot problem, the test pass locally.

67: 111702-replyto-67.patch queued for re-testing.

Can't believe this issue still around (was checking for a project in 2010)... So this won't cover the case where we want to set custom from field. e.g. want to send from no-reply@site.com and reply-to: sales@site.com where site default is contact@site.com.

One more thought is, why can't we make a generic $headers arg with default overwrite something like:

function drupal_mail($module, $key, $to, $langcode, $params = array(), array $headers = array(), $send = TRUE) {

  // Build the default headers
  $headers += array(
    'MIME-Version'              => '1.0',
    'Content-Type'              => 'text/plain; charset=UTF-8; format=flowed; delsp=yes',
    'Content-Transfer-Encoding' => '8Bit',
    'X-Mailer'                  => 'Drupal',
    'Sender' => $default_from,
    'Return-Path' => $default_from,
    'From' => $site_config->get('name') . ' <' . $default_from . '>'
  );

}

I was trying to manually test the patch in #67 on simplytest.me but the following error would show up during installation - Configure site. The service definition "field.info" does not exist.

May be it is the patch or some D8 code outside the patch causing this?

I am completely new to doing anything with/in the core, so please correct me if I am thinking on the wrong lines, but doesn't the latest patch in #67 prevent the FROM address from being anything other than the SITE_EMAIL? Wouldn't that be a very bad idea? Because there are use cases where the FROM address needs to be different at different times (eg: different newsletter categories when using the simplenews module).

Shouldn't we be passing both FROM and REPLY-TO addresses to drupal_mail() and use SITE_EMAIL as the default or the fall back.

@Sumeet.Pareek Welcome to core development :)
Can you test again with http://simplytest.me/project/drupal/8.x?patch[]=https://drupal.org/files...

Now its working for me so could be an environmental random failure.

Simplenews could still override the From address if needed, just in their hook_mail_alter. And any contrib module could provide the D6/D7 behavior. We are just providing better default behavior, not disallowing at all doing whatever you need.

@vijaycs: Not sure if we want to change that. As said, we can override if needed, but this IMHO should be the responsibility of a mail-related contrib/custom module.
The scope is just improving the deliverability of Drupal mails for a default installation of Drupal 8, taking into account newcomers + shared hosts.

This was RTBC on July 1st, and it is almost 7 years since the original report. Let's move forward and try to get this in, or postpone or won't fix if there are enough reasons.

@penyaskito makes sense to me... we should get this in and worry about further improvements as follow up... +1 to RTBC

penyaskito's patch looks right to me. thanks!

67: 111702-replyto-67.patch queued for re-testing.

67: 111702-replyto-67.patch queued for re-testing.

HEAD was broken

+++ b/core/includes/mail.inc
@@ -140,15 +140,13 @@ function drupal_mail($module, $key, $to, $langcode, $params = array(), $from = N
+  $headers['Sender'] = $headers['Return-Path'] = $headers['Errors-To'] = $site_mail;

It looks like we're doing more than just adding the reply-to header and setting the from header to site mail. We're also adding the Errors-To header which is deprecated according to https://commons.apache.org/proper/commons-email/userguide.html

Specifically, the "Errors-to:" SMTP header is deprecated and cannot be trusted to control how a bounced message will be handled.

I think we should not be setting this header - especially since there is zero discussion of why this should be added on this issue. OH IT'S A REGRESSION :) ... searching of existing issues reveals #1397270: Errors-To header makes email sending via Amazon AWS SES impossible, #1197376: Remove Errors-To header for Amazon SES, and #1062616: Failed integration with Amazon's Simple Email Service (SES) - which actually removed the Errors-to! It'd be nice if we could add an assertion to ensure that the Errors-to header is not set when using drupal_mail()

+++ b/core/modules/system/lib/Drupal/system/Tests/Mail/MailTest.php
@@ -78,19 +78,19 @@ public function testCancelMessage() {
-    $this->assertEqual($from_email, self::$sent_message['headers']['From']);
+    $this->assertEqual($reply_email, self::$sent_message['headers']['Reply-to']);
...
+  public function testFromAndReplyToHeader() {
     global $language;
 
     // Reset the class variable holding a copy of the last sent message.
     self::$sent_message = NULL;
-    // Send an e-mail with a sender address specified.
-    $from_email = 'someone_else@example.com';
-    drupal_mail('simpletest', 'from_test', 'from_test@example.com', $language, array(), $from_email);
-    // Test that the from e-mail is just the e-mail and not the site name and
+    // Send an e-mail with a reply-to address specified.
+    $reply_email = 'someone_else@example.com';
+    drupal_mail('simpletest', 'from_test', 'from_test@example.com', $language, array(), $reply_email);
+    // Test that the reply-to e-mail is just the e-mail and not the site name and
     // default sender e-mail.
...
 
     self::$sent_message = NULL;
     // Send an e-mail and check that the From-header contains the site name.

There are two emails sent by this test I think that with the first email sent we need to assert on both the Reply-to and From headers. On the second we need to assert that the From header is set correctly and the the Reply-to is not set. Actually does this bring up an issue with the patch - what happens if the $reply value is NULL in drupal_mail()?

Wow, thanks for catching that!

The attached patch fixes it and adds assertions for verifying the Errors-To is not set.
Also added requested assertions for From and Reply-to.

Because I was there, also added assertion messages to those.

If $reply value is not given in drupal_mail(), we don't add the Reply-to header, so the mail client will response back to the sender, who is going to actually be the site email.

Great! finally ready

Committed 8e92d3b and pushed to 8.x. Thanks!

Suggested change notice:

drupal_mail() now uses 'reply-to:' instead of 'from:' header

In the fight against spam more and more ISP's require mail send though the php function to originate from the main domain. The contact form however uses the senders email as "from:" address.

This has been changed so that the 'from:' header is set to the site-wide email address, and the senders email is set in a 'reply-to:' header.

Drupal 7:

drupal_mail($module, $key, $to, $language, $params = array(), $from = NULL, $send = TRUE)

Drupal 8:

drupal_mail($module, $key, $to, $langcode, $params = array(), $reply = NULL, $send = TRUE)

YAY! Big thanks everyone involved in reviewing this!

@kim.pepper: Looks good to me, I just created https://drupal.org/node/2164905.
I only appended that you can still modify the From: by implementing hook_mail_alter().

@all: Is the first Change notice I ever create and I'm not really aware of the process, so please review and mark this issue as Fixed.
If I did anything wrong please tell me for next time.

One thing I thought of when posting this issue for the Subscriptions module...

If I am understanding this patch correctly, for modules that have an override email address to use as the "from" address, the system default address will be used for "reply-to", is that correct?

If so, it will not be possible to do what I'm attempting to do in Subscriptions, which is to have it use "no-reply@mysite.com" instead of the system default "from" address. Sure, the user will see that it came from "no-reply" and probably won't even attempt to reply, but I'm getting out-of-office autoreplies - which (it seems) would go to the system default as a result of this patch.

ExTexan wrote:

If I am understanding this patch correctly, for modules that have an override email address to use as the "from" address, the system default address will be used for "reply-to", is that correct?

no; with this the system default will always be used as the "from" address, and the override email address (if set) will be used as the "reply-to" address. (this is necessary so that SPF & other anti-spam measures which are becoming more common will still work.) so what you're planning to do in the subscriptions module will still work, since the out-of-office replies will go to no-reply@mysite.com or wherever you want.

Looks like this issue got closed accidentally... but some of us are still running D7 sites, so re-opening.

The patch from https://www.drupal.org/node/111702#comment-7000314 seems to still apply to 7.31, so re-uploading that here to kick off testing again.

looks like the issue metadata didn't save, trying again.

I'm using this simple patch and its works fine!

go test bot.

Updated tests to expect "reply-to" instead of "from".

Been using patch #99 for some time in production. +1 from me.

RTBC from me as well.

Random testbot fail.

seems like testbot wasn't stable. It's green and cleanly applies to current HEAD

I don't understand the rationale behind this patch, which looks like a regression to me.

The original code looks perfectly correct:

• The envelope (Sender and Return-Path) is set to the site address;
• The message (From) is set to the sender address.

SPF / DKIM and friends act on the envelope, not on the message itself.

Could we get more details from people affected by this on what they are seeing that is not what they would expect?

For example, I cannot reproduce Jen issue described in #28. Emails sent to me via Drupal.org's contact form have all the headers that I expect and pass all the SPF checks in Gmail.

(Both the change notice and the original issue seem to misunderstand basic concepts about email.)

In response to #112, I am having a significant issue with this related to people using the contact form and who use a yahoo.com mail account. See the "bounce" message I am getting from gmail. I have full SPF and DKIM correctly configured on my mail server. Mail is being signed, but GMAIL sees the FROM address coming from YAHOO and marks it as spam.

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of its
recipients. This is a permanent error. The following address(es) failed:

@gmail.com
SMTP error from remote mail server after end of data:
host gmail-smtp-in.l.google.com [64.233.177.27]:
550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to domain's
550-5.7.1 DMARC policy. Please contact administrator of yahoo.com domain if
550-5.7.1 this was a legitimate mail. Please visit
550-5.7.1 http://support.google.com/mail/answer/2451690 to learn about DMARC
550 5.7.1 initiative. s43si6580166yho.201 - gsmtp
@gmail.com
SMTP error from remote mail server after end of data:
host gmail-smtp-in.l.google.com [64.233.177.27]:
550-5.7.1 Unauthenticated email from yahoo.com is not accepted due to domain's
550-5.7.1 DMARC policy. Please contact administrator of yahoo.com domain if
550-5.7.1 this was a legitimate mail. Please visit
550-5.7.1 http://support.google.com/mail/answer/2451690 to learn about DMARC
550 5.7.1 initiative. s43si6580166yho.201 - gsmtp

------ This is a copy of the message, including all the headers. ------

Return-path: .com>
Received: from www-data by with local (Exim 4.80)
(envelope-from .com>)
id 1YPBV9-0005bc-5e; Sat, 21 Feb 2015 09:57:51 -0500
To: @gmail.com,@gmail.com
Subject: [other requests] Delete
X-PHP-Originating-Script: 33:system.mail.inc
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed; delsp=yes
Content-Transfer-Encoding: 8Bit
X-Mailer: Drupal
Sender: admin@.com
From: @yahoo.com
Message-Id:
Date: Sat, 21 Feb 2015 09:57:51 -0500

=======================================
I have replace some real names with FQDN, my-mail, my2-mail, , site-domain-name.

Note that gmail rejects the mail because of Yahoo's DMARC policy that is strict, and doesn't let yahoo mail come from my server. And yes, it is reading the FROM addr, not the RETURN path or the SENDER, all of which are in sync with the site's DKIM and SPF records.

The YAHOO address is the user using the contact form. the two GMAIL accounts are the intended recipients of the contact form.

(using fully up-to-date DRUPAL 7.34, Debian Wheezy, Exim4, php54-fpm, apache2.2 on a large KVM VPS). I am a very experienced linux/BSD sysadmin and have been using drupal since 5 for many custom sites.

Hope that helps.
George

@gorlov: Ah, yes, DMARC broke email. And it is certainly a good reason for the change, but it's really the real reason is mentioned in this issue.

Let's move forward with this for 7.x.

I've said this before in this issue queue, but for anyone who's new here, it's possible to work around this problem with the contrib module contact_reply_to until this change gets pushed to core.

Is this issue fixed and pushed to Drupal 7.34 ?
I can see some of the code in the new Drupal mail.inc code. but not all changes.

Rewarded work :)

@RajabNatshah this issue will be marked fixed when it is committed to the 7.x-dev branch, and will appear in the next stable release of Drupal 7 which comes out after that. You need to apply the patch manually until then.

Thanks Matt .

I do have the patch in a project, but I'm updating Drupal for that project. I will apply the patch then.

Rewarded time :)

So off the bat, this is missing some things from the Drupal 8 patch (the watchdog message and some tests). Also, the documentation of the new reply-to parameters is a little lacking.

But a bigger problem is that this is a very heavy-handed fix, especially for a stable release. As others pointed out above (e.g. #72 and #74) this will break various use cases. For example if your site mail is webmaster@example.com but you want to send something from Judy Smith <president@example.com> this will prevent that despite the domain being the same, and the display name will be removed from the email too. Plus, the Contact module becomes less user-friendly since without an indication somewhere in the "From" field of who actually sent it, it's hard for the recipient to tell from scanning their inbox - they'd basically have to open up each message. (I'd think we at least want the From to be something like sender@theirdomain.example.org via Site Name <webmaster@yoursite.example.com> to avoid that problem?)

All of this can be dealt with in hook_mail_alter(), but it's odd to require doing it there, and regardless I don't think we should be breaking existing code for Drupal 7.

The Webform module fixed this recently in a way that deals nicely with all the above considerations (see #2236237: Use Reply-To instead of From e-mail headers (Google/Yahoo/etc anti-spoofing policy marks e-mails as spam)) and this patch would break their solution. Shouldn't we try to implement something similar to what Webform did in Drupal core, and not change the API?

Like Damien I am also a little confused about the history of the issue; this was committed to Drupal 8 in late 2013 but the serious problems with this didn't start until April 2014, I believe (when Yahoo changed their policy).

A lot of the earlier reports in this issue, while it's true the patch here would help with those kinds of situations, I think in most cases the root cause would be solved by adding an SPF record for your site to indicate which servers are authorized to send mail for your domain. For example, something like #28, where Gmail marks the message as "This message may not have been sent by: someone@notmydomain.com", what Gmail would like to do in that instance, I believe, is mark it as "someone@notmydomain.com via yoursite.com" (that's what it does for mail sent via the drupal.org contact form). But it will only do that if it can verify via an SPF record that the mail was actually sent by yoursite.com.

So what should one use to fix this unbelievably backwards behavior of Drupal? The Webform module? The https://www.drupal.org/project/contact_reply_to module? I keep reading these bug reports, some of them are marked as "Closed", but there seems to be no definitive write-up on what to do.

seems that issue #111702 still not pushed to the Drupal 3.37 or the dev .. Still we need to apply the patch
I will have a look at the contact_reply_to mdoule . https://www.drupal.org/project/contact_reply_to
I will keep following this issue until we have it in Drupal or have the right module to work around.

Attached is a re-work of the patch that was used for Webform. I personally object to providing users with a checkbox for "Do you want the bug fixed?" even if we default the answer to "Yes". However, it is late in the D7 cycle and I get that we don't want to break contrib modules that affect mail, so let's give this a go.

1
down vote

There is one e-mail address that is used as From-address for all mails sent automatically by the system. By default, this is set to the same as the site administrator email. To change it, do the following:

Navigate to Administration » Configuration » System » Site Information.
Locate the field "E-mail address". Change it to whatever you want it to be.
Press Save configuration.

Note that will also be used as the From-adress for password resets and other generated mail.

If you require fine-tuning beyond this (to distinguish between different types of auto-generated email messages), you need to write a custom module. In that case, you may use hook_mail_alter to alter the From:-address, based upon some criteria (e.g. the subject field.

For avoidance of misunderstanding: The three steps listed above will not change the site administrator email id. To inspect and change the site administrator email id, click on the "My account" link while logged in as site admin, and then press the Edit tab to see it. The site administrator email id is by default the same email id as the From-address used for email sent by the site itself - but they do not need to be the same.

Summary is if you want to change the from and reply to address you can either change it from the above 3 steps mentioned which will be changed for all email which are going from system. But if you want to do it for some specific emails then you can do it from hook_mail_alter by writing own module.In this case you will have more power in terms of customizing each and every mail going from the system. Contact me if you need help in this or any calrifications.

1
down vote

There is one e-mail address that is used as From-address for all mails sent automatically by the system. By default, this is set to the same as the site administrator email. To change it, do the following:

Navigate to Administration » Configuration » System » Site Information.
Locate the field "E-mail address". Change it to whatever you want it to be.
Press Save configuration.

Note that will also be used as the From-adress for password resets and other generated mail.

If you require fine-tuning beyond this (to distinguish between different types of auto-generated email messages), you need to write a custom module. In that case, you may use hook_mail_alter to alter the From:-address, based upon some criteria (e.g. the subject field.

For avoidance of misunderstanding: The three steps listed above will not change the site administrator email id. To inspect and change the site administrator email id, click on the "My account" link while logged in as site admin, and then press the Edit tab to see it. The site administrator email id is by default the same email id as the From-address used for email sent by the site itself - but they do not need to be the same.

Summary is if you want to change the from and reply to address you can either change it from the above 3 steps mentioned which will be changed for all email which are going from system. But if you want to do it for some specific emails then you can do it from hook_mail_alter by writing own module.In this case you will have more power in terms of customizing each and every mail going from the system. Contact me if you need help in this or any clarifications.

prateek_drupal: every Drupal site needs this if they are using the core Contact module, or any other module that sets the "From" address to be that supplied by a site visitor.

The reason is DMARC, which, against all RFC advice, looks at the From address for an email. So if you want Drupal to send emails where a reply will go to the visitor's email, you have to set the From address to belong to the site, and the Reply-To address to be the visitor's. This is what this patch does.

Without this patch, or a contrib module that does the same thing, mail from Drupal can get blocked by large email providers like Yahoo! that use DMARC.

For this reason Drupal 8 has this behaviour in core.

I created a Drupal 7 contrib project that simply swaps From email with site email, and uses senders email as Reply-To. https://www.drupal.org/project/reply_to

Patch in #126 still applies cleanly.

thx, jenlampton. still works for me.

I tested the patch and it didn't work for me:

1. The default value of the checkbox doesn't match the default value of the variable_get(), so nothing worked at all until I visited the Site Information page and clicked the Save Configuration button.
2. After that it still didn't work, since the user-friendly version of the From header (e.g., "someone@example.com via Site Name <siteaddress@site.example.org>") never made it through in the email. I think that's because the code sets $message['from'] to the user-friendly value, but doesn't set $message['headers']['From'] to the same thing?

I would also lean towards not having the checkbox, personally. One of the reasons for borrowing the Webform solution is that it bends over backwards to only change the behavior when necessary and to change it in a user-friendly way when it does... and if someone really doesn't want the new behavior for some reason I'm pretty sure they can still change it in hook_mail_alter().

Also, based on a quick code review:

1. This could use tests. Maybe some of the Drupal 8 ones can be borrowed, although I guess they are testing a different thing.
2. The function documentation for the $from parameter of drupal_mail() needs updating for these changes.
3. The t() call needs to use the language that was passed in to drupal_mail() - otherwise it won't necessarily be in the correct language.

Would be really good to get this issue fixed though! (And then there could be a followup issue to discuss forward-porting this solution to Drupal 8.)

In drupal-7.51 this is still not fixed? Why?

@bigfatguy: Please read the documentation on what the different issue status values mean: https://www.drupal.org/node/156119

Is the intention to still back-port this to D7? Reason I ask: I became aware of the issue from using the contact forms on drupal.org to write to other people, and ticking the 'Send myself a copy' box. I adjusted my SPF record but obviously that doesn't help with DMARC, so I wonder if there's a significant amount of community mail being lost – to Gmail accounts for example – because people may not check spam folders.

Note that the comments link to two different modules that propose a solution to this problem whilst waiting for the D7 commit.

• https://www.drupal.org/project/contact_reply_to has a stable release and 1700 reported installs
• https://www.drupal.org/project/reply_to has an alpha release and 45 reported installs

Here's a re-work of the patch in #126 without the setting since @David_Rothstein and I seem to be in agreement on that. The code in this new approach is more similar to the work we are doing over in https://github.com/backdrop/backdrop/pull/1823/files for Backdrop. Both still need tests.

Noticing this recent issue popping up all over hundreds of websites I run. Core contact form is no longer usable and I have been porting them all to webform. Even webform you must use a valid email on that domain as the sender. Any solutions down the road for this?
Thanks

Setting NR for test queue.

Patch #142 works fine with Yahoo, but not with Gmail, since the 'My Name ' format is interpreted as multimple adresses. Here's the mailer-daemon message: gmail-smtp-in.l.google.com SMTP

@mlangenfeld you can put <code> tags around things like that so they don't get stripped out of your comment. I am copying it here now with that done so others can see the mailer-daemon message you're referring to:

gmail-smtp-in.l.google.com <gmail-smtp-in.l.google.com 5.7.1 smtp; 550-5.7.1 [xxx] Messages with multiple addresses in From: 550 5.7.1 header are not accepted. lxxx - gsmtp> SMTP

This is very curious and it is not obvious why it would occur. Could you provide some more details (such as what the actual "From" header looked like in this message)?

Looking at the patch it is not obvious to me what would cause this, although I am curious what would happen if e.g. the site name has a " in it, since it looks like that could cause trouble with the From header.

@David_Rothstein I'm sorry for the tag, this is my first post in the Drupal forum. The From line reads, as it should:
"[email of sender] via [Sitename]" <[standard site email]>
But the string has been exploded after 45 characters, so it reads:
<"[email of sender] via [part of Sitename]>, <[rest of Sitename]" <[standard site email]>>
which Gmail interpretes as 2 adresses. I have to say that our sitename is quite long (32 characters), but there are no " in it.
Update: it seems the problem is already known in Drupal 8: #2863601: GMail rejects mail because long 'from' field is split by Drupal 8

Thanks - that is definitely helpful research! Sounds like the patch needs work then. Also, I confirmed that a site name with a single " in it breaks things in a similar way (not specific to Gmail) so it needs work for that too, although it's possible that's just a different (and less likely to encounter in reality) version of the same problem.

The patch looks pretty good otherwise, but it still needs to pass the correct language into t(), and I'm adding the "Needs tests" tag since a couple people above mentioned that. Also there are a couple minor misspellings ("specefied" and "reformated").

Here's my rework of the patch. I just added a mime encoding for $headers['From']. This seems to fix the issue as discussed in #2863601: GMail rejects mail because long 'from' field is split by Drupal 8 and proposed in #2717965 Site name is not UTF-8 encoded in email headers . It works fine on my production site.

Interesting this issue has been around for a decade "and" drupal.org doesn't even have DKIM or DMARC setup. At all.

Anything coming from security-news drupal.org is being sent from IP's that aren't even in the SPF records.

I recently got on the bandwagon for becoming DMARC compliant and within a week, was blown away at the number of emails being sent on behalf of my domains. Legit were on the order of maybe 15 - 25 per day on some low end domains but illegitimate? In the 100's, issued on my domains behalf from all over the world... I only have 2 outgoing email services, the webserver and G Suite and they're both US based servers. In otherwords, there's no reason I should have email being sent from servers in India, or China.

Here's an idea of where those security-news drupal.org emails are going:

[Security-news] H5P - Critical - Reflected Cross Site Scripting (XSS) - DRUPAL-SA-CONTRIB-2017-071 
security-news@drupal.org Aug 30 (3 days ago)

Why is this message in Spam? It's similar to messages that were detected by our spam filters.

SPF: SOFTFAIL with IP 140.211.10.49

Not only is the SPF failing, but neither DKIM nor DMARC are in place for Drupal.org.

Looking at the SPF record for drupal.org, it's only pointing to the mx records (smtp1-4.osuosl.org) and servers.mcsv.net, none of which contain the block for the server generating security-news on IP 140.211.10.49.

Considering Drupal.org isn't even compliant, I guess it shouldn't come as any surprise this issue for core is taking... a while?

Patch in #149 is also working for me. Thanks @mlangenfeld!

@philsward makes some very good points. @jenlampton is on the ball too.

@jenlampton, can you please confirm if this patch still works even if you aren't supporting DMARC? I looked at the patch and it looks ok to me but I haven't tested with and without DMARC.

@philsward, I wanted to respond as I am re-taking over the d.o. infra maintenance currently and working through a backlog. DMARC/DKIM is on the agenda, but we currently use relays that don't support that (We will either be vacating them or supporting their upgrade). The SPF fail is an entirely different issue likely related to a recent migration of the lists server. I will be looking into and fixing that today or tomorrow morning.

-N

@nnewton thanks for the heads up. Super important stuff and at the time of writing my comment, I wasn't sure where to go to reach out to the folks necessary to get it done. My hope was to reach some attention from someone in this thread that might be privy to talk with the right folks. I later found the contact form for the "Drupal Association" which is where I should have gone in the first place, but wasn't aware of it.

Glad you're looking into it! I was floored at how "important" those items are for proper email setup when I went through the journey of setting it up on my end. I was shocked to see Drupal related emails going to spam for this very reason.

Sorry everyone for distracting from the original topic :-/ I was merely trying to be a squeaky wheel.

---

Glad to see this thread is moving forward though! It's been a long time coming!

The patch #149 looks good and fixes the issue.

Please see https://www.drupal.org/core/issue-priority and ensure you provide valid reasoning of 'critical' according to that criteria before changing issue priority.

@kim.pepper: I agree that this shouldn't have been bumped to "critical" but it's been flagged as "major" since @alexpott set it that way in 2013 and no-one who's looked at it since then has objected. Let's leave it there.

Will this patch work with the 'smtp' module? I am not certain if 'smtp' bypasses all this logic or sits on top of it.

Corresponding ticket created at 'smtp' https://www.drupal.org/project/smtp/issues/2941007.

ANSWERING MY OWN QUESTION:

Yes this patch works great because the needed logic occurs before the 'smtp' hook or any hooks are applied.

Thanks for a great patch!

RTBC!

Patch in #149 applies cleanly to 7.58.

Patch #149 is working really well for me on 7.57 and now 7.58.

What is stopping this from being committed now?

Not sure if its right place here
but i have a problem
since i have the patch 149 in
it will transmitted the user email in the Personal contact form
and this should not happend because

Allow other users to contact you via a personal contact form which keeps your e-mail address hidden. Note that some privileged users such as site administrators are still able to contact you even if you choose to disable this feature.

Ah, that sounds like it needs work then. (changing status)
For those who are not using the per-user contact form, the patch in #149 still applies to 7.59.

Patch #149 doesn't work for me in 7.59 and I solved using:

function HOOK_mail_alter(&$message){
  if('contact_page_mail' === $message['id']){
    $message['from'] = variable_get('site_mail', ini_get('sendmail_from'));
    $message['headers']['From'] = variable_get('site_mail', ini_get('sendmail_from'));
  }
}

I hope it helps!

Bumping to 7.70. This didn't make it into 7.60.

Shouldn't that be 7.61?

yes 7.61 please.

This is classified as a bug so yes it could be in the 7.61 release. The core maintainers will have the final say.

I've just reread the recent issue history: besides the points in #164 and #166, we still haven't written any new tests (#135), and it'd probably also help to get an update from someone on the d.o infrastructure team.

Just a quick word to the wise - we've been using #99 in production for a couple of years with no worries.

But recently a change in backend email providers has highlighted that this patch can create multiple reply-to headers (which AWS SES does not like...), when used in conjunction with the Webform versions (> 7.x-4.3) that contain the "webform_email_replyto" variable (discussion in https://www.drupal.org/node/1462192). Setting this variable to 0 is an effective workaround for now.

We'll test the same scenarios against the most recent patch and report back.

Patch in #149 still applies cleanly to 7.63.

Would it help to say that this has been production-tested in Backdrop CMS since version 1.1.0 (2015?): https://github.com/backdrop/backdrop-issues/issues/305

@klonos thanks for testimony.

this is one of the last patches should be committed in the Drupal 7 release before EOL

We need to backport the test updates committed 6 years ago to D8
git show 8e92d3b1f16d
see attachment

Updated with a check the 'From/Reply_to header' in 'mail.test' - backported from D8

   * Checks the From: and Reply-to: headers.
   */
  function testFromHeader() {

Please check attached interdiff file

Here is a test, and the test-only patch is the interdiff.

Tests that email headers for contact message are correctly set.

Two new checks are added in #183 based on #149 regarding test

backported from D8:

- A new check for 'From/Reply_to header' in 'mail.test'
- A new check for personal contact email headers in 'contact.test'

Please check interdiff file for details.

I'm coming in pretty late here, but why is Drupal setting the Return-Path header? The recipient's mail server sets this to the value of the envelope sender. Drupal can influence what gets set in the MAIL FROM: command but it probably shouldn't set the header directly.

https://dmarc.org/
its the keyword
some Mail provider use this.
and drupal use the email adress (Mail From) from the registered user to send emails to staff or anything else.
if Drupal setst the from mail to any addres the mail-demon will not send the mail cause the adress ist unknown.

When Drupal sends email via SMTP it sends the SMTP command MAIL FROM: email@domain.com. The SMTP server can then set the Return-Path header. If you set this header directly the SMTP server may at best ignore it and add its own or simply remove it.

It should be possible to configure the sender (as opposed to From address) within Drupal but not the Return-Path header.

@imclean - you are confusing the Return-path: header (which is also known as the "envelope sender") with the normal Reply-To: header which can be set by the sender of the message.

This patch sets the Reply-To: header to be the email address any replies should be sent to, and the From: header to the email address configured for the website. This allows mail to pass DMARC checks which would otherwise fail if the From: address domain implements DMARC and the web server is not authorised by DMARC to send mail for that domain.

@fonant I'm not confusing them, although I've probably brought up the problem with Drupal's handling of the "Return-Path" header in the wrong issue. This is the most thorough discussion I can find of mail headers but I probably should've started another issue.

On the topic at hand, as has been mentioned DMARC seems to have caused some problems. It probably should consider the "Return-Path" header or at least "Sender" or "Reply-To" instead of "From".

To pass DMARC on my server, I found that I had to set the message 'From' to an email address matching the originating domain, as well as ALL of the relevant headers:

$headers['From'] = email@originatingdomain.com;
$headers['Sender'] = email@originatingdomain.com;
$headers['Return-Path'] = email@originatingdomain.com;

And then set the $headers['Reply-To'] to the original sender's email address.

I realize that Return-Path is supposed to be set by the receiving server, however it seems that it is used by DMARC as well, at least in some configurations. The above is what worked for me.

I got a new challenge: a malformed From header when using non-asci characters (like ä, é and so on) in the site_name:

SMTP error from remote mail server after end of data: 554 5.2.0 MXIN600 Message is not RFC 5322 compliant: 'From' header is missing or malformed ;id=YTdni2kWspxBdYTdnigqw5;sid=YTdni2kWspxBd;mta=mx7.tb;d=20191123;

My working solution for these common European characters is transliterating the site_name before mime_header_encode(). The code starts with 2 existing lines and end with 1 existing line:

      preg_match('/^"?([^<]*?)"? *(?:<(.*)>)?$/', $from, $matches);
      if ($matches) {
        $sitename = variable_get('site_name', 'Drupal');
        if (function_exists('transliteration_get')) {
          $sitename = transliteration_get($sitename, '');
        }
        $from_name = t('!name via !site_name', array(
          '!name' => empty($matches[1]) ? $matches[2] : $matches[1],
          '!site_name' => $sitename,
        ));

I don't know if this will work as wel for other non-asci chars.

@PROMES I think your issue is #1419874: "From" mail header incorrectly encoded if (a) has name and e-mail address and (b) name contains non-ASCII characters

@longwave: Thanks. I was following / using this issue for so long time I assumed it fits in this issue.
I will move my reply to issues/1419874.

reapplying to Drupal 7.69.

Updating target, so the issue is not forgotten.

The current latest patch works a expected. But in non-English countries we have sometimes sitenames with special characters.
If module transliteration is installed I transliterate the sitename before this name is assembled in $from_reformatted.

Currently my Netbeans program refuses to make pach files so I changed the attached patchfile manually with my changes:
if ($matches) {
+ $sitename = variable_get('site_name', 'Drupal');
+ if (function_exists('transliteration_get')) {
+ $sitename = transliteration_get($sitename, '');
+ }
$from_reformatted = t('"!name via !site_name" ', array(
'!name' => empty($matches[1]) ? $matches[2] : $matches[1],
- '!site_name' => variable_get('site_name', 'Drupal'),
+ '!site_name' => $sitename,
'!site_mail' => $default_from,
));

Hello,
got still the issue with emails, see
https://www.drupal.org/project/drupal/issues/111702#comment-12567413
at the moment it's not DSGVO conform so i have to disable the contact forms

if a user sends a message from the users contact form so its the email address in reply-to from the user who sends the message and thats is wrong. in this case should the "no-reply" address used or remove reply-to adress.

Is this still an issue now that https://www.drupal.org/node/3185877 is in D7?