[META] Adopt the symfony mailer component

As an FYI: https://drupal.org/project/swiftmailer

Moving to 9.x.

Yes. I think Email still important for a CMS. SwiftMailer is easier to adopt other transactional email services.

I think we can add it as new backend

This might be a dupe of #1346036: Add an alternative to hook_mail() and deprecate it, but the fact is we need to start somewhere. I think the first big benefit would be getting the library into Drupal core. Not having HTML support is going to kill momentum for contributed developers trying to build rich experiences on Drupal (like Drupal Commerce.)

We can start by adding the library and finding ways to incorporate it along the way, but this way modules can easily use it.

Here's the first step.

I was discussing with @bojanz that we should probably make this issue a meta/plan with a bunch of issues, the first being your patch.

And yes, we have a few related issues, there's also one from @joachim about adding a config entity based system to manage and send mails but I think that's something that will live on top of the actual mail API.

A second library to look at would be html2text/html2text I think. The swiftmailer module uses that and it might allow us to deprecate all or most of \Drupal\Core\Mail\MailFormatHelper and leave that to an external library that cares about that stuff.

And the obvious next step is then a Swiftmailer-based PhpMail replacement. A first step could just be a drop-in replacement and then another issue to adds the settings UI that swiftmailer.module offers.

Another angle is if and how we'd add things that mailsystem.module currently provides. Or if we in the long-term plan to deprecate our own pluggable mail API entirely and have the extensibility on the swiftmail level.

Oh, and HTML mails of course ;)

I believe we can get the php_mail plugin to wrap around Swiftmailer and introduce deprecations without breaking anything, based off of #9.

The Swiftmailer module does include html2text/html2text. So we can follow the Swiftmailer logic inside of php_mail core plugin. Negotiate whether to strip HTML to text or send as HTML based on the provided content type headers in $params.

This way we can support HTML mails out of the box without configuring/changing the current mail plugin (which doesn't even have a UI currently.)

Another angle is if and how we'd add things that mailsystem.module currently provides. Or if we in the long-term plan to deprecate our own pluggable mail API entirely and have the extensibility on the swiftmail level.

I was going to make an argument of "What about Mandrill and SendGrid", but both have documentation that shows how to setup the SMTP settings. So I feel that getting plugins into Swiftmailer w/ some of the configuration options from the Swiftmailer module is best end result.

Looks like a lot of people are fans of swiftmailer, but I haven't looked at it yet and don't know what benefits it brings. So let's make this a plan with reasoning first.

Could you document in the issue summary:
* what does swiftmailer better than our current mail system?
* what new features would swiftmailer enable that we currently don't have?
* why is swiftmailer the best library for that? Are there alternatives?
* what are the drawbacks of integrating swiftmailer?

Per #11, I have wrriten a response and updated the issue summary.

@mglaman now that this is a plan, I think we should have a separate issue to add the swiftmailer library.

Also created #2830384: Deprecate MailFormatHelper in favor of html2text/html2text library to get the ball rolling there, but that could be more complicated than I anticipated.

Swiftmailer updated to 6.0.1 and I think we need to add library & use it in initial issue

> And yes, we have a few related issues, there's also one from @joachim about adding a config entity based system to manage and send mails but I think that's something that will live on top of the actual mail API.

That's correct, and I think this issue and #1346036: Add an alternative to hook_mail() and deprecate it touch each other but don't overlap.

From what I can tell reading here, swiftmailer replaces our mail backend system for sending mail. It doesn't replace hook_mail(), which is a system for putting dynamic values into hardcoded or configurable message text, and for making that message text translatable.

Let's turn this into a meta. The first child issue could be adding html2text/html2text and swiftmailer/swiftmailer into core.

The first child issue filed: #3165762: Add symfony/mailer into core

Wouldn't be the symfony/mailer component better now ?

@pounard, thanks! +1 to use the symfony mailer component.

Two links for whom are interested in the symfony mailer component:

• Sending emails with Symfony: Swift Mailer or the Mailer component
• Symfony Mime / Mailer: Introducing the new Symfony Mime and Symfony Mailer components

Adding "Needs issue summary update" tag for updating a plan to IS.

s/majority/some

As stated above, I don't see how this can replace all the hook_mail() stuff in our mail system.

> As stated above, I don't see how this can replace all the hook_mail() stuff in our mail system.

If we adopted symfony/mailer could we deprecate hook_mail() and replace it with Twig templates?

Actually, symfony mailer can replace hook_mail() but it's a long and complicated path. hook_mail() is the build part of sending the e-mail of our current API.

IMHO, the first step would be replacing/reimplementing \Drupal\Core\Mail\Plugin\Mail\PhpMail, not MailManager. And maybe the mailhelper or at least parts of it for html to text conversion and so on.

However, the next step would then be to either make symfony mailer or some adapter on top of it the API that is used directly to send e-mails, basically bypassing MailManager entirely. hook_mail() is a horrible API that's completely out of place in our service and object oriented architectures. We'd lose absolutely nothing if we'd just pass the subject/body/headers directly to whatever is sending the mail. If you want to make it reusable, put that in a service. No need to force having the mail build code in a completely different place.

Imagine being able to build an e-mail with a value object like the examples on https://backbeat.tech/blog/sending-emails-with-symfony/ instead of the current, undocumented magic array keys that often vary between which contrib module you use.

That said, as interesting as being able to replace our whole pluggable MailManager would be, that's a massive change that has so many BC breaking implications that I don't even know where to start ;) hook_mail() is not an issue, each module on its own could decide when to migrate to the new API. But hook_mail_alter? We'd need to provide a BC layer for that hook that takes a modern mail value object, converts it into the current array, runs alter hooks on it, and then transforms it back into the value object? Considering that you can basically put whatever you want in that array, that sounds more than scary. And we wouldn't just break a single hook or API call for a whole ecosystem of modules like mailsystem, maillog, mimemail, smtp and current swiftmailer module, we'd basically do it like Thanos. SNAP them out of existence. I have no idea how that would fit in our BC policy. We've moved modules into core before, but this is a whole new level ;)

So IMHO before we start on the details we probably need a plan to address this topic as a whole and have framework/release manager approval. Basically making it an iniative.

So IMHO before we start on the details we probably need a plan to address this topic as a whole and have framework/release manager approval. Basically making it an iniative.

Tagging "Needs framework manager review" for this.

Moving to "Drupal core ideas"

> If we adopted symfony/mailer could we deprecate hook_mail() and replace it with Twig templates?

hook_mail() handles translation, but then Commerce has switched to Twig templates for mail, so they must have a way to do that.

> hook_mail() is not an issue, each module on its own could decide when to migrate to the new API. But hook_mail_alter? We'd need to provide a BC layer for that hook that takes a modern mail value object, converts it into the current array, runs alter hooks on it, and then transforms it back into the value object? Considering that you can basically put whatever you want in that array, that sounds more than scary. And we wouldn't just break a single hook or API call for a whole ecosystem of modules like mailsystem, maillog, mimemail, smtp and current swiftmailer module, we'd basically do it like Thanos. SNAP them out of existence. I have no idea how that would fit in our BC policy. We've moved modules into core before, but this is a whole new level ;)

I have already posted a plan to convert hook_mail() to a new system (using config entities), which includes BC handling. Needs feedback!

> I have already posted a plan to convert hook_mail() to a new system (using config entities), which includes BC handling. Needs feedback!

Hi @joachim, did you mean the 3nd paragraph in #34? Or you posted it somewhere else?

It's on #1346036: Add an alternative to hook_mail() and deprecate it, mostly in this comment: https://www.drupal.org/project/drupal/issues/1346036#comment-11933661

I know #1346036: Add an alternative to hook_mail() and deprecate it but IMHO that should/could be a layer above the current/future mail API. Not every mail is something that needs to be managed through configuration, and a low-level API like sending a mail should IMHO _not_ require a config entity.

> hook_mail() handles translation, but then Commerce has switched to Twig templates for mail, so they must have a way to do that.

No, they didn't. They render something _in_ hook_mail(), like everyone else who sends HTML mails. The only special thing is that they pass the result through the mail system through mail options and they generalized it so they don't need a custom hook_mail() implementation for every mail by just passing it through to render a certain Element. But it's still hook_mail().

> IMHO that should/could be a layer above the current/future mail API. Not every mail is something that needs to be managed through configuration, and a low-level API like sending a mail should IMHO _not_ require a config entity.

Agreed. If we use Symfony mailer, it would go below what #1803948: [META] Adopt the symfony mailer component proposes, which is a system for configurable mail, and not about the actual sending of mail. IIRC in my module the mail service uses a slim interface (called something like MailMessageInterface IIRC?), so for code that just wants to send a mail and not bother with user-configurability, we could provide a class that's a slim implementation of the interface.

It's unfortunate, that Symfony Mailer is not a standalone library. It has its own dependency and is integrated with the framework. I'm wondering if that introduces any implications for us?

It doesn't require anything that we don't already have, you can composer require it and it only installs symfony/mailer:

requires
egulias/email-validator ^2.1.10
php >=7.1.3
psr/log ~1.0
symfony/event-dispatcher ^4.3
symfony/mime ^4.4|^5.0
symfony/service-contracts ^1.1|^2

Coming from #84883: Unicode::mimeHeaderEncode() doesn't correctly follow RFC 2047 which is the type of low level bugs we have in our mail system that are quite hard to fix because of the architecture. I'm really interested in exploring #31 because I think that'll give us the quickest and cleanest way to use more reliable code for sending emails. It'd be great to separate out that from the larger and more over-arching piece about the suitability of hook_mail.

+1 for #31/#41.

Also +1 for #31 / #41

I find https://symfony.com/doc/current/components/messenger.html is more promising instead of just a mail

Removed outdated information from the issue summary.

Added related issue.

I posted a PoC patch here: #3165762: Add symfony/mailer into core.

I'm not sure what the process is to get this idea accepted. I'm not even sure it requires an idea issue? We do need approval for the component, but given that it is part of symfony, that should be relatively simple? We also have the approval from @alexpott above.

As quickly mentioned in that other issue and also per #31, this can be split in many partial steps. We can start with a basic, kinda 1:1 replacement implementation, and then later consider HTML support, transport configuration (could for now be pushed into mailsystem.module, but I'm happy to get that into core too). And then start with the real work: deprecating hook_mail() and (most of?) MailManager.

Happy to just unpostpone the core issue and move forward there.

I am an active maintainer of simplenews (a popular module that sends complex HTML mails) and swiftmailer (a module that can send HTML mails, using the library of the same name). After fixing various issues these two modules now work reasonable well together for sending HTML mail over different transports.

However in my view the core MailManagerInterface and MailInterface are not really suitable for modern day email sending. Having just now seen this issue, my initial feeling is make a clean start, ignoring the entire existing mail system component. This would then allow a simple migration path without back-compatibility issues. The new mail system could start life in contrib, then an experimental core module, then stable, then deprecate and finally remove the old mail system.

Off the top of my head, here are some requirements that a replacement system should provide:

• Plain text or HTML mail with optional plain text alternative
• Automatic conversion between plain text and HTML as required, using modern supported library code, using the existing MarkupInterface to distinguishing markup from plain text
• Multiple supported transports
• Flexible configuration of sending options, allowing different settings for different mail "channels" (module/key in current terminology)
• Attachments, including "inline images", with proper security to control attaching of protected files
• Theming of HTML mails, with automatic conversion to inline CSS
• Error reporting covering transport failure, temporary and permanent errors
• Tracking/blocking of permanently failed addresses
• ?Bounce handling (or API to support it)

Possibly some could live in contrib, but even so the core mail system needs to allow for them, designed with the necessary interfaces.

It's true we could create in implementation MailInterface based on symfony mailer. It's less obvious to me that this is useful, but I'm only on the fringes of the Drupal community so I might be wrong. This option could eventually be a modest improvement on the swiftmailer module, being based on a modern codebase. However it would still suffer from the major limitations imposed by the current interfaces. I'm not sure how much of the work would be useful for the full rewrite option. We should consider whether to introduce such a component into core knowing that it would likely be deprecated soon after.

Here is a new contrib module Symfony Mailer that prototypes the approach from #48. Comments welcome😃.

As maintainer of swiftmailer, i also thought a bit about that. I came, totally indepently from @AdamPS, to roughly the same conclusions:

However in my view the core MailManagerInterface and MailInterface are not really suitable for modern day email sending. Having just now seen this issue, my initial feeling is make a clean start, ignoring the entire existing mail system component. This would then allow a simple migration path without back-compatibility issues. The new mail system could start life in contrib, then an experimental core module, then stable, then deprecate and finally remove the old mail system.

A strong +1 on that from me.

And kudos for letting code speak and creating the contrib module. I groked the code and will thow all praise and dissent in an issue there.

Swiftmailer is EOL, only Symfony Mailer is supported from November 2021: https://symfony.com/blog/the-end-of-swiftmailer

I'm all for exploring to use symfony mailer directly in a contrib module.

> This would then allow a simple migration path without back-compatibility issues.

It's not quite that easy though. hook_mail_alter() and using the existing mail system in general is an API too. If user.module suddenly directly uses symfony mailer to send its mails then that will break all sites that use the swiftmailer and hook_mail_alter() to convert them to HTML mails or any other change like that. And the second core deprecates the old API and switches to symfony mailer directly then a whole bunch of _heavily_ used contrib modules (mailsystem, swiftmailer and any other module that provides mail plugins) are basically dead. I'm not sure how and when we'd do such a change? We can't do it in 9.x but I also don't see how we can do it in 10.0 either.

That's why I proposed to start with a symfony mailer based mail plugin, that is a clearly defined, achievable first step that gets us symfony mailer in core and makes it easier to experiment with an approach like what the symfony mailer module proposes.

@longwave: good to know, that's only relevant for the swiftmailer module though I think. As an intermediate step, we either need a 3.x version that actually uses symfony mailer or create a new project for that. Maybe the new module could both? Provide both an integration with the existing mail API and same feature set as swiftmailer module and also offer a more direct and modern API that contrib and custom code can use if they explicitly depend on that. Then sites that use it only need to configure it once and for now, all mails will use that configuration.

Thanks @Berdir and @longwave - the EOL deadline is rather sooner than I had realised!

What I mean when I say a "simple migration path" is that we can provide a replacement mail system (initially contrib, later core), and let it sit side-by-side with the old one. Each site owner can choose when they want to switch from the old to the new - maybe they enable/disable modules or there is a config option. Then we deprecate the old mail system and finally in D10 (or D11) remove it entirely.

It's true that we could add a core symfony mailer based mail plugin. A fully working and usable implementation would likely be similar to swiftmailer module which is around 3000loc (there is lots of complicated conversion code between the two different interfaces). It would presumably depend on the contrib mailsystem module to enable it. And we're still stuck with the existing API. The part that seems most useful to experiment with is the new API because a symfony version of swiftmailer module is not really ground-breaking.

In comparison, I believe the new mail system approach could be much simpler to develop as it can mostly rely on the symfony component which it exposes fairly directly.

What do you think @Berdir?

Absolutely I agree about the "intermediate step" - the Symfony Mailer module plans to add a back-compatibility layer. This could be an implementation of MailInterface as you already imagine. Or maybe the new module could replace the entire MailManager with a shim that calls the new mail system - this would help avoid some bugs and limitations of the existing MailManager.php.

The Symfony Mailer module would very much welcome more developers or reviewers. Hey it's a potential chance to be an author of a new component of Drupal Core😃.

Latest commit to Symfony Mailer module adds a back-compatibility layer.

Note this does not keep compatibility with hook_mail_alter() - that would tie us to a D7-era array of rendered headers that then have to be "unrendered" back to structures only for symfony to render them again. So we need to make a break - anyone who switches to the new mailsystem has to rewrite their hooks. It's a pain I realise but I see no alternative.

What it does is allow for module A to write a BC plugin on behalf of module B - without touching any code in module B. The plugin allows module A to send mails the old way and maps them to the new mail system interface. The Symfony Mailer module includes plugins for all the core modules. They are quite easy to write - ranging from 32-82 lines of code including blanks and comments. In this way we can get some active testing of the proposed new mail system without needing any changes to core or other modules.

> that would tie us to a D7-era array of rendered headers that then have to be "unrendered" back to structures only for symfony to render them again

Is this at all feasible or is there something irreversible in the transformations? Because if we can do this and have a BC layer that still goes through hook_mail_alter(), we can keep supporting it for the time being, while notifying users via deprecations that hook_mail_alter() is going away and pointing them to a change notice that tells them how to upgrade to the new way of doing things.

@longwave it's a good question to ask.

Background: IMO the overall goal is to switch from old interfaces, old features and old libraries to new versions of each. I propose that switching any part of the system is likely to be non-BC. For example, if we switch the default Drupal Core implementation from PhpMail to symfony mailer (keeping the same interfaces), that will change the content of the mails sent which could break important features on some sites. So I feel that any switch would need to be controlled by a config setting rather than occurring automatically.

Personally I don't see much advantage in "partial" switching; instead I would rather let go of all that's bad and make a simple, feature-rich, clean new mail-system. Developers will be attracted to the advantages and will happily choose to switch (rather than being "forced to" by annoying deprecation notices). Eventually yes, the old mailsystem would be deprecated in entirety once the new one is popular and stable.

If we keep supporting the existing interface including hook_mail_alter() I feel that we would end up with something quite similar to swiftmailer module: hacky, complex, hard to maintain, hiding the many useful features of the symfony mailer library from application code. If anyone feels inspired to create a stable version of such a module, then it would have some benefit - sites using swiftmailer module could migrate once swiftmailer library becomes unsupported. However if a new mailsystem project succeeds, sites might move on to that.

I don't really think this needs to be an ideas issue - the ideas queue is more for product decisions, whereas this is refactoring/replacing a couple of low level APIs. So a core meta issue linking all the sub-issues would be plenty. Could potentially be this one if we move queues and add child issues.

Done as @catch suggested in #57.

I don't think this is postponed on anything. Now 10.1.x is open we can figure out how to move forward with this.

Symfony Mailer is now close to Beta with 3000 sites reporting usage. Still there are feature gaps, and the testing is very light. From my position as maintainer I feel it needs more time to develop and stabilise in contrib. I would be very happy for other maintainers/developers to join in.

It would be very interesting to hear more comments here about whether there is general support for adding something similar to Symfony Mailer into core.

The Criteria for evaluating proposed changes was lasted updated October 2020, but seems relevant to this discussion:

1. The changes support and enhance Drupal project aims.
2. The proposed changes are current.
3. The proposed change doesn't introduce any significant issues or risks.
4. The changes are well coded.
5. There is demonstrated demand and support for the change.
6. The change will be used by a substantial part of the installed Drupal base.
7. The benefits of the change justify additional code and resources.

IMO, 5 and 6 above are the most difficult. How has "demonstrated demand and support" been objectively determined in the past? A survey/poll? And how do we quantitatively define "substantial part"?

Also, looking at other open source CM systems/frameworks might provide valuable insight.

@Chris Matthews

So there are a couple of considerations:

1. If we're 1-1 replacing always-on (or nearly always-on) core functionality with an external library, then de-facto that library will end up being used by every core install. So it just comes down to whether introducing a new dependency and having to deal with updates outweighs maintaining our own code. This just requires a dependency evaluation https://www.drupal.org/about/core/policies/core-dependency-policies-and-...

2. For new features (at least anything that requires a new module), we don't have a defined threshold, but there is some discussion in #1273344: Establish heuristics for core feature evaluation and #3158669: By default deprecate non-experimental modules that are used by less 5% of sites before the next major version about what that would look like.

For demand, if we look at contrib mail modules, it gives an idea relative to overall install base, only including 8/9 installs:

SMTP: https://www.drupal.org/project/usage/smtp - 60,000

Swiftmailer: https://www.drupal.org/project/usage/swiftmailer - 40,000

Mailsystem: https://www.drupal.org/project/usage/mailsystem - 90,000

The Symfony Mail module is very new and not stable yet, so still at 3,000 installs.

This doesn't really give a conclusive picture since we don't know exactly what functionality we'd end up providing in core yet. However, if we made the SMTP and swiftmailer/symfony mailer contrib modules completely obsolete (and possibly others?), that's over 100,000 installs, which is approx 1/5 to 1/4 of reported Drupal 8/9 installs.

@catch, thank you, all of the above makes sense. FWIW, I'm primarily a site builder and currently am unsure which direction to take re: adding HTML email functionality in contrib: (1) "Risk" continuing to use SwiftMailer; (2) go ahead and switch to Adam Shepherd's excellent Symfony Mailer module, or (3) switch to another contrib module like HTML Mail (8.x-3.0-alpha3 released February 2021) or Mime Mail (8.x-1.0-alpha4 released December 2021). It would indeed be helpful to have core support some or all of what Symfony Mailer is working towards. In order to get the necessary backing of the community, I think @Berdir's comment in #31 is right on

So IMHO before we start on the details we probably need a plan to address this topic as a whole and have framework/release manager approval. Basically making it an initiative.

This is partially off-topic as it's more about the current contrib situation with swiftmailer and symfony_mailer, but also somewhat affects moving things into core and many who are interested in that topic are following here already.

The situation we have right now is that the swiftmailer contrib module is deprecated and unsupported since Nov 21, since the swiftmailer library went EOL. Despite that, since then, the usage of the swiftmailer module went slightly up and then stabilized around 40k. The reason for that is that there is no drop-in replacement available for what it does. Unlike the SMTP module, it doesn't just provide a mail plugin but also the ability to send HTML mails including a template and settings for text formats. For many sites using that project, those are crucial features.

I have to admit, I've barely looked at the API and internals of the Symfony Mailer project but I know enough that it replaces the entire mail system from core with a completely new API. Even if that is all flawless and bugfree, it is not stable yet (still in alpha and active development per #63) and switching from swiftmailer to symfony mailer if you have a bunch of contrib modules, custom mail hooks, templates, maybe some mail altering to customize some mails is a lot of work. It comes with a BC module, but that too replaces basically the entire core/mailsystem mail handler from what I understand and requires you to replace lots of things (alter hooks, templates, ...).

IMHO, what would be helpful to get those 40k of an unsupported and possibly insecure component, which is essentially a ticking time bomb (there have been severe security issues with sending mails before and there might be again) is to have a straight port of the swiftmailer project to one that uses the Symfony Mailer component under the hood. I'm considering to create that and will have a closer look at what that would exactly look like. Not quite sure yet how to name that and where it should live (number of options, separate project but I could also imagine a 3.x branch of swiftmailer to keep the transition minimal).

Just a FYI for now.

Back to core. @AdamPS, re #56: The problem is that core doesn't really have the luxury of asking the question of whether we should provide BC or not. We have to, unless some external thing that we have no control over prevents us from doing so (which is not the case here). I do agree with you that a BC layer for hook_mail_alter() would be extremely challenging but there isn't really an alternative. That's exactly why I said back in #52 that the one thing that's fairly easy to do and already comes with benefits is to include symfony/mailer and just replace the mail plugin. That said, my patch was just a proof of concept, we do not have to use the outdated transport, we could also add some basic configuration that contrib could provide a UI for as a start, happy to add that to mailsystem, anyone who wants to configure mail sending already uses that anyway), but we can make all of that opt-in and keep existing sites on the current default for now. With no BC concerns whatsoever.

If we do not have a plan that includes a BC layer, then this will most likely not go anywhere and we'll be stuck with hook_mail() forever.

@Berdir, great summary of the current situation. FWIW, a 3.x branch of Swift Mailer seems to make the most sense (to me) given the contrib landscape.

Alternatively:

• symfonymailer
• symfony_mailer_integration

Thanks @Berdir it's always useful to have your knowledgeable comments.

I have to admit, I've barely looked at the API and internals of the Symfony Mailer project

I would value it very much if you did😃. As far as I see it, the core mail API is still mostly stuck in the world of Drupal 7 and 1990s plain text emails. The swiftmailer module is an attempt to bring in modern features as well as possible while being stuck with the old API. The symfony_mailer module is an attempt to create a new API that's fully embracing modern email features and Drupal 9/10 mechanisms. Similar to the transition from Drupal 7 to 8, it takes some work for site owners. However in return it gets a lot of benefits IMO.

Back to core. @AdamPS, re #56: The problem is that core doesn't really have the luxury of asking the question of whether we should provide BC or not. We have to,

As far as I understand, the core BC strategy allows for this (the version numbers being just an example - could be +1 throughout):

• 10.x introduce a new mailer component alongside the existing one - sites can enable either one at their choice
• 11.x deprecate the old one
• 12.x remove the old one

And without that we really are stuck with hook_mail() forever - because so long as we have a BC layer we are still tied to it.

to have a straight port of the swiftmailer project to one that uses the Symfony Mailer component under the hood

Yes I can see that would be useful to many sites. After a period of maintaining and bug fixing swiftmailer, simplenews and raising issues against core, I was determined that things could be better so my personal interest was the new mail system direction.

However it seems a shame to divide the efforts of the Drupal community into 2 quite different projects and directions. Another possibility is to enhance the existing symfony_mailer project with full back-compatibility. I wonder @Berdir if you would be interested in that? For a start there is #3263884: Create a policy element that calls the legacy hook_mail_alter(). However we could go even further - for example there could be a configuration option per module that sends mail whether to be BC or modern:
- legacy hook_mail() and hook_mail_alter(): uses old style templates/config and old style hooks
- legacy hook_mail() only: uses old style templates/config but new style hooks
- new interface(proxy support for modules that don't implement it yet) uses new style templates/config/hooks

In this way we can potentially have the best of all worlds - fully BC, fully maintained libraries, seamless option to modernize at the choice of each site owner.

I said back in #52 that the one thing that's fairly easy to do and already comes with benefits is to include symfony/mailer and just replace the mail plugin.

I agree it's easy, but I don't grasp how it helps anyone. Functionally it seems about the same as PhpMail. What use cases did you have in mind?

Yeah we would need something like #3263884: Create a policy element that calls the legacy hook_mail_alter() for inclusion in core.

If you can send e-mails using the old way, alter e-mails that are currently defined by hook_mail(), then that will keep most things working while allowing us to deprecate hook_mail(), hook_mail_alter() and sending mails defined with hook_mail(). This would need to cover for example moving the user registration e-mail from the old to new system, so that attempts to alter the 'old' mail still work. We can then deprecate all uses of hook_mail_alter() for removal in a major release along with the bc layer.

> As far as I understand, the core BC strategy allows for this (the version numbers being just an example - could be +1 throughout):

You can actually shorten that together. We can introduce the new and deprecate the old in the same major version. But...

The challenge here is that it's not just a specific API doing one thing and replacing it with the new one. Those are the "easy" deprecations. file_create_url() and its related functions as an example. There's even an alter hook there too, but it's a simple data structure that we could keep. (the issue had 250+ comments over 5 years, "easy").

In this case, there are many parts/API surfaces that all play together:
* mail arrays to mail objects
* MailManager::mail() + hook_mail()
* hook_mail_alter()
* The existing mail plugins with mail formatting and sending
* Speaking of formatting (which really is about converting html to text.. or not), I see that symfony/mailer optionally supports https://github.com/thephpleague/html-to-markdown (https://symfony.com/doc/current/mailer.html#text-content), which is yet another library.
* The config/ability to have different plugins/transports based on the mail being sent. Do we want to keep support for that? I'm not sure, but I can see use cases for that, like sending newsletters through a mail service but send password recovery mails directly.

For BC to work, we need to support basically every combination of those in the new and old version
* For new sites using the new mailer, existing modules sending mails need to keep working
* For existing sites that customize their mails through hook_mail_alter(), often to work with a specific mail plugin
* Sites using swiftmailer, smtp, mailjet, and similar modules to send HTML mails, to send them through SMTP and other API's, also when initiated through the new API.
* There's also the mailsystem module, there's no way that's going to keep working, it's way too deep inside the mail api, but I can handle that, we can do corresponding releases there.

A quick search for @Mail plugins finds a about 3 pages of results, lots of commercial services (amazon ses, mailjet, mailgun, mailchimp, and many more), various logging modules, test plugins as well of course. For all of those the D11 "port" (assuming we'd manage this in the D10 cycle) is going to be interesting.

A very rough, not fully thought through idea is that we do tie it to the plugin configuration in system.mail (and mailsystem can do it's thing with that as well). We define a plugin that really is the new API. If that is the one that's configured, for old code that uses the hook_mail() approach, we call it, then build a Mail object from it and pass it to the new service. For the new API, we convert it to an array structure (or implement ArrayAccess?), call the alter hook, then send it. If an old plugin is used, we then pass the same array-representation to the old plugin.

Using the old plugin results in a status report warning. Maybe not initially, so we have time to root out issues before telling existing sites to move over. And then in the next major version, we remove that config and old API.

> I agree it's easy, but I don't grasp how it helps anyone. Functionally it seems about the same as PhpMail. What use cases did you have in mind?

It is now, yes, again it was just a PoC. But we'd just need to add support to configure the Mailer DSN and poof, smtp support in core (for text mails only, but we have to start somewhere). The smtp module could become just a UI to configure that. Or we could put that in mailsystem or even core. And on the other end, we have a reason to put symfony/mailer in core and can start to use and test it. I'm not sure yet if this same plugin should then also be the backwards compatibility layer or if that would be a separate one, but maybe it could be.

About swiftmailer plans I'll reply later.

Thanks @Berdir for a long explanation. I added some answers/clarifications where I could.

I imagined any changes to core as a long-term project. First we could get stable and widely used code in contrib. I saw this issue as a place to discuss type of thing might be welcomed into core. Is that what others are expecting too?

I raised #3315358: Allow full management of override plugins (fairly easy to implement). With that and #3263884: Create a policy element that calls the legacy hook_mail_alter() (more complex, but easier than swiftmailer 3.x) then the Symfony Mailer module is fairly highly back-compatible. I would welcome any feedback on what is then missing if we do these two.

As I see it, @Mail plugins would be replaced by Symfony Mailer transports. Already there is support for 9 3rd party transports. The Drupal community would no longer need to develop/maintain transport code, it can be done in the Symfony community scope. Symfony Mailer module already has a plug-in based mechanism to wrap these using @MailerTransport, presenting a GUI for them to be configured.

Symfony Mailer module replaces mailsystem module. Symfony Mailer allows detailed configuration of "Mailer Policy" with fine granularity, based in @EmailAdjuster plug-ins. This can specify what transport to use (TransportEmailAdjuster), and many other things too: addresses, theme, HTML/plain body, subject, etc. Yes you can send password recovery emails via a different transport, with a different from address, and different CSS-based styling.

But we'd just need to add support to configure the Mailer DSN and poof, smtp support in core

So yes, great idea, even if we just add this much to core, then it's a significant step forwards. The configuration code already exists in Symfony Mailer as I just described.

OK I thought some more and believe I understand better now. The key here is the "ticking time bomb": swiftmailer is unsupported and so thousands of sites are under pressure to change to something. They aren't necessarily ready to test a major change so need something safe, the minimum possible change.

My new idea is #3315541: Create a Mail plug-in that emulates swiftmailer. Instead of replacing the entire mail system (removing mailsystem module), this allows them to selectively replace mails (controlled by mailsystem module). The aim is to copy swiftmailer closely in terms of function/behavior, but switch to modern code, allowing the symfony mailer library to do as much work as possible.

I'm still in favor of keeping the different approaches in a single framework (although it could be a separate project, dependent on symfony_mailer) because:

1. It allows the community to all pull together, sharing of a substantial amount of code, especially transport configuration, and also $message array to Email conversion.
2. It allows sites to have a smooth transition starting from the drop-in replacement then gradually adopting new features. For example they might decide to switch user emails to HTML, but still keep other emails like before.

Comments very welcome.

The key here is the "ticking time bomb": swiftmailer is unsupported and so thousands of sites are under pressure to change to something. They aren't necessarily ready to test a major change so need something safe, the minimum possible change.

IMO, ^^ is 100% correct. And, given the available options, I think #3315541: Create a Mail plug-in that emulates swiftmailer is the best route to take.

For clarify, does this issue need to be moved to the "Ideas" issue queue so that the decision to officially "adopt" this "child" can be made? If not, should this issue be a 'feature request' or a 'plan'? Seems like in order for it to be a 'plan' it must first become the community agreed upon path forward.

For clarify, does this issue need to be moved to the "Ideas" issue queue so that the decision to officially "adopt" this "child" can be made?

Several people expressed their support for #31 (e.g., @alexpott in #41 and ff). Also #51 already argued against turning this into an idea. Plan is the correct category for this umbrella issue.

The logical first step is to get #3165762: Add symfony/mailer into core in. That issue has a small enough scope to actually have a chance to succeed.

Drupal Symfony Mailer module now has a stable release with 8k sites using it. Compatibility with swiftmailer is quite good. The "ticking time-bomb" has substantially been prevented. In the end we chose a different approach and #3315541: Create a Mail plug-in that emulates swiftmailer was not implemented.

This has given a reasonable insight into how to integrate Symfony Mailer into Drupal. However of course that module is complex and so not in itself a logical first step to be added to Core.

The logical first step is to get #3165762: Add symfony/mailer into core in. That issue has a small enough scope to actually have a chance to succeed.

I agree that we need to make a first step. However It feels like we're trying to take a first step without agreeing where we're heading😃. This issue has category = plan. I suggest that the first thing we need to do here is write a plan with a list of goals, and a sequence of steps to achieve them.

It feels that some work is needed to show that #3165762 is the right first step:

1. State a clear purpose (the IS is mostly empty).
2. Work out how we would later add new function. Currently the sending code cannot access the Email object, there is no way to configure the transport, there is no handling of HTML emails.
3. Check compatibility. The current patch locks all sites using CoreRecommended to a specific version of symfony/mailer which might cause a conflict with other contrib modules or break existing sites.
4. Allow for further development. We don't want to promise BC which might make it difficult to add new function. This could be resolved by marking it as experimental.
5. Review details. The code has likely been simplified rather too much - in comparison swiftmailer is much bigger. E.g. splitting a list of emails by php explode fails if the display name contains a comma.

I added a proposal for a plan.

Thanks @AdamPS.

I got the chance to moderate a core conversation session recently at Drupal Dev Days in Vienna (recording, slides).

The discussion touched the following topics:

Why introduce the symfony mailer component in the first place?

• Important mail transports out-of-the box (SMTP, etc.).
• Maintained in symfony repository.

What is the procedure of introducing new dependencies?

New dependencies can be introduced in minor versions. Breaking changes (remove old APIs) are only possible in a major release.

How to suppress delivery during development or force redirection in test environments?

• The NullTransport can be used to disable delivery (DSN: null://null).
• Headers can be modified in a subscriber to the MessageEvent.

How to provide backwards compatibility for hook_mail_alter?

This was probably the most complex topic discussed. It became apparent, that converting message arrays (as required by hook_mail_alter) to Email objects and vice versa would be quite a challenge. Mainly because message arrays are extended by contrib and custom code with keys and values core doesn't have any knowledge about. During the discussion, two approaches to this problem were examined:

1. Core starts to restrict the set of key-value pairs which may be present in the message array by deprecating certain combinations. In order to make that work, it would be necessary to provide new APIs for those use cases before.
2. Core introduces the symfony/mailer component, but without switching right away its own mails to the new API. From that point in time contrib modules implementing hook_mail_alter should be extended to additionally provide a MessageEvent subscriber mirroring the logic in their hook_mail_alter() implementations. After contrib and custom modules had some time to adapt, core will switch its own transactional mail to the new API and deprecates hook_mail and hook_mail_alter.

Multiple participants favored the second options.

How to pass context to MessageEvent subscribers?

Problem: Implementations of hook_mail_alter can access the id, module and key properties in order to check where a mail came from. But there doesn't seem to be a way to pass context via the MessageEvent.

• Symfony seems to resolve that with custom headers on the mail message (e.g. X-Transport for transport selection).
• Another way to make messages distinctive would be to leverage the PHP type system. Modules would subclass the Email class for each transactional mail. However, requiring subclasses which do not implement any logic would be a bit pointless.

How a message is built in the absence of hook_mail?

Pass an Email subclass which accepts a render array into the mailer. The symfony mailer MessageListener then calls a Drupal specific implementation of the BodyRendererInterface which turns the render array into markup.

This way the full power of the Drupal theming system can be leveraged to build, extend and alter the body of mails.

Could this be built in contrib and ported into core once its finished?

Contrib already provides a big selection of mail related plugins. Getting a contrib module into core has proven to be a big effort in the past. We have the tools in place to iteratively develop things in core in manageable steps.

Would core need to provide support for multiple transports

I.e., is there a use case for mails from one module sent via SMTP, all other mails sent via another transport)?

Very likely core doesn't need to cater for that case.

> How to pass context to MessageEvent subscribers?

I also proposed to have a standard subclass that essentially keeps the concept of module/key identification by enforcing it through a create/factory method. A service factory to get an e-mail object could also allow contrib/custom to replace it with a different class with additional functionality if there are use cases for that.

> Would core need to provide support for multiple transports

As discussed, actually different real transports seem like a very unusual use case. There are some interesting cases on top of that, I saw that there's an async option, where the actual sending (and rendering) is delayed, that could be interesting for simplenews and other bulk e-mail use cases. As well as rerouting and so on. But more than happy to let contrib explore on this, we can always add more features to core later on if they are widely useful.

> How to provide backwards compatibility for hook_mail_alter?

I still have plenty of questions about the second option, but we can hopefully figure that out as we go. The next steps are clear, defining and getting a new API into core and using it at least for tests that don't need to provide BC. One idea on top of the second option that I just had that could be very interesting is to introduce a setting. We've done this in the past for behavior changes. Basically you'd opt in to core using the new mailer API or not. Contrib is free to respect that or not, though they are of course welcome to do so as it would help sites transition to the new API. The setting could at first be opt-in and experimental and at some point we could switch it to on by default for new installs, deprecate the old setting and eventually remove the setting together with the old API.

Sure, would be a bit of work to maintain both implementations at once, but core only has a handful of mail implementations, that seems quite manageable.

> How to provide backwards compatibility for hook_mail_alter?
In Drupal Symfony Mailer, we do option 2. We provide a controlled migration process from the old system to the new, allowing some modules on old and some on new. The old system calls hook_mail_alter(), the new system calls the new hooks. There is no hook BC - that seems hard/impossible.

> How to pass context to MessageEvent subscribers?
In Drupal Symfony Mailer, we have extended the Email class, adding $type and $subType equivalent to module and key in the old API.

However we don't use MessageEvent instead we created something else. Drupal\symfony_mailer\Email extends \Symfony\Component\Mime\Email using the decorater pattern - it has a protected $inner; of type \Symfony\Component\Mime\Email. This was done to allow changes to make things more Drupal like. For example the original class has function subject(string $subject) which forces translation of any string potentially before the correct language has been set. Also the original class limits extension with private members, Drupal names set-accessors setSubject() rather than subject(), it allowed us to create an interface, etc.

> How a message is built in the absence of hook_mail?
In Drupal Symfony Mailer, we have @EmailBuilder plug-ins.

> Could this be built in contrib and ported into core once its finished?
It already exists in contrib in Drupal Symfony Mailer😃. We can use that as a reference as we build up function in Core step-by-step.

The migration path from old system to new is quite complex. We need to migrate at least 4 things:

1. email builders: hook_mail()
2. email altering: hook_mail_alter()
3. email config/forms: user.mail.yml etc: allow HTML body and all the other possibilities of a modern email interface
4. email transports: implementations of Drupal\Core\Mail\MailInterface

The only solution I can see is a phased transition, where the two mail systems exist in parallel in the code. First the new system is experimental, then it becomes stable, then the old is deprecated, then the old is removed.

Core modules would provide both old and new mail systems during the transition. Contrib modules could do the same, or switch from old to new with a major version change.

In Drupal Symfony Mailer we have MailManagerReplacement which implement the old API and converts to the new (calling the old hooks). This allows removal of the old MailManager and old transports whilst continuing to support modules using the old building/altering code.

We can have a config setting for which mailsystem to use overall, and another for each module (for those that provide both old and new email building). Site builders can first switch mailsystem/transports, get that working then switch modules one-by-one or all at once depending on the stability of each one in each environment.

This issue is starting to veer into #1346036: Add an alternative to hook_mail() and deprecate it, where I posted a detailed plan for deprecating hook_mail().

Key design summary of Drupal Symfony Mailer.

Naturally core will want to simplify, but still most of these classes exist for a really good reason😃. As a minimum I would expect to need Mailer, Email, EmailBuilder.

This issue is starting to veer into #1346036: Add an alternative to hook_mail() and deprecate it, where I posted a detailed plan for deprecating hook_mail().

It's true, both issues will add an alternative to hook_mail() and deprecate it. I looked at your module as part of writing Drupal Symfony Mailer, and I like the ideas there.

The key is that this issue has some huge advantages: we totally get rid of our own mail system in favour of a high-quality, well-supported, third party one which has loads of useful features especially HTML mail and a wide range of transports out-of-the-box. For all this, people are willing to undertake the effort of migrating the entire mail ecosystem. The community seems to have enthusiasm to complete this issue.

To make it work, we need to closely match what symfony does.

• Email is a class to match symfony rather than being a config entity (also a class is good performance for modules like simplenews that can send 50k emails in one go). This automatically gives us body, subject, address and many things.
• However, we can extend the symfony Email class to add tokens, languages and many other Drupal things. Drupal Symfony Mailer has method to render a content entity, which picks up the idea you had of mail_message entities.
• Email transports (back ends) are already defined by symfony. We can extend them with a config GUI (maybe in contrib?)

I'm maintainer of Drupal Symfony Mailer, and it's a good template for what can happen here. I'm sharing what I've learnt. I expect that some of it will be taken into core, some not. You are maintainer of Mail. It's less close to what's happening here, but still many of the ideas can be adapted, so you can share ideas too.

> The migration path from old system to new is quite complex. We need to migrate at least 4 things:

IMHO the main challenge is hook_mail_alter() and the behaviour of the system as a whole which is more or less the combination of all 3-4 points customized to a specific site. The hook_mail() implementations themself can change on its own whenever they want to convert to the new system. But hook_mail_alter() is unpredictable, you will have custom and contrib implementations, for example we alter user mails to a set a format so they can be sent as HTML with swiftmailer. Once user module switches to the new API, that just stops working.

3. is IMHO a separate system on top, see also below.

4. Those modules will then get left behind or will have to reimplement themself, depending on how much we get into core. For example, the smtp module could just be a UI to configure the symfony smtp transport, or we have a new flexible UI in core or a new contrib module that supports many different transports.

I still think my idea in #83 might work, basically a feature flag, and each site can decide on its own, at least for core and collaborating contrib modules, when they switch to the new system. Once the flag is switched, all modules that respect the feature flag will switch over to the new API, any that don't will either still use hook_mail() (which still works until we rip it out in D12 or whatever) or will already start to use the new API whenever they feel like it.

Fully transparent BC seems pretty much impossible, especially for 4, in the end, either you use the old mail API to send the mail or the new, there's no both.

> This issue is starting to veer into #1346036: Add an alternative to hook_mail() and deprecate it, where I posted a detailed plan for deprecating hook_mail().

I think that was always the scope here. In my opinion, I don't think we need to have/enforce the concept of a mail builder aka hook_mail(), in a way, the scope of this issue in my opinion is to replace the current mail plugins and just drop the concept of hook_mail(). Your module could then just rewrite EntityMail::mail(), and instead of bypassing MailManager and calling the mail plugin directly, you'd create an Email object (either the one from Symfony or ours, to be defined). Everything else would stay as-is.

IMHO, mail builders/config entities to manage mails are useful many cases of e-mails where e-mails should be configurable things, user mails are the perfect example for that. But there are also many cases that don't fit into that, from very basic one-off custom e-mails to things like Simplenews on the other end of the spectrum, which already has its own email management.

In short, that other issue could, instead of trying to replace hook_mail(), become a new, optional API on top of the new mail API that you can use if it fits your use cases (configurable/translatable/extensible mails).

I would say technically, we don't need to guarantee that any particular e-mail goes into hook_mail_alter(), e.g.:

10.1.x - user registration e-mails are in hook_mail
10.2.x - user registration e-mail has moved to the new system and no longer appears in hook_mail_alter().

If you're specifically altering the user registration e-mail, then yes this is going to break your module, but that's the same for form alter, preprocess etc. which we allow changes for here: https://www.drupal.org/about/core/policies/core-change-policies/frontend.... There used to be a more generic section about alter hooks in the main bc policy but seems it got moved to the front end one, will look at adding one back.

If we convert core e-mails one by one over time without a feature flag, we're not changing the actual API of hook_mail_alter(), it just won't get that particular e-mail any more. It's like if we changed an entity-specific form to a generic entity-wide one and the form ID changed, the alter wouldn't fire for that either.

I'm not sure how a feature flag would work with contrib. Let's say core maintains both hook_mail() and new versions of every e-mail, and 15 contrib modules also maintain both hook_mail() and new versions of every e-mail. And 10 modules implement both hook_mail_alter() and whatever the new alter system is for a few different e-mails (or to consistently alter every e-mail). But some contrib modules will only have hook_mail() e-mails, and some contrib modules will only have new-style e-mails - what happens with those?

Feels like we're talking about something similar to #3371619: Implement utility method for invoking backward compatible code except with a feature flag rather than by version, and then all the 'well behaved' modules will send new-style or old-style based on the flag and everything else uses only new or old.

I'm not sure if 'everyone has to maintain two versions of an e-mail and site owners have to decide which one they prefer' is going to be less work for everyone than 'contrib and custom modules have to check their hook_mail_alter() implementations every six months in case an e-mail has been converted.

However I don't maintain any contrib modules really, so pinch of salt and everything.

> Email transports (back ends) are already defined by symfony. We can extend them with a config GUI (maybe in contrib?)

> IMHO, mail builders/config entities to manage mails are useful many cases of e-mails where e-mails should be configurable things, user mails are the perfect example for that. But there are also many cases that don't fit into that, from very basic one-off custom e-mails to things like Simplenews on the other end of the spectrum, which already has its own email management.

Agreed that not every module that sends mails needs config entities - it would be overkill in cases like simplenews and webform notifications.

But I really think the config UI should be in core - core needs it for User module emails for one thing. If we don't provide it in core, contrib will reinvent that wheel over and over again. And as demonstrated by Mail module, making config and a UI for this is fairly straight-forward.

Regarding #90, so long as change notices are written that document the changes necessary for contrib code and custom code, I think that would be completely fine.

> I would say technically, we don't need to guarantee that any particular e-mail goes into hook_mail_alter(), e.g.:

I was going to say hook_mail_alter() is more about custom stuff, like converting user mails to HTML, but there are actually quite a few contrib implementations, there's modules like https://www.drupal.org/project/bcc for example.

The feature flag idea isn't just about the alter hook though. Many sites today use modules like smtp, swiftmailer and so on to send HTML through SMTP and other transports. If we change user mails in 10.x to be sent through the new API, they will likely no longer be sent at all. This is possibly not something that sites think of when testing. Is that really OK? I'm not sure.

> But some contrib modules will only have hook_mail() e-mails, and some contrib modules will only have new-style e-mails - what happens with those?

We can't force contrib to to follow the pattern, but we can do it for core and we can provide a change record that does it in the same way. I would expect quite a few contrib modules will follow that, especially big ones that send a lot of critical mails like commerce and webform. Yes, it's similar to the issue you mentioned, it's also not just a feature flag but also allows for core BC, if a feature flag/config key isn't active, it might also be an older core version, and then we can fall back to hook_mail. Especially as I expect this will not get in with 10.3 but 10.3 or 10.4 even?

Realistically, most contrib won't immediately switch over, many smaller ones struggle to have the changes committed in time for a new major version, not too concerned about that. So when core starts to switch over with a feature flag and for example has a requirements message about this but for now continues to work as-is then both sites and contrib maintainers gain some time. Sites will need a replacement/new version of modules like smtp/swiftmailer that allow to configure the required transports, that will need to exist first. (As maintainer of mailsystem, I'm open to getting in patches that at least add some basic UI options, whether or not mailsystem will continue to exist or not, possibly not, but there are some features that like theme selection which might or might not get into core).

> But I really think the config UI should be in core - core needs it for User module emails for one thing. If we don't provide it in core, contrib will reinvent that wheel over and over again. And as demonstrated by Mail module, making config and a UI for this is fairly straight-forward.

I agree with that, having something generic for what user module does could be useful in core, I definitely built very similar UI's in custom modules too. But I do think that it can easily be a separate issue once the new API is in place and then build on top of that. We shouldn't have to touch the configuration and UI of user mail configuration for the low-level API transformation, I'd expect we just convert _user_mail_notify() + hook_mail() into a new service, with the feature switch.

What I would suggest as next step is then an implementation issue to starts to define the mailer service(s) and APIs and we can start to implement at least a few core implementations for example user and contact. Then we can also look at the changes and discuss just how much standardized builder API (aka hook_mail()) we need or if we're ok with one-off services for that in those modules.

We mention swiftmailer, but #3359916: Conflicts with drupal/core-recommended 10.1.x-dev requires egulias/email-validator ~4.0.1 is kinda requiring an upgrade to symfony_mailer as of Drupal 10.1. Does that change any of this discussion?

Opened #3379794: Add symfony mailer transports to Dependency Injection Container which demonstrates how I think we should integrate the transport layer (i.e, mail delivery, nothing about mail building yet).

> We mention swiftmailer, but #3359916: Conflicts with drupal/core-recommended 10.1.x-dev requires egulias/email-validator ~4.0.1 is kinda requiring an upgrade to symfony_mailer as of Drupal 10.1. Does that change any of this discussion?

swiftmailer was just an example for one of the various mail plugin modules that are currently used. FWIW, the conflict is only with drupal/core-recommended, not drupal/core.

> Fully transparent BC seems pretty much impossible, especially for 4, in the end, either you use the old mail API to send the mail or the new, there's no both.
>I'm not sure how a feature flag would work with contrib.

Drupal Symfony Mailer module provides a lot of flexibility for handling the transition with good BC:

• Global switch for which mailsystem to use (enable Drupal Symfony Mailer = use the new system). We have a MailManagerReplacement mapper that allows the new mailsystem to support the old API, including hook_mail_alter(). So even without any changes to contrib modules, you can switch the global switch. The transports change, which may require action from site-builders (however automatic migration is a possibility). The actual emails should stay exactly the same, giving full BC to modules and hooks.
• Modules may choose to support both APIs (core does, contrib can choose). Switching API may change email configuration and contents if the module took the chance to adopt a new UI or new features such as HTML support. In this case we should have a per-module switch for which API to use, to allows site-builders to choose when to change (given that a contrib module might introduce support for the new API in a patch version).
• OR Contrib modules may create a new major version that adds support for the new API and drops support for the old. In this case, the new major version should have a hook_requirements() that the new mailsystem is enabled. Site-builders would upgrade to the new major version at the same time as switching on the new mailsystem.
• An easy option to provide the global switch is to put the new mailsystem in a module, which could be experimental at first. Contrib module releases that require the new mailsystem can simply add a dependency on the new module.
• We can deprecate and remove the old mailsystem, whilst keeping legacy support for the old API for an extra Drupal major version to ease transition.

> IMHO, mail builders/config entities to manage mails are useful many cases of e-mails where e-mails should be configurable things, user mails are the perfect example for that. But there are also many cases that don't fit into that, from very basic one-off custom e-mails...
>Then we can also look at the changes and discuss just how much standardized builder API (aka hook_mail()) we need or if we're ok with one-off services for that in those modules.

The MailerBuilder plug-ins in Drupal Symfony Mailer seem to work well, for example:

• Call the building code in a new render context to avoid leakage. We can write code to build a mail directly with causing this problem provided we do it carefully, leaving all rendering, token replacement, etc to occur inside the mailer.
• Encapsulate the code - a site builder can swap the implementation using a hook.
• Plug-in annotation provides metadata, e.g. human-readable names for different mail-keys for use the config UI.
• Saves creating an extra service for each module that sends emails: _user_mail_notify('password_reset', $account) becomes simply \Drupal::mailer()->mail('user', 'password_reset', ['user' => $account]) or $this->mailer().

However I agree they shouldn't be compulsory.

> Simplenews on the other end of the spectrum, which already has its own email management.

With Drupal Symfony Mailer, this custom email management becomes redundant. See #3275129: Use Symfony Mailer API to send mails which removes 2000 lines of code. Similarly commerce has invented its own system (less complex) that could be removed.

> But I really think the config UI should be in core - core needs it for User module emails for one thing. If we don't provide it in core, contrib will reinvent that wheel over and over again. And as demonstrated by Mail module, making config and a UI for this is fairly straight-forward.

Drupal Symfony Mailer also has a config UI which supports approx 20 fields (not just body/subject) which can be set for specific mails or all. Once we have the base mail system in place then we can look at options for the config UI.

> What I would suggest as next step is then an implementation issue to starts to define the mailer service(s) and APIs and we can start to implement at least a few core implementations for example user and contact.

I have created #3380041: Create new mailer service based on symfony. I'm willing to write an initial patch. However first can people provide feedback and give opinion on two questions that I asked.

I created two more issues #3380269: Create a plug-in system to build emails with Symfony Mailer and #3380273: Create a system for configurable emails in Symfony Mailer.

Feedback and comments very welcome please. I've written them based on "Drupal Symfony Mailer" which has been proven - but of course it's just one possible way of doing things. Any other ideas very welcome it's all open for debate.

I raised 3 more issues: #3380470: Migration strategy for moving to Symfony Mailer, #3380472: Create a BC mapper for MailManager to convert to Symfony Mailer, #3380476: Change core modules to use Symfony Mailer. I'll wait a week or two now to allow people to comment.

Don't know if this is relevant but I want to make you aware that Devel module offers a file mail catcher, see https://gitlab.com/drupalspoons/devel/-/blob/5.x/src/Plugin/Mail/DevelMa.... As it's a popular module we should be careful not to break its functionality

> Don't know if this is relevant but I want to make you aware that Devel module offers a file mail catcher, see https://gitlab.com/drupalspoons/devel/-/blob/5.x/src/Plugin/Mail/DevelMa.... As it's a popular module we should be careful not to break its functionality

Depends on your definition of "breaking". It will not break, it will simply cease to be used, like any other mail plugin using the current mail plugin, the whole concept/plugin system will be deprecated and eventually removed.

_Many_ modules in their current form will either no longer be needed or will need to completely rethink their approach and just become a UI for symfony mail transports for example. And any site that doesn't use the default mail plugin will need to make some adjustments, but we're doing our best to make this as smooth as possible.

In regards to the issues, I'm unsure how much we can split it off and in what form. As there's no guarantee when issues will land, it would for example be pretty weird if the transports are in but nothing is using them. So both from that perspective and also to be able to review and discuss the impact, we might need to split vertically with a minimal feature set instead of horizontally by layer. For example, #3380041: Create new mailer service based on symfony proposes to add a ton of features to the mail service, like theme switching. Which I find super important and a critical feature of the mailsystem module, But it's also something that both doesn't exist yet in core and is hard to test and review without anything using it.

So IMHO an approach where we do a minimal implementation of the mailer, just as much as we need to use it for the current use cases in core and start to convert them so that we can see how the whole thing will work for example for user module. At first without a BC layer. We can still try to keep different systems/layers and the actual module conversions in separate commits in a merge request with a bit of git rebase -i juggling, Then when we're happy with how the conversions look, we can get the mailer/transport and whatever other classes we need in in their primary issues.

On experimental module vs core I'm unsure. Core would be nicer of course but we would need to figure out how to define an experimental API in the Drupal\Core namespace, I think we haven't done that yet, would probably require input from framework/release maintainers. An experimental module would eventually require a move and deprecation.

(...) will need to completely rethink their approach and just become a UI for symfony mail transports

Trying anticipate that, I've proposed upstream a file mailer transport https://github.com/symfony/symfony/pull/50962, unfortunately rejected.

Trying anticipate that, I've proposed upstream a file mailer transport https://github.com/symfony/symfony/pull/50962, unfortunately rejected.

Shame, but presumably it can live somewhere else?

@Berdir - yes I see what you mean, it does make sense. So we could make a prototype working system as a vertical split of the existing issues just taking the minimum from each for a working system.

Do you imagine we would actually commit this prototype?

==

Key question from #3380041: Create new mailer service based on symfony - do you have any thoughts?

Should the new class Email extend the base symfony one, or decorate it (meaning have a class variable $inner containing the base class)?

• Advantage extend: we can pass our class everywhere where the base class is expected.
• Advantage decorate: we can make a clean start, renaming/removing functions or changing their parameters.

OK, I posted a patch on #3380476: Change core modules to use Symfony Mailer.

It may seem strange to have this without any patch for the underlying services/interfaces. However I feel it gives a pretty good idea how the whole system will look, and it's easy to deduce how the interfaces will be, especially as I wrote really detailed comments.

Key point: I am assuming a new mailer service in Core, with capabilities to send via the old mail service.
Advantage: We can make a clean switch in the core modules to call the new API only. We discard the code that calls the old API and avoid the need for an if (new_mailer) statement for every bit of code that sends a mail.

Then I propose a new issue "Create a new mailer service/API ready for symfony, but still send using the old mail service", which would be a fairly small amount of simple code. Together with the above:

• Core modules use the new API
• with calls to exactly the same hook_mail() and hook_mail_alter()
• generating exactly the same mails
• meaning the existing tests all pass without any changes
• so these two could be committed without relying on any other issues.

I can also make a usable prototype in contrib (perhaps v2 of the existing Drupal Symfony Mailer) that replaces the mailer service with one that allows the above code to actually send via Symfony Mailer.

I welcome comments please:

• Comment here about the proposed strategy
• Comment in #3380476: Change core modules to use Symfony Mailer (can use GitLab) about the details of the API and code changes

> Should the new class Email extend the base symfony one, or decorate it (meaning have a class variable $inner containing the base class)?

I don't know. I was hoping that we would be able to stick to using the symfony component as directly as possible without a lot of Drupalism on top, but you made a few good points about problems with that like the strict types that conflict with our translation system.

I think what I'd like to see is that we try to work directly with Symfony as much as possible first and then look at the result and introduce APIs to improve that where we think it's worth doing. Your work in #3380476: Change core modules to use Symfony Mailer is interesting, but it's not really a vertical split based on what we have, it's more the icing on a multi-layer cake :)

One thing to keep in mind with BC is not just the current state and transition to that new system but also the future. The downside of working directly with certain symfony components like the serializer is that those were some of the most annoying BC issues between D9 and D10, but that was partially because we skipped a major version. If we decorate/use adapters, then we can deal with symfony API changes for example in the e-mail class most likely in nicer ways.

you made a few good points about problems with that like the strict types that conflict with our translation system.

We've tried to persuade Symfony to use Stringable type hints instead of string for validator, but so far that's been rejected. #3255245: [Symfony 6] Revert 3231603 to use our own TranslatorInterface and especially https://github.com/symfony/symfony/pull/44911 have details. I'm all in favour of them getting even more feedback that preventing the use of Stringable is an unreasonable API limitation so maybe we can have another push via the mail system. However given how that went last time, decorating seems a good way to deal with unreasonable API limitations and future changes that they introduce.

Great thanks.

it's not really a vertical split based on what we have

True. As I understand it, your vertical split is a working system for one module. The difficulty is that the only issue that is module-specific is #3380476: Change core modules to use Symfony Mailer so it would mean writing nearly everything. So what I've tried to do is create a strategy that is

1) Easy to commit (relatively speaking) - two manageable size issues together make a significant step forward in a fully-BC way, with the chance to test the future strategy works using contrib.
2) Meaningful to review - that one issue already gives a good idea of the way forward and people can see if they like it.

decorating seems a good way

Also we do decorating already in contrib so it seems like the way to go.

I've updated some of the child issues in response to feedback so far. This leads to simplication and I've added a lot more detailed explanation. Further comments very welcome.

Next step I will aim to create a minimal usable patch as described in #109 - except I can do what Berdir suggests and include only one module, likely user. I created #3385025: Minimal usable Symfony Mailer prototype.

First step done!! #3165762: Add symfony/mailer into core = Fixed

Next step: #3379794: Add symfony mailer transports to Dependency Injection Container

Added 2 new issues to stage 1.

My current idea is to implement features for this initiative if possible using the stage 1 @Mail plug-in. These issues are each fairly simple and self-contained, hopefully easy to commit. At the same time, we can create new releases of Contrib Drupal Symfony Mailer that remove the matching function and use Core instead.

This makes stage 2 more approachable - otherwise it seems like a huge block of code chunks that all depend on each other.

There are ways in which I feel I can help this initiative:

• I've developed some specialism in Drupal mail. I identified an area that could be improved and did something about it. Drupal Symfony Mailer module can be a template to help add similar features to Core.
• @znerol created some really nice quality patches, and I reviewed them based on what I Iearnt from developing Contrib.

However when it comes to this META issue, I don't feel I'm succeeding to help. It seems that the majority of comments I make and issues I raise meet an objection from @Berdir😃. I'm only on the fringes of the Drupal community and I'll freely admit that I'm not the best one to predict what Core would like. Maybe someone else would like to step in and take the initiative? There's code from DSM, some patches on these issues that I already attempted. Plus I'm still keen to keep an eye on things with reviews based on what I already learnt.

In any case, looking after the mailer in Contrib is already keeping me quite busy, and I can continue to work on it to adapt to what happens in Core.

It really wasn't my intention to push you away, I'm sorry about that. I was just trying to help guide it in a direction that I personally think has better chances of getting into core in a reasonable timeframe. And sometimes it might just not match what my idea was, but I think #3395814: Framework for testing Symfony Emails shows that we are hopefully on the way to common ground on at least some of these issues :)

Don't worry it's OK - my comment was also a knee-jerk reaction sorry.

However I think there is a good underlying point: it would be great to have a clear direction from Drupal Core "product management" for what the Core committers would accept. Or perhaps we need a team of people willing to work on this initiative with a range of different backgrounds and skills?? Perhaps that team should include a Core committer?

IMHO Stage 1 is succeeding. There is a step-by-step sequence of things we can add to Core that are each fairly small and self-contained. We can then adjust Contrib (DSM, DSM-L and new Mailer Transports module) to match - using what's in Core, and extending it. I guess that's partly why I'm attracted to add more things to this stage, but I'm certainly not fixed on that approach.

Stage 2 is harder for me to see how to move forward. It's a whole new mail system, which while not huge, feels to big for a single issue. Then if we split it, how does half a mail system make any sense? I proposed #3385025: Minimal usable Symfony Mailer prototype, but I feel it's more of a prototype that something we'd actually commit. Comment #109 describes the minimum thing I can see that makes a useable system - two issues, each quite big but not huge, they feel possible to commit.

My time is already quite full with Contrib DSM. Another way we could move forward is to get some key people from Core to become advisors/reviews for that, and we can move it in the direction that would be attractive to Core. Contrib is much more flexible than Core, so it might be easier to get the shape mostly right there. Then we can come back to offer Core something that's already close to being acceptable, and with validated on 1000s of live sites.

Agreed, stage 2 is quite hard. And I think we agree on #3379794: Add symfony mailer transports to Dependency Injection Container being the next main step, but that too is fairly complex already, don't feel like I can contribute much there, no clear opinion for once ;)

I pinged @longwave in Slack and he'll bring it up with the other committers as well. I can imagine that @alexpott or @catch might have opinions on the DIC issue.

#3385025: Minimal usable Symfony Mailer prototype does seem like a good place to then experiment on stage 2, I'll comment a bit more there.

Discussed the above with @alexpott, @larowlan, @catch and @lauriii. We previously agreed that the mail system does indeed need a complete overhaul, and from prior discussions nobody can see a way of doing this with full backward compatibility. We are generally excited about the current approach and the progress made so far. However, we noted that to date the problem has been approached from a mostly technical point of view, and we should first of all take a step back and define our goals here.

From a product perspective, we have questions such as:

• What email features do users expect Drupal to have?
• What features do our competitors provide?
• What features do contrib modules provide that we should move into core?
• What features should remain in contrib?

Tagging for product manager review to try and define where we should be going with this. It seems pretty clear to me that one goal should be "HTML emails out of the box" as we are providing a complete rewrite of the mail system, but other than that I'm not 100% sure on decisions we have made from a product point of view.

I think support for file attachments would be good as well, if that's not covered by "HTML emails out of the box".

Thanks @longwave.

It seems pretty clear to me that one goal should be "HTML emails out of the box" as we are providing a complete rewrite of the mail system, but other than that I'm not 100% sure on decisions we have made from a product point of view.

Mostly so far we've not made product decisions. We've made a 99.9% certain technical decision is to use Symfony Mailer. Technically it seems like a very strong choice, and I don't see how it would limit our choices for product features. Symfony includes HTML, attachments and many other things. Even if we want a feature it doesn't have, we should be able to add it.

In Contrib there is Drupal Symfony Mailer. It provides two big features that don't necessarily belong in Core:

• "Mailer Transport" GUI for selecting and configure the symfony transport (including a page to send a test email)
• "Mailer Policy" GUI - fully flexible configurable mails: body, subject, addresses, headers, etc.

Some questions it would be useful to answer:

1. The rough consensus of comments so far is that the 2 above GUIs should remain in Contrib. It would be useful to have clarification of that.
2. There will be a lot of detailed features questions, for example should Core support setting the theme used for sending emails? (Currently no - it's done in mailsystem.) What would be the mechanism to answer these?
3. On the boundary of product management and technical there will be some "architecture" type questions such as: How will the new mail system use templates, themes/libraries, tokens, etc? What would be the mechanism to answer these?

It seems pretty clear to me that one goal should be "HTML emails out of the box" as we are providing a complete rewrite of the mail system,

Just to connect some dots, I'd like to mention here that I read a different opinion with concerns explained from @berdir in #3392879-5: Add support for HTML email to Symfony Mailer plug-in and #3392879-7: Add support for HTML email to Symfony Mailer plug-in

Just to connect some dots, I'd like to mention here that I read a different opinion with concerns explained from @berdir

I believe the concerns from @Berdir are about whether to support HTML in the "Symfony Mailer plug-in", meaning the @Mail plug-in which was created in #3165762: Add symfony/mailer into core. I'm fairly sure that we all want to support HTML in the new mail system API which will eventually be created in #3380041: Create new mailer service based on symfony.

ahh, mindblowing... sorry for mixing things. Increasing my followed Symfony Mailer-related issues with one more... =]

The Needs Review Queue Bot tested this issue.

While you are making the above changes, we recommend that you convert this patch to a merge request. Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

bot was a bit too eager, sorry for the noise