Problem/Motivation

I've never been fully satisfied with Feeds, partly because I just don't think the concept of a "feed" completely applies to a mailbox. A feed implies an ordered, unilateral (one-way) read operation between the feed and the Importer. But a mailbox is inherently bilateral- we have to write back to mark emails as read, etc... This makes some types of error-handling almost impossible to accomplish with Feeds.

The complexity of Feeds gives it an unforgivingly steep learning curve, which I think turns a lot of people off from Mailhandler 2.x, but the rigidity makes it hard to break out of the "fetch->parse->process" workflow for any reasonably interesting edge case.

Proposed resolution

Use Rules instead of Feeds. Develop a proof-of-concept in D7, then migrate it to D8. I don't want to support too many major branches simultaneously, so the plan would be to maintain 6.x-2.x and 7.x-2.x for the foreseeable future, work on developing 7.x-3.x/8.x-1.x (Rules-based branches) during that time, and when D8 lands stop supporting 6.x-2.x and only support 7.x-2.x (Feeds-based) / 8.x-1.x (Rules-based).

Remaining tasks

Tasks required for proof-of-concept:

  • Start a 7.x-3.x branch to develop a proof-of-concept
  • Implement Mailbox configurations and retrieval plugins just like in 7.x-2.x, as well as permissions to administer mailboxes
  • Create Mail entities that represent imported emails. The question here is should all reasonable headers be broken out into fields (From, To, Message ID...) or should there just be a single Headers field, and command plugins can provide tokens for from/to/etc based on the headers. I really like making things granular, but the Token-based solution sounds better since we'll already be providing tokens for User ID.
  • Write importer code (mailhandler_collect_mail()) to turn new emails into mail entities on cron.
  • Write code to provide tokens for authenticated User ID (From address) and other common headers
  • Create a quick-start set of rules to turn new emails into articles.

Big issues still to consider:

  • How to implement command plugins
    • Probably just like they are now- text boxes on Rules that define available plugins. Problems with this: is this even possible, and how can you turn these entries into tokens for use on the same Rules? How can you define default values at import time?
  • How to accomplish workflow proposed in original report

Other features currently provided by Mailhandler that need to be reimplemented in 8.x-1.x:

  • Help pages
  • Filters plugins (distinguishing comments and nodes)
  • Input Filters
  • Authentication via Tokenauth and Multiple Email

User interface changes

Example import flow:

  1. hook_cron or Rules scheduler fires mailhandler_collect_mail()
  2. mailhandler_collect_mail() checks all mailboxes, and turns any new mail into Mail entities. This will be accomplished using the existing Mailbox configurations and retrieval plugins. This process should be pretty "dumb" - Mail entities should match the original emails as closely as possible, with a bare minimum of processing/sanitization. Mail entities will have the following fields:
    • Headers (To/From/Message ID/etc...
    • Body (HTML)
    • Body (Plain text)
    • Attachments
  3. A trigger "After mail save" allows Rules to act on new Mail entities. Authenticate plugins provide Tokens for Authenticated User ID (From address), Authenticated User ID (Tokenauth), etc...
  4. A Rule creates a new node (or whatever else you want), and sets the field values using these provided Tokens, replicating the mapping functionality currently in Feeds

Original report by heylookalive

This is an idea I'm working through as I'm thinking it could be really good and figure I should bounce it off other people.

Instead of using Feeds to process emails we could use rules instead, this means that you could perform pretty much any action including creating nodes (though would need to work out a mapping solution). It wouldn't be without complexities but would be inherently more powerful as the current solution assumes that we want to create.

I'd think that this would need to be a new branch (7.x-3), any interest?

An example flow for a site:

  1. User comments on a node (requires approval first)
  2. Site sends an email with rules to admins with the identifier "[c:9999]" in the subject (comment entity, entity id 9999). This email has a reply-to of the mailhandler email address
  3. Site admin replies to this email with either the text "approve" or "reject"
  4. Drupal draws in this email, triggers any rules with the action "mailhandler processes email", in this case the rule would approve the specified entity if the message body contains "approve"

Comments

Dane Powell’s picture

It's an intriguing idea. I've never been fully satisfied with Feeds, partly because I just don't think the concept of a "feed" completely applies to a mailbox. A feed implies an ordered, unilateral (one-way) read operation between the feed and the Importer. But a mailbox is inherently bilateral- we have to write back to mark emails as read, etc... This makes some types of error-handling almost impossible to accomplish with Feeds.

On top of that, the complexity of Feeds gives it an unforgivingly steep learning curve, which I think turns a lot of people off from Mailhandler 2.x.

On the other hand, there are so many plugins and tools around Feeds that turn it into a veritable Swiss-army knife. Mappings being the most important one, like you pointed out. Also, the ability to have separate fetchers, parsers, and processors, so that for instance you could just use the Mailhandler fetcher if you wanted to, and use a custom parser to do something crazy like read CSV attachments and turn them into new users in bulk or something (I think there's an issue for something like this in the queue).

Swapping feeds out for rules is definitely not something I'm able to hack on at the moment, but I'm open to hearing more.

heylookalive’s picture

Hi! After some thought I had a stab at doing the following.

1. Leave Mailhandler as it is.
2. Use the feeds rules module to glue together both sides.
3. Create a custom module that serves provides the action "Fetch entity by identifier".

To explain 3 (if interested), I started out with a basic identifier as previously suggested ([node:123]) although decided that this would be too easy to mess with, you could increment the identifier to effect nodes you wouldn't necessarily want to. This moved me on to use of a hashed identifier - this requires a data store and the ability to unhash. So, in the form of a hashed identifier we need to supply this as a token for use in the email copy so we can retrieve it from the reply response. Also had the idea that two tokens should be offered for an entity, one time use and persistent use. Issue I hit here is that tokens aren't generated on use necessarily - which would mean having a hash for every entity in the site which seemed too heavy.

This solution didn't work out exactly how I wanted in a generic sense as for in order for Rules to grab an entity it needs to know the entity type ahead of time.

It would be possible to create a rule when needed specifying that entity is of type but I halted work when it wasn't so generic and also the inefficiency of the token problem. Could post the code up, but to achieve what I want if it's acceptable to have a non hashed identifier and crete a rule specifying entity type (which is what you'd do anyway in my comment use case) would just require the ability to scan a string (email body or subject) using regex and create a variable to be used by the core Rules fetch entity action. Or you could use feeds tamper to provide the identifier from that side.

I do agree that the use of feeds is a barrier but one worth overcoming as it's great :)

Dane Powell’s picture

I've been thinking about this more, and it's appealing to the point that I'd almost like to code up a proof-of-concept. Rules just seems like a more flexible and mature platform than Feeds, especially in Drupal 8.

Basically, Mailhandler would just check mailboxes (we'd probably implement a cron job), and fire Rules hooks and provide tokens for messages. Rules could then create a new entity using those tokens / variables, to replace the functionality currently provided by Feeds. I think you can set fields on an entity by 'setting a data value', which would replace the mapping functionality.

The only thing I'm not sure about is how to handle commands and authentication plugins.

Dane Powell’s picture

Just thinking about this more... I'd probably create a new entity type ("email") with fields for header, body, attachments, and source mailbox (or maybe a little more granular, with fields for sender, date, etc...).

On cron, Mailhandler would simply check mailboxes and create email entities for new messages. This is nice as it would decouple the fetching and parsing process, so if fetching succeeds but parsing fails, the message doesn't get black-holed (as is currently the case with Feeds).

Rules could then react to those new emails, by transforming them into a node or whatever the user wants to do.

I'm still not sure how the other custom functionality (authentication, commands, Mail Comment) would get integrated into the workflow.

oadaeh’s picture

To uniquely reference the mail message, you could use the message ID, and then you wouldn't have to hash & unhash and keep a separate data store. Making it a field on an entity type would make it searchable w/EFQ.

heylookalive’s picture

@dane, yep emails as entities sounds good. That way rules can fire on the event email entity created, and you can just access the data via fields/variables as you normally would. I didn't think of this which is why I was hard bolted to mailhandler + rules, but a viable way could be to use: mailhandler + feeds + eck (create email type) + rules + entity rules.

The real trick is having a way to understand what entity and id we're trying to effect if wanting to do something with an existing one (instead of create). You could have multiple methods e.g. ID by sender email address, by ID token ([n:123] - would want a mapping of n = node).

Auth and the actual process of connecting and grabbing the emails, auth etc I have no idea! :) Feeds might still be the way to go. Not knowing mailhandler too well could commands/comments be done with rules?

Dane Powell’s picture

I also considered using Feeds + Rules, but that just seems like an extra unnecessary step when Feeds already accomplishes what 90% of people need. Additionally, I'm not thrilled about sticking with Feeds- it seems pretty 'greedy' to me, in that it tries to pull in as much content as possible but does very little error-handling, allowing messages to fall through the cracks. That's okay for something like an RSS aggregator, which is what Feeds was built for, but not so good for editors posting content by email.

Commands / auth plugins are just ctools plugins, so we can apply them pretty much anywhere as long as we have something to hook into.

Dane Powell’s picture

Title: Transition to using rules instead of feeds. » Plans for 8.x
Issue summary: View changes
Dane Powell’s picture

Issue summary: View changes

I updated the issue summary with my latest thoughts on how this should work. Feedback is welcome.

Dane Powell’s picture

Title: Plans for 8.x » Plans for 8.x / transition from Feeds to Rules
Dane Powell’s picture

Issue summary: View changes
Dane Powell’s picture

Issue summary: View changes
Dane Powell’s picture

Issue summary: View changes
Dane Powell’s picture

Issue summary: View changes
Dane Powell’s picture

Issue summary: View changes
Dane Powell’s picture

Sorry for spamming the hell out of everyone, I'm just trying to get the issue summary cleaned up.

cvining’s picture

You had me at "unforgivingly steep learning curve." Rules is powerful, flexible and (for me) has become a must-have resource. It's very appealing to me to re-use all that power for a mailhandler too. If you guys want/need a use-case tester, I'm very happy to help.

-- Cronin

Dane Powell’s picture

Issue summary: View changes
Dane Powell’s picture

Hmm, instead of a one-time import-and-delete, let's actually associate created entities with mailboxes, and sync the status of messages in that mailbox. This could allow Mailhandler to be used as a base for some pretty awesome modules, like webmail.

MtRoxx’s picture

I will help with testing, documentation and training videos if you'd like.

sonicthoughts’s picture

As a system builder looking for solutions, I'd like to point out that lack of upgrade path to D8 is starting to inhibit adoption of D7 modules. Seems like this issue remains in limbo. Feeds does have a D8 branch now, so some clarity here might helpful. Perhaps the main D8 blocker is lack of a co-maintainer?

Dane Powell’s picture

Yes, a co-maintainer would help. I agree, at this point it makes a lot more sense to keep the Feeds-based architecture for D8 and consider switching architectures later on as part of a new major branch or different module altogether.