Digest Email

I am about to implement this in a custom module. I will add all the messages of a certain type. these messages are not send by notify On cron run, I aggregate all messages releated to the same node type and the same user to one new message. This message aggregates the text of all the other messages and will be send as digest mail. I din't find another solution to do it with the message notify module out of the box.

You will find it in the ERPAL Projects Module https://drupal.org/node/1995398 https://drupal.org/project/erpal_project its the module erpal project notify but it is not a module for general usage yet.

Would love to have this as part of the Message Notifications module. It would be a killer feature for popular communities like Drupal Commons.

Also requested in Commons: #1743014: Support activity digest emails

> It would be a killer feature for popular communities like Drupal Commons.

I agree, and actually have some ideas how to implement it - so now it just a matter of finding someone to sponsor Gizra's work.

> I agree, and actually have some ideas how to implement it

Message notify should declare a new CTools plugin type called digest. It will have a render function that returns a digest message, and can be replaced by other implementing modules.

We want to add this functionality to Open Atrium 2 also and will work try to with the community to get this done in a way that will work for everybody using the Message module.

Thanks Mike, that great news. Would love to test it once you're ready.

I would be interested in helping to partially sponsor this work, if that will help get the ball rolling. let me know what it would take, Thanks.

If @mpotter needs help, we may join forces to build a general module for that feature

This is something I'd like to help with as well.

I'd like to get the ball rolling, so I'll summarize the issue and see if we can get traction. So far we have offers of:

@mpotter: development
@manuelBS: development
@loze: sponsor
@Dustin@PI: sponsor (from Commons thread)
@BarisW: testing

Summary:
- It looks like manuelBS has started work on something, but not quite the way @Amitaibu had envisioned?
- mpotter, it sounds like you're actively developing a solution? Is there a link or issue in Atrium we could check out?

Has anybody been in touch with anybody else? Looks like there's sponsorship money on the table... is it enough? ;)

I can test too.

I think before starting development (right you are, I may help) we need a concept about how these digest will be created. Who will do that. Maybe @Amitaibu will do this

I am still able to sponsor a small amount of the work and help with testing (I had more budget before but we spent it on other parts of the project).

@mptotter Do you want to publish the concept for OA2 so the community can help?

I can sponsor a small amount as well.

I have just created a sandbox project for a custom module i have been working on for one of my projects. Its based on field values for determining if the item should be sent.

It supports digest emails based on a per user basis. The module is early in it development, and at the moment is strictly an API module - no UI. I think it could probably be used here for the digest ability.

https://drupal.org/sandbox/chertzog/2077787

Sorry I got distracted with the OA2 Beta4 release and haven't been back here.

1) For people able to sponsor, can you contact me via email so we can work out the details of what that might look like?

2) We really want this functionality for the Public Release of Open Atrium and plan to start serious development when I get back from vacation the week of Sept 9th. We'll have one other developer here at Phase2 helping with the development.

3) Where do people want to start discussing the architecture for this? Sounds like Amitai had some ideas. I want to make sure the architecture supports the core Message API and is something anybody using Message notify can use. My thoughts are around a new notify plugin so the user can select which notification method to use on a per-group basis.

The Open Atrium requirements for this functionality are:

a) Daily, Weekly, (other?) digests of the email notifications they would normally be getting individually on a per-Group basis. For example, Group A is set for Daily digest, Group B for Weekly, and Group C set for normal immediate email notifications.

b) An option for combining multiple Group Digests into a single email message. Need some work on wireframing how this email digest would look and be structured.

c) Ideally would use existing Message settings (Subject/Body partials) rather than adding a whole new slew of settings. Perhaps some settings for global template settings, but each message in the digest should somehow summarize using their existing config. Otherwise the setup of digests gets cumbersome when working with dozens of existing messages.

d) Works with the Message API (message, message_notify, etc). No need to reinvent the wheel given that message_notify already supports plugins. We want this to be done in a way that any distribution currently using Message can just turn it on and use it.

My current ideas are around creating a database table to keep track of notifications that *would* have been sent to the user, except that they have digests enabled. So notifications accumulate in the database until the digest time expires (daily, weekly, etc) and cron aggregates them using a template and sends the digest, then clears the database table for that user/group combo.

I'm not sure about Open Atrium but Drupal Commons has an Activity Stream. Could this be leveraged to generate an Digest Email?

https://drupal.org/project/simplenews and https://drupal.org/project/simplenews_scheduler

I think all you need is a token that renders an email enhanced view (in Drupal Commons, that view might include the previous day's activity) from the perspective of the subscribers account (by either passing an argument or masquerading as the subscriber?). Put that token in a newsletter, and schedule it to run every day (sending a few each cron job). You could also set the newsletter to auto-subscribe users on registration if you like.

You could also create 3 tokens (weekly, bi-weekly or monthly newsletters?) and 3 newsletters, so people can choose the frequency.

One thing to remember is to ensure that simplenews doesn't cache the token and send the same digest to everybody, this thread seems to describe the solution https://drupal.org/node/700308#comment-5344532 Originally I think with simplenews it would only allow a few unique tokens per newsletter and the rest were cached for performance reasons, but looks like that's changed now.

Personalised digests are always going to be pretty resource hungry, but if you consider how many notifications they're replacing, it doesn't seem that bad.

One thing to remember is to ensure that simplenews doesn't cache the token and send the same digest to everybody, this thread seems to describe the solution https://drupal.org/node/700308#comment-5344532 Originally I think with simplenews it would only allow a few unique tokens per newsletter and the rest were cached for performance reasons, but looks like that's changed now.

I was remembering wrong, if all you want to do is embed a view (digest), can just use a token, don't have to disable caching the newsletter.

Quick update here, I'm working with mpotter on developing this feature at https://drupal.org/project/message_digest using the basic method laid out in #7.

it's nowhere near working yet but should be testable by this time next week. I will update this thread at that point.

The current plan has IMO too many assumptions about its use. A notification can be something other than an email. Intervals can be according to business logic and it won't make sense to have a plugin per interval.

Here's how I think we should tackle this problem:

1. Message digest module (MG) should be dependent on Message subscribe module.
2. MG should declare a new CTools plugin called Message Digest. Those plugins would be responsible of getting the relevant items (e.g. posts from last week, comment from last day, etc)' and formatting it (e.g. a list of items, list of teasers, etc'). MG can probably come with two plugin for different format types, that share the same interval logic
3. MG should hook_message_subscribe_get_subscribers_alter() to remove users from getting a notification. Those users can be identified by a field that would be attached to them called message_digest_interval that can hold day, week, month, etc'. There would be another field called message_digest_last_processed that will hold the timestamp of the last time MG tried to find content to send to each user.
4. MG would implement hook_cron() and get a range of users by the right interval, and the last time they were processed.
5. MG would format the digest notification and pass it on to Message notify - this part is time and resource consuming, so we would probably want a Drush integration that would be able to act from the CLI without timing out.
6. Tests are a must to get it right ;)

Thanks Amitaibu, this is absolutely awesome advice.

MG should hook_message_subscribe_get_subscribers_alter() to remove users from getting a notification.

So what about messages that aren't coming from Message Subscribe? Just have them be delivered directly and don't put them into the digest?

Those users can be identified by a field that would be attached to them called message_digest_interval that can hold day, week, month, etc'.

I think we should give the user the ability to choose interval per-message/subscription type instead of globally. What are your thoughts on that?

MG would format the digest notification and pass it on to Message notify

Would it need to declare a new Digest message type for this to be possible? Because Message Notify is specifically built to deliver one message at a time right, meaning you can't easily just pass it arbitrary content that isn't a message?

Also probably goes without saying, but we want the user to be able to control the digest on a per-group basic. e.g., for Group A the user wants a daily digest, for Group B the user wants a weekly digest, and for Group C the user just wants the standard immediate notifications.

Then, in addition to this, we want an option to aggregate the daily and weekly digests so the user can choose between getting a digest per group, or just getting a single digest that contains all of the groups.

btw, Why did I give the initials MG instead of MD I have no clue, probably sleep deficit :P

> So what about messages that aren't coming from Message Subscribe?

Like what kind of messages? You should probably let them be, if they were not sent as subscribe's notification.

> I think we should give the user the ability to choose interval per-message/subscription type instead of globally. What are your thoughts on that?

Any customization makes it a little harder, but you can probably do it with a field collection holding the message type and the interval.

> Also probably goes without saying, but we want the user to be able to control the digest on a per-group basic

In that case those field collections would be attached to the OG membership entity. Wonder how you gonna deal with the UX to not make the settings cryptic ;)

> Would it need to declare a new Digest message type for this to be possible?

You don't have to - you can use View modes. So declare a new view mode called "daily digest", and then you can populate it differently for each message type. You would probably use tokens are arguments to replace the formatted digest inside the message.

Good luck :)

Moving to Message Digest queue.

I just tagged a dev release and added some simple docs to the project page. It should be ready for testing at this point -- it's working for us in Open Atrium 2.

Please do give it a whirl and post anything that is confusing or doesn't work for you. Also note the remaining todo list on the project page.