We want to be able to just "Do CRM Mail" well with Party. Exactly how this will be done and what "Doing CRM Mail" means is complicated. So rather then figure out from the start where Party Mail will go I think its best to just start where is obvious and see where we go from there.

In this issue Summary I will try and keep a list of related issues updated. Currently the issues are also tagged Simplenews

Step 1: Integrate Fully with Core Simplenews
#1547516: Simplenews - Create a "Sycronize with Party" Checkbox for Simplenews that is analagous to "Syncronize with User"
#1547552: Simplenews - UI for seeing lists of parties in a newsletter List
#1547582: Simplenews - Subscription settings on individual Newsletters
#1547606: Simplenews - Simplenews "Field" on the User
#1547620: Simplenews - Adding a Party into a list

Step 2: Figure out whether to add features to simplenews or to something else.

Step 3: Integrate what we've done with simplenews with the drupal mail chimp module (Or http://drupal.org/project/emf

Step 5: Make a "Party Mail API" that can "just work" with both mailchimp and simplenews in a consistent manner but also provide the ability for people to make it work with other stuff

Step 6: Pull features that are more generic then just mail out of Party Mail and into something else (this might happen with Party Lists)

Comments

joachim’s picture

joachim CreditAttribution: joachim commented

I really don't want us to reinvent any wheels here.
At the same time, we definitely need to be able to plug into a variety of things: Simplenews for small sites, of MailChimp for large numbers of users. (As for Newsletter module, I don't know it at all).

yautja_cetanu’s picture

yautja_cetanu CreditAttribution: yautja_cetanu commented

We've just had a meeting about Party Mail.

Summary

Our Step 1 is to build a "Party Messaging API" module that interfaces between party and simplenews. It is not going to be very complicated and will work with whatever simple news does out of the box. Our aims is to make:
1) Simplenews' UI work with Party
2) Party's UI (Views) work with Simplenews. Obviously most of this will be there but we might need to fill in a few gaps.
Step 2: Is to make this API work with other modules. This can be via Messaging or via EMF (http://drupal.org/project/emf) and so people can then use mailchimp and the mailchimp module
Step 3: Finally we need to make this work with a Party Activity API. In reality this Messaging API may actually turn into the Activity API as maybe all the ways that party interacts with mail is just an activity. Alternatively this module may work alongside the activity API.

Primary use case we're trying to solve:
If I send an e-mail to a party via any method available to Drupal and the e-mail bounces. I want Party to know that the e-mail has bounced and act accordingly. (This should scale for 100,000 people newsletters and sending single e-mails)

(This may involve looking into porting http://drupal.org/project/simplenews_bounce to D7 although work has been started on github)

Random Meeting Notes

	• Discussed out clients:
		○ We're working with student management
		○ Integrated thingy
		○ David Connerth - Drupal Site
	• 
	• Really important to record bounce rates. 
	• Really important to include opt-out in Mailing
		○ Need to include opt out with CRM. Need to know
	• Scale for 500,000 e-mails
	• Simplenews can throttle outbound e-mail and prioritise.
	• We just need a bridge between CRM and simplenews.
	• If I'm on the phone to a prospector,  and I say "want to join our newsletter?"
	
	• Discussion of simplenews vs php-list
	• Freelocke talks about Amazon Simple E-mail Conduit.
		○ Uses the Messaging module for handling unique e-mails.
	• Definitely need to eventually support simplenews, mailchimp
	• Andrew - Behind phplist is php Mailer.Handles Batching and bounce rates.
	• Freelocke switched over to Swift Mailer instead of php mailer.
		○ E-mail marketting framework - EMF - general mail framework for talking to services.
	• Party Mail API is probably needed. And then Modules that connect that to simplenews, or mailchimp.
		○ Possibly do it with ctools plugins
		○ Standard way to find out if things have bounced.
	• Do we connect up with Messaging?
	
	• Party Mail API
		○ Working with Simplenews, Messaging
		○ Work with EMF- http://drupal.org/project/emf
	• Simplenews - Does it include open tracking (embedding a unique image that tracks whether that specific message)
		○ Uses Simplenews statistics
		○ Freelocke says this is important
	• Conversation about Newsletter vs Simplenews
		○ We can't really tell! It seems simplenews came before Newletter, can do everything (work with multiple newsletters). Simplenews is on 40k sites
	• Jamie: Party Messaging API - To be able to scale to newsletter and single e-mails always recording bounces.
		○ Maybe thing between messaging and newsletters.
	• David Connerth - Party Messaging API
	• Freelocke - We talked about whether an activity API can completely replace the Messaging API
	• David Connerth - We will build a module called Party Messaging API that works with simplenews for now.
		○ As long as Simplenews handles  bounce tracking.
		○ Next step Messaging
	• Bounce tracking - Someone has started it in Simplenews - http://drupal.org/node/1418418
	• Party - Simplenews Intergration
		1) We want to add Parties into a newsletter
		2) We want a view of Parties to send a set of parties to a newsletter
moreonion’s picture

moreonion CreditAttribution: moreonion commented

Sorry we missed the meeting but we're looking into party as well for a few projects and we're excited about the progress so far. Great job folks!

Regarding email I have a few ideas that might be a bit far-fetched but maybe worth considering (you tell me :-) )
Disclaimer: i'm not a developer, rather use crms from a strategic point of view.

We're planning on using party for digital campaigns which means we have actions (e.g. petitions, webforms) and then want to profile our audience in the CRM. From there we want to create segments that we also email to, send out micro-targeted emails (based on various filters like activities, location,etc.) , send follow up emails and let supporters organically profile themselves through "dynamic groups" (just a set of filters that keeps the group up to date).

Certainly an integration with a newsletter-style module is great for "newsletter type" emails but neither simplenews, nor "newsletter" module are great for targeted email marketing as is (have used both for lots of projects).

Third party integration solves some of those problems but I think an integration can barely be as powerful as an email component that is designed to work on top of the CRM.

Things on my wishlist for a messaging component that works with party:
* track sent emails along with CRM record (maybe even activity?)
* track opening of emails along with CRM record
* track click-through along with CRM record
* create "filters" (= dynamic groups?) of CRM records that can be saved and emailed to
* create "message templates" that can be assigned to each group
* create contextual emails (e.g. CRM record has been "added" to a "dynamic group" --> send email)
* create follow up email functionality (CRM record of mailing X did not react after 48 h, follow up email is sent)
* have strong token integration so that all fields can be used in emails

I deliberately leave out other important features that you've mentioned and discussed already (like bounces, etc.).

Looking forward to further discussion and hope that our developers will be able to join you folks in the battle field soon!

Cheers
Florian

yautja_cetanu’s picture

yautja_cetanu CreditAttribution: yautja_cetanu commented

"Certainly an integration with a newsletter-style module is great for "newsletter type" emails but neither simplenews, nor "newsletter" module are great for targeted email marketing as is (have used both for lots of projects)."

Regarding that comment. I'm currently researching pure simplenews integration to see how far that would get us.

What about simplenews makes it not so good for targetted email marketting. Is it just the inability to have dynamic groups?

joachim’s picture

joachim CreditAttribution: joachim commented

> Is it just the inability to have dynamic groups

Dynamic groups for simplenews was raised back in the 6.x days. There's at least an issue open for this, where the idea was (last I saw) to add an API for any module to say 'I provide the list of recipients of this newsletter'. Simplenews itself would then become an implementer of that API, to provide a list of emails based on its member management and sign-up UI.

If that's still at the planning stage, it could be something that we could move forward perhaps?

yautja_cetanu’s picture

yautja_cetanu CreditAttribution: yautja_cetanu commented

This was the original OP. I just thought I wanted to keep it in the discussion

This is a placeholder task for defining what I think a mailing in party could look like. I'm writing this as something that someone else could pick up and work on in parallel to the rest of party.

Step 1 - Basic Mailing on your own server
Need something that works in a similar manner to CiviMailer which you can host on your own server.

    Need the ability to e-mail groups (including future smart groups, possibly use OG for now)
    Need the ability for a wysiwyg editor to write the e-mails and attach styles to the e-mails (hopefully this is done by another module)
    Need to attach the e-mails as activities to individual parties (so work alongside the not yet created party activity sub-module)
    For reference. http://drupal.org/project/newsletter, http://drupal.org/project/simplenews, http://civicrm.org/civimail

Step 2 - Generic Party mailing api (if thats what it is?)

Here we'd create something that will work with both step 1 and mailchimp or potentially any other alternative.
yautja_cetanu’s picture

yautja_cetanu CreditAttribution: yautja_cetanu commented
Title: Party Mailing Integration » [Meta] Party Mailing Integration and Integration with Simplenews
Component: Code » Ecosystem modules

Changing this thread to be a placeholder for all the issues related to party mail integration and the discussion surrounding it.

yautja_cetanu’s picture

yautja_cetanu CreditAttribution: yautja_cetanu commented

So I've put forward my suggested roadmap for developing this integration above in the OP. I think the temptation will be to start with 6 but I think we should get intimate with Simplenews and mailing lists first by trying it out before moving to mailchimp and before planning for building generic list builders without understanding the use cases.

However regarding dynamic lists I took Joachim's advice and spent a bit of time reading the simplenews issue queues to get an idea of where they are going. They do want to eventually include "Enterprise features" so I think working with them might be a good idea. I've started the conversation here: #1547642: [meta] SimpleNews and the Party CRM

moreonion’s picture

moreonion CreditAttribution: moreonion commented

Sorry for the late reply.

Dynamic groups is clearly "must have" that simplenews currently lacks. I'm not sure about those other features I currently miss - would they just need a change in interface to be really used along with a nice CRM or would it also require changes in code / concepts.

Something like contextual emails / message templates is quite important too. The newsletter module has message template feature but it doesn't really allow "contextual" emails (like CRM record X just entered dynamic group Y). This can be super helpful if you want to plan an "experience" for your customers/supporters. For an example you can plan in advance which stuff people receive via email/snail mail once they've first signed up. You can literally evolve the relationship based on their reaction, whether they click on links in specific emails and depending on where they are in their "journey" (how new customers/supporters climb the ladder and become super awesome brand advocates). Each of these segments deserve a totally different treatment, right?

Random question thrown in here:
Wouldn't be "groups" and "dynamic groups" be a part of the CRM rather than the newsletter module? They could also be used to create other stuff then email like direct mail or website personalization (when linked to user). If they are not part of the CRM - can those filters use all CRM data?

At the moment I'm a bit concerned with keeping data in different places. My hope regarding a drupal CRM is to get rid of the "webform data, user data, profile2 data, simplenews, other CRM systems" mess. But maybe that's just me as a non-developer.

yautja_cetanu’s picture

yautja_cetanu CreditAttribution: yautja_cetanu commented

Thanks again for the reply.

Regarding whether this dynamic groups stuff goes int simplenews or party. We don't know yet. I think, if we do it, we're going to make it for simplenews first and in the process see how much stuff should be made more generic. I found it difficult to figure out what other uses for "dynamic lists" there would be but I quite like "website personalization".

Keeping data in different places is definitely a concern. Party is an attempt to solve that by being a module that essentially pulls all that data into one place when you're dealing with it but allowing you to store it in the seperate places of all the modules that create the data. I think we have to do things this way with Drupal because there are so many modules that will store their own information about a party (such as Commerce storing orders) that we have to work alongside other modules.

moreonion’s picture

moreonion CreditAttribution: moreonion commented

<<>>

Other use cases for "dynamic groups" or saved queries (how ever you want to call it) are:
* populate listings for simpler CRM record updating and bulk editing (creating a group should be less complicated than configuring a view)
* populate listings for public exposure (recent clients, recent backers of our petition)
* use it for reports and analytics (such as "how many people have been emailed this week and have bought item X (logged as activity)")
* use it for contextual emails / snail mail (such as "person X has just bought their second product")
* store "segment" info in session cookies and personalize product display ("person X has bought products in the category Y")
* better engagement tracking through adding url suffixes based on segments and tracking of offline activities ?from=facebook or ?from=brochure (used with analytics tool it allows super detailed reports)

<<>>

Ok, I understand your point. I suppose we'll have to actually try to find out whether using pre-existing modules such as simplenews will limit the flexibility provided by party too much or whether there are enough use cases covered. The other question is whether a "party addon" module can be developed in a sustainable fashion. But I suppose we'll give it a shot anyhow...

moreonion’s picture

moreonion CreditAttribution: moreonion commented

I just quickly wanted to add http://drupal.org/project/manymail as a candidate for the integration. Just stumbled over it and it looks promising.

yautja_cetanu’s picture

yautja_cetanu CreditAttribution: yautja_cetanu commented
Assigned: Unassigned » rlmumford

That does look like a good module! Wonder how it compares to simplenews.

rlmumford can you start bringing in simplenews into the rest of Party.

rerooting’s picture

rerooting CreditAttribution: rerooting commented

Theres also http://drupal.org/project/views_send and http://drupal.org/project/mailchimp's new campain functionality.

I feel that, in order for something to qualify as part of the core party module, it needs to either very directly extend core CRM related functionality such as introducing new data sets or extending hats in ways that impact drupal core related functionality - such as assigning roles on hat assignment, etc.

Personally, I like all of these modules, but I feel that, to follow best practices, it would make sense if this was accomplished via modules like party_simplenews and party_mailchimp.

This leads us to the task of extending the api controllers!

rlmumford’s picture

rlmumford
Primary language English
Location Manchester
CreditAttribution: rlmumford commented

I've just lumped all of the party_simplenews code into the Party project as a submodule, It's still rough around the edges but atleast we now have code that everyone can see/play with/bug hunt.

I agree with rerooting. It seems like the best practices way of doing this is to have separate modules for each of the mailing providers, while having core provide a useful API that makes those modules lives easier.

I've added party_simplenews to the project now however, as I think in the process of working on it and building other mailing integration modules, we'll find that there parts of the simplenews integration that are really more generic than simplenews, and this way we can move code around more easily. I expect that once what is in "generic party mail api" is decided party_simplenews will be moved to it's own project page to be maintained there.

rlmumford’s picture

rlmumford
Primary language English
Location Manchester
CreditAttribution: rlmumford commented

See #1680050: Bring party_get_all_emails and party_get_email_fields into core Party module (from party_simplenews.module)

rerooting’s picture

rerooting CreditAttribution: rerooting commented

Indeed, that'd be splendid. Let's keep our friends close for now until we can build solid API's for their use cases and let them leave the nest :)

rerooting’s picture

rerooting CreditAttribution: rerooting commented
Issue summary: View changes

Changed the first post to summarise a roadmap