Reverse Bounty Idea: email bounce handling module for Drupal

This module would be very useful. The following use case "EBH will provide a way to list users with more than X bounced emails" would be the most used by me and many others so I wish it wasn't an optional one.

I think that mailman dependency isn't a good idea. It would prevent the use of this module by a lot of users.

Thanks.-

EBH would be a great companion to Subscriptions, because bounces are a pain for every Subscriptions installation. I'd be very interested in providing some integration, for example suspending subscriptions to bouncing addresses.

I wrote Subscriptions (with chx) for free and would not recommend a module that comes with a license fee, but you might be able to find funding for its development. Putting up a solid project and offering services might be a better strategy in the long run though.

A dependency on Mailman would likely be a turn off for many potential users.

Please try to talk to the Bounced Email maintainer about taking over his project. He's still active, but he hasn't touched Bounced Email in two and a half years. He'll probably turn it over to you and you can start from scratch in the proper slot.

While PiP itself have not seen a release in six years, the PECL extension relatively recently did. Might worth a look http://pecl.php.net/package/python
--
The news is Now Public | Drupal development: making the world better, one patch at a time. | A bedroom without a teddy is like a face without a smile. |

CiviMail (http://civicrm.org/civimail), a component (add-on) of CiviCRM, handles bounces. We used version 1.8 on drupal 4.7 and neglecting some little issue on multiserver environment, works quite well. It can works either with MTA access (amavis) and imap. We test with gmail for domains (imap support from september 2007) without problems.

ema

You could also look to http://www.phplist.com/ for an example of bounce handling code. I nearly used phplist instead of simplenews for one site because they wanted bounce handling, but it was too painful so we went without the bounce handling. In short, I'd use this module if it existed.

Please add to your requirements that it fire actions each time it processes a bounce. So that third-party modules can automatically act on the bounce.

Since everyone thinks their preferred brand of "engine" is better and because there are different transports (e.g IMAP,POP) my only request is that the design is pluggable. E.g you can completely rewrite the engine module and replace the one used by default. I can assist with the design if you want to sound things out.

This will make it possible for your engine module implementation to be dependent on mailman yet other people can make theirs dependent on say enemies-of-carlotta instead.

--
Sammy Spets
Synerger Pty Ltd
http://synerger.com

As the maintainer of Simplenews I can say that there is a fair demand for EBH.
EBH should ideally be independent from a mail backend and have an API which various systems van use. But I realize this will not be easy.

Recently AjK started an issue with code to make an EBH for simplenews based on VERP. See the issue for more info: http://drupal.org/node/242137

-- Erik

I'd be keen to see this. I have a forthcoming project that will use email lists extensively so better bounce handling would be useful. I can test your functionality, and perhaps contribute some code, and a few $. Can I suggest you follow up the suggestion from salvis and look to take over the Bounced Email module?

There may be sites that heavily use email, newsletters, notificacions, etc. so I think it would be nice to have an option to process bounces.

On the other hand... it would be nice if there was a way to promote some kind of standard for bounce notifications.

I'm currently dealing with a site with more than 300000 registered users, not in Drupal but it will someday, and for bounces I created a small PHP script that simply removes all that comes to a bounce-catcher email address used for the purpose. This is executed daily via cron. I can still check via webmail what is there in case this is needed, but only what remains between crons. We have aroung +20000 bounces daily. But not all mean permanent errors. Many are temporary, kind of quota exceded, etc. The problem is that it is impossible to automate some kind of analysis. Every email provider has its own format for the messages they send, you have to read manually to get an idea of what they are trying to tell... so that's why my process removes all bounces. Of course, this is not scalable solution, good practice either, but it works. At least, until a better solution can be applied.

Hi what happened with this???

I'm also looking for "bounce detection" solution in Drupal.
Or actually it could be universal to benefit all :)

Mailman and CiviMail are indeed good ideas for code reuse.

Doing googling found two more aspects which can be concidered:
* loop detection
* sending probes

Also found some pieces that havent mentioned
in this thread before. Maybe useful for someone.

OpenPSA
http://www.openpsa.org/version2/documentation/bounce-detection/

Lamson Project
http://lamsonproject.net/docs/bounce_detection.html

Email Bounce Detector
http://bouncedetector.riaforge.org/

Perl Module: Mail-DeliveryStatus-BounceParser
http://search.cpan.org/dist/Mail-DeliveryStatus-BounceParser/

PHP List
http://docs.phplist.com/PhpListConfigBounces

Moodle
http://docs.moodle.org/en/Email_processing

EZMLM (written in C)
http://www.ezmlm.org/faq/How-bounces-are-handled.html#How-bounces-are-ha...

Majordomo
(No good "bounce detection" link)

As I need real solution for bounce handling I did a step forward.
I guess Mailman in most mass-tested, mass-deveolped and ripe software to re-use.
Today I took look into Mailman and CiviMail.
Here are my findings.

I understood that there are 2 layers processing done.
1) Identifying email address and bounce reason
2) Decision algorithm that decides when to disable user

Pros

• Has 2-layer system
• First layer has MUCH richer/diverse patternset to find email from bounce
• Second layer is VERY developed to handle temporary delivery problems
• Its very easy to "rip out" first layer processing for own needs. Tested. (see code later)
• Mass-tested, mass-deveolped and ripe software
• Code written in Python. Its pro because the code is more strict.

Cons

• Code written in Python. Its con because need to use 2 different languages in system (PHP and Python). Hence needs some interfacing/bridging.
• Second layer probably more rooted into the framework

Pros

• Written in PHP, simpler to deploy/integrate.
• Probably possible to "rip out" first layer functionality. Not tested.

Cons

• Has 1-layer system. Almost.
• Keeps simple patterns in database - more complexity
• CiviCRM bounce processing (L2) usually just disables user after 1 bounce which is very annoying to constantly enable users whose mailboxes have been full for a moment or they have been on vacation (notification) etc.
• A lot of different bounce reasons (vacation, mailbox full, etc) - more code needed to decide yourselt how handle which cases.

There could be one more option to handle bounce detection and handling/processing.

• Install almost any list software which code can be accessed freely.
• Create dummy list where will be all emails.
• No email is never sent to list and all notifications will be turned off
• Send mails to users from custom code as needed
• Feed all bounces to the dummy list
• From time to time synchronize enabled/disabled status between the list and the drupal system

Proof of concept.

I have Debian Lenny System.
Mailman version 1:2.1.11-11+lenny1.
Installation is at /var/lib/mailman

In proof of concept dir have following links:

email -> /var/lib/mailman/pythonlib/email
paths.py -> /usr/lib/mailman/bin/paths.py

Its needed to make easier to include needed files.
In same dir create following Python script (modification of /var/lib/mailman/tests/onebounce.py):

#! /usr/bin/env python

import sys
import email
import getopt
import paths
from Mailman.Bouncers import BouncerAPI

# print usage
if len(sys.argv) <= 1:
        print "Usage:",sys.argv[0],"<bouncefilename1> ..."
        exit

# loop through arguments
for file in sys.argv[1:]:

        print 'Processing',file

        # read in bounced message from file
        fp = open(file)
        msg = email.message_from_file(fp)
        fp.close()

        # loop it thourgh different specs
        # (Yahoo, Qmail, Postfix, Exim, ...)
        # we could actually break at first match
        # but for demo's sake check all of them
        for module in BouncerAPI.BOUNCE_PIPELINE:

            modname='Mailman.Bouncers.' + module
            __import__(modname)
            addrs = sys.modules[modname].process(msg)

            # sometimes we get None instead of []
            # these modules just have different return vaules
            # when not matching
            if addrs is None:
                addrs = []

            # report analysis result
            if addrs is BouncerAPI.Stop:
                print module, 'warning, not a bounce'
            elif len(addrs) == 0:
                # pass
                print module, '...'
            else:
                print module, 'bounce', ', '.join(addrs)

        print

It's important to keep code stepping for Python.
chmod 755
It reads commandline args as filenames.
Reads file and reports emails or if further processing is not needed.
Output:

user@host:~/mailman-bounce-handler% ./proof-of-concept.py /var/lib/mailman/tests/bounces/postfix_01.txt
Processing /var/lib/mailman/tests/bounces/postfix_01.txt
Postfix bounce xxxxx@local.ie

user@host:~/mailman-bounce-handler%

Actually /var/lib/mailman/tests/bounces is full of testcases.
So you could run to test

./proof-of-concept.py /var/lib/mailman/tests/bounces/*

This case was not so interesting to me,
so I didn't bother to create proof-of-concept code.

Here are just good starting hints
if somebody wants to develop something on his own.

Important file: sites/all/modules/civicrm/CRM/Mailing/BAO/BouncePattern.php
Important class: CRM_Mailing_BAO_BouncePattern
Important function: match(&$message)

Important db table: civicrm_mailing_bounce_pattern
It holds handful of most important bounce detection patterns.

Important db table: civicrm_mailing_bounce_type:

+----------+-----------------------------------------------+
| name     | description                                   |
+----------+-----------------------------------------------+
| AOL      | AOL Terms of Service complaint                |
| Away     | Recipient is on vacation                      |
| DNS      | Unable to resolve recipient domain            |
| Host     | Unable to deliver to destintation mail server |
| Inactive | User account is no longer active              |
| Invalid  | Email address is not valid                    |
| Loop     | Mail routing error                            |
| Quota    | User inbox is full                            |
| Relay    | Unable to reach destination mail server       |
| Spam     | Message caught by a content filter            |
| Syntax   | Error in SMTP transaction                     |
+----------+-----------------------------------------------+