Listhandler for 5.x

Just another user that would love to see how this would work for my site running Drupal 5.

I'm running a site trying to consolidate a lot of different sources. Both blogs, site rss feed, and more than one mailing list. Being able to feed the mailing lists into individual forums would get great.

Shiny; I'd love to review some patches.

I currently don't use the module myself, so I don't have plans for an update.

code underway.

Shiny, this may or may not be useful.. I spent a few hours going through the upgrading from 4.7 to 5.1 documentation and took a stab at getting this to work.. but then found out the server doesn't have the php imap module..so mailhandler isn't able to pull in messages - which I think is required.
Also..what I thought I would find but didn't -
I couldn't figure out how to tell it which "forum" to look for "mailhandler messages" that it would catch and send the web-forum responses back to the list..or perhaps by the sql query it does it "knows" which are? Not sure quite yet..

When saving options in the admin menu, it says they save, but they don't. I'm sure this has to do with my attempt at modifying the forms to upgrade them to 5.1

It was an attempt (that isn't working yet), so it may or may not assist you!

patch posted in next comment

Maybe a patch file is a better way of going about this.. note: I changed listhandler_admin($form) from above to listhandler_admin() because it didn't belong there

Attachment	Size
listhandler_1.patch	10.11 KB

thanks jriddiough.
My employer has given me all next week to work on any FOSS project i like -- i'll knock this task off my list then, including merrging testing what you've provided here.

This update:

-Includes the .info file
-Replaces deprecated $_POST['op']
-Completes a clean run through coder.module

Attachment	Size
listhandler_2.patch	16.19 KB

I took a glance at the patch, hopefully I will have some time to test it out and see what more needs to be done. Is there anything you're aware of that needs work, Dave? I think I'll check out that coder.module :)

i've merged this with what i've got
(it's all in my darcs repo, http://darcs.coffee.geek.nz/contributions/modules)

attaching my .install
fixes bug with missing {} around tables and adds postgresql defs.
(should be merged to drupal 4.7 probably)

Attachment	Size
listhandler.install.patch	1.4 KB

I'm just waiting for this too.

Subscribing.

Subscribing

darcs repo

Either a directory was forgotten or the site was rearranged a little.

http://darcs.coffee.geek.nz/drupal/contributions/modules/listhandler/

Subscribing. We are waiting for this as well.

Please help all the 5.x people by making this available..this is desperately needed for the Forums...it will improve Drupal's image a lot...pls do this someone.

Bump, we need this realy !!

I took some time to test out this set of patch, using the code available from the darcs repository. Main bug : the listhandler_admin_form function needs more work, to print description/help, or at least labels, for every field, which is necessary to put in the right values.

Here's a patch incorporating partches submitted earlier in this thread. It might be easier as a starting point, for people who can't use darcs, along with a few path corrections.

Attachment	Size
listhandler_D5.patch	11.96 KB

It would be much appreciated if the owner could step up now and take a look to get this resolved. Listhandler is a very important module, and lot of people are waiting for this.

Humm.. after testing, it seems like the last patch needs to be improved a lot. Mailhandler does it's thing, but the message is not imported/published in the forum.

@shiny : can you confirm that your code does import messages successfully?

When attempting to install Listhandler on Drupal 5.1, using the Darcs version, I can't even get Listhandler to show up in the module page so I can install it. Is there something else I need to be doing to test this here?

Here's a new version of the patch .. this is still incomplete, but it's working better. This patch applies against HEAD (14/02/2007) and still needs more work.

Attachment	Size
listhandler_3.patch	16.82 KB

I've reviewed the patch and applied it to CVS so can create more patches from there.

I was able to patch against HEAD fine (couldn't wait for the CVS to update). However, there's a test done on $node->tid in listhandler_mailhandler(); which is not being set by Mailhandler or Listhandler. I post this here as I didn't know if this was a change in Mailhandler that has happened since its Drupal 5.x version, that might in turn require some changes in Listhandler.

A small documentation patch that might help a lot of people test this out .. even though it's only about getting mailhandler to work.

Attachment	Size
listhandler_4.patch	438 bytes

Tried the latest HEAD version -- IT WORKS -- except that the first forum post originated in a forum comes back to the forum and is not filtered out, as it used to be in 4.6.7.

Weird thing is, the comments/replies are NOT duplicated as well. It only seems to be the initial forum post.

Oops, sorry... An update to my previous post: ANYTHING that originates in the forum IS duplicated when it returns via Mailhandler. The message is NOT filtered out.

i hope this comes for Drupal 5 soon :)
anxiously await it

Attached is a patch that corrects the display on the Listhandler configuration page.

1: Displays " | " in tabulated form
2: Saves updates to mailing list prefixes
3: Doesn't display mid values
4: Doesn't display duplicated "The configuration options have been saved" message

Attachment	Size
settings.diff_0.txt	2.61 KB

Correction,

1: Displays "[mailing list] | [prefix]" in tabulated form

This patch works nicely for me, thanks!

subscribing, I'd love to implement this.

Joris

subscribing... would like to see this working for 5.x (and soon 6.x)

Any updates on this?

For me it works perfectly in 5.3 when receiving from mailing list.
The thing i'm not sure of is that it sends correctly the mail from forum to mailing list.
From a quick look at the code it seems to me that:

should be changed in:

hi
cld u kindly make an official release or a dev release for 5.1

thanks

My patch should solve a bug related to the headers of mails sent from drupal when user posts a message in the forum. It also provide an attachment feature

1) Now listhandler should dected loop mails (mails generated from forum posts) and duplicates mails should be discarded.
2) Client mail programs should display correctly mails in hierarchy mode.
3) Forums attachments are converted into file links at the botom of mail body.

The attachment feature should be easy extended in order to convert them in real mime attachment (listhandler option to choose between the two methods?).

It applies against listhandler HEAD release and it contains all the previous patches of this thread.

It needs work, so any testing and feedback is well appreciated.

Attachment	Size
mailfromforum.patch	8.05 KB

I applied this patch to a new copy of HEAD from CVS, but it stumbled so I fixed it by hand. Here is a patch that works for me.

HTH

Attachment	Size
listhandler-head-to-5.patch	5.05 KB

Hi! Looks not bad.

The update to the mailhandler table does not happen, I had to run it from Mysql. Tomorrow I'll test it fully and report.

I dnloaded from CVS and applied listhandler-head-to-5.patch on top of it.

Basically nothing happens, apart that email subjects do get stripped of what should be removed. When one mails in content from the mailing list it gets posted (as with mailhandler alone), yet if I make comments on the forum nothing gets to the list. Any suggestions? Maybe I should have first patched with mailfromforum.patch and THEN applied listhandler-head-to-5.patch? It's not very clear from the previous postings.

One more problem.... when user answers to a topic from the mailing list, his answer does not become a comment. It becomes a new topic. Let alone breaking threads, this has heavier consequences, as it means that if you want to accept content from people who are subscribed to the list you either:
1) manually add accounts for each of them and give them a role for them to be enabled to create these "topics" (safe but heavy) OR
2) allow anonymous users to create the nodes on their own...

As I write this I am starting to understand that I haven't seen listhandler running at all. This is what MailHandler does. So when is ListHandler supposed to run, and how? based on cron?

I think you patched it correctly, listhandler-head-to-5.patch should be applied to HEAD. It includes the attachments code from mailforum,patch
If the comment is not getting to the list then this needs to be fixed, obviously.
Which listmailer are you using?
Did any messages get through to watchdog? In particular for listhandler, starting with 'Error sending.....'
Have you got tid: xxx in the commands field in the mysql mailhandler table?
I have managed to get it to work with ezmlm but I had to add some code:

This goes in place of

on line 313
It's the 'Return-Path' that drupal_mail() appears to strip out if the request comes from a module, but I'm not sure about this. 'Return-Path' is used by ezmlm to establish where the mail comes from, not 'From' as you might expect.
If you have access to your mail logs you might see a refusal, then you will at least know that an email was sent but not accepted.

You said:

As I write this I am starting to understand that I haven't seen listhandler running at all. This is what MailHandler does. So when is ListHandler supposed to run, and how? based on cron?

Listhandler does not use cron, when a comment is submitted, the listhandler function 'listhandler_comment()' or 'listhandler_nodeapi()' should be run and they run 'listhandler_send_mail()'. They also put a message into watchdog.

I am using the following function to help in debugging:

edit the path to your log and use it with lines like this:

Your other points are I think more mailhandler related and are working on my test rig so I can't really say why it's not working for you. Check that you have 'type: forum' in the commands field in mailhandler.

So... basically we need a better set of docs and a nicer UI... as it turns out, the current docs for mailhandler ask you to write commands for mailhandler in terms of:
taxonomy: ['catname1', 'catename2',...]

This is very nice but infact it disables listhandler.

If you are to use this module you should write in the form:
tid: 22
type: forum
promote: 1
status: 1
comment: 2

It would be much handier if we really could write catnames as strings, although this indeed can lead to trouble, as there is no unique index on catnames... Maybe we can put more tids on different lines, can we? I haven't checked but I'd say yes, given the way REGEXP works in the select.

There seems to be something going wrong on function listhandler_mailhandler, though. I keep receiving the number '2', instead of the text in the comments sent from the mailing list... will have to trace this later on.

I'm using mailman on debian etch, with mailhandler and mailman_manager. It seems to work fine so far, I only have to trace it and find out how the text of the mailing list comment gets to become a number... but okay, this evening I'll give it a couple of hours and find it out.

I really think it should getter a better explanation and we should update the mailhandler docs to explain the potential conflict in command style.

One more thought: I'm positive I can add more fixed taxonomy marks on automated input by putting it in sequence, I wish I could have a similar command for audience: (working with organic groups), so I can drive a dedicated forum into a given group. Yet this is quite an overlapping solution, as OGs do a lot af mailing themselves, which should definitely NOT happen when people are already subscribed to mailman... which means patching the OG module, too. To a fairly wide extent this kind of forums make OGs redundant, so probably I'll simply get rid of OGs altoghether... less stuff being executed is always good news :)

Hmmm... one thing we really may want is a follow-up to the comment/topic creation forms, that requests to people who submit new content but are NOT subscribed to the list whether they want to subscribe. It would save a lot of admin trouble and make better marketing for the lists. There is a lot of returning traffic from OGs because people get mailed about new stuff, if I am to get rid of OGs I'd rather have an equivalent traffic driver in place. I guess that would work.

Do you think it's possible to make it?

Listhandler only uses tid: nn
it ignores the rest. It uses mysql REGEXP to pull the right record from mailhandler table. See line 235 for an example:

Any other entries in mailhandler.commands are used by mailhandler itself.
I have just tested listhandler with taxonomy: ['this','that'] in commands and it runs ok

I have to say that i'm using now my patch on a production site and it does not work as it seemed to work on a fresh drupal installation.
Posts in the forum still gets duplicate.
Test site uses php5 whereas production site uses php4. Don't know if it could be the reasons, i don't think, but I'm going to investigate it.
Listhandler does not discarded duplicated msgid, even my patch seems to correctly generate them.

Just a quick addition. Duplicate comments are correctly discarded, but not open thread posts.
Hutch, does the patch correctly work for you?

Comments are working, I *think* new posts work too, I will test tonight and come back with results.
.........
The result is that I have it working both ways for both new topics and comments, and there are no duplicates, the Drupal log shows them being discarded. The code for this is in function listhandler_mailhandler(). It compares the Message-Id in the incoming mail to check if it has already passed through the system.

I suspect that the problems people are having are to do with the sending (listhandler) and receiving mail (mailhandler) The rest of the code is working.

It is possible that Message-Id is being over-written somewhere, which would have the effect of duplication.

I think i've found the reason with the new topic problem.
I run the cron.php with the php command line and in that enviroment the $_SERVER["SERVER_NAME"] is empty.
This leads the ereg function in listhandler_mailhandler() to check for matching, which obviously returns false.
For who needs it and can't change how the cron.php has to run, the quick workaround should be to override $_SERVER["SERVER_NAME"] with the basename of your site (i.e:www.mysite.com)

I would imagine running cron.php from the command line might have other ramifications for your site - why don't you just hit the webpage from cron instead of running it from the command line?

I think the command I use is just "wget example.com/cron.php" which will also return any errors cron pops up with such as missing db connection or whatever.

Long story. However, yes, you're right. And now it runs with wget :-)

As soon as i'll be free, i'll try to write a patch for handling mime attachment generated from forum messages. In the meanwhile, i'd suggest to change in the patched listhandler.module the line:

with

in order to avoid broken links in mails generated from posts where attachment name contains spaces.

This should be corrected in the install:

key (lid) is redundant, since you already have a primary on it.

Dear All,

Gerhard has kindly passed the development of ListHandler to myself and Ser.

Currently there is a 5.x-1.x-dev branch (http://drupal.org/node/202136) that is tracking the work for Drupal 5 compatibility. Everything in that branch will be ported into Head as well.

So far I've committed my patch (http://drupal.org/node/126126#comment-524188) and all subsequent patches are under review.

I plan to release a beta version in the New Year - end of January at the latest.

Please continue to post patches to this task; please diff them against the DRUPAL-5 branch.

Thanks.

Phil L.

Welcome Phil :-).
I'm glad that this module is still developped.

This patch solves the space bug with the attachment feature (upload module) from my previuos patch and adds support also for comment_upload module.
It does not support uploadpath module or any other module that changes file attachments path after listhandler processing (link in the email could be different from the real one).
If we want support more types of upload modules,i think that the attachments link implementation should be modular, with a plugin for every module.
It also adds, at the end of mails with attachments,a notice message with the link of the post/comment that generated it.
It contains all previous 5.x patches and it applies against DRUPAL-5.

Attachment	Size
mailfromforum2.patch	3.87 KB

This is good news ;-)

I am using ezmlm and have found it necessary to alter things a bit to get it to work. I have done this by expanding on the 'include' feature, which optionally allows a file to be included. It uses the mailhandler commands by adding 'list_type: ezmlm'. Listhandler looks for this and if found prepends the value to the filename to be included (if it exists), eg ezmlm_mail.inc which contains the following:

I have done this because I cannot get drupal_mail() to work, the mail is sent but has had some headers stripped out and ezmlm refuses the message.

It does not of course interfere with existing usage and provides greater flexibility for dealing with different listmailer software.

The attached patch contains all of mailfromforum2.patch plus new code to read in the mailhandler commands and handle the include.

Attachment	Size
listhandler.module-fileinc.patch	5.6 KB

Hutch, just to know, what's the MTA used by your drupal?
Looking at drupal_mail:
// Note: if you are having problems with sending mail, or mails look wrong
// when they are received you may have to modify the str_replace to suit
// your systems.
// - \r\n will work under dos and windows.
// - \n will work for linux, unix and BSDs.
// - \r will work for macs.

If you can find that this why drupal_mail does not work for you, i think that your patch should refer to a general solution for every Windows MTA servers instead of for a given listmailer.

The server running Drupal is linux Debian sarge running a qmail MTA.

I will experiment some more with drupal_mail() and see if I can get it to work.

The ezmlm listmailer requires the Return-Path header to be set the same as From:

I don't think the problem is linefeeds, the reason for using \r\n to separate header lines is so that Windows based MTAs can handle it. Qmail handles it either way. The mail arrives but is rejected by ezmlm, I can see this in the qmail send logs.

If the .inc file does not exist and/or no list type is defined the default drupal_mail will be run instead so there shouldn't be any problem there.

Anyway I will experiment some more and try to figure out why drupal_mail is mangling the headers, the drupal_mail code suggests I should be able to get the Return-Path header set the way I want it.

Hi!

now working perfectly. My lost comments came from having set a:
comment: 2
command in mailhandler, which was supposed to state that newly imported messages could be commented.

What I am using is:
Debian etch
Postfix with mysql based virtal domains
Mailman

Great module, btw.

Added full support for mail attachements in mails generated by forum posts.
Listhandler configuration page has now the option to enabling the convertion of forum attachments in links or send them as mail attachments (the default).
Again, it contains all previuos patches, it applies on DRUPAL-5 tag and it's reviewed with code review module.
Hutch, it does not contain your patch because i can test it and i fear to loose something in appling it, so i leave you the task (consider to remove your debug function).

Attachment	Size
mailfromforum3.patch	3.82 KB

Your patch applied perfectly, I will try to persuade drupal_mail() to work with ezmlm as soon as I get a moment.
I presume that this does work with Mailman.

Yes, I've tested it with mailman.
I was in hurry in my previous post so, in addition to my english grammar errors, i attached an incomplete patch.
This is the corrected one with the missed attachment configuration option.

Attachment	Size
mailfromforum4.patch	4.14 KB

Patch mailfromforum4.patch is working fine and it is now working with drupal_mail()
I had to add the following though:

Immediately above the drupal_mail line.
Without it drupal_mail sets the Return-Path to the default mail for the site and ezmlm rejects the post as that email address is not a member of the list.

So far so good ;-)

Hey Guys,

I've been watching this thread for awhile, and I'm really impressed with the work being done to bring this module up to snuff for 5.x. Thank you!
I know its not quite finished yet, but I'm just curious how hard it will and whether there will an effort to upgrade this to 6.x once the efforts here are finished. I'm only asking because of the 6.0 release candidate announcement. If its a "not any time soon" or "it'll be hard" that's fine since i probably won't be upgrading to 6.x soon anyways, but just wanted to ask. Can also put this into a new issue/thread if necessary.

Hey rcross,

Ser has taken on development for Drupal 6, however he's currently waiting on me to finish my work on a Drupal 5 version.

I expect a 1.0 stable release to take a few months as the beta release will need a bit of time for 'being out there and available' for adequate testing.

Please wait until 5.x-1.0 stable is available before creating a new task for a Drupal 6 version.

Thanks.

Thanks for that hutch.

I've put that in CVS and an updated .tar.gz will be along shortly.

Phil, Hutch patch is modelled for my patch, where $mid is an array, and i don't think it will be useful without mine.
You should probably write it as:

but in this case it'll not work anyway because the wrong drupal_mail usage does not process mail headers.

Just to confirm that, $mid needs to be an array to work with drupal_mail().
'References', 'In-Reply-To', 'Message-Id' and 'Return-Path' are the 4 keys.

could go just below

so that it would get picked up by the _listhandler_mail_attachments() function. Not tested by me but I don't see any reason why it would not work.

Thanks for pointing out my schoolboy error!
I'll work on getting samuelet's patch in next.

Hi Samuelet,

Can you re-roll your patch as a unified diff format please?
The line numbers are now out of sync and your patch now puts code in the wrong places :( .

Thanks.

P.

I have attached two patches, one for ezmlm and one for samuelet's attachments. They should be applied in that order. They were built from listhandler.module-5.x-1.x-dev of 2008-Jan-05
I have tested it and it has sent mail, not tested the attachments feature though.

Attachment	Size
listhandler.module-20080106-ezmlm.patch	1.84 KB
listhandler.module-20080106-attachment.patch	4.81 KB

Just curious - would it make sense to write this module to utilize the functions from mailhandler.module? It seems like there is already quite a bit of duplication efforts going on. (Note - i haven't looked at the code, just following this thread with interest, so apologies for any ignorance)

Hi rcross,

I'm not sure where the idea of duplication of effort comes from. Mailhandler handles the incoming email, Listhandler send outgoing email. Decoding of email and creation of email are different processes.

However, feel free to look at the code of both Listhandler and Mailhander and suggest any benefits or improvements you can see. Please create a new task (for a 2.0 version) if that is the case as this task is specifically for providing a Drupal 5.x-1.x version of Listhandler.

Thanks.

P.

I agree that mailhandler is aimed at handling incoming email - but my understanding is that listhandler also does this by creating new nodes based on email from the mailing list (and filtering out those messages it has sent). I just remember seeing a post or two about making another module "compatible with mailhandler", so I figured it might be worth investigating. Apologies for possibly hijacking this thread - just didn't seem to make sense to post a new issue if it was pointless, and there isn't a forum for this module or anything. I'll try taking a look at the code when I get a chance (probably won't be soon though)

Listhandler-5.x-1.0-beta1 is now out.

Please download and test.

P.

I've installed Listhandler-5.x-1.0-beta1 and I have found one issue. When I hit reply in an email no matter what I put in, what shows in the post is a number 2 and that's it. Just 2.

I'm not a programmer so I really can't tell you anymore than this. Any ideas on this?

Not sure if anyone else has this problem, but here it is and what i did to fix it:

I was using mailhandler and listhandler to keep a forum and a mailing list in sync. The problem is, when people would respond to the forum *comments* with an email reply, it was creating a new forum topic. i.e. wasn't finding the comment as a parent. Seems there were 2 problems: on line 496, it didn't need the +1, and on line 506, i added some code to check the comments table for the title. Here's what I have from lines 495 on:

Hope that helps someone out.

subscribing - this is much anticipated... thanks!

Seems there were 2 problems: on line 496, it didn't need the +1

Oh, interesting.
I went through the logic on that line and the code looked right.
But I put a little bit of debugging in there and sure enough, I could get it to skip the first character of the real subject.

Committed.
Thanks.

and on line 506, i added some code to check the comments table for the title.

I'm not sure how I can test this. Surely the Listhandler logic should have caught it if it was a reply to a forum thread. What specific problem did this code solve for you?

Thanks.

P.

The ETA is:

The current 5.x-1.x-dev is feature complete and stable as far as I am concerned.

Unless there are any substantiated bug reports then there will be a 5.x-1.0-beta2 release on the 31st of May.

Pending no issues with beta2, then there will be a 5.x-1.0 (stable) release on the 28th of June.

Regards.

P.

Apologies for the lack of release this weekend.
I'm currently preparing for a job interview (not Drupal related ;) ) so that is my #1 priority right now.
Expect a release by the end of next Saturday.

Regards.

P.

A Drupal 5.x stable release of Listhandler has been made:
http://drupal.org/node/287804

Regards.

P.

Automatically closed -- issue fixed for two weeks with no activity.

thanks a lot!