Mailing lists and sympa

Hi again!

So sorry I didn't write earlier, I guess I'll have to review my subscription options on drupal.org (I thought I would be notified, but well... =D)

Thank you for all your suggestions, let me explain :

- The mail server of the university only accepts a certain number of e-mail (something like 50 or 100) every 10 minutes or so, but these 10 minutes are not reset unless the sending mail server has been quiet for 10 minutes. I'm not too sure how exactly their spam filter works, but it's something like that. What this means is that using any web server in the world (excepted a whitelisted one), it takes about 2 or 3 days to get some 200 e-mail over =D The rest are rejected (because most of the webservers stop trying after that).
- As we're using public web hosting for the webserver, the university would not whitelist it (which is understandable).
- The problem is to find a mail server that is using sendmail and not qmail, for instance. Why? Because what the mail server of the university allows is one single e-mail with a lot of Bccs. Perfect, you might think, in that case, I just modify the notification and everyone gets the same e-mail. Unfortunately, qmail and some other programs of it's kind take the e-mail with all the Bccs and split it :) As a result, the mail server of the university get's the same amount of e-mails. Sendmail doesn't do this, but virtually no one uses sendmail (google for instance isn't, if my last test was correct).

Basically, to summarise, I will never win a fight against the super high end spam filter of the university if I don't manage to do one of the following :

- send e-mail directly to the mail server putting everyone in BCC... which is not possible, unfortunately, because my web hoster blocks port 25...
- use their list server.

The issue with what salvis called quite correctly "legitimate users" is that in case the webserver gets hacked (yes, yes, you always have to start from the worst case =D), it would not be a legitimate user anymore, and as they can't control our webserver (and don't want to), they have to protect their precious e-mail system against this kind of threats. :)

Now the thing with the "time conflicts". Cron is run every hour (that's enough for notifications). My plan was to "rotate" through the list addresses to send everyone his notification. Now, if I have, let's say, 10 lists, and 11 people are posting, in the same hour, something which needs a notification sent out, then the subscription module would use for the first notification list1, for the second notification list2, for the third list3..., and for the eleventh list1 again. That would mean that the first notification and the eleventh notification would be sent to the same people (the ones that were supposed to get the eleventh).

I don't know if I'm really clear about that, if not, I'll try to explain it again =D

Anyway, supposing there was no way around this =D (or, actually, :( ), do you think modifying the mailer module of subscriptions would be a good place to start? Or should I rather modify another module (like notifications, or some other module...)? I'm asking this also for "future development", i.e. because I'd like to make sure subscriptions will be ported one day to drupal 6, etc.

Supposing I decide to move on with the subscriptions module, where would you store the information about the list subscriptions should use next? In a new entry in the database?

Well then, thank you very much for your ideas and your support, I hope I'll manage to find a good solution for everyone =D

Cheers and have a nice day!

Felix.

Hi Salvis!

Thanks for your complete and quick answer, I'll try to give you answer to everything =D

That seems perfect. If you run cron every 12 minutes, and you limit it to sending 50 messages, you can get 250 messages out per hour. Yes, you can set the maximum number of messages per cron run.

Oh, cool, that might be an acceptable workaround for the moment (even if that means that not everyone gets the messages at the same time...). Another limit is my provider, that is limiting cronjobs to 1 per hour (but don't worry, webcron.org will be able to help me out =D)

It takes about 2 or 3 days to get some 200 e-mail over

Something's not right here...

You're so right :) That's the problem... Everything that comes from outside the university is suspicious...

You mean you'd gather the recipients of the notification for node 123 into one list, send that to the listserver to reconfigure its subscriber list, send the notification to the listserver for distribution, gather the recipients of the notification for node 124, reconfigure the listserver again, etc.?

Yes, nearly. Actually, except for the step "sending the list to the listserver" because the listserver would actually retrieve that list itself. (sympa can "link" one list to one textfile that it retrieves every time it sends an e-mail).

I don't think this can work reliably, because if two emails are sent in rapid succession, there's no guarantee that the first will arrive before the second. You'd have to be able to put the recipient list into the actual message and have the listserver separate the two.

That's where the ten lists would come into play. To avoid this kind of problem, I wouldn't use the same list every time. The notification for node 123 would be sent, for instance, to list1@listserver.com and the recipients written in list1.txt. The notifications for node 124 would similarly be sent to list2@listserver.com and the recipients written in list2.txt. And after 10 notifications, we start over from list1 again, hoping (knowing, because users can only subscribe to taxonomy terms, and only to some of them) that there won't be ever 10 nodes posted between two crons (in an hour's time).

Is your IT infrastructure department in the service business? Why don't you just let them help you...?

No, you have to understand, our IT department is extremely busy :) It's already a big concession from their side to let me use their list server...

I will certainly port Subscriptions to D6 — I need it myself. :-)

Cool =D

I hope it is clearer now. Maybe the "workaround" solution you mentioned (limiting the number of e-mails sent out - a feature I had overlooked because it was not in the first version of subscriptions I had installed) will be quite helpful in the meantime...

Please tell me what you think of the idea with the lists, or any other idea if you think what I was suggesting is simply too stupid... :)

Cheers, thanks a lot and have a nice evening / day, wherever you are right now,

Felix.

Hi Salvis,

Thanks for all your help.

The BCC issue you might have been referring to might be this one http://drupal.org/node/182450 and was created (and closed right now) by myself... Now this is not a good solution because of the reason I mentioned above (most of the servers are using something like qmail which splits the message and sends every e-mail separately, thus loosing the advantage of the BCCs, and plus, you actually can't put that many BCCs in one mail). Anyway.

Now, I haven't got a forum installed or anything where you can post comments that will be sent out to everyone, so there won't be any "hot topic" and tons of replies in one hour (at least something positive =D).

The custom mail exploder would be a really nice idea, indeed. But... (always a but...) the university doesn't want me to do that, they prefer their list server. What can I do?

Oh well, I'll try to dig into the code and find out the best way to make a list out of what I get from the subscription_mail module. The problem is the following: in 1.9, the e-mails for a node were processed together, so I could group them easily.

Now, I have no idea how the new module will check the subscriptions... Will I have to create 10 global variables and check every time if $body == $bodyGLOBAL1 ? And at the end, send all the e-mails ? That's not very clear. Basically, each of those global variables would contain one body of a subscription that needs to be sent out between two crons. After processing all the subscriptions (running through the loop), I would have one array of user IDs (and e-mail-addresses) for each "type" of e-mails sent out, and would then send the e-mail and write the list with the user IDs in a text file.

Sound feasible, but not exactly simple. Would you recommend sticking to 1.9 (although it has the known issues with rights management and will probably not be ported to Drupal 6)?

I'll try to post some code as soon as I've come a bit further.

If you have any other idea like the one you posted before (with the custom mail exploder), please tell me, I'm not too much fond of the idea of programming something is not necessarily ideal...

Well then, cheers, thanks a lot again, and have a nice day,

Felix.

Hi Salvis,

Sorry for the long silence.

The content types will be a couple of different ones, but they're all very simple content types (some text, a body, that's about it).

All the subscriptions will be "triggered by taxonomy", i.e. people are only able to subscribe to taxonomy terms (but there again, different terms from different categories...)

If you're willing to bet that there will never be more than one subscription item created at the same time, then all pending queue records for the same item will have consecutive sqid's.

No :) I'm not =D That's a dangerous bet to make...

If 1.9 works for you, then feel free to keep using it, but I've suffered too much from it (as a user) to recommend it for anything, and I won't ever touch it again. So, unless you do it, the chances for 1.9 to get ported to D6 are nil.

I guessed so, and I understand why (although I quite liked it, it did most of what I wanted without "complaining" =D).

Basically, you seem to be saying that the task is nearly impossible, unless I want to recode the whole module :)... If that is so, I'll try to look for some other solution, if you have any good idea, please tell me (for instance another module close to subscriptions that you know might be easier to recode ?) :)

Thanks for your time and effort,

Felix.

Ah, cool :)

Comments are disabled nearly everywhere, so comment subscription won't be needed. And update subscription might be nice, but is not absolutely necessary.

Thanks for your help!

Felix.