Improve simplenews throttle

I second this and like to ask if this can be implemented in 5.x as well? Please.

Kind regards,

Frank

yes I have alaways the same problem , some people receive the same email 2 or 3 times
I will try these solution for drupal 5

@GiorgosK, please provide a patch for a watchdog message as you described. This is a good simplenews core functionality.

I made the attached patch for this functionality. The elapsed time is reported in the Log, the available time is added to the description of the cron throttle setting. GiorgosK thanks for the idea.

Attached patch is committed.

I do not want to add (semi) static information to a watchdog message. If it is always the same, it will be ignored. The Send email settings is the place where the admin sets the cron throttle and needs to know the max time.

Perhaps the user should be warned if sending time take more than 80% of the max execution time. But this percentage is arbitrarily. Do you have any experience with a save margin (as percentage or seconds)?

Alternatively sending should be stopped if 80% of the max execution time is spend. However this will cause a problems with sending newsletters without cron because not all emails are send, and there may be no cron to send the remaining. But on the other hand exceeding the max time will cause problems any way.

You are right but
1. there are more reasons to limit the number per cron job. Like a max number of emails per hour set by a hoster and other cron jobs that also need time.
2. this is theory ...
After a few hours investigation I found out that the PHP max execution time is approximately the sum of user time and system time as you can get with getrusage(). It does not correlate with time() because "Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running." (see: set_time_limit()). So getrusage() is the way to go. But getrusage() is not available on windows platforms, so we need a fallback.

time() is not suitable to measure the execution time at all. Sending 1500 emails from my test site takes 35 seconds (time()) and 2.1 seconds PHP execution time.

Many scripts prevent the max execution time problem by increasing the max execution time limit or disabling it using set_time_limit(). But there is no real need to do this here. We send discrete emails and can break any point we want, unlike of example uploading a large file. And further safe mode prevents changing the time lime.

A fall back is also required for the situation where cron is not used, but emails are send immediately.

Attached patch does what I described.

Patch committed.
Note that this functionality does NOT work on windows platforms. I expect that the elapsed time in the log will be 0.0 seconds. Feedback from users with drupal on a Windows platform is highly appreciated.

This is a different subject, the initial request was about max execution time.
All my sites and my customer's run on shared hosts. I have only encountered one host who restricted to max 20 smtp connections at once, this forced me to send in batches of < 20. I remember one similar issue in the queue. Are you talking assumptions or actual problems? Please open a new issue, if this proposal fixes actual problems.

Sending a mailing on a one-email-per-6-seconds host is like filling a bath tub with a thee cup. My advice: get another host.
Still, I can see a need for this, but I'm not sure if simplenews is the right place. It is not just simplenews who suffers from this problem, all mass mailing applications do. Perhaps a downstream is a better place.

Implementing this in simpelnews may not be too difficult. The biggest problem I see is how to create a trigger that fires every x seconds fully in the background. If you use a function with sleep() or usleep() and you call it with poormanscron, the user will see a white screen. This should not happen.

If you can build a prove of concept, I would be interested for simplenews integration.

Since PHP has no multi threading, perhaps this can be done in the manner of cron.php. A php file called by a separate crontab process. For example every hour crontab calls example.com/sites/all/modules/simpelnews/mail_cron.php. This starts a process which runs for a bit less than one hour. The process calls simplenews_mail_spool() every x seconds to send y emails. It just has to make sure that simplenews_cron() or any other calls to simplenews_mail_spool() are inhibited. This process should monitor is't own PHP max execution time.
This can only be used in combination with crontab (or other external cron triggers) and not with poormanscron.

Sleep() is the trick but it has important side effects. No poormanscron is one.
Perhaps this is the time to start looking at the Drupal engine after driving it for so many years ;)

I like to see a feature in simplenews that can send emails in with frequent small batches like phpList does, using the same three parameters: batch size, time between batches, max duration. This should be a cron driven but separate from drupal's cron. i.e. mail_cron.php instead of cron.php. Who wants to develop this feature?

Attached is the first version to improve the simplenews throttling system. This patch is based on ideas of phplist referred in #16 and the observation that simplenews cron should not be run with poormanscron.

Instructions:
- patch simplenews HEAD with the attached
- Move/copy the mail_cron.php file to your web root
- Setup cron(tab) to call mail_cron.php
- Setup simplenews mail settings as admin/settings/simplenews/mail
- Set 'Cron mail limit' to the limit set by your host. e.g. 600 emails per day
- Set 'Mail cron duration' equal to the duration set for cron(tab). e.g. 1 hour
- Set 'Mail method':
In the above example Batch sends 25 emails (= 600/24) and wait for 1 hour (= cron duration).
Throttle sends single emails with 2.4 seconds (average) in between the mails, resulting in 25 emails (= 600/24) in one hour (= cron duration).

If your system sends other emails apart from simplenews, you should reduce the cron mail limit. For example to 580 emails, which leaves space for other applications to send 20 emails per hours.

duplicated by #399352: Simplenews and throttling

Hi there

Great module, but I am quite a newbie when applying patches.

So my questions are:

- how do I apply this patch and
- where (which folders) should I copy this patch too?

I have set up my contributed modules structure in sites/all/modules/contributions/

Is there anything else I would need to do to get this up and running (ie. run update.php?) Would I expect to see any new options etc in the Simplenew tab etc?

Thanks for your help

To apply a patch see: http://drupal.org/apply/patch

New patch attached.
Changes:
- Old cron sending method no longer triggered by cron.php.
- Status results changed in simplenews_mail_spool().
- Documentation changes in simplenews_mail_spool().

Hi,

I don't see this committed anywhere in the CVS messages. Checked the source code, and it's not there... why was it closed?

I'm needing this feature since Hostgator requires the sending in batchs of no more than 500 emails per hour each batch, throttled down to 1 email per 8 seconds.

Should I apply this patch and review/test? Thanks.

yes, please apply to HEAD and report you results.

Thanks. Ok, we've applied the patch against the Oct 25 dev version available for download in the module page.

3 hunks failed, and had to be applied manually. One minor issue when doing so with the simplenews.module file:
There are 2 lines that doesn't appear to be modified, but are not in the -dev version nor in added the patch. After this

// Update results for reporting in newsletter admin panel.
simplenews_send_status_update();

comes

variable_set('simplenews_last_cron', time());
variable_set('simplenews_last_sent', count($mail_sent));

but the variable_set() calls were not marked as "added" in the patch. I just added them, hoping it will be fine. Please confirm this is how it's supposed to be.

    // Update results for reporting in newsletter admin panel.
    simplenews_send_status_update();
    variable_set('simplenews_last_cron', time());
    variable_set('simplenews_last_sent', count($mail_sent));

We'll be testing and reporting the results soon.

After cleaning the theme registry, we got the screen attached at admin/settings/simplenews/mail

Usually, cron.php (not poormanscron) will run every hour. We DON'T have to setup a scheduled call to mail_cron.php do we?

A more general example:
Suppose we have a list of 1200 subscribers. We have cron.php firing each hour. The shared host imposes these restrictions:
- Blasts should happen overnight
- no more than 500 emails per hour
- throttled (1 email each 8 seconds approx.)

We want to use simplenews, how would we configure it?

Thanks in advance.

Well, with cron.php running every hour, emails are not being sent...I could use a little bit of help here...TIA!

I'm interested in trying the patch from #26 but as per #30 it's not applying cleanly. Is there a new version of the patch so I can test it?

Pointing to related
#361071: Use locking to eliminate duplicate emails from concurrent simplenews cron runs.

subscribe

As we're close to an agreement on #361071: Use locking to eliminate duplicate emails from concurrent simplenews cron runs., I'm proposing to close this one as "duplicate", waiting for the completion of the other issue in 6.x-2.x before porting it to 7.x.

@miro_dietiker, @Sutharsan, any opposing thought ?

Well, throttling is not realls the same as the concurrency issue.
But still Closing.

We have implemented elysia_cron to allow you fine-grained controle over cron tasks.
If you need further controle over the time simplenews sends, add custom rules to elysia_cron.
#1010684: Support Elysia Cron hook_cronapi

If we need some further tweaking in the throttle algorighm, please reopen.

I could even recommend you to use drush to call the simplenews cron separately and disable it on the regular cron run. You'll have perfect controle over it that way.

Drupal itself should improve its cron / background task halding... instead of every module again...

Has anyone successfully implemented Simplenews throttling to comply with email restrictions common to cheap shared hosting (example #31)? After reading through this thread, it does not seem like there have been clear successes. I'd love hear some strategies that worked.

I've used PHPList in the past, but I'd love to be able to integrate newsletters more closely with drupal data.