Message throughput?

There are two things that will affect your throughput: your MaxSendRate (as determined by Amazon) and the number of requests your server can make simultaneously. Amazon starts you at 1 message/second to prevent spammers from using SES and increases the number as you prove the quality of your emails. (See their doc pages for info on sending limits). There is also an option to petition the AWS powers-that-be to increase your limits by reviewing your use-case in person. (Sorry, don't have the link to that on hand...)

The number of simultaneous threads issue is because SES takes about 2-3 seconds to process a single call to AmazonSES::send_email(). There was some talk about how to deal with this using CFBatchRequest to help with queuing messages on the Amazon forums but I haven't tried those options yet.

As mentioned, Amazon will limit you to 1/second and 1000/day until you prove to them you're not a spammer. I believe they will raise your rate automatically, and this is something you can monitor from the Amazon SES module backend. It's worth noting that you could potentially send emails faster by sending the message to multiple recipients. They are billed as separate messages but would get sent all at once, I believe.

If your needs are great, you may need to send messages into a queue multiple threads can attack. I'm afraid job_queue for 6 and the built-in queue for 7 probably won't buy you much (except batch processing) since they run on single-threaded cron. If you would like to place a small bounty on this, I'd be more than happy to write it up as a separate module to enable using something like Amazon SQS, but for now it's not in my pipeline.

http://aws.amazon.com/sqs/

Ryan Palmer

// Lead Developer, Drupal Connect
// drupalconnect.com

I haven't personally tested the time it takes to send messages at that rate, so I can't confirm or deny the "2-3 seconds per message" observation. But it would not surprise me if this was the case, considering round-trip network latency and so-on. Do you host with Amazon? It could potentially be faster if you did.

Nope, that wasn't me! I didn't read the thread yet. What's the problem?

1 email call every 2-3 seconds = 20-30 emails per minute, or 1200-1800 emails per hour?

Let me clarify that those number are what I was seeing when calling Amazon's servers from either my localhost or the server my site is running on (cheap-ass shared hosting). As Ryan mentions, I assume you'll get better results if you on an EC2 instance. Though my latency to other AWS services usually isn't very high, so who knows...

I've had some success using CFBatchRequest (see the link in #1). Theoretically, you could batch all your calls and execute them in parallel, but I haven't played with it too much so I don't know the ramifications this would have on you hitting your MaxSendLimit. When I get that figured out, I'll submit a patch.

Attached is a patch that allows bulk sending without multiple calls to send_email(). It involves a bit of refactoring, so I would appreciate testing and reviews...

Here's what I did:

• Much of amazon_ses_send() was refactored into amazon_ses_prepare_message() which converts a Drupal hook_mail_alter message into an AmazonSES message
• Added amazon_ses_send_bulk() which accepts an array of emails and a Drupal-style message array. send_bulk will queue up all the messages and send them with a single call to AWS.

With an array of 100 email addresses, I was getting a turnaround from send_email of around 1.8 seconds (180 seconds total processing time) vs. about 7.5 seconds using this method.

Please see the note at approx line 65 in amazon_ses.mail.inc regarding hitting your MaxSendRate using this method. I haven't hit it in my (fairly limited) testing, but I'm interested in hearing how others fair. Specifically post your MaxSendRate and the number of messages successfully sent via this method. In particular, post any Throttling errors you get.

Eg:

MaxSendRate: 5.0
Sent 350 messages successfully with 6 failures (all address blacklisted errors)

Without any confirmation from Amazon, it seem they have doing the right thing w.r.t. queuing messages and sending them at whatever your allowed MaxSendRate is. Either that or they allow short bursts over your MaxSendRate as long as you don't do it too often. Either way, I believe getting some real numbers would be instructive.

Thanks.

Without any confirmation from Amazon, it seem they have doing the right thing w.r.t. queuing messages and sending them at whatever your allowed MaxSendRate is.

Scratch that... Just ran another test with a total of 485 messages:

MaxSendRate: 5.0
Max24HourSend: 10000
Send 270 successfully
215 returned errors
209 were throttling errors

I had not sent any messages in the previous 24 hours so all the throttling was due to too many messages per second.

OK, here's another patch -- same refactoring as #10 -- but it gets the user's MaxSendRate from Amazon and uses that for the queue size. I've had better luck with this method, sending to basically the same list in #11 but without the throttling errors. 'Course it took 70 seconds to process the request...

In order to truly maximize the rate we can send emails via SES, we would need to track the time it takes for each batch to process and use that to adjust the queue size (eg: 2 second round trip means we use 2 * MaxSendRate in order to make as close to MaxSendRate requests per second). Anyhow, the refactoring makes it easier to do something like that in the future.

Coo, thanks for that. Is this for the D6 branch?

Not trying to be contrary, just curious: what does this get you versus just bracketing the actual send in locks and variable_get/set to ensure each process waits long enough?

That way it will work even if you need to send a bunch of different emails. For example, trying to use SES to send the account creation email after using the user_import module still wouldn't work. I only mention this because it happened to me yesterday. ;)

In any case, it seems like a good idea to guarantee that you never exceed the send rate, and just log whether or not it was an issue, regardless of whether the module is being used correctly.

Something like this:

     //wait until no other process is sending to protect 1s buffer
     while (!lock_acquire('amazon_ses_send', 0.1)) {
       ;//just wait
     }
     $result = $mailer->get_send_quota();
     $max = (int)$result->body->GetSendQuotaResult->MaxSendRate;
     //make sure we wait 1s from the last send, or at least 1s if we don't know
     $wait = variable_get('amazon_ses_last_send', time()) + 1/$max; //need to get a decimal, not 0, but you get the idea
     while (time() <= $wait) {
       ;//just wait
     }
     $mailer = new AmazonSES();
     $status = $mailer->send_email($source, $destination, $ses_message, $opt);
     //log our completion for the next process, and release the lock
     variable_set('amazon_ses_last_send', time());
     lock_release('amazon_ses_send');

@Ryan, yes it's a D6 patch... I don't have a D7 project using SES yet. ;)

@wamilton, the issue I'm trying to solve is that SES can take 2 seconds to process a single send_email() call, at least from my localhost and cheap shared hosting account. This patch allows you to essentially send send_email multiple requests to SES in parallel. If your MaxSendRate is 10 you can get 100 emails out in 20 seconds (sending 10 in parallel) rather than 3+ minutes (sending one at a time).

Though this raises the point that if you're getting a round trip from SES in less than 1 second, you may get throttled. That's a problem I'm hoping to have one of these days... Or if you have two processes that trigger send events, you may get throttled. Adding the semaphore lock would be a good idea in both the bulk and non-bulk send functions.

Nice thread, but old.

SES initial max send limit is now supposed to be 10,000 emails per day. Which is increased automatically if quality is sufficient.

The other numbers are quite nice, but I would like to get more updated info regarding Drupal throughput.

Sorry, Now my code base does not support for drupal 6.