Last week a client made a bulk update to their Salesforce Contacts which triggered 44,000 outbound messages to the Drupal site. While about 90% of the messages have been processed in the last 7 days, there's still 4,000 messages that are jammed up in outbound message queue with the error message java.net.SocketTimeoutException: Read timed out
. I imagine this has happened to other folks too.
A suggestion from a Salesforce engineer was to create a buffer on the Drupal side - return TRUE immediately to Salesforce and store the outbound messages in a table until they can be processed.
The code in sf_notifications to receive an outbound message looks something like this currently:
$content = file_get_contents('php://input');
$dom = new DOMDocument();
$dom->loadXML($content);
$resultArray = _sf_notifications_parse_message($dom);
$ret = _sf_notifications_handle_message($resultArray);
// Sends SOAP response to SFDC
if ($ret) {
_sf_notifications_soap_respond('true');
}
else {
_sf_notifications_soap_respond('false');
}
exit;
Instead of sending the SOAP response after parsing and handling the message, what if we had something like this:
$dom = new DOMDocument();
$dom->loadXML($content);
if ((variable_get('sf_queue_incoming'), FALSE) && function_exists('sf_queue_add_incoming')) {
_sf_notifications_soap_respond('true');
sf_queue_add_incoming($dom);
exit;
}
$resultArray = _sf_notifications_parse_message($dom);
$ret = _sf_notifications_handle_message($resultArray);
sf_queue_add_incoming would take the DOMDocument and store it in a table until it can be processed. SF Queue would then have some admin configurable settings about how frequently to process items in the incoming queue as well as how many items to process at once. We would also have to provide some admin settings or at least a hook that allows admins/developers to do something if there was an error with parsing or handling the message.
What do you all think?
Comment | File | Size | Author |
---|---|---|---|
#5 | incoming-queue-1127166-5.patch | 6.47 KB | kostajh |
Comments
Comment #1
AaronBaumanI have just been dealing with this myself.
By the time sf_notifications gets the message, the bulk of the PHP processing has been done in the Drupal bootstrap.
In order to respond to Salesforce in a timely manner, we need to avoid bootstrapping altogether.
In other words, create a standalone php script to handle messages from Salesforce and queue them.
Then, modify sf_notifications to process the queue on cron runs.
Any approach that relies on, e.g. menu callbacks, will not solve this issue.
Comment #2
kostajh CreditAttribution: kostajh commentedGlad to see I'm not the only one with this problem.
Would implementing hook_boot in sf_notifications be an option?
Comment #3
AaronBaumanI really think the best approach is going to be handling outbound messages outside of a Drupal environment altogether.
Comment #4
kostajh CreditAttribution: kostajh commentedI think it would be good to support both hook_boot and a script that would reside outside of the Drupal environment.
In either case, sf_notifications needs to have its own table for an incoming queue and an implementation of hook_cron with some options for processing the messages.
I have a working implementation that I'm going to put through the paces tomorrow, I'll post a patch here after testing.
Comment #5
kostajh CreditAttribution: kostajh commentedHere's a work-in-progress patch that does the following:
I tested this with a bulk update to 250 records, and it performed just fine, no timeout errors on the Salesforce side.
This patch still needs:
Testing and feedback would be very helpful! Be sure to run update.php or re-install sf_notifications so that the new database schema is installed.
Comment #6
EvanDonovan CreditAttribution: EvanDonovan commentedWill have to make sure that the implementation we do in 7.x includes this also.
Comment #7
brianV CreditAttribution: brianV commentedAssigning to myself.
Comment #8
EvanDonovan CreditAttribution: EvanDonovan commentedWe (brianV & myself) added this in 7.x-2.x using the Drupal queue.
We won't have time to backport to 6.x-2.x, but someone else could, if they wanted to make that a 6.x-2.x dependency.
Comment #9
kostajh CreditAttribution: kostajh commentedCould you post the commit hash where this was added to 7.x-2.x?
Comment #10
brianV CreditAttribution: brianV commentedThese are the three relevant commits:
http://drupalcode.org/project/salesforce.git/commitdiff/99af9d7f1b713439...
http://drupalcode.org/project/salesforce.git/commitdiff/c80ccc7359da382f...
http://drupalcode.org/project/salesforce.git/commitdiff/0d1f8f338dde3f2f...
It could probably be backported more or less directly to D6 if desired with a dependency on the Drupal Queue module: http://drupal.org/project/drupal_queue
Comment #11
kostajh CreditAttribution: kostajh commentedCool, thanks Brian. Have you all tested how well it holds up with thousands of outbound notifications coming in rapid succession? This was the concern that Aaron and I had in earlier comments on this thread.
Comment #12
brianV CreditAttribution: brianV commentedkostajh: We haven't tested in under extremely heavy load yet.
Our view is that this is much better than the non-queue implementation, but can still use room for improvement if it turns out to be warranted in practice.
There is no reason we couldn't add an additional PHP file to use as the endpoint that simply writes new queue items to Drupal's queue table, and returns an 'OK' to Salesforce.. If you look at SystemQueue::createItem(), you can see that the required query is very simple:
http://api.drupal.org/api/drupal/modules--system--system.queue.inc/funct...
Comment #13
EvanDonovan CreditAttribution: EvanDonovan commentedHmm....interesting thought, Brian. That way a bootstrap would be avoided.
When we get into the testing phase, we can see if this would be necessary.
One thing to note is that we already have made it so that the entire cache doesn't get cleared each time an insert is done in response to an outbound notification. That would've been a huge performance hit.
Comment #14
kostajh CreditAttribution: kostajh commentedThanks for the notes. I think it is worth implementing Drupal Queue for both sf_import and sf_notifications in the 6.x-2.x branch.
This should be in another issue, but wanted to mention that there is also the problem of processing the queue. When there are tens of thousands of records, trying to process them on cron run means that it will take weeks for a queue to get processed. I've been looking at using drushd (http://drupal.org/project/drushd) to help with that, but do you have any other ideas on how to handle this?
Comment #15
kostajh CreditAttribution: kostajh commentedThere is no patch for 6.x-2.x, so setting this to needs work.
Comment #16
kostajh CreditAttribution: kostajh commented