Last week a client made a bulk update to their Salesforce Contacts which triggered 44,000 outbound messages to the Drupal site. While about 90% of the messages have been processed in the last 7 days, there's still 4,000 messages that are jammed up in outbound message queue with the error message java.net.SocketTimeoutException: Read timed out. I imagine this has happened to other folks too.

A suggestion from a Salesforce engineer was to create a buffer on the Drupal side - return TRUE immediately to Salesforce and store the outbound messages in a table until they can be processed.

The code in sf_notifications to receive an outbound message looks something like this currently:

  $content = file_get_contents('php://input');
   
	$dom = new DOMDocument();
	$dom->loadXML($content);
	
	$resultArray = _sf_notifications_parse_message($dom);
	$ret = _sf_notifications_handle_message($resultArray);

	// Sends SOAP response to SFDC
	if ($ret) {
		_sf_notifications_soap_respond('true');
	}
	else {
		_sf_notifications_soap_respond('false');
	}
	exit;

Instead of sending the SOAP response after parsing and handling the message, what if we had something like this:

$dom = new DOMDocument();
	$dom->loadXML($content);

	if ((variable_get('sf_queue_incoming'), FALSE) && function_exists('sf_queue_add_incoming')) {
	  _sf_notifications_soap_respond('true');
    sf_queue_add_incoming($dom);
    exit;
	}
        $resultArray = _sf_notifications_parse_message($dom);
	$ret = _sf_notifications_handle_message($resultArray);

sf_queue_add_incoming would take the DOMDocument and store it in a table until it can be processed. SF Queue would then have some admin configurable settings about how frequently to process items in the incoming queue as well as how many items to process at once. We would also have to provide some admin settings or at least a hook that allows admins/developers to do something if there was an error with parsing or handling the message.

What do you all think?

CommentFileSizeAuthor
#5 incoming-queue-1127166-5.patch6.47 KBkostajh
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

AaronBauman’s picture

I have just been dealing with this myself.
By the time sf_notifications gets the message, the bulk of the PHP processing has been done in the Drupal bootstrap.

In order to respond to Salesforce in a timely manner, we need to avoid bootstrapping altogether.
In other words, create a standalone php script to handle messages from Salesforce and queue them.
Then, modify sf_notifications to process the queue on cron runs.

Any approach that relies on, e.g. menu callbacks, will not solve this issue.

kostajh’s picture

Glad to see I'm not the only one with this problem.

Would implementing hook_boot in sf_notifications be an option?

AaronBauman’s picture

I really think the best approach is going to be handling outbound messages outside of a Drupal environment altogether.

kostajh’s picture

I think it would be good to support both hook_boot and a script that would reside outside of the Drupal environment.

In either case, sf_notifications needs to have its own table for an incoming queue and an implementation of hook_cron with some options for processing the messages.

I have a working implementation that I'm going to put through the paces tomorrow, I'll post a patch here after testing.

kostajh’s picture

Status: Active » Needs review
FileSize
6.47 KB

Here's a work-in-progress patch that does the following:

  • Provides new settings in admin/settings/salesforce/notifications for how to process outbound messages: process each message as it is received, or queue for processing on cron run
  • Adds a basic table, salesforce_notifications_queue, to temporarily store outbound message data
  • Implements hook_boot and hook_cron to store the incoming SF outbound messages, and processes up to 100 records per cron run

I tested this with a bulk update to 250 records, and it performed just fine, no timeout errors on the Salesforce side.

This patch still needs:

  • Configuration options for processing queue items, like what is provided by sf_queue
  • Better processing capability. Current implementation assumes that one outbound message is stored per row in the salesforce_notifications_queue table, when in fact dozens might be bundled together in one row.
  • Another option in admin/settings/salesforce/notifications that specifies a PHP file, i.e. sf_responder.php that would contain the code in hook_boot. And instructions on how to modify the htaccess file to redirect requests from sf_notifications/endpoint to sf_responder.php
  • Testing with bulk updates to 1,000, 10,000 records, etc

Testing and feedback would be very helpful! Be sure to run update.php or re-install sf_notifications so that the new database schema is installed.

EvanDonovan’s picture

Issue tags: +architecture, +7.x-2.x

Will have to make sure that the implementation we do in 7.x includes this also.

brianV’s picture

Assigned: Unassigned » brianV

Assigning to myself.

EvanDonovan’s picture

Assigned: brianV » Unassigned
Issue tags: -7.x-2.x

We (brianV & myself) added this in 7.x-2.x using the Drupal queue.

We won't have time to backport to 6.x-2.x, but someone else could, if they wanted to make that a 6.x-2.x dependency.

kostajh’s picture

Could you post the commit hash where this was added to 7.x-2.x?

brianV’s picture

kostajh’s picture

Cool, thanks Brian. Have you all tested how well it holds up with thousands of outbound notifications coming in rapid succession? This was the concern that Aaron and I had in earlier comments on this thread.

brianV’s picture

kostajh: We haven't tested in under extremely heavy load yet.

Our view is that this is much better than the non-queue implementation, but can still use room for improvement if it turns out to be warranted in practice.

There is no reason we couldn't add an additional PHP file to use as the endpoint that simply writes new queue items to Drupal's queue table, and returns an 'OK' to Salesforce.. If you look at SystemQueue::createItem(), you can see that the required query is very simple:

http://api.drupal.org/api/drupal/modules--system--system.queue.inc/funct...

EvanDonovan’s picture

Hmm....interesting thought, Brian. That way a bootstrap would be avoided.

When we get into the testing phase, we can see if this would be necessary.

One thing to note is that we already have made it so that the entire cache doesn't get cleared each time an insert is done in response to an outbound notification. That would've been a huge performance hit.

kostajh’s picture

Thanks for the notes. I think it is worth implementing Drupal Queue for both sf_import and sf_notifications in the 6.x-2.x branch.

This should be in another issue, but wanted to mention that there is also the problem of processing the queue. When there are tens of thousands of records, trying to process them on cron run means that it will take weeks for a queue to get processed. I've been looking at using drushd (http://drupal.org/project/drushd) to help with that, but do you have any other ideas on how to handle this?

kostajh’s picture

Title: Create incoming queue for Salesforce outbound messages » Use drupal_queue for Salesforce outbound messages
Status: Needs review » Needs work

There is no patch for 6.x-2.x, so setting this to needs work.

kostajh’s picture

Status: Needs work » Closed (won't fix)