Allow salesforce to initiate imports

The way I've accomplished this in the past involves Salesforce outbound messaging and Drupal hook_xmlrpc.*

For a good primer on SF Outbound Messaging, I suggest Mike Simmond's blog, specifically this thread -- make sure to read the whole thing because there are several revisions of the code sample.

In a nutshell, you need to setup outbound messaging via the Salesforce web gui and direct it to your drupal install at e.g. http://example.com/xmlrpc.php . Then tweak the sample code into your own module's hook_xmlrpc and you're rolling.

It might take you a while to refine everything to work with your specific use case.

I happen to be working on an sf_xmlrpc module that uses your salesforce.module's field maps to automate the synchronization from Salesforce to Drupal, but it's not quite ready for prime time. I'll try to get a version of that up soon...

* I should mention that an alternative route would be to set up a polling function (ie. a hook_cron) in Drupal to ask Salesforce every so often if it has any updates. Some Drupal developers will prefer this method because it can be administered solely through Drupal. However, this method imho introduces unnecessary complications like race conditions and consistency issues that you can mostly avoid by using outbound messaging.

Here's the module i've got so far.
I'm using it on 2 development sites so far, and have had no issues.

If you've never used it before, Salesforce outbound messaging can be the hardest part to set up.

You need to set up the Outbound Message, then set up a Workflow to actually fire the outbound message...
I don't remember where i found the tutorial, so dig around a bit if you can't figure it out right away.

Let me know how you fare.

DON'T USE THIS ONE. USE THE NEXT ONE.

Sorry - there were some extraneous debugging calls that will probably break that last upload for you.
Use this one

Bump: Awesome feature! More testing and review is needed before this can go in SF core though.

Minor updates, including multiple fieldmaps per salesforce object, further documentation.

I had problems using the above module so I updated the code and rolled a patch (and a tgz). Main changes are:

- Changed fieldmap load so that they didn't try to load based on a range. If the next field map was 4 it would try to load fieldmaps 1 through 3, but if you had deleted fieldmap 2 it would error out.
- Updated the code to use the salesforce_object_map table, rather than salesforce_ids which doesn't seem to exist.
- Changed the data reading to use stream_get_contents rather than a while loop.
- referenced SforcePartnerClient to access SObjects, was erroring out when tyring to create new SObjects without the reference.
- Fixed minor bugs confusing the $info and $map in the update section.
- Converted tabs to 2 spaces for sanity while editing.

sf_rpc module I was using was the 2009/09/04 version above and the patch is based on this version.
salesforce_api version was from 2009/09/01.

Can you help me setup this module ? Do I need to install the module attached to comment #7 as well as apply the patch in comment #7 ?
In the outbound messaging configuration, what end point URL do I need to use ?

This module should probably be updated to not use xmlrpc.
xmlrpc is typically used to expose web services, not to service one-way incoming communication.

As far as I can see, there's no reason that incoming SOAP messages from Salesforce should go to xmlrpc.php,
instead of a regular old path, e.g. /admin/settings/salesforce/sf_import/request

In the meantime, TimAlsop: You should use only the .gz file from #7.
Your endpoint url is http://example.com/xmlrpc.php -- replace example.com with your drupal site's url.

Thankyou. If this works for us, we might re-write the module as you suggested so it does not use xml rpc.

I am getting a org.xml.sax.SAXParseException: Premature end of file. error in Salesforce when using this.

I am trying to get SF to send changed contact info to Drupal, to update the Drupal user related to that contact.

Sounds like a problem with the network.
What is the result when you load your drupal site's xmlrpc directly?
e.g. example.com/xmlrpc.php

Make sure your outbound message is pointing to the same place and the sf_rpc module is enabled.

When I go to XMLrpc direcly I get...

DOMDocument Object
(
)
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> 
    <soapenv:Body> 
      <notifications xmlns="http://soap.sforce.com/2005/09/outbound"> 
        <Ack>true</Ack> 
      </notifications> 
    </soapenv:Body> 
</soapenv:Envelope>

I assume that

DOMDocument Object
(
)

is something browser specific, as it's definitely invalid XML.

At any rate, the rest is as expected.
I surmise either the endpoint is not set correctly on the outbound message,
or salesforce cannot connect to your webserver for whatever reason.

I am now seeing the "org.xml.sax.SAXParseException: Content is not allowed in prolog." error in SF too.

I have added a few watchdog()s to dump $dom and $result_array, they both return 1 in the log.

So SF is connecting, but something isn't right.

Maybe

DOMDocument Object
(
)

is actually being returned by the server? It should not be.

not really sure what I did, but this seems to be working, and DOMDocument Object() is not in my XML now.

Magic!

Thanks Aaron.

So it appears that when this error happens the $dom object doesn't actually get loaded up with the xml when $dom->loadXML is called, but no error is returned. When I call a debug right after line 36 (using #7) as the base

   $dom->loadXML($content);

I get this back from my

print_r

DOMDocument Object
(
)

I'm expecting to see the

$dom

variable loaded up with the contents of $content, so that the script can start manipulating it in the later parse and handling functions. Because there's nothing in there, I belive that's the trouble. The XML file that SF spits out is attached. Anywhere where you see the word REMOVED I took out the data for privacy purposes, but there was data there and it was at least in the format I was expecting (dates in fields that should be dates, strings in strings, etc.)

If you have any ideas about what I can try next to debug, it would be much appreciated.

TT

I should clarify and the xml is the output of what I receive in

$content

when I receive the SOAP post from Salesforce.

Re: #19, #20
Calling print_r on a DOMDocument will not give you what you expect
(assuming you're expecting to see something other than what you report in comment #19).
See http://www.php.net/manual/en/class.domelement.php#86224 or google "php print_r domdocument".
(My comments in #15 and #17 were made in ignorance of this fact.)
If you want to see a better representation of the parsed XML data,
call print_r on $resultArray or muck around inside the "parse_message" function.

That said, this module needed an overhaul.

In this patch

one patch in 2 versions:
- sf_notifications-DEV.patch applies to the D6.x-2.x-DEV revision as of this post
- sf_notifications.tgz is the tar-gzipped version of sf_notifications-DEV.patch for those who would prefer such a format.
- sf_notifications-ALPHA1.patch applies to the D6.x-2.0-ALPHA1 release
All notes apply to both unless noted:

• ALPHA1: backwards compatible updates to salesforce_api, sf_user and sf_node
• DEV: only adds sf_notifications module files - no updates to any existing files.
• Renamed sf_rpc to sf_notifications, since this module was abusing xmlrpc into responding to SOAP notifications.
• Changed endpoint from "http://example.com/xmlrpc.php" to "http://example.com/sf_notifications/endpoint"
• Fixed some outstanding issues that may have been the root cause of issues in #19
• Moved responsibility for creating Drupal->SFID mappings on import into sf_user and sf_node
• Added watchdog messaging
• Added placeholder for administrative settings

I have found it somewhat easier to test this module using curl:

curl -d @filename.xml http://example.com/sf_notifications/endpoint

You'll have to capture an Outbound Message to stick into filename.xml, but after you do it once you shouldn't have to again. Alternatively, you can use or edit accountsample.xml_.txt from comment #19.

Caveats

• If you set up an outbound message and have an active mapping between Drupal and SF,
  deleting a SF record will delete the mapped Drupal object.
  In the future, administrative settings will be available to finer control over this behavior.
• These patches wholly contain the module "sf_notifications".
  You should update your salesforce version to D6.x-2.0-ALPHA1 (according to the update instructions in README.txt)
  or to D6.x-2.x-DEV (from CVS, again according to README.txt),
  and then apply the appropriate patch to the salesforce directory.
• You should disable and remove sf_rpc from your Drupal instance before applying this patch.
• You need to change your SF Outbound Message endpoints after applying this patch.
• You will only need one of the following patches:

Thanks Aaron, I'll get that installed and put it through its paces. Thanks also for the tip regarding

print_r

. That article makes sense.

Cheers.

Thanks, trevor.

Instead of using XML why not http://www.salesforce.com/us/developer/docs/api180/Content/calls_for_dat... ?

mikestefff:
This feature request is to allow the drupal module to leverage SalesForce's outbound message capability by receiving inbound notifications from SalesForce. You have linked to an API that would allow clients to poll SalesForce for changes. (think "pull" vs. "push" -- this feature request is to accomodate "push" capability).

It might not be a bad idea to take advantage of the getUpdated and getDeleted API, and it might address your comment about importing on cron, but it's a different issue.

At any rate, the patches in #21 need to be udpated @ DEV.

Hi

I am testing out the sf_notifications module using the latest dev version.

I have setup my outbound messages and workflow rules on SF.

It does not seem to work at the moment, when my workflow rule is triggred I get this watchdog error message.

SalesForce Notifications: No import handler defined for . No record created.

SObject Object
(
    [type] => Contact
    [fields] => stdClass Object
        (
            [Id] => [SalesForce Notifications: No import handler defined for . No record created.

SObject Object
(
    [type] => Contact
    [fields] => stdClass Object
        (
            [Id] => 003A000000UCxYuIAL
            [LastName] => bob1
        )

    [fieldnames] => Array
        (
            [0] => Id
            [1] => LastName
        )

    [Id] => 003A000000UCxYuIAL
)
]
            [LastName] => bob1
        )

    [fieldnames] => Array
        (
            [0] => Id
            [1] => LastName
        )

    [Id] => 003A000000UCxYuIAL
)

And on salesforce when I check the status of the message I have this error

"org.xml.sax.SAXParseException: The processing instruction target matching "[xX][mM][lL]" is not allowed."

Any Ideas??

Thanks

t14: see my comment #25 - the latest patches need to be updated for the latest DEV

feel free to take a stab at this if someone has more time than I do

subscribe - this looks like code we'll be needing before too long.

Well, I've taken a stab at updating the sf_notifications.module file for 6.x-2.x-dev version of Salesforce, in which the salesforce_object_map table no longer has a fieldmap column, and I have gotten workable results.

The first change is in the function _sf_notifications_parse_message():

233c252,255
<     'SELECT oid, sfid, fieldmap, drupal_type FROM salesforce_object_map
---
>     // there is no fieldmap column any longer in the table
>     // so we remove it; other changes are necessary to be able to load the field map of course
>     //'SELECT oid, sfid, fieldmap, drupal_type FROM salesforce_object_map
>     'SELECT name, oid, sfid, drupal_type FROM salesforce_object_map
238a261

As a result further changes are needed in the function _sf_notifications_handle_message():

84c84,86
<     $fieldmap = salesforce_api_fieldmap_load($object_record['fieldmap']);
---
>     // We don't have fieldmaps anymore! So we added name
>     //$fieldmap = salesforce_api_fieldmap_load($object_record['fieldmap']);
>     $fieldmap = salesforce_api_fieldmap_load($object_record['name']);

And further in the same function, to accommodate the new signature of the sf_node_import() function:

121c126,129
<           $drupal_id = $function($obj->fields, $object_record['fieldmap'], $object_record['oid']);
---
>             // ain't no fieldmap anymore! import functions take name, not fieldmap id
>           // $drupal_id = $function($obj->fields, $object_record['fieldmap'], $object_record['oid']);
>           $drupal_id = $function($obj->fields, $object_record['name'], $object_record['oid']);

I am attaching a very rough (includes unnecessary comments and commented out log_debug statements) but tested and working copy of the module file itself (the only one that had to be changed) for reference, hopefully the community can move towards a new patch for the current ALPHA and DEV versions of the salesforce module after testing and feedback.

Here's #30 in patch form for easier evaluation

OK - a few updates on #30, #31:

• added per-fieldmap activation / deactivation. important: fieldmaps default to inactive. in order for Drupal to respond to Outbound Messages, you must set a fieldmap to "active" via Notifications local task (admin/settings/salesforce/fieldmap/%/notifications) or Notifications overview (admin/settings/salesforce/notifications).
• added per-fieldmap conditional. right now this is a simple text area that accepts PHP. If a given fieldmap's Salesforce type matches the Salesforce type of a Notification, the PHP gets evaluated to see whether or not to proceed with the operation. This is more than sufficient for me, but some folks might prefer to choose specific fields and set conditions based on fixed or PHP values for those fields. If anyone wants to pursue this as an option, it might complement the open-ended php field nicely.
• some refactoring for DRY and readability
• updated README

The biggest outstanding issue now is authentication.
We are opening up a gaping security hole, so we *have* to support some kind of authentication given the sensitive nature of the operations we're exposing.
Since SF Outbound Messaging doesn't actually support traditional authentication, we are in a bit of a corner.

Here are some ideas I've seen after a little searching:
- "custom URL" e.g. security through obscurity
- firewall / IP access rules
- SSL client certificates

Of these, the only truly reliable method seems to be SSL client certificates. The implications of this are (at least) two-fold:
1. Since SSL Client certificates must be configured at the web server level, secure outbound messaging cannot be implemented on the Drupal layer alone
2. If site admins do not set up SSL Client certificates and choose to use sf_notifications over port 80, Drupal sites will not be secure

reviews of the patch and input on security issues appreciated.

Here is a stab at the IP access rules implementation. This patch should be used after the patch in #32 is applied. It is incomplete and needs some work.

The patch provides a textarea in admin/settings/notifications where the user can enter a list of allowed IP addresses. Salesforce provides a list of allowed IPs. This patch is incomplete because it does not exactly match what Salesforce specifies. For example, SF says to whitelist this block of IPs: 204.14.232.0/23. The patch allows traffic from any IPs that begin with 204.14.232. Perhaps someone else with a better understanding of IP addresses can read through their Knowledge Article and fix the code.

An access callback to a new function, sf_notifications_allowed_ips, is added for the sf_notifications end point. The function checks to see if the IP is within the block of whitelisted IPs, and if so allows the outbound request to continue for processing. Otherwise it's rejected.

Let me know what you think.

Here's a calculator that might help you suss out a formula for CIDR notation:
http://bonomo.info/coyote/cidr-calculator.php

It gets pretty wonkish, but here's a more detailed explanation of CIDR:
http://www.pantz.org/software/tcpip/subnetchart.html

if i have a chance i'll take a whack at it, but it should be possible to accomplish this elegantly with bitwise operations.

a little more searching revealed this CIDR matching function:

function cidr_match($ip, $range)
{
    list ($subnet, $bits) = split('/', $range);
    $ip = ip2long($ip);
    $subnet = ip2long($subnet);
    $mask = -1 << (32 - $bits);
    $subnet &= $mask; # nb: in case the supplied subnet wasn't correctly aligned
    return ($ip & $mask) == $subnet;
}

using that function, I rerolled the patch so that the "whitelist" setting accepts either:
- a full IP address
- a valid CIDR range
- any combination of the two, separated by newlines

Unless we want to get into a discussion of IP spoofing over tcp/ip, I think this is a decent Drupal-level security measure for sf_notifications.

I would like to get this into DEV relatively soon, unless there are reasonable objections or critical bugs or security issues.

Looks good! Thanks for finding the cidr_match function, that is certainly helpful.

A few more updates after some dogfooding:

• typo in cidr_match() prevented it from ever returning TRUE.
• sf_notifications_endpoint() was require_once()ing SforcePartnerClient.php from the old location
• inserts were not working - the "fake" $object_record in the insert loop was not correctly defining 'drupal_type'
• one debugging message was calling print_r() without the second parameter

The drupal_set_message in hook_enable was providing incorrect instructions on the endpoint to use in Salesforce, fixed it here:

+ '!base_url' => $base_url . '/' . SALESFORCE_PATH_NOTIFICATIONS_ENDPOINT,

See attached patch.

Another tiny fix. !message should be !messages in hook_enable.

How do people feel about moving this into 6.x-2.x-dev? Doing so should result in more user testing and feedback which would allow for bugs to be identified quicker as well.

sf_notifications seems mature enough to be included in dev - i added "Experimental" after the module name to absolve myself of any responsibility ;-]

I'm also adding a new Component to the issue queue for sf_notifications

Thanks for everyone's work on this