Use SOQL queries instead of getUpdated() to improve efficiency of sf_import

Salesforce Import implements hook_cron() and calls the Salesforce API getUpdated() function with three parameters: object type, start date, end date. Salesforce responds with a list of Salesforce IDs that have been updated for the time range.

The problem is that this can result in obtaining SFIDs that our import process doesn’t care about. For example, if you are using SF User to sync Drupal users with Salesforce Leads/Contacts, you might not care about Salesforce records without an e-mail address.

For a current implementation I'm working on, 50-60% of the records returned by getUpdated() are objects that we do not want to import. If you're dealing with a few hundred records that's not a big deal; if there are 100,000+ it becomes a bigger concern.

Proposed solution

Salesforce Import should use a more versatile implementation for selecting items to import.

This would be done in two parts. The first part is to start storing the last_import and last_export time for Salesforce records #1147180: Store time of last import, last export and initial creation for items in salesforce_object_map table.

Once we have that information, we can use it when querying Salesforce for updated records in hook_cron.

Example SOQL query:

	$query = $sf->client->query("SELECT $fields FROM '%s' WHERE LastModifiedDate > %d", $object_type, $last_cron_run);
	

In the example above, $fields would be “Id, LastModifiedDate”, and whatever other fields an admin wanted to specify (i.e. Email).

The next step would be to run a query against {salesforce_object_map} and confirm that the last_import date is older than the LastModifiedDate coming from the Salesforce query. If that date is older then the SFID would be added to {sf_import_queue} table.

Finally users with the ‘administer salesforce’ permission should also be allowed to create custom queries so that additional WHERE clauses could be added.