I'm syncing over contacts and accounts and I'm running into the fact that I'm syncing over thousands of accounts that have no data in the fields that I need to sync with. It just plugs up the queue as well as created a bunch of blank nodes on me.

Can I suggest adding a required field to the admin form that will go into the query as a !=null to only pull items from salesforce that have that field and ignore those that don't? This allows me to effectively "assign" something to the sync when creating it in salesforce.

Right now I'm working on a patch that'll use "key" as that field since it seems that if you designate something as Key it should be required, right?

CommentFileSizeAuthor
#9 required-field-1992170-9.patch3.18 KBcellar door
#1 1961760-2-requirekey.patch1.04 KBcellar door

Comments

cellar door’s picture

cellar door commented
Status: Active » Needs review
StatusFileSize
new 1961760-2-requirekey.patch1.04 KB

Alright I've got a patch for this. Still not sure what your intended use for the Key designation is but if it's key it's probably required. I'm using this to filter through thousands of records for only the ones that apply to this sync. Hopefully this helps!

kostajh’s picture

kostajh commented

Thanks for the patch, I'm just not sure this is the best way to do this. "Key" is being used for the externalId for upserts to Salesforce.

I mentioned in #1967258: Add hook to stop an individual SF object from being processed on pull that we might try to use a Rules based approach for processing items, but that wouldn't help with filtering out records from the salesforce_pull SOQL query.

Perhaps there could be an admin interface at /admin/config/salesforce/pull where you could customize the Salesforce Pull SOQL query?

kostajh’s picture

kostajh commented
Title: Required Field » Add required field to SOQL query for Salesforce Pull
Component: Code » salesforce_pull.module

See also #1969034: Create admin interface for Salesforce Pull

cellar door’s picture

cellar door commented

Yeah I actually ran into an issue with the externalID by using the Key field. I'm going to re-roll into a patch where it will allow for a radio (could even be a checkbox) that will do the same but with a new setting.

Yeah processing isn't a bad idea to hook as well but for me it was that there were 7k objects pulled into the queue and at 50 a time it takes a long long time to process through all of them. This keeps them from being pulled in the first place if you don't want them to be.

I'll re-upload a new patch when I get it finished. Should be this week.

levelos’s picture

levelos commented
Status: Needs review » Needs work

The SOQL query is now an alterable object. Ref. http://drupalcode.org/project/salesforce.git/commit/83578cbe00883ee2cc52.... Doesn't preclude the change being discussed here, but might make it easier. Marking this as needs work for a subsequent patch from Cellar Door.

kostajh’s picture

kostajh commented

In my opinion the solution here is a bit too brittle. The "key" field is required for upserts but not necessarily for deciding on what is pulled down from Salesforce to Drupal.

cellar door’s picture

cellar door commented

Kostajh - Can you explain more for me what the purpose of the "key" field is? I've been looking for documentation etc. in the code but haven't been able to find much. My thought on this is that I'm going to make another selection on the mapping for required that will then put it into the query. It'll be separated from the key field though.

kostajh’s picture

kostajh commented

Sure, you can read more about it here #1951728: Allow those using soap to use both "externalId" and "idLookup" and https://www.salesforce.com/us/developer/docs/api_rest/Content/dome_upser.... Basically, "key" is a Salesforce field that has the property of "externalID" and can be used for doing upserts (update / insert) when pushing data to Salesforce from Drupal.

Because its usage is limited to exporting from Drupal to Salesforce, I don't think it makes sense to use this as the field for filtering objects that are imported to Drupal from Salesforce.

cellar door’s picture

cellar door commented
Status: Needs work » Needs review
StatusFileSize
new required-field-1992170-9.patch3.18 KB

Man just as I got this working dev went and had some major overhauls on the mapping and pull modules!

I took my code from the beta 2 and refactored it to the latest dev but I'm not sure it'll be 100% working because I have to update my current codebase before I can test. This should get the point of what I was trying to do - add a radio for the selection of a required field (could be a checkbox for multiple I guess if we want to get complex) then just adds it as a !=null condition.

Let me know what you think

tauno’s picture

tauno commented
Issue summary: View changes

The simplest, most flexible way of handling this would be to use hook_salesforce_query_alter() to add fields to your query. Not sure if it makes sense to add to the UI.

cellar door’s picture

cellar door commented

I think the reason I wanted to add it to the UI rather than the hook query is to make it available to the site admins to not pull all data into the query. What this came out of was a salesforce integration that would otherwise pull down 15-20k records then process them in the queue when only 3-4k were needed. By marking a field as required, I was able to filter down that query to just what was needed.

amaisano’s picture

amaisano commented
Priority: Normal » Major

How is this not a bigger deal? There is no other way that I know of to 'limit' what gets pulled into Drupal. We only want to sync Contacts that meet certain criteria - that's about 1/100 users. Drupal pulled 4000+ Contacts and created user accounts for all of them - I only wanted a fraction - the ones with a certain value for a certain field.

Unless there is a way to do this without writing custom code?

amaisano’s picture

amaisano commented
Related issues: +#2268695: adding condition via hook_salesforce_query_alter problematic, +#1969034: Create admin interface for Salesforce Pull
aaronbauman’s picture

aaronbauman
he/him
commented
Status: Needs review » Needs work

Does SOQL "!= null" work for all field types?
What does null mean for a checkbox?
Does SOQL differentiate between empty strings and null?
What about whitespace?

The proposed hooks in #2186153: Provide additional Salesforce push/pull hooks for pre_export changes. will address the functionality concerns raised here about implementing custom / arbitrary business logic for sync.
Putting this into UI might actually lead to more confusion.

mariacha1’s picture

mariacha1 commented
Status: Needs work » Closed (won't fix)

I agree with aaronbauman. These sorts of restrictions should be added through a hook_salesforce_query_alter hook.

/**
 * Implements hook_salesforce_query_alter().
 */
function my_module_salesforce_query_alter(SalesforceSelectQuery &$query) {
  if ($query->objectType == 'Contact') {
    $query->addCondition('Email', "''", '!=');
  }
}

More at http://cgit.drupalcode.org/salesforce/tree/salesforce.api.php#n177