Implement "Object Setup" Feature

Initial patches attached, based soley off the old patches. Couple notes:
- The column "modified" in the salesforce_objects table is not there (see note on line 694 of salesforce_api.admin.inc), I *think* it should have been added by function salesforce_api_update_6200 in salesforce_api.install but I'm not sure. I removed the column but didnt' add it back to the patch.
- the variable "salesforce_api_enabled_objects" is not there, hence added objects do not appear in the Fieldmaps -> Add -> Salesforce object dropdown
- Everytime I save the Object setup, objects are saved multiple times, regardless of if it's been saved before. Hence, my salesforce_objects table currently has several rowes of redundant data

Ive reworked these patches and rolled them into a single file taking into consideration the feedback provided on the Create solid data structure issue. Ive removed a lot of the surplus code and refactored the object definition storage to utilise the system cache table.

The patch will:

• set the default enabled objects to 'Campaign', 'Contact', 'Lead'
• add an 'object setup' menu item in the Salesforce settings page where admins can select the SF objects available to drupal when creating fieldmaps, these objects (+ schema) will then be cached and saved in the system 'cache' table
• add an 'object settings' fieldset to the admin settings page where the cache lifetime can be specified, this also features a button to directly rebuild the object cache

Ive tested the changes a number of times and everything seems to be working ok for me using the latest 6.x-2.x-dev version. Im happy to work on feedback / changes so let me know.

The patch should be run from the Salesforce API module root directory and for reasons unclear to me you have to specify the -p0 option when applying the patch, eg: "patch -p0 < salesforce_api_476978.patch"

This looks good but needs a more extensive review, which I don't have time for right now. Have you checked that the code complies with Drupal's coding standards, with Code Review module? This isn't a blocker for review, but necessary for committal.

Also, I assume you meant to mark this "needs review"?

Ive updated the patch to comply with the coding standards using the coder module.

Have rerolled the patch using the latest 6.x-2.x-dev version (Commit #257840) as the existing patch fails using this version. Would be good get this reviewed and into the API soon

Just to check, the latest patch is #7 right? Was getting a little confused with the dates of the comment (#6 is 9/23, #7 is 9/16)

yes, the latest version of the patch is in #7. Let me know how you get on using this, im keen to get this tested and in to the API asap.

I reinstalled the module with the patch and that worked fine. I went into the Object setup section and selected my desired object. I got a torrent of the same warning:

warning: Invalid argument supplied for foreach() in D:\git\ambassador\trunk\sites\all\modules\contrib\salesforce\salesforce_api\salesforce_api.module on line 281.

But then I also get a notification stating:

Salesforce object cache has been refreshed.

I saved again without changing any of the objects, and it was fine-- same notification, no warnings. I added some other objects and saved, removed some objects and saved, it was all fine.

Maybe this happens the first time you try to refresh the cache? The error is happening in salesforce_api_cache_build().

Edit: my cache lifetime was set to none at the time.

This sounds like a bug introduced by the patch. Correct me if I'm wrong. Were this issues reproducable without the patch?

This is a bug introduced by the patch. Without the patch, there is no "Object Setup" tab, nor is there an Object settings section in the "Settings" tab.

Ah, of course! I should have realized that.

I believe I have more detail about what's going on.

In short, if you only want to activate one item, the API does not return an array with one object-- it only returns the object. So the for loop doesn't work because it's iterating over the fields of the one object, rather than iterating over the array of all the objects, and then iterating over each object's field.

I kept running into this problem with this use case:

Install a FRESH install of Salesforce.

Enable sf_node and sf_user.

Go to the SF settings tab and put in your credentials.

Then jump immediately to the Object setup table (instead of going to fieldmaps first). Try to activate only one object (which it does because I believe the salesforce_api_enabled_objects variable doesn't get set). If you go to fieldmaps first, it sets the variables to the default list of Campaign, Contact, and Lead, so I don't think you run into this issue.

You will get those warnings, and you won't be able to set the list to one object.

Then try activating two objects-- that will work.

EDIT: I can try to submit a patch for this, but it's not going to work for most people, I think, because I have quite a few sf patches in my codebase, and I'm afraid it won't apply correctly to what's in -dev right now.

Also, is it not possible to export/import changes from Salesforce? All of the fields on the right-hand (Salesforce) side of the "Mapped field values" area are blank for me. And the only "Unmapped fields" I see is the Id.

Here is a new patch that fixes a few things I noticed:

• "page callback" is spelled "page callbacck" for $items[SALESFORCE_PATH_OBJECT .'/%']
• Refactored salesforce_api_cache_build to use salesforce_api_object_to_fieldmap_fields
• Clean-up salesforce_api_object_to_fieldmap_fields to return the item to be added to the aggregating array, instead of creating a dummy array to aggregate. Hopefully this is what the intent of that method was. Also added t() around the label.

Unfortunately there is still the issue regarding being able to export/import fields with objects outside of the "Campaign, contact, lead" default. When DrupalSalesforce->retrieve() is called, it's only returning the Id. So none of the attributes appear on the Salesforce side of the "Mapped field values" table.

This patch encompasses legion80's from #16 and adds:

• function salesforce_api_describeSObjects: Wrapper for SOAP SforceBaseClient::describeSObjects. Given an array of sf object type, return an associative, normalized array of SF object definitions, indexed on machine-readable names of SObjects.
  This addresses the "torrent" of errors mentioned in #10. Also, keeps a static cache of SObject descriptions to minimize SF API calls during a single HTTP request.
• function salesforce_api_describeSObject: Similar to the above, but describe a single object and return a single object.
• Disable checkboxes for SF Objects that are in use by fieldmap(s). Addresses #587054: Object setup: deleting object type doesn't alter fieldmaps
• Add salesforce_api_object_options to hook_theme
• A couple minor code convention issues.

Ack - sorry folks -- new patch gets rid of some garbage from previous.

Another minor adjustment - Salesforce soapclient is very picky about its function arguments, which caused the cache to be empty on rebuild.

and again.

First, sorry for my poor english.

I test your patch on a new install of Drupal 6.14. And I have a problem. I see the new tab "Object Setup" on salesforce module configuration and i can select salesforce objects that i want use.

These objects are visible when i add a fieldmap, in a select box. I choose a drupal node and a salesforce object. But when i submit the form, system display an error : "Salesforce object field is required".

i have been trying to apply this patch and run into two issues:

1. when applying the patch from command line, i receive the following error: 'malformed patch at line 612'.
2. i manually applied the patch and when i go to edit my fieldmaps it throws this error: "This fieldmap cannot be edited, because an object definition could not be found."

has anyone else experienced this and if so found a workaround?

EDIT: actually i am experiencing the same behavior as the previous poster where 'Salesforce object field is required.' is displayed even when the object is selected.

Try disabling and uninstalling salesforce, deleting all salesforce_* and sf_* tables and Drupal variables, then re-installing and re configuring salesforce module.

I'm seeing the same error message when trying to add a field map with the latest patch. It appears the SF object checkboxes never retain their 'checked' state under the new Object selection page (ie. make several selections, click 'Save' at the bottom of the page, it then appears to save with an updated alert message at the top of the page but nothing remains checked and the SF object table appears to be still empty). Perhaps that has something to do with the problem? I'm not sure I located everything such as the 'Drupal variables' mentioned above however uninstalling several times and deleting the SF tables didn't appear to allow selection of other objects. The single object that appears in the Add Field Map drop-down no matter how many were selected on the Object-select page seems to always be the last one that was checked.

Hopefully this helps track down the issue -- this seems like an incredibly useful feature and I'm looking forward to testing it out some more.
Thanks in advance for any assistance you can provide.

i am experiencing the same behavior as others have reported even after uninstalling and clearing the db of salesforce columns and variables.

On a clean d6.14 install + salesforce-6.x-2.x, I was getting errors when this patch was applied.
I could not use the object setup feature because of conflicts between the data structures used in salesforce_api_cache_build and salesforce_api_admin_object.

The attached patch against salesforce-6-x.2.x addresses this issue.... I hope.

Aaron, this one is working for me, started from a fresh copy of salesforce-dev. A generated wsdl was necessary for it to work, did not work with the default wsdl's. I think it was a difference in the data format returned by the sf toolkit, the default returned the objects in $objects->types while the generated uses $objects->sobjects.

After an uninstall/reinstall, Salesforce table/variable purge and patch #26 applied, the Object Setup page reads:

"There was an error retrieving the list of SalesForce objects. Please verify that your SalesForce instance is properly configured."

I'm using a generated enterprise wsdl and can run the Examine Data Structure demo on the Test/Demo page.
Is anyone else seeing this message? Perhaps I'm missing a step?

a_square:
anything in watchdog or php logs?

I may be too much of a newbie to know where to look for the php errors; I don't see any errors in drupal's syslog though.
Would it be a worthwhile exercise to start from a fresh install of Drupal + SF module + this patch?

My guess is your issue is related to comment #27 and you're experiencing #473822: Need a "Clear Cache" option for Salesforce/SOAP cache. In brief, you may have updated your wsdl, but if you didn't clear the PHP wsdl cache, your soap client is probably still using the old one (even if you restarted apache). Until #473822 gets patched, you need to clear the cache manually by inserting the following somewhere in your php code (before the first line of function salesforce_api_admin_object should work just fine):

ini_set("soap.wsdl_cache_enabled", "0");

If this doesn't fix things for you, then double check your wsdl file:

1. Open the file "enterprise.wsdl.xml" in your salesforce_api/toolkit/soapclient directory.
2. Search for <complexType name="DescribeGlobalResult">
   (mine appears around line 540)
3. This element should have one child element <sequence>, which itself has 3 child elements <element>.
4. One of the <element> elements should have a "name" attribute of either "sobjects" or "types".
   It will look something like this:

   <element name="types" type="tns:DescribeGlobalSObjectResult" minOccurs="0" maxOccurs="unbounded"/>
   

   or

   <element name="sobjects" type="tns:DescribeGlobalSObjectResult" minOccurs="0" maxOccurs="unbounded"/>
   

5. If your wsdl matches the former, therein lies the issue, and you should generate a new wsdl and clear the wsdl cache again.

If neither of these fixes work , we're back to square one and we'll go from there.

There's a pretty steep learning curve here, a_square, so thanks for your patience.

Thank you for the detailed response, Aaron. I had the cache clearing in place and was able to locate those 'sobjects' rather than 'types' references in my generated wsdl.

It turned out I need to move the wsdl under the "salesforce_api/toolkit/soapclient" directory you mentioned above, rather than "salesforce_api\wsdl" for it to pick it up. It seems to be saving the selections and populating them in the mapping drop-down now, so I'm looking forward to getting some new nodes populating in SF today. Again, your assistance is much appreciated.

Not sure if this is the right thread, but have been working with masande and we've run into this error:

ubercart-salesforce export: no user fieldmap found. Not exporting

I've created a fieldmap between an UC order and the Salesforce Contact object.

...and it tells me to check the watchdog for more information, but there is none to be found in the watchdog tables.

Any suggestions would be greatly appreciated...or directions to a more proper thread.

thanks - hart808

hart808, please create a new thread for this support request and include details about your drupal / salesforce setup.

I have applied this patch against the dev release with a SF instance using the Enterprise WDSL. Works as expected.

I have installed this patch (from #26) as it was required to make my salesforce module work. Without this patch I was unable to select the salesforce object in the fieldmap tab. I then started to wonder how I can select an object other than Lead, Contact or Campaign. I read the reset of this thread and noticed that there is supposed to be a new tab where I can configure which objects I want to be available. I do not see any tab, other than Settings, Fieldmaps and Test/Demo.

I wondered if I needed to reinstall this module to make the object tab appear, so I uninstalled the module, deleted the salesforce tables from database, installed the module again and reconfigured it, but the object tab is still not showing. Does anybody know why this might be the case ?

I found the problem - when I applied the patch I used Tortoise SVN and didn't save the patch made to all 3 files, so ony one of the files modifed by this patch had been used by the module. I applied the patch again, so that all 3 files are modifed and now the Object Setup option appears, as it should.

pearlbear: I've never heard of that behavior and can't recreate it on a clean D6.15 install.
Are you using the patch from #26?
Do you have any errors? (Is error reporting on?)
Are you using the latest dev release?
Did the patch apply correctly?
Are you working on a clean drupal install?

Let us know if you can provide explicit steps to reproduce.

nevermind

Hello,

I've had to stumble around a little in order to find a configuration that allows this patch (#26) to work. I've noticed that the only way I get the correct behavior is if I insert my own custom enterprise.wsdl.xml, as exported directly from SF, in /salesforce_api/toolkit/soapclient (instead of the "default" one that comes with the soapclient download) AND if I disable PHP wsdl caching via the patch at http://drupal.org/node/473822. If either of those conditions are not met, I get "There was an error retrieving the list of SalesForce objects" when accessing the "objects setup" tab.

This has lead to some conceptual questions and confusion that I was hoping someone might be able to address (I just don't know the php well enough to distill this directly from the code):

1). How many levels of caching are involved here? It seems that there is wsdl file caching (apparently done directly at the php level), but there also appears to be some other cache control options introduced by this patch (in the "object settings" pane). Are these different types of caches, caching different things?

2). Where is the list of available salesforce objects actually stored locally? Is it stored statically in /salesforce_api/toolkit/soapclient/enterprise.wsdl.xml, or is it extracted on the fly directly from salesforce? In other words, if I add an object in Salesforce, will it be detected automatically or do I need to manually overwrite my /salesforce_api/toolkit/soapclient/enterprise.wsdl.xml file again? I originally thought the former, but my testing has left me confused on this.

I'm really just trying to understand how new SF objects are managed and what cache hurdles might obscure changes. That picture seems to be rather complex so I was hoping to step-back from my trial-and-error testing and see if someone can just point me in the right direction.

Any info is really appreciated!

Cheers,
Ryan

Ryan, it sounds like you're getting a pretty good grasp of the module.
First, the wsdl cache is indeed at the php level. There's a patch in to allow admins to clear it manually.
#473822: Need a "Clear Cache" option for Salesforce/SOAP cache

The caching introduced by this patch caches the Salesforce Object schema returned upon saving the settings on the "object setup" tab. This patch also introduces a fieldset on the admin/settings/salesforce page to set the caching interval or clear the object cache.

Second, the list of SObjects is stored in a variable, salesforce_api_enabled_objects. The global list of objects is never cached. Each time the "object setup" tab is loaded, the list of available objects is fetched directly from Salesforce. (@see salesforce_api_admin_object and salesforce_api_admin_object_submit).

Caching is a catch-22 with Salesforce. On one hand, we want to keep our limited SF API calls few. On the other hand, we want to keep our data fresh. These 2 things are fundamentally at odds, and we'll have to keep working towards a balance.

Thanks Aaron!

Ok, I think I'm getring things. So the "Salesforce Object schema" is basically just the field structure of ENABLED (not global) SF objects, as selected on the "object setup" tab. Furthermore, the list of which objects are available is simply stored in a simple text variable (in the DB) as salesforce_api_enabled_objects.

This would also imply that having a fully updated enterprise.wsdl.xml (to reflect the "Salesforce Object schema") is no longer necessary with this patch, as the code will "bypass" that wsdl file, and fetch/maintain it's own "Salesforce Object schema". Furthermore, the patch extends caching controls over this "locally" maintained schema this to the user.

I had to state that in my own words.... just hope I'm still getting it right.

If I'm on track, than the only thing I don't understand is why I have to disable the php wsdl cache AND have a custom (case-specific) wsdl file installed in order to access the options in the "object setup" tab. The logic above would imply that wsdl info (cached or not) is not really needed in order for the patched code to structure a correct "Salesforce Object schema".

Hope I'm not over-reaching with my questions... I just want to be sure it's all clear.

Thanks for all your help!

Ryan

Ryan,
I think you've got the basics down.

For some reason the WSDL that's distributed with SF's PHP toolkit is always different from the generated one.
For example, the definition of "DescribeGlobal" returns "types" in toolkit's wsdl, and "sobjects" in the generated one.
(I don't understand why something so fundamental would ever change, but alas.)

The (SOAP) cache must be disabled/cleared only if the distributed WSDL was used by PHP.
I believe you're correct that the WSDL is not used for sobject schema once it's been (drupal) cached.

Thanks again Aaron for all this conceptual clarification. This helps a lot!

I've found that everything works fine if I follow your advice from #31 and just use my own custom wsdl. As you noted, the custom wsdl should not be necessary, but given some differences between this generated version and the "generic" version (above and beyond the object schema) I now see why this is needed. I also realize there is the php wsdl cache disable patch, but I have had trouble implementing it. Disabling that cache really only seems necessary when I am accessing the "object setup" tab anyway (in my own case I have not yet seen the need to disable it site-wide).

Cheers!
Ryan

committed patch from #26 http://drupal.org/cvs?commit=330868

Even though this was not RTBC, there are so many other contribs that rely on this patch i found it essential to roll this in.
Also I got tired of responding to issues with "apply the object setup patch".

If you find issues with object setup, please do not reopen this issue.
Instead create a new one please.