I tried to run this command:

drush sf-import 3124860e3cb9faa393e014471460eced --count=10000

and I got this message:

drush: Start/count exceed possible rows. Number of rows in Salesfoce : 1000
Drush command terminated abnormally due to an unrecoverable error.                                                                                                                  [error]

The thing is, there are over 100K records matching this map.

Basically, the SELECT performed around line 247 in sf_import.drush.inc in my case returns the first 1000 records (which is my limit of returns per API call). Since I am calling for 10,000 records, it thinks I'm asking for too many records. The code is not taking into account the fact that it needs to loop and get more records if there are any more to retrieve.

CommentFileSizeAuthor
#6 1205720-sf_import_various_fixes-6.patch6.37 KBdpearcefl
#4 1205720-sf_import_various_fixes-4.patch5.5 KBdpearcefl
#1 1205720-sf_import_various_fixes.patch4.17 KBdpearcefl

Comments

dpearcefl’s picture

dpearcefl commented
Title: Drush sf-import of large number records will fail » Drush sf-import of large number records will fail (plus other fixes)
Status: Active » Needs review
StatusFileSize
new 1205720-sf_import_various_fixes.patch4.17 KB

This patch addresses the following issues:

  1. The previous code assumed that all of the records to be retrieved would happen in one SOQL query. The number of records per query is limited and a flag is set to indicate that not records were returned. This patch adds the code to execute the needed queryMore commands. The number of records retrieved so far per query is printed to indicate script activity.
  2. Made a change on the $function call to pass in the entire record rather than just the Id. This stops the function being called from having to do a lookup to retrieve the information we can now pass in. The result is we can process records 20 times faster. The same change was made to the SF Suite bulk import a few months ago with the same massive improvement.
  3. This patch adds a "--quieter" parameter to disable the "Created object ###" and "Updated object ###" messages. Defaults to showing the messages as before.
  4. Corrected a spelling mistake in a message (Salesfoce -> Salesforce)
  5. The variable $imported was incremented but not used anywhere. Completed the implementation and added it to the 'report' section.
  6. Formatted the report section a little
joeybaker’s picture

joeybaker commented

Thank you!

But, I get a
warning: Invalid argument supplied for foreach() in sites/all/modules/salesforce/sf_import/sf_import.module on line 130.

dpearcefl’s picture

dpearcefl commented

joeybaker, are you using Drush? That line number corresponds to the function sf_import_settings_form().

This patch only applies to drush and the command sf-import.

dpearcefl’s picture

dpearcefl commented
StatusFileSize
new 1205720-sf_import_various_fixes-4.patch5.5 KB

Fixes a bug in the above patch having to do with the $count and $start variables. Refined the messages a little more.

dpearcefl’s picture

dpearcefl commented
Status: Needs review » Needs work

OK, I found some other counting issues. Hold on...

dpearcefl’s picture

dpearcefl commented
Status: Needs work » Needs review
StatusFileSize
new 1205720-sf_import_various_fixes-6.patch6.37 KB

OK, here's the best patch. Catches all manner of weird command line options.

EvanDonovan’s picture

EvanDonovan commented
Issue tags: +7.x-2.x

Tracking also for 7.x-2.x.

dpearcefl’s picture

dpearcefl commented
Status: Needs review » Needs work

Sigh. I'm going to have to rework some of the looping. With my 108K+ Opportunities records, an sf-import can take over 500MB. Not useful.

Alice Heaton’s picture

Alice Heaton commented

Hi,

With my 108K+ Opportunities records, an sf-import can take over 500MB. Not useful.

Do you mean 500MB of RAM ? A bit heavy indeed :)

It is unfortunate SOQL does not provide a proper LIMIT statement.

What do you think about :

  1. In your patched version, modify the loop that performs the 'queryMore' queries
    so that it takes into account the 'start' and 'count' parameters.

    As it goes drop entries earlier than 'start', and stop querying once it reached 'start' + 'count'.
    Hopefully this should free up the memory we don't actuall use ;
  2. Loop over all entries externally, by calling the Drush function itself in a loop, setting 'start'
    and 'count' accordingly. The advantage of doing this externally is that it can be spread out,
    for instance doing one chunk every cron run.

I don't have such a large data set, so it's a bit hard for me to test whether this would actually help.

kostajh’s picture

kostajh commented
Status: Needs work » Closed (won't fix)