My (agentrickard) session on getting external data into Drupal went pretty well. You may download a PDF version of the presentation from my website.

The main points are basically:

• You don't need to batch import data into Drupal as long as users can find what they want
• "Lazy instantiation" is great for dealing with large data sets
• Drupal supports several methods of lazy instantiation
• Cache aggressively to reduce network latency
• Pipes is very cool.

In all, I think you'll find that Drupal can be a powerful integration framework, if you examine the APIs.