While it's possible to get to a reasonably large Drupal install on a single server where the webserver, Solr, and the SQL server are running on the same box, at some point you will need to separate these pieces to scale the site up.

Generally the Solr search server would be totally decoupled from the webservers for a large-scale deployment. You would not have it run on any of the web nodes so that you can maximally utilize them to deliver web pages.

The way Solr works, you need a single master server to send index updates to. You can mitigate the single point of failure by replicating the index to one or more slave servers. see: http://wiki.apache.org/solr/SolrReplication

The Drupal Apache Solr Search Integration module doesn't have any built-in facility for using different URLs for updates and searches, so the simplest approach to spreading requests across multiple Solr servers would be to have a load balancer (e.g. nginx) that routes your requests among Solr servers, with updates going to the master, and searches spread across the master and any slaves. The advantage of using nginx (or Varnish, etc) to load balance is that failed servers are automatically removed from use and/or another server tried instead.

For sites (such as drupal.org) serving certain very popular pages based on search results from Solr, it is useful to cache search results using Varnish (or possible nginx, etc) as a HTTP cache. Solr supports such caching by supplying ETags in responses and handling 'if-modified-since' headers sent by the HTTP cache.
Example : solrconfig.xml

<httpCaching lastModifiedFrom="openTime" etagSeed="Solr"never304="false">
  <cacheControl>max-age=43200, must-revalidate</cacheControl>
</httpCaching>

More caching tips can be found
http://wiki.apache.org/solr/SolrCaching

As an alternative to using a separate load balancer, if you have a static set of Solr servers and are using Apache httpd, you could potentially configure each web server to proxy and load-balance the search requests see http://httpd.apache.org/docs/2.2/mod/mod_proxy.html and http://httpd.apache.org/docs/2.2/mod/mod_proxy_balancer.html

While not recommended, you could also write custom Drupal code to alter the host for the search before it's run using function $solr->setHost($host). In such a case you could just send all searches to one slave, or among all Solr servers based on some list, but you'd likely want a way to detect failed requests and would end up building your own load-balancer in PHP.