I have a large hierarchical taxonomy and can see the the LEFT JOINs are killing the pager_query during page load.

I posted a feature request to Drupal core for a node count table per term SQL table which would help as well..
Will try this module now.