Core slows down when few modules are missing

Not sure if this is bug but this feature can cause a lot of headache.

Problem is in module_load_all() function called in _drupal_bootstrap_full(). module_load_all() leads by different paths to drupal_get_filename() which is trying to be smart about locating files. If module file doesn't exists in location stored in system table function calls drupal_system_listing() to find .module file. If this happens 4 - 5 times drupal page load time grows to 15 - 20 seconds, where most time is spent in listing function.

Shouldn't this happen only on module admin page to try to locate those files and update system table? If this behavior is ok at least should be documented.