A common question, particularly by relative newcomers to the Drupal community, is "Why isn't feature X in core?" Feature X is often either a "utility" module used by most Drupal sites, such as Views or Pathauto, or a module that is only used for specific types of websites, such as Fivestar or LoginToboggan.
Innovation vs. stability
One of the biggest arguments for not adding a feature to core, is that once it is part of core, it has more potential to stagnate. For example, the Profile and Poll modules have seen little change since they were first introduced to Drupal.
For comparison, the Views module has seen multiple iterations since being introduced in Drupal 5. Its original incarnation could only handle creating lists of nodes. The second generation extended the concept to taxonomy, users, and other entities. The third generation added even more flexibility and a complete overhaul of the UI.
This is possible because contributed modules do not have to meet the same release, testing, review, and documentation requirements of core, for better or for worse. A contributed module maintainer can choose to do a complete rewrite at any time, rather than waiting until the next major release of Drupal core.
Maintenance
People often lament the fact that when you download a package such as Joomla! there is much more functionality out of the box. Drupal core is relatively simple; while you can build a small community site with the tools in Drupal core, building most sites requires the use of contributed modules.
However, remember that adding more code to Drupal's core requires more time for the development team to get it properly tested and released. Keeping core to its essential functions, and encouraging contributed modules to "hook" in and extend it, is the Drupal way.
Some people might argue that longer release cycles can only be a good thing, since they dislike having to update their websites every 18-24 months. But shorter release cycles are an advantage of Drupal. By quickly adapting new technologies and ruthlessly cutting outdated ones, each release opens new possibilities.
Time and resources
It takes a long time to get things into core. Unlike contrib, which is a "wild west" where each module maintainer plays in his own sandbox with relative impunity, every single line of code that goes into core is meticulously scrutinized. It has to be, because Drupal core is used by millions of websites, and the community's attention to detail is one of Drupal's key strengths. Most changes to core go through dozens, if not hundreds, of revisions before they are deemed worthy by the larger community.
Even on ubiquitous modules such as Views and Pathauto that are used by most sites, there are usually only a few people doing major development work. Maintainers of these modules often have their hands full keeping up with bug reports and feature requests for their module, often during their free time. Time spent getting a module into core is not spent providing support for the hundreds or thousands of sites using the module right now. Often, efforts to bring these modules (or important aspects of these modules) into core requires additional developers to step up, or funding of the module maintainers.