Over use of Views/Panels?

Panels and Views seem pretty amazing, but before I go overboard with them I just wanted to run my idea by the community. My site design is a 3-column layout (with a header and footer) with multiple distinct sections in each column -- just like the Drupal default theme. Maybe sections are used several places on the site, but there is never content that is not within a section.

For the sections on the left and right columns I was thinking about using blocks, perhaps generated via the Views module. I've noticed that I couldn't put blocks in the content column since actual content is lighter than the lightest weight of the block and I couldn't hide the content in the View. So realizing this I turned to Panels.

I saw that the '3-column stacked' layout in Panels is exactly how my site is setup. So now I am thinking, should I just hide the left and right columns with blocks in them (and the footer and header) then just use the '3-column stacked' layout that Panels provides? Or should I keep Drupals default header, footer, and side columns then just add the Panel as a single column in the middle?

Is this just a matter of preference? When people use Panels do they go all out and just panel everything with mini panels to display blocks in lieu of using Drupals core modules?

Thanks in advance.