Disable &nbsp; and more than <br> tags in a row

I'd say you'd need a special filter designed to do that if you need to stop multiple breaks from being rendered. I don't think there are any editor settings that could be tweaked to do that, but a plugin could certainly try to remove multiple break tags. The problem with that approach is that anyone can disable the editor an add as many breaks they want.

There's also the problem of what the user expects to happen when hitting enter or shift+enter multiple times vs what they get, both with an editor plugin that ignores the keys and/or removes the tags when entered and with a filter based approach where users see the extra breaks in-editor but not in the rendered contents. Only education could prevent such a confusion, but that might not be easy to do depending on the type and number of users.

I see these problems a lot too. Just about every MS Word socument I've ever recieved is "formatted" this way...

I've seriously considered using an editor like WYMeditor to make it clearer to users what they're actually doing to the content on a markup/semantic level. I've not tested it on my clients yet so I'm not sure how they would react.
An editor like that still doesn't "physically" prevent users from making breaks, spaces or paragraphs everywhere though.

Maybe the easiest solution (if there isn't already a plugin for this somewhere out there) would be to write a page with content guidelines, and/or set up a review workflow where only a few users are allowed to publish content others have written. If that's possible or doable depends on your site and users/community of course.
Teaching everyone HTML, CSS and separation of content and presentation is probably not an option, but maybe all they need to know is that better tools for the job than the enter and space keys.

Yes, that would indeed be a sort of hack, but at least you're not hacking Core or a module. ;)
If you need a long-term soltion, I suggest looking up how to create your own input filter. It would be the "proper" way of handling this during rendering, if it can't be done during actual input or pre-db-storage, as it would allow for the changes to be cached etc.