In upgrading the audio module on a couple sites recently, I found myself going through the extra step of making sure the getid3 dir didn't get overwritten within the audio module directory.

Is there a downside to creating a sites/all/lib directory, and using that to store 3rd party libraries?

This keeps the libraries separate from the module code, which could have some advantages when it comes to maintaining modules that use 3rd party libraries that cannot be distributed from d.o for security reasons, or for licensing reasons.