I've looked a bit at the bundles that get generated. Generally, think there are too many that include jquery.js (or jquery.min.js).
The existing bundler module allows one to restrict the number of bundles that are put into one page.
What we want instead is a limit on the maximal file size in addition to that. That is, all files that are over a certain size limit (e.g. 100k) will never be added to a bundle but always will constitute a bundle of their own, possibly simply a link to the file. This would potentially increase the limit of the bundles per page but would make sure that big files are only downloaded once by a partiular user for the whole site.
To do that we need to:
1) change the main advagg module to also record file sizes, not only md5 hashes etc.
2) rewrite the bundler module (or perhapes create a custom module) that copies most of the existing module but has the altered functionality that I described.