IMCE gets really slow, often does not render if too many images (500-error)

When the files directory contains 5000+ images IMCE begins to fail. Script timeout messages pop up in the user's browser and occasionally we get http/500 errors. This can *probably* be solved by changing server settings but my site is already very generous with how much memory/throughput/etc is available.

Upon disabling the wysiwyg IMCE button and converting to filefield (with filefield_sources enabled) the site performance seems to improve... very unusual. Of course when clicking the IMCE button on the filefield exhibits the same errors as noted above.

As mentioned this might be more of a server config thing, but just in case there is a memory issue here I wanted to report this. Also, some sort of "please wait" might be a suitable solution to this so that end users know to wait if lots of users experience this issue.