Role-based access to filefield_sources elements

Hi there,
I want to restrict other sources (imce, attach...etc ) to those who have sufficient permissions (i.e. prevent #process callback from running if user does not has enough permissions). Is this doable?
Thanks