I'm looking to remove some of the dropdown items on the panels_content page that are provided by CTools.

The links are provided by panels_edit_panel_get_links() in panels/display-edit.inc, and it's easy enough to hack them out at this point. However, I'm more interested in cleanly removing them and avoiding altering a contrib module to preserve an easy upgrade path. I can think of a few ways to do this, but am not sure the best way to hook into this functionality to change it given the stack of things it rests under.

Is there an easy way to make modifications to these links using the hook system?