Entityreference is a great module and the fact that you can call a view in its widget it is even better.

What I would like to have in the Entity Reference views display, is the same "Argument Input" option from the Content Pane display.
Such option allows the view to be driven by contextual filters coming from the page the view is used on. Very useful with panel, that can pass tokens and context to the view.

I think is just a matter of adding the proper $option[argument_input] lines, and CTools should handle the rest, as it does for the Content Pane. I had tried that myself, but being no coder I accomplished nothing.