Can't save panel page because varnish returns error

I get the following error:

http://mysite.com/structure/pages/edit/site_template|

Recoverable fatal error: Argument 1 passed to _esi_panels__display_uses_esi() must be an instance of panels_display, null given, called in /esi/modules/esi_panels/esi_panels.module on line 218 and defined in _esi_panels__display_uses_esi() (line 261 of /esi/modules/esi_panels/esi_panels.module).

Once I disabled esi I was able to save the panel page