ESI does not restore original request url on exit so page cache only stores esi fragment for each page

The ESI panels and blocks module, do override $_GET['q'] and $_SERVER['REQUEST_URI'] with the original values of the page it was included. That's needed for proper content generation, for sure. Problem is that on delivery, esi_deliver_esi_component calls drupal_page_footer. Depending on your various cache settings both drupal and varnish sides, this will lead drupal to cache the content of the ESI under the full page path. Result obviously is that a visitor hitting a page with an ESI include that has been cached that way will be served the ESI content itself instead of the page.
Resetting the original context in hook_exit does prevent that from happen (patch provided for the panels submodule), but it should probably be handled globally by the main esi module. This would mean adding the original url to the esi url from the esi module itself instead of having that done for each submodules. Page callback esi_handle_component would then retrieve it at the page callback, and reset the original values before passing back the renderable array to the the delivery callback. Does that make sense ?