What the Field UI does currently is still modeled after the D7 architecture:
- define routes for "Manage display [view_mode]" pages,
- use access callbacks to only show those where the view modes uses "custom settings"
- in the page callbacks, show a form collecting various stuff and save them where appropriate on submit.
The D8 situation is that those "Manage display" pages really are "where you edit an EntityDisplay entity".
So the EntityDisplay object should be loaded by the route as part of parameter upcasting (in a way that the upcast would fail if status = FALSE --> 404), and received as an argument in the form builder (instead of currently: the form builder receives the raw URL parts, and at some point goes "I'll load an EntityDisplay object because I'll do stuff with it").
(and same for form displays of course)
An additional step might be to make those forms be the actual, official "Entity forms" for the EntityDisplay entity types (in the Entity API sense), but that might be a bit harder since the entity types are provided by entity.module while the forms are provided by field_ui.