Views caching of rendered entity field misarchitected, leading to unexpected results.

I have a view with some rendered entity fields (which by the way is the coolest thing since the thermos jug!).
Also i have a special view plugin that shuffels around my views rows in the pre-render.

Expected result: All fields are shuffeled.
Experienced: Rendered entity fields stick to their row.

Further analysis shows that rendered entities are cached in field_handler->wrappers with row-id as index.

They should be cached with entity [revision] id as index.