Rewrite file field output breaks with contextual links

Case:
Content type with file field and link field. Using media module and file_entity for displaying images.

View block with excluded link field outputting link as text, file field showing image with formatter file_rendered with rewrite to output field as a link, using link field token.

Now with contextual links core module enabled, this view works in preview mode, but not when shown in block on the site. The image gets moved out of the "" tag. When I disable the contextual module, the view works in both situations.

I'm not sure if this is a views, media or file_entity problem. But I assume it belongs here.