Language Inconsistency in contextual filter for (Content entity referenced from field_section_reference) Content: Nid

I have a view that shows content with an entity reference field referencing the currently displayed node.
This is a scenario often used, just a block (pane in my case) that shows content (child nodes) that belongs to the current node (parent node).

This uses the contextual filter (Content entity referenced from field_section_reference) Content: Nid
with 'provide default argument': content ID from URL'

The view is filtered with 'Field Language:Current user's language'

This works fine for the default site language, showing all content of the language (url language selection)
But when I use the language switcher, the view does not show anything any more. Translated content of current language is not shown.

I removed the contextual filter and added a normal filter, selecting manually a language other than the default site language. This worked, translated content shows fine.

If you are reading this and have entity translation enabled in a similar setup, please do me a favor and test if I'm right. I have spend quite some hours trying to find alternatives and workarounds, but can't display a view of translated content that is referencing the currently displayed node, when using a not-default language.

Thanks