Language switcher does not switch back to default language after having switched languages

I am using language domains "lang.example.com" instead of language prefixes "example.com/lang". I left my English (default language!) language domain empty as suggested my Drupal ("Leave blank for the default language.") and set my French language domain to "fr.example.com".

When I acces my site via "example.com" the language switcher links "English" to "example.com" and "French" to "fr.example.com". Up to here, everything is fine. However, when I switch to French (i.e. when I access my site via "fr.example.com") then the language switcher links "English" to "fr.example.com" instead of "example.com". I suspect that this is the case because Locale module can't find any language domain assigned to English (remember, I left it blank) and hence uses the currently active domain instead. In fact, when I do set my default language domain to "en.example.com" then the language switcher on the French page links "English" back to "en.example.com". However, I don't think that this is by design as the default language should always be accessible/linked to via the default website domain "example.com".