While the options for users to select which should be displayed on the Custom Contact List are toggleable on the Custom Contact Management page (admin > user > user > contact), it doesn't work on the user edit forms (user > [user] > edit#contact).

It's a bit strange since the toggle-links itself seem to be ok, e.g.:
http://[domain]/flag/unflag/anonymous_contact/?destination=user/[user]/edit&token=[token]

But if I click on these links, the option doesn't visibly toggle and seems not to be saved.