Adding a text field to user reference

I am trying to add a content type that is very simple and just has a title, body and set of users.
Easy right, well there is one slight issue though, I want to use the user reference field and set it to multiple values so I can add a number of users to this content type. Bound to each user I want to have a role in this type.

Title
Body
User 1 - Role
User 2 - Role

I could put the Role as a optional field in the user and thereby reference it in my view but I would like to have it really in the content type itself since this field is only needed for a vast minoroty of users on the site. Anyone has any good solution for this ?

Best Regards