for now just doing this...
myModule_form_alter( **if particular code** unset($form['field_my_user_reference'] );