Allow the reference field to be edited

Currently it seems that if a user ads a node via the Nodereferrer link on the referenced node, that node reference field cannot later be edited. Is that correct? This is a problem because users will sometimes add nodes in the wrong place so I would like to edit the node reference field so the referring node references the correct node. That's a mouthful.

Example) On my site, I've used Node Referrer so users can add a "Photo" (referring node type) for "Listings" (referenced node type). They usually add it on the correct Listings, but sometimes they add it on the wrong Listings and I want to edit it...but can't.