Any user who can edit a node should be able to remove it from a group

At the moment, a user may remove a node from groups only if he/she is an admin of those groups or is both the node's author and a member of those groups.

Actually, anyone who has the permission to edit the node should have that right.

For non-admins, we need to check node_access('update') rather than $node->uid, and ignore the user's group subscriptions.