Edit links appear on taxonomy term views even if user doesn't have "access contextual links" perms

If a user has "administer taxonomy" permissions but not "access contextual links", errant "edit" links can appear on the page. The problem has to do with not have sufficient checking of permissions before modifying the $links variable in contextual_preprocess_views_view(). Patch is attached.