Need to manually clear cache for term to display

When I enable the "list" permission for a term in taxonomy_access, the term doesn't display in the directory module until I manually clear the cache.

-Nate