It's good that filefield_file_download() prevents a file from being seen when the user doesn't have permission to view the node it is attached to. However, if a file is attached to several nodes, and only one of them is restricted, then the file is unviewable in all scenarios. Allowing files to be reused on nodes is the explicit purpose of filefield_sources, but the same thing can also happen with imported data or other uses of files like taxonomy_image.
I wonder if the node_access() check really belongs in this hook. The context of the file can't really be gained just from the filepath. Instead, it comes from whatever is causing the file to be viewed, whether it's through a link, or whatever. It's not enforceable except by calling it a best practice, but the calling context should call node_access() before displaying the file.