Great and easy setup, once it is all understood how Drupal handles private files.

In my site config I added an error_page directive so that if someone tries to access the file directly they get a Drupal 404 from the site instead of the generic nginx error.

location ^~ /sites/default/files/private/ {
  internal;
  error_page   404  /index.php;
}