By andrewbenkard on

Some users of a D6.x extranet that I built are encountering an authentication problem. It's intermittent and hard to reproduce.

Symptom:
An authenticated user clicks to download a PDF. The server times out.

Let's go to the logs.

1. Drupal dblog says Page Not Found, message is 403.shtml, no referrer. Usually there are a few instances of this as the user encounters the problem.

2. Apache error log:
[Mon Oct 24 15:17:12 2011] [error] [client 000.000.000.000] client denied by server configuration: /home/sitename/public_html/sites/default/files/some.pdf
[Mon Oct 24 15:17:10 2011] [error] [client 000.000.000.000] client denied by server configuration: /home/sitename/public_html/sites/default/files/some.pdf
[Mon Oct 24 15:16:23 2011] [error] [client 000.000.000.000] client denied by server configuration: /home/sitename/public_html/sites/default/files/some.pdf

...In some instances, there are 20 or more of these rejected requests. Perhaps that is a user re-clicking in frustration.

Environment:

  • D6.22 with (partial module list) Content Permissions, LoginToboggan, Pathauto, Path Redirect, Persistent Login.
  • PHP memory limit 96M.
  • System is on a well-provisioned VPS, plain vanilla LAMP stack.
  • File system is Public download method
  • Only Authenticated Users are permissioned to see content.
  • Site traffic level is low - 95% of the day there are <2 concurrent users.

More clues:

  1. In the case of one user, his Internet Explorer aleays freezes. However he can always download PDF's fine with Firefox. Weird.
  2. Restarting the browser often helps the user avoid the problem.
  3. It's happened to me once, when logged in as uid1. Otherwise I don't see it.
  4. The PDF's are all properly permissioned, 644, in the files directory which is 775.
  5. Deactivating mod_security doesn't help.
  6. Persistent Login alerts me that session.cookie_lifetime PHP setting is 2000000, and that it should be 0.
  7. Persistent Login "Days to remember the user" is zero i.e. no expiration.
  8. Possibly related: we are occasionally encountering the charming "PHP Warning: MySQL server has gone away" error. Working on a fix for that separately.
  9. Possibly related: One user complains about having to re-login unexpextedly. Details sketchy on this.

Any suggestions on what component to troubleshoot next?

Thanks in advance
Andrew
NJ, USA

⋅ Categories: Drupal 6.x

Comments

andrewbenkard’s picture

andrewbenkard commented

FWIW I bumped PHP memory limit to 256M. No change; still have the problem.

andrewbenkard’s picture

andrewbenkard commented

I dug deeper into the apache logs and rediagnosed the problem.

PDF's were on occassion being broken up into lots of 206 requests by apache's byterange processing aka byteserving. Normal. But something was introducing 404's and that seized up the browser's Reader plugin. Drupal reported bunches of 403 in dblog.

The sysadmins at my host now report:

However, I did notice multiple denied errors while loading the file, as well as random occasional 404s. It seems that this chunking behaviour was triggering mod_evasive, which was causing the browser to not load some chunks (due to the number of request), and then hammer the server until the file finally downloaded after the timeouts reset.

mod_evasive is now off and we seem to be in the clear. I don't know if my host (WiredTree, which I recommend) has security cranked up any more than normal.

Anyhow, there you go.