Nginx, Fastcgi, PHP, rewrite config for Drupal
Afternoon.
So I've been getting stuck into making Drupal 4.7 (and 5.0) work with Nginx, which is a bit like Lighttpd except without the firehose-esque memory leaks you get with Lighty and actual web traffic busier than a trickle.
This has worked for me for the last several days on (! NSFW !) Cliterati.co.uk, which roars through about 30 HTTP requests per second.
Credit: This page was completely invaluable, and everything Drupal-ish here is merely minor edits to that earlier work.
Why would you want to do this?
Because if you're running Apache/Apache2 with mod_php on a dedicated server with 1Gb of memory, and you have a lot of traffic, and more than about 50 of your visitors are logged in and posting to forum.module most of the time, then your dedicated server can't run Drupal. This is nuts. While people are raging against the non-existent caching for uid>0 in Drupal, you may want to cut your static memory requirement by about 85% at a stroke by showing Apache the door.
Nginx (and lighttpd) do more than just this, in performance terms, but even if they didn't I'd still need to run one of them to keep such a server afloat.
Nginx
Firstly you have to install nginx, which is not going to be covered here. I went for compile-from-source because the versions in Debian repositories are ancient. Next, here's the configuration details I've found to work with Drupal and URL aliasing. I'm posting an example of an entire http{} section from the config file. It passes everything PHP-related to one or more PHP fastcgi processes listening on port 8888, which you'll be setting up after this.
In nginx.conf:
http {
include conf/mime.types;
default_type application/octet-stream;
server_names_hash_bucket_size 128;
#log_format main '$remote_addr - $remote_user [$time_local] $request '
# '"$status" $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 20;
tcp_nodelay on;
#gzip on;
server {
listen 192.168.0.1:80; # Replace this IP and port with the right ones for your requirements
server_name example.com www.example.com; # Multiple hostnames seperated by spaces. Replace these as well.
#charset koi8-r;
#access_log logs/host.access.log main;
location = / {
root /path/to/drupal; # Again, replace this.
index index.php;
}
location / {
root /path/to/drupal;
index index.php index.html;
if (!-f $request_filename) {
rewrite ^(.*)$ /index.php?q=$1 last;
break;
}
if (!-d $request_filename) {
rewrite ^(.*)$ /index.php?q=$1 last;
break;
}
}
error_page 404 /index.php;
# serve static files directly
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
access_log off;
expires 30d;
}
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
location ~ .php$ {
fastcgi_pass 127.0.0.1:8888; # By all means use a different server for the fcgi processes if you need to
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME /path/to/drupal$fastcgi_script_name; # !! <--- Another path reference for you.
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
}PHP and Fastcgi
I'm not mentioning PHP 4 or 5 because this part is exactly the same for each. I'm referring to 'php5-cgi' because that's the name of the right binary for the right PHP version on my Debian-based server. Your mileage may vary.
This shell script will launch a few fastcgi PHP processes bound to port 8888 for Nginx to talk to. I launch it as root - it starts the processes as the Debian apache user and exits.
#!/bin/bash
## ABSOLUTE path to the PHP binary
PHPFCGI="/usr/bin/php5-cgi"
## tcp-port to bind on
FCGIPORT="8888"
## IP to bind on
FCGIADDR="127.0.0.1"
## number of PHP children to spawn
PHP_FCGI_CHILDREN=5
## number of request before php-process will be restarted
PHP_FCGI_MAX_REQUESTS=1000
# allowed environment variables sperated by spaces
ALLOWED_ENV="ORACLE_HOME PATH USER"
## if this script is run as root switch to the following user
USERID=www-data
################## no config below this line
if test x$PHP_FCGI_CHILDREN = x; then
PHP_FCGI_CHILDREN=5
fi
ALLOWED_ENV="$ALLOWED_ENV PHP_FCGI_CHILDREN"
ALLOWED_ENV="$ALLOWED_ENV PHP_FCGI_MAX_REQUESTS"
ALLOWED_ENV="$ALLOWED_ENV FCGI_WEB_SERVER_ADDRS"
if test x$UID = x0; then
EX="/bin/su -m -c \"$PHPFCGI -q -b $FCGIADDR:$FCGIPORT\" $USERID"
else
EX="$PHPFCGI -b $FCGIADDR:$FCGIPORT"
fi
echo $EX
# copy the allowed environment variables
E=
for i in $ALLOWED_ENV; do
E="$E $i=${!i}"
done
# clean environment and set up a new one
nohup env - $E sh -c "$EX" &> /dev/null &Initial results
I've moved two phpAdsNew ad servers, a fairly-busy Wordpress blog, the above-mentioned Drupal site and my own Drupal site from Apache2 into Nginx in the last week. Processor use has increased on the server, but critically the ongoing web+database memory use has come down by over 300MB. This means that the box hasn't gone into swap for a week (normally it was paging out every day), server response times are up and (because I'm also using APC to store Drupal's source files in memory) page download speeds are also up considerably.
Basically, it's saved a server.
I've tested it with Drupal 4.7 and Drupal 5.0, and with PHP4.3.3 and PHP5.2.0. All those configurations work.
Caveats
Drupal can serve static files either directly or via Drupal itself - it's a configuration option (in Drupal 4.7 it's in admin/settings - I don't yet have 5.0 seared into memory). This configuration requires that you serve those files directly. If Drupal's configured the other way your site will look incredibly odd, and image.module in particular breaks horribly. Me, I don't care. If this is an issue for you, it might be time to get into that in the comments.
Enjoy... and if you've got suggestions to improve this, I'd love to hear them.

Server Specs & Site Traffic
handelaar,
Thanks for putting this information out there. We're looking at getting our own server and consolidating all of our Drupal stuff onto it as well as some misc traffic. Would you mind providing the hardware configuration of the server (RAM, disk(s), CPU(s), etc) as well as the type of traffic you get (hits per day/week/month whatever)? We're trying to figure out if a Celeron would do us to start or do we need something higher powered and if so how much more do we need.
Your time putting this together is appreciated.
Server details
It's a dedicated box at Servermatrix. Intel(R) Celeron(R) CPU 2.40GHz, 1Gb RAM, 80Gb HDD.
Like I said, it's doing 30-odd HTTP requests per second. The majority of the traffic at any given moment is coming from one of two sites, which between them serve 1.3m pages per month. A bunch of other co-hosted Drupal installs bring that total over 2m between them.
Interestingly, I found that locking MySQL down a little (using my-medium.cnf) and restricting its memory footprint works rather better than giving it my-huge.cnf and running the risk of sending the whole box into swap during peak periods. And both the high-traffic sites have had their databases converted to InnoDB to avoid the problem of one posted comment causing the site to be unable to show any pages to anyone for several seconds, because 400 SELECT queries are stalled for every 1 second spent on that INSERT (and the time spent waiting for table locks before the INSERT can begin).
"MyISAM considered harmful", indeed.
See also:
Right under this page on a Google query for nginx rewrite examples, Scott Yang draws my attention to the part of the nginx docs which I couldn't see staring me in the face.
Where Apache uses !-f then !-d in sequence, I was looking for a way to ape it. Turns out I didn't need to. So above where I wrote this:
if (!-f $request_filename) {
rewrite ^(.*)$ /index.php?q=$1 last;
break;
}
if (!-d $request_filename) {
rewrite ^(.*)$ /index.php?q=$1 last;
break;
}
...you should instead use this...
if (!-e $request_filename) {rewrite ^(.*)$ /index.php?q=$1 last;
break;
}
...because Nginx has the "-e" directive which matches against a file, directory or symlink all at once, rendering my attempts to replicate the Apache way unnecessary.
You really want this instead...
Try this instead:
if (!-e $request_filename) {rewrite ^/(.*)$ /index.php?q=$1 last;
break;
}
Drupal is expecting something of the form "node/1" and not "/node/1" after the "q=". There is at least one module, GlobalRedirect, that does not tolerate the leading "/". The rule above fixes the problem.
Current Drupal site: Inventor Spot
drupal in a subdirectory
I haven't used drupal under nginx in a subdir, until the other day. Found out the above won't work. You have to do something like:
if ($request_uri ~* ^.*/.*$) {rewrite ^/(\w*)/(.*)$ /$1/index.php?q=$2 last;
break;
}
for your rewrite instead.
For example, if your subdirectory is called mysub a request of http://example.com/mysub/admin/report/status will be rewritten correctly as http://example.com/mysub/index.php?q=admin/report/status instead of http://example.com/mysub/admin/report/index.php?q=status. Remember that .* is greedy.
There is probably a better way to write this, but this works for me for now.
Nginx Performance & Feedback
handelaar, thanks for the information on your server. Based on the fact that most of our sites will be running semi-static content (i.e. usually updated by us and not the users) I think we'll be okay with a lower end box.
As an aside I ran some performance numbers using http_load and the difference between Apache2 and nginx was pretty dramatic. Granted this isn't completely an apples to apples comparison but I did do some tweaking with both to get the numbers more inline but nothing too drastic. This was on a fresh Debian Stable install using a 'as delivered' from Dell Precision 360 w/1GB of RAM and a 3GHz Intel P4 inside my local network where both machines were connected to a Linksys WRT54G. Also I repeated this test several times to verify the results were consistent. The Drupal install was the latest 4.7.X branch with 7 pages set up and referenced in the urls.txt file.
Server: Apache2
Apache2 caching=On
Drupal Caching=On
$ http_load.exe -parallel 50 -seconds 30 urls.txt
547 fetches, 50 max parallel, 3.62053e+06 bytes, in 30.015 seconds
6618.88 mean bytes/connection
18.2242 fetches/sec, 120624 bytes/sec
msecs/connect: 122.124 mean, 3250 max, 0 min
msecs/first-response: 2337.75 mean, 5016 max, 141 min
HTTP response codes:
code 200 -- 547
Server: nginx
PHP_FCGI_CHILDREN=6
PHP_FCGI_MAX_REQUESTS=250
eAccelerator=On
Drupal Caching=On
Browser Load Time = 1
$ http_load.exe -parallel 50 -seconds 30 urls.txt
2748 fetches, 50 max parallel, 1.83063e+07 bytes, in 30.015 seconds
6661.68 mean bytes/connection
91.5542 fetches/sec, 609905 bytes/sec
msecs/connect: 6.52766 mean, 3265 max, 0 min
msecs/first-response: 215.662 mean, 3625 max, 0 min
HTTP response codes:
code 200 -- 2748
I'm still playing around so I'm sure the numbers from nginx will only get better.
I dont think that this would
I dont think that this would be a fair comparison, we all know that eAccelerator makes apache 2 very much faster, yet you didnt test apache2 with it here, but tested nginx with it.
Addition to the nginx.conf
I added
location ~* /(modules|themes|misc|sites|profiles|scripts)/ {return 404;
}
to prevent download of the non .php extension files.
themes dir?
Are you sure about the themes directory?
To test this make sure to restart nginx and clear your browser cache.
Sorry
I'll blame lack of sleep. This doesn't actually work, didn't realize how many .css files are under the modules dir now. Here is the better config.
location ~* /(modules|themes|scripts|sites)/ {if (-f $request_filename) {
rewrite \.(module|inc|info|engine|sql|sh)$ / permanent;
}
}
My /admin/build/modules page
My /admin/build/modules page wouldn't submit after using this bit. Why not just copy how .htaccess does it?
This worked nicely for me:
# hide protected fileslocation ~* \.(engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(\.php)?|xtmpl)$|^(code-style\.pl|Entries.*|Repository|Root|Tag|Template)$ {
deny all;
}
--
Scott Nelson
This by Them, LLC
Services Module
Debian style init script
Here's a Debian-style init script for starting and stopping the PHP FCGI instances. The one thing it doesn't do is clear the environment like the original does. You could probably interleave env into this somehow to get it to do that, but I didn't go that far on this first pass. Note that, as with any use of start-stop-daemon and --background, you don't get much error checking on startup.
#!/bin/sh -e
# Start or stop PHP FastCGI handlers
#
# based on Postfix's init.d script
SSD=/sbin/start-stop-daemon
DAEMON=/usr/bin/php5-cgi
NAME=phpcgi
FCGIPORT="8080"
FCGIADDR="127.0.0.1"
export PHP_FCGI_CHILDREN=5
export PHP_FCGI_MAX_REQUESTS=1000
USERID=www-data
PIDFILE=/var/run/fastcgi.pid
TZ=
unset TZ
test -x $DAEMON || exit 0
. /lib/lsb/init-functions
DISTRO=$(lsb_release -is 2>/dev/null || echo Debian)
running() {
if [ -f ${PIDFILE} ]; then
pid=$(sed 's/ //g' ${PIDFILE})
exe=$(ls -l /proc/$pid/exe 2>/dev/null | sed 's/.* //;')
if [ "X$exe" = "X$DAEMON" ]; then
echo y
fi
fi
}
case "$1" in
start)
log_daemon_msg "Starting PHP FastCGI Handler" ${NAME}
RUNNING=$(running)
if [ -n "$RUNNING" ]; then
log_end_msg 0
else
if ${SSD} --start --chuid ${USERID} --background --pidfile ${PIDFILE} --make-pidfile --startas ${DAEMON} -- -q -b ${FCGIADDR}:${FCGIPORT}; then
log_end_msg 0
else
log_end_msg 1
fi
fi
;;
stop)
RUNNING=$(running)
log_daemon_msg "Stopping PHP FastCGI Handler" ${NAME}
if [ -n "$RUNNING" ]; then
if ${SSD} --stop --pidfile ${PIDFILE} --signal 15; then
rm -f ${PIDFILE}
log_end_msg 0
else
log_end_msg 1
fi
else
log_end_msg 0
fi
;;
restart)
$0 stop
$0 start
;;
force-reload|reload)
$0 stop
$0 start
;;
*)
log_action_msg "Usage: /etc/init.d/fastcgi {start|stop|restart|reload|force-reload}"
exit 1
;;
esac
exit 0
Init script submitted to Debian
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=426780 has yet another init script that is being proposed for inclussion in Debian's php5-cgi.
imagecache module mods to nginx.conf
If you're using the imagecache module, serving static files directly like the original poster does, nginx bypasses the image rescaler. In order to make image cache work with the following,
# serve static files directlylocation ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
access_log off;
expires 30d;
}
you need to add the code below to force the imagecache images through PHP. My code assumes that your imagecache image dir is set up under /files/imagecache.
# imagecache needs to have php read any files that it's planning to manipulate
location ^~ /files/imagecache/ {
index index.php index.html;
# assume a clean URL is requested, and rewrite to index.php
if (!-e $request_filename) {
rewrite ^/(.*)$ /index.php?q=$1 last;
break;
}
}
Current Drupal site: Inventor Spot
Can't get rewrite to work with nginx
I am not able to get the rewrite working correctly...
Without the rewrite rule all works as expected. (No rewrite but the main site shows, just no links work because of no rewrite)
When I have the rewrite section in my conf file, I get no CSS, pictures, just straight text with a darn funny looking white page.
Here is what I have in my conf file:
===============================================
# serve static files directly
location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$ {
access_log off;
expires 30d;
break;
}
if (!-e $request_filename) {
rewrite ^(.*)$ /index.php?q=$1 last;
break;
}
error_page 404 /index.php;
location ~ \.php$ {
include /usr/local/nginx/fastcgi.conf;
fastcgi_pass 127.0.0.1:57102;
fastcgi_index index.php;
}
===============================================
Very strange indeed as it seems like the above would work. When I comment out the rewrite section the home page displays fine but just not rewrite (as expected).... As soon as I put that rewrite section back in I get the page with straight text, no css formatting, no pics, etc.
ANYONE have any ideas? Spend all day on trying to figure this out and I am completely stumped.
Shell Script Modifications
The shell script as shown above did not work for me. (Debian Etch circa 5/2008). I had two issues:
1. It did not spawn anything.
2. When I tweaked it some more, I got it to spawn a parent, but no chldren.
3. When I finally got it to spawn, there is another issue where the user account that started the script (not www-data) owned the process and threw permission errors.
So, I revised the punchline of the script a bit. Instead of:
if test x$UID = x0; then
EX="/bin/su -m -c \"$PHPFCGI -q -b $FCGIADDR:$FCGIPORT\" $USERID"
else
EX="$PHPFCGI -b $FCGIADDR:$FCGIPORT"
fi
I used sudo with as few modifications to /etc/sudoers as possible. If you spent some time modifying /etc/sudoers, you could probably come up with a more elegant approach:
if test x$UID = x0; then
EX="sudo -u www-data env PHP_FCGI_CHILDREN=4 $PHPFCGI -c $PHP_CONFIG_FILE -q $
else
echo "Sorry, can't start. Must start as root"
fi
Pay careful attention to the "env..." bit. This allows the php5-cgi to make children. To run multiple versions of php, I added the $PHP_CONFIG_FILE stanza as follows:
##DIRECTORY to *find* the php.ini
PHP_CONFIG_FILE="/etc/php5/cgi"
I still have a problem where the listener dies with peak loads. php5-cgi sucks all of the cpu up and then dies. I don't care about the cpu running near 100%. it seems like it can't queue very well. Does anyone have any experience with nginx instance as load balancer to multiple php5-cgi listeners?