Hi

I'm becoming convinced that

a) Drupal (particularly 4.6) with mod_php and apache/apache2 simply cannot scale to >2 pages per second on a site which has fast-changing content like busy and large forums, if you only have one server to run it

b) There's a decent chance that replacing Apache in the chain and using lighttpd/fastcgi has a good chance of increasing that performance to enable a single box to cope.

So I'm working out a recipe to describe the steps required to move from one to the other. If you have recipes which work for you, please add your comments here and help me get a good manual into the handbook.

[Note: I'll want to cover "fixing" superglobals, enabling mod_rewrite, and adding an SSL server to the mix for ecommerce use.]

Thanks...

jh

Comments

kbahey’s picture

John

I know that you run one of the highest traffic Drupal out there.

But is 4.6 as a start worth it?

Note that there is a patch for HEAD (i.e. 4.7) about fast path / file caching that will speed up access enourmously by caching content for anonymous users.

I looked briefly at lighttpd/fastcgi, and it required more time than I was willing to spend that week. It is not a pressing need for me since performance for my sites is not an issue at present.

This is where I last looked:
http://www.lighttpd.net/documentation/fastcgi.html#preparing-php-as-a-fa...

I think Gerhard has done some work on FastCGI, not sure if it was Apache or Lighttpd though.
--
Drupal development and customization: 2bits.com
Personal: Baheyeldin.com

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

handelaar’s picture

Sadly there'll be many sites which are stuck with 4.6 until all their modules are in place.

4.7 fixes a number of things, performance-wise. The url_alias table patch is in, for example. And we've finally got rid of the nutjob schema which puts node bodies in the node table (which rules, even if importing data did just get exponentially more irritating :) ).

Of course I'm interested in doing a recipe which applies to both versions - or two recipes, if need be.

I did see mention the other day from someone telling us he 'always' uses fastcgi for customer sites. I'm just not 100% certain it was Killes...

killes@www.drop.org’s picture

Wasn't me, I try to avoid non-Drupal parts of jobs, ie I don't usually set up servers. However, as of yesterday I have switched my development machine to lighttpd. The fastcgi that was mentioned was done on Apache by a friend of mine: http://hilluzination.de/php-fastcgi/
--
Drupal services
My Drupal services

Lausch’s picture

hi, maybe interesting as testing environment for you.

i created my first version of WLMPD

Windows
Lighttpd
MySQL
PHP
Drupal

http://www.bundle.at/node/1

regards

Lausch

Roberto Gerola’s picture

Hi, I'm trying to configure lighttpd to work with drupal under OpenBSD.
I'll use the the last stable version of lighttpd.
I'll be happy to contribute to your idea.

ryanc’s picture

I've installed Drupal 4.7.0-rc with PHP4, Mysql 4.1.10 and lighttpd 1.4.8 + FastCGI on Ubuntu Hoary. So far it seems to work fine. It took about 5 minutes.

Amazon’s picture

Hi, we are going to be releasing a recipe with the file caching patch to put a static web page server for serving cached pages and a dyanmic site in the back for handling PHP pages generated by Drupal.

I am hoping to distribute it this week.

Kieran Lal

j0rd’s picture

Have you looked at installing lighttpd infront of apache first? You use lighttpd to serve static files like CSS/GIF/JPG/JS and then proxy dynamic requests to apache like PHP. This is a simple solution which frees up apache to work on more important tasks. It can help reduce server load and speed up your pages. I would suggest you try this before attempting to port drupal to lighttpd.

It only takes an hour or so to get running. http://www.google.com/search?q=using+lighttpd+proxy+to+apache

Take a look. There are many ways to implement this.

killes@www.drop.org’s picture

There is no need to "port" Drupal. Drupal 4.7 seems to work fine with lighttpd. The only thing that needs changing are the rewrite rules if you want clean urls.
--
Drupal services
My Drupal services

Arkimedes’s picture

I started playing with Drupal a little while ago. I'm currently running the following

12" powerbook, 1.2 gb ram, 12gb development partition, Mac OS X 10.4.6, XCode 2.3

lighttpd 1.4.11 (running under supervise)
mod_rewrite
mod_fastcgi
and plenty others

PHP 5.1.4
safe mode off
APC version 3.0.10
libmemcache-1.4.0.rc2
running as fastcgi
and plenty others

MySQL 5.0.21-standard

memcached 2.0.4 (2 instanced running under supervise)
I'm writting patches to support this for caching

I don't have any problems with it. It should be quite scalable if you've got
the resources. With fastcgi you can run multiple instances and have the
requests distributed to them. If needed you could even run a proxy like
squid or lighttpd mod_proxy, or even some packet forwarding code to
distribute incoming requests to multiple web servers.

I compiled everything from source and made adjustments for OSX when I
could find the information online. I did have to grab some pre-requisite files
to get everything installed, but there is usually information on how to do that,
plus package managers often have everything for you. You may want to
look for binary packages (like the one for MySQL) and write up installs based on
those.

I'm just playing around, so I didn't really document what all I did during the install,
but usually I just followed the information in the INSTALL files for each package.

Arto’s picture

Lighty can definitely run circles around Apache, for most normal configurations. I've been using it (and PHP with FastCGI) on all my own sites for a year now, and have also migrated several clients' infrastructure over to it. I'm very happy with Lighttpd, and cringe whenever I have to work Apache (especially its baroque configuration syntax).

As for boosting Drupal's performance in general, I've almost completed a module that adds support for static page caching in Drupal. (And though superficially similar, this differs significantly from the proposed core patch available for 4.7 in that it cuts PHP out of the loop altogether.)

Arkimedes’s picture

I have noticed that lighttpd doesn't support .htaccess files.

Add or change this line:
url.access-deny = ( "~", ".inc", ".module" )
in your lighttpd.conf file to block downloading of the module source code.

There may be other changes needed for security, but I haven't looked into it enough.

kbahey’s picture

I plan to write a detailed configuration guide for Lighttpd / PHP FastCGI, but in the meantime, here are the rewrite rules:

This assumes you run Drupal in the Document Root of the server. If you run Drupal in a subdirectory, then change accordingly.

# Clean URL
server.modules += ( "mod_rewrite" )

url.rewrite-final = (
  # Clean URL test for 4.6
  "^/system/test/(.*)$" => "/index.php?q=system/test/$1",

  # Clean URL test for 4.7
  "^/system/test-clean-url/(.*)$" => "/index.php?q=system/test-clean-url/$1",

  # feed alias
  "/rss.xml$" => "/index.php?q=rss.xml",

  # Search
  "^/search/(.*)$" => "/index.php?q=search/$1",

  # More than one argument
  "^/([^.?]*)\?(.*)$" => "/index.php?q=$1&$2",

  # No arguments
  "^/([^.?]*)$" => "/index.php?q=$1",

  # Exempt .html suffixes. Uncomment this if you have your path aliases end in .html
  #"^/([^.?]*\.html)$" => "/index.php?q=$1"
)

These also help:

# Handler for 404
server.error-handler-404  = "/index.php"

url.access-deny = ( "~", ".inc", ".engine", ".install", ".module", ".sh", "sql", ".theme", ".tpl.php", ".xtmpl", "Entries", "Repository", "Root" )

The main annoyance with lighttpd is that there is no equivalent to .htaccess per user/directory, and hence you cannot easily make some sites private and others public and other niceties of .htaccess.
--
Drupal development and customization: 2bits.com
Personal: Baheyeldin.com

--
Drupal performance tuning and optimization, hosting, development, and consulting: 2bits.com, Inc. and Twitter at: @2bits
Personal blog: Ba

James Andres’s picture

Although not exactly the same situation, I suggest you look into the tutorials on configuring lighthttpd for RoR.

Similarities include:

  • All page requests going through a single index.php file (dispatch.fgci in rails's case)
  • Clean URL's achieved through url rewriting which all point to a single file (again index.php instead of dispatch).
  • Hiding various files by extension (.rb, .inc .. etc).

The challenges faced by both platforms are the same.

-- edit --
Before I get flamed yet again ....... I'll rephrase that: The challenges faced by both platforms are quite similar in some cases.

;-)
-- /edit --

James

Lead Developer on Project Opus
www.projectopus.com

magico’s picture

There is only the problem when we try do do searchs that contain a dot :(

----------
Fernando Silva
Openquest - Sistemas de Informação, Lda

edrex’s picture

actually this works.

lots of special casing tho. I could see it breaking given other unforeseen circumstances, but we shall see.

edrex’s picture

Thanks for bringing all this together in one place.

I wrapped everything above in a conditional like:
$HTTP["host"] =~ "^(www\.)?domain\.com" {
...
}

Seems to work. Testing visibility of sensitive files etc. Will reply with any problems.

edrex’s picture

lots of problems. I don't think this is going to work properly unless lighttpd gains an equivalent to

  RewriteCond %{REQUEST_FILENAME} !-f
  RewriteCond %{REQUEST_FILENAME} !-d

Will hunt around some more and if nothing surfaces sulk back to apache (I want my clean urls)

digits’s picture

You can do that, not with strict "translation", but with some tinkering and lighty logic...
I have personally no idea what I'm speaking of, I just read this on the lighttpd forum which is why I dare to do that statement ;-)

http://forum.lighttpd.net/topic/275#857

ajwwong’s picture

Khalid,

These rewrite rules are very helpful and they work great, thanks!

**My question**: Are there equivalent rules to force "www" or "no-www" prefix in the web address? In the old apache .htaccesss, I think you can put these commands in to force either www or no-www in the prefix.

OLD APACHE REWRITE COMMANDS TO FORCE / UN-FORCE WWW PREFIX

RewriteEngine on
RewriteCond  %{HTTP_HOST}  ^example\.com  [NC]
RewriteRule  ^/?(.*)  http://www\.example\.com/$1 [R=permanent,L]

or:

RewriteEngine on
RewriteCond  %{HTTP_HOST}  ^www\.example\.com  [NC]
RewriteRule  ^/?(.*)  http://example\.com/$1 [R=permanent,L]

Is there an equivalent rewrite command to achieve the same effect for lighttpd? Thanks!

Blessings,
Albert
www.ithou.org

***EDIT:*** I FOUND THE ANSWER HERE: http://textsnippets.com/tag/www

Good luck everyone! :-)

ajwwong’s picture

For those of you reading from home... here's the answer... to the question of forcing lighttpd to prefix www to the domain or not:

Reference: http://textsnippets.com/tag/www

Good luck, all!

edrex’s picture

just a quick pointer for anyone reading this thread:
http://www.morphir.com/Lighttpd-Install-and-configuration-for-Drupal-wit...

nezroy’s picture

Almost working. Unfortunately you still get back a Status 200 - OK header on 404 errors, which is a Google no-no.

kalashari’s picture

Does anybody have an idea for the rewrite rule in order for imagecache module to work on lighttpd?

One user came up with this:

url.rewrite-once = (
"^/files/imagecache/(.*)$" => "/index.php?q=files/imagecache/$1"
}

This "works", but it actually makes a file everytime it is called, so it doesn't actually cache anything, but rather it uses imagecache calls on every load.

j0rd’s picture

Lullabot has a great article relating to this subject. If you scroll down to the bottom of the comments and look for "j0rd" i've provided the info for getting imagecache/apache and lighttpd to play nice. I also provide some other speed improvements for lighttpd.

http://www.lullabot.com/articles/using-lighttpd-static-file-server

---
Drupal Freelancer, Drupal Themes and eCommerce with Ubercart

ajwwong’s picture

I'm not sure if this is a problem with lighttpd rewrite or audio module, but here's the deal...

Audio module works fine for uploading and normal "playing" using the Flash Player.

However, when you try to download the file, it **redirects** to the default front page.

For example:
[Try: http://www.ithou.org/audio/download/896/02_track_02.mp3 ]

[This is based off of the node here: http://www.ithou.org/node/896 ]

You can see that it simply redirects to the front page.

Interestingly, however...

http://www.ithou.org/?q=audio/download/896/02_track_02.mp3 works ... I'm not sure why this is. I believe that it is because of the lighttpd rewrite rules which I have been using... essentially kbahey's rewrite rules a couple of lines up... .

I will post here as I discover more. Thanks for your help.

Dunedan’s picture

I've nearly the same problem. Everything works fine with lighttpd + audio-module + clean-urls expecting the file download. Are there any news on this topic?

ajwwong’s picture

Hi Dunedan... Just an update... I haven't been able to figure this one out myself. I think it has to do with some of the ways that lighttpd is rewriting... if you figure this out, please post! My users have been asking me about this one for weeks. :-) Thanks!

Dunedan’s picture

Ok. I just talked to the lighttpd-guys, and they told me that these rules can't work, because they exclude urls, which includes a dot, from being rewritten (what's obviously necessary for css-files and so on). The solution would be to add a audio-specific rewrite rule like:
"^/audio/(.*)$" => "/index.php?q=audio/$1",

That works like a charm for me.
They also told me that it would be better to use the mod_magnet (http://trac.lighttpd.net/trac/wiki/Docs%3AModMagnet) module for rewriting. But atm I've no time to have a look at it.

ajwwong’s picture

Dunedan... Awesome! Thanks for the message and I am happy to confirm that this totally works like a charm. Happy holidays!

Blessings,
Albert