Lighttpd rewrite rules wanted

As I don't myself need Lighttpd support presently, I have no plans for this right now. It will have to wait for somebody who is interested in running Drupal + Boost on Lighttpd and volunteers to figure it out.

I use this in my lighttpd.conf:

$HTTP["host"] == "drupal.mydomain.tld" {
    server.document-root = "/var/www/drupal"
    url.rewrite-once = (
        "^/(misc|modules|files|themes)/(.*)$"=>"/$1/$2",	    
        "^(.*)$"=>"/index.php?q=$1"
    )
}

Hope it helps ;). You should enable mod_rewrite in the server.modules section. This should work with any version of Drupal, I tested on 5.
Cheers.

I forgot to add a rule for the home link, "^/$"=>"/index.php?q=node", so here it is:

$HTTP["host"] == "drupal.mydomain.tld" {
    server.document-root = "/var/www/drupal"
    url.rewrite-once = (
        "^/$"=>"index.php?q=node",	    
        "^/(misc|modules|files|themes)/(.*)$"=>"/$1/$2",
        "^(.*)$"=>"/index.php?q=$1"
    )
}

Cheers.

Hi, has anyone made these rules? I have setup lighttpd using this tutorial:

http://joshhuckabee.com/drupal-lighttpd-and-xcache-debian-2

I would like to add the boost capability for it.

I am also looking for lighttpd rules for the boost rewrite rules. I tend to think it is possible since lighttpd does have mod_rewrite and using the mod_magnet we might be able to get to some logic in a lua extension.

ok here is some lines of code I added to my existing lua rules for lighty to work with boost.

Lighty Env Assumption:
==================
you are running lighty enabled with fast-cgi and with mod_magnet module enabled and using lua to process conditionals and/or clean URLs.

Need to patch:
===========
There is one additional patch you will need (unless the current version already has it) to get a handle on lighty.env["request.method"]. Due to a bug this table was rendered empty in mod_magnet. I used the patch from this link : http://trac.lighttpd.net/trac/changeset/2137.

So after patching to add the steps for MAGNET_ENV_REQUEST_METHOD and MAGNET_ENV_REQUEST_PROTOCOL don't forget to recompile lighty again with the lua path.

The LUA Conditionals and approach
===========================
Some of this code especially the part that hands control over to drupal to render physical path at index.php, I already had from an earlier source and you are probably using it too. The additions I made are to handle the boost rules. So far its working nicely for me after some hair pulling .. and I have tested for as many conditions I can think of .. essentially accomplishing the functionality similar to the .htaccess rules that boost comes with. Pls provide comments / opinion to improve this further or maybe even another better approach.

A Word on performance
==================
I have researched a bit on the string functions and how costly they are as opposed to the REGEXP rules processing .. lua should be much more efficient as long as its not processing/reading in a big files for input. Also in the rules conditions matching I tried to follow the least travelled path to get from point A to B. So first, I evaluate for static files like jpg, css etc. and if it finds it get lighty to serve it up. Next I apply the first combined filter "anonymous user" ? AND "not a query string" ? AND "GET" ? and in that order since I want to try to avoid any unnecessary evaluations and thus using the most restrictive filter criteria first (assumption is dependent on site type). Since this is an AND'ed evaluation I need to find the likelihood of any of the 3 criteria that will MOST likely give me false at the earliest point in time and help me avoid the other two evals .. for me that is the logged in user, since query strings will be quite prevalent and GET paths will be MOST prevalent. The next is the PATHS type filter for "/user" ? OR "/cache" ? OR "/admin" and in that order again based on the path most likely to be true first.. and then onwards..

I may consider the use of LPEG as another lua library for the expression matching , but given the simplicity of the patterns we are after here, the string functions should be quite efficient. In any case unless you are looking at trying to serve up > 1000 req/sec I dont think that lua performance will hold us back . If someone using lighttpd have a way to benchmark I'd love to hear more on this.

So here is the code attached .. If you see ways to improve for performance/functionality I would most appreciate it. I compacted the code as much as I could at this initial pass but there may be more iterations needed to get to an even smaller footprint. Also I am hoping that any bugs will come to surface as you try out different conditions. I can guarantee some bugs will be site dependent that is related to caching rules and some erratic behavior with caching (you have to test and tweak .. one thing I learnt about boost ) and may require individual tweaking for your own site but it would help to know if there any universal type bugs (fundamental logical flaw or any other type of performance improvement).

Hi drupdrips,

thank you for this great work, it's good to know that someone cares about Lighty ;)

I tried to get this to work on Debian GNU/Linux "Etch" for several hours; to save others the tedious work, some notes: "Etch" comes by default with Lighty 1.4.13-4 that does not include the Patch mentioned above. However drupdrips' modified "drupal.lua" won't harm Lighty, but Drupal doesn't use the Boost cache either.

Lighty version 1.4.19-5 - which should include the crucial patch - is available in Etch Backports; this package is broken at the time of this writing and includes several critical bugs; it can neither be easily fixed (dpkg --configure -a) nor overriden by the usual options (--force etc.). As far as I can see at the moment, running Drupal with Boost and Lighty on Etch requires currently to build it from source - if there's no fix for Lighty 1.4.13-4.

Happy hacking! -asb

If you can't get the patch to be able to do eval for lighty.env["request.method"] == "GET" or using another way to check the request method within the lua code environment (if you are using lua for conditional evals and clean URLs) , the boost rules will not work with lighty. From my observation, the patch did not seem to be Linux distro specific, although I could be wrong.

One thing I would say, nothing worked for me initially and I had to debug the hell out of it to get it to work. What I mentioned above is the packaged solution that is working for me, but you may have some other tweaks you have to invest some time in, but I do not think it should be unworkable.

You probably already tried debugging, but if not, try putting in a bunch of print statements at the beginning (under local prefix = '' line) to see if the variables values are as you expect them to be and at the end try to print the Final path also. Something like this is what I was using but also print statements in between wherever there is a transition or an important assignment taking place.

print (" ")
print ("********************************************")
print ("Initial Variable Values are: ")
print ("env.uri.query is : " .. (lighty.env["uri.query"] or "nil"))
print ("env.request.uri is : " .. (lighty.env["request.uri"] or "nil"))
print ("env.request.method is : " .. (lighty.env["request.method"] or "nil"))
print ("request.cookie is : " .. (lighty.request["Cookie"] or "nil"))
print ("request.Host is : " .. (lighty.request["Host"] or "nil"))
print ("uri path is " .. lighty.env["uri.path"] .. " -- physical path is " .. lighty.env["physical.path"])
print ("Referer is " .. (lighty.request["Referer"] or "nil"))
print ("Cache-Control is " .. (lighty.header["Cache-Control"] or "nil"))
print ("Expires is " .. (lighty.request["Expires"] or "nil"))

Without the mod_magnet patch the value for lighty.env["request.method"] will be empty and then boost rules will not work. So one way or the other you have to be able to patch it or use a version that already includes it. (I am using 1.4.19) and I still had to patch it. Not sure if the latest 1.4.20 has it or not.

BTW pls don't modify the Issue Title. I changed it back.

I'm using 1.4.20, drupal.lua works fine for me,
but when I add "boost_lighttpd_luaRules_ver1.txt" inside drupal.lua I've got "500 - Internal Server Error"
is it seem that I need to patch it?

To make it clear to me,.. how to use the patch?
is it used to patch the lighttpd tar ball source? so I should re compile and re install the lighttpd?

thanks in advanced

Note that if you want the power of Boost and you are willing to still run Apache for your dynamic pages, you can set up a ProxyPass that directs all requests for pages in the cache directory to Lighttpd. Just put something like the following in the VirtualHost record:

ProxyRequests Off
ProxyPreserveHost On
ProxyPass /files http://0.0.0.0:81/files
ProxyPass /cache http://0.0.0.0:81/cache #for Boost

Check your memory limit in php.ini file. Is it at least 16mb ? The default 8mb is too small especially if you are enabling CCK and Views etc. Not having enough memory will be a cause for the 500 error.

you can try putting the print statements in the lua file and check for the debug output from lighttpd.error.log file to see what values you are getting.

If you do patch the lighttpd to include the patch for mod_magnet so you can get a handle on request_method, then yes you have to recompile lighty. I just ran sudu, make, make install after the patch is applied. Also make sure the config includes path to lua .. You can do the following :

./configure --with-lua PKG_CONFIG_PATH=/path/to/lua/
sudu
make
make install

Hi, does anyone confirm that http://drupal.org/node/150909#comment-695427 works for D6 version? Thanks.

If someone wants to write the rules for lighttpd, here's the bare minimum apache rules. I've been able to get everything down to this one rule, if your using the 6.x-1.x-beta1 or above.

  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html -f
  RewriteRule .* cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html [L]

If you want gzip support here are the rules for that

  #gzip
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{HTTP:Accept-encoding} gzip
  RewriteCond %{DOCUMENT_ROOT}/cache/gz/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html.gz -f
  RewriteRule .* cache/gz/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html.gz [L]

  #normal
  RewriteCond %{REQUEST_METHOD} ^GET$
  RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
  RewriteCond %{DOCUMENT_ROOT}/cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html -f
  RewriteRule .* cache/%{SERVER_NAME}%{REQUEST_URI}_%{QUERY_STRING}.html [L]

Looking at #6 and it's not supper clear on how to modify it to work with the new rules.

Just adding the lines suggested in #10 doesn't work (using 127.0.0.1:81, where lighttpd is listing my home files)
It gives me this error

Forbidden
You don't have permission to access /cache/localhost/0/family/white-eyes-zosteropidae.html on this server.
Apache/2.2.8 (Ubuntu) PHP/5.2.9-0.dotdeb.2 with Suhosin-Patch Server at localhost Port 80

but the URL at that page is still the correct one: http://localhost/family/white-eyes-zosteropidae
Removing these 4 lines and reloading results in apache serving the boosted html page.

It looks like I need to change the .htaccess file, not just the httpd.conf.

@ferrangil
your error is related to permissions. Try this patch #500944: internal umask overriding system defined umask & set your file permissions to something like 0644.

@ferrangil
Looks like your using 5.x so the above patch will not work in your case. Sorry about that.

Yes, I'm on 5.x. I'll try to look at the patch and adapt it to 5.x if possible. Thanks

After a file is written @chmod($filename, 0664); is all you need to do.

This would be nice to add to the release archive as something like boosted/lighttpd/boosted1.txt with a little documentation.

@andrewsuth
What's the standard way to do it?

@mikeytown2: Well, I actually didn't read the thread closely enough, I thought they'd already come up with a working and tested script for lighttpd.

I guess the stard procedure would be to make sure the community is happy with the code then add it to the CVS for future releases.

The solution in http://drupal.org/node/150909#comment-997460 is for those wanting to run webserver purely on lighttpd which is the original feature request of this thread. Running lighttpd for static files while apache serves up dynamic content is better to an extent, but if you are wanting to avoid apache altogether and go with lighttpd with fastcgi for smaller memory footprint and faster performance, you will still need to configure the rules that work with lighttpd.

BTW , the complaint by some of lighty having memory leaks has been patched in later versions. Also some people who complain of memory leaks do not understand how the php FCGI children work. When the php-FCGI initially spawns the children, not all all of them get touched or used by the php invoked by drupal, at first, maybe one or two, but over time and especially with heavy concurrent traffic, all of them will get touched at one point creating the appearance of memory leak as each children occupy the resident memory of an amount which is dependent on the number and type of drupal modules invoked for the authenticated user. (Some of this will be shared memory and some of it will be private). So that is matter of having correct expectations of your memory upper limit given the configs of the # of children * resident amount of memory that will be assumed for each thread over time .. .... if that is very close to your real RAM reduce # of children, or add more RAM or try reducing other components that might be eating server memory or reduce the # of drupal contrib modules (possibly with custom written module that does exactly what you want done). Make sure you are using an opcode cache like Xcache or APC. I have not seen lighty leak in my configuration even under heavy load and I know that because I do not see it crossing a threshold that is within my expectation (set right to begin with).

I have digressed in this thread into performance areas.. but what else is boost other than a performance solution.. when it comes to performance it is better, I think to take a holistic approach .. some things make a big difference, some things don't. .. for the ones that do make a big difference in my opinion are: boost / memcache(if you can do it), php op code cacher (I use xcache and happy with it), lighty for web server configured with php-fastcgi (unless you need features that are only available with apache web server), aggregation for JS and CSS and if you got too many modules in use, possibly look at opportunities to write a custom module that just applies your business/application rules to reduce the php memory footprint and to avoid unnecessary code execution.

I assume that this will also work for the 6.x branch of Boost too, right?

I will test it this week when I get a few hours. If others can also test the great work done by drupdrips, it would be really helpful. Then maybe we can try and get it submitted as part of the module archive, as I mentioned in #21.

@andrewsuth
5.x rules do not work for 6.x. The 6.x rules went from 3 down to 1, so it should make it simpler to do; if I could find detailed documentation.
Found this: http://redmine.lighttpd.net/projects/1/wiki/Docs:Configuration
Are there any more configuration pages out there? Finding documentation for lighttpd is almost impossible.

Subscribing. I'm using lighttpd 4.22 with drupal 6.13 and attempting to use the dev version of boost.

Could I get some comments on a proposal I have, as this could potentially effect Lighttpd integration.
#537186: Better prevention of URL collisions.

lighttpd rules attached.

this doesn't include everything as i don't need rss to be cached and stuff. This will work if you want to cache dynamic html content.

for some reason enabling gzip doesn't create file .gz extension. (Edit : it was created like gz/(host)). Fixed and updated the rewrite rule.

couldn't update the attachement on the previous one. please check this one which handles gz

@humble.techy Big Thanks!

Can I get someone to test his work on 6.x?

I tried Boost Lighttpd Rewrite lua file (via humble.techy-Aug. 10) on a local machine.
I renamed this boost.lua and moved it to /etc/lighttpd
In lighttpd.conf, I enabled mod_magnet and added the line:
magnet.attract-physical-path-to = ( "/etc/lighttpd/boost.lua" )

Using a different browser, I visited the local site. Boost is working in that the files are being created in cache/[site]

However, it does not appear that I am using the cache. The page source has no boost legend on the bottom.

Lighttpd=1.4.22, lua=5.1.4

Any suggestions?

the main problem with lighttpd mod_rewrite is that it doesn't check whether file exists or not

I've reviewed lighttpd bugtracker for such a feature request (file / directory existance check) and found that:

http://redmine.lighttpd.net/issues/985

after long debates (~1 year(!!)) the patch was written for 1.4.19. it was never (and it seems that it won't be) included to the official mod_rewrite.c. I don't know why. developers advice to use mod_magnet instead. пидоры блядорогие.
I use 1.4.24 now, but as far as I see there were no changes to mod_rewrite still 1.4.19, so it sutes for me too.

so I'll try it and if it will work fine I'll post the improved lighty rewrite rules for boost / drupal.

but!
there is still a problem with RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
it seems that some guys, who want lighty to have apache rewrite syntax, should gather and cr8 something like mod_apache_rewrite

is there any way to avoid serverside drupal cookie detection?

cookies let the server know if your logged in or not. without that check, it would send cached pages to logged in users.

You could try enabling Turn off clean url's for logged in users and disabling Cache pages that contain URL Variables... that might work, but it's an ugly solution. This should allow you to skip the cookie test and have boost work most of the time.

cookies let the server know if your logged in or not. without that check, it would send cached pages to logged in users.

As I type this I suspect that I am setting myself up for a "Duh." Nevertheless, exactly what is wrong with serving cached pages to authenticated users?

depends on your site, some sites will look and behave normal if your logged in. but in general if you have users who login and anonymous users, the html that is served will be different from the anonymous; if the html is exactly the same, whats the point of being logged in?

what is wrong with serving cached pages to authenticated users?

it's the solution only if u display user-specific data via ajax. unfortunaly standart drupal core doesn't behave this way. u should install tonn of ajax modules then. maybe it'll be not bad.

mikeytown2, so there is no way to check cookie via drupal script and tell server which page to take... it's a pity.

http://drupal.org/project/ajaxify_regions

Explain in more detail what you mean with the cookie check

the check for presence in cookie of drupal user id

thnx 4 the link

The magic of boost is it doesn't invoke php. If you want to start to do php tests, to figure what what page to serve, then you should switch to
http://drupal.org/project/fastpath_fscache
or
http://drupal.org/project/authcache

See http://groups.drupal.org/node/21897 for more options.

ajax regions is very nice project
even better then cookie check

did u think to integrate this module into boost and forget about serving different (dynamic/static) content to registered/anonymous users? what we really need to be dynamically served are admin pages. am I right?

The magic of boost is it doesn't invoke php. If you want to start to do php tests, to figure what what page to serve, then you should switch to

No, I don't think it is good idea to check which page to serve by php, or any other script. It is not good for perfomance, cuz we should do this check every time the page is served. Then we'll have logic:

[ Server -> Script -> Server ]

on the other hand we don't invoke any script and therefore need less cpu and less memory. I have a site with over 120k hits / day. It'll be critical 4 me.

This is also the reason I don't want to use mod_magnet.

@tntclaus
#289674: Supporting cached pages for authenticated users - It's in the game plan.

I've checked patched mod_rewrite.c. It works just fine and there is no need for now in tonn of rewrite rules for each local file. So if one feel himself to serve static cache to logged in users - he's free to do this without mod_magnet.

The second way not to use mod_magnet is to disable clean url for logged in users.

The third way is to use mod_magnet ;) , but without checking file existence. We should check whether mod_rewrite modified url or not then. I think this case has almost no difference with the way lua is checking files. It's ugly, and there is no performance benefits too.

To make boost work with lighty in normal way we need to check cookies inside mod_rewrite. I won't have much time to try solving this task. Does anyone want to try?

btw, lua scripts above ( http://drupal.org/node/150909#comment-1907374 ) do not work properly

upd: sry, it doesn't work with mod_rewrite ;) w/o mod_rewrite it works fine
upd2: except / page

Good news - it looks like they finally added a -f equivalent to the Lighttpd mod_rewrite: http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModRewrite. (This means a 2 year long feature request is closed - http://redmine.lighttpd.net/issues/show/985.)

I'll have to recompile Lighty and see if I can use this - here's the info on the new release: http://blog.lighttpd.net/articles/2009/10/17/pre-release-lighttpd-1-4-24.... I'll get back to you once I have my ruleset - I only need it for CSS & JS though so it might not be adequate for most people.

For people who still want to use mod_magnet, after much searching I also found this - http://redmine.lighttpd.net/projects/lighttpd/wiki/AbsoLUAtion. The external-static and Complex rewrites rulesets seem the most helpful to me.

Heads up, nginx+Boost beats varnish... would be interesting to test lighttpd as well. http://groups.drupal.org/node/26485#comment-101296

Glad to hear lighttpd is getting their act together in terms of rewrite support.

Good news - it looks like they finally added a -f equivalent to the Lighttpd mod_rewrite

It's good for drupal, but not for boost, cuz it requires more complex conditions (e.g. cookie check) for making decision about which rewrite rule to use.

The only way is to use nod_magnet and lua

The attached above drupal.lua is good enough to use with boost and drupal. After some slight nodifications I use it at my high traffic site. It differs from the one attached above is that it handles both files and directories.

After migrating from apache+mod_php to lighttpd + fscgi I've got about 50% less cpu usage. I have about 200-300 users online per second with peaks about 1k-4k at a time.

@tntclaus: which is the comment with the drupal.lua that you are recommending? And does that handle the js & css, because that's the part that I actually need.

I see your point that -f won't be enough for Boost as a whole, but since I only use Lighty to serve up JS, CSS & images, I think I can use the mod_rewrite.

My ineptitude is staggering and growing.

At this point, I confess that I am lost. There have been several threads on Lighty that originated many months ago. Is there now a Lighty-Boost recipe? Could someone with the erudition to do so, kindly summarize this somewhere?

@tntclaus: which is the comment with the drupal.lua that you are recommending? And does that handle the js & css, because that's the part that I actually need.

Check this one:
http://drupal.org/node/150909#comment-1907374

It doesn't handle directories though and has some minor bugs too ;) It doesn't handle css and js too.

I see your point that -f won't be enough for Boost as a whole, but since I only use Lighty to serve up JS, CSS & images, I think I can use the mod_rewrite.

If u use lighty only for images, css and js u doesn't need boost at all ;)

Drupal core caches js / css static files at "^/sites/((yoursite)|(default))/files/((js/js_.*\.js$)|(css/css_.*\.css$))".

You can enable it at admin/settings/performance page

@tntclaus:

I do use Boost. I use Boost to serve up the cached HTML of the PHP, as well as CSS and JS. It's just that, as I said earlier in the thread, I use the Apache ProxyPass directive to pass the /files directory to Lighty. But the Boost /cache directory is served by Apache.

I already have Drupal's native CSS & JS caching enabled. But I need to get rules that work with Lighty so it does the rewriting properly when the CSS/JS cache is cleared.

Thanks for the suggestion on the patch, but I think that I'll have to use a very different approach since my use case is so different.

I'm wondering if we can even settle on a standard set of Lighty rules since it seems like many of us who have commented here have different use cases.

So as a summery of this thread, using lighttpd to serve everything else besides dynamically generated text (html, json, xml, js, css) by the Apache ProxyPass directive is how to do it with 6.x

I remember reading that you can't use lighttpd to serve boosted files because you don't know if the file exists. I found this snippet of code; maybe this would be useful?

attr = lighty.stat(lighty.env['physical.path'])
if (not attr) then
  lighty.header["Location"] = "http://www.example.com"  .. lighty.env["request.uri"]
  return 302
end

via: http://www.thingy-ma-jig.co.uk/blog/14-11-2009/drupal-imagecache-perform...

-- little helper function
function file_exists(path)
  local attr = lighty.stat(path)
  if (attr) then
      return true
  else
      return false
  end
end

Boost 5.x is fully supported if following these rules http://drupal.org/node/150909#comment-997460

If someone wants to get 6.x fully supported that would be ideal; the rules are less complicated with 6.x.

It seems like an ideal situation to serve the logged in users from Apache and the anonymous users from Lighthttpd. Has anybody got the 'ideal situation' working?

Seems like this config would take Boost to the next level.

Call it an early Christmas Present... new rules based off of #6

I don't run lighttpd so these rules are not guaranteed to work; hopefully they do. Has support for html, xml, json, js, css; no gzip.

Looking at http://redmine.lighttpd.net/wiki/1/Docs:ModMagnet I can set the content type
lighty.header["Content-Type"] = "text/html"
Something I should do once someone can verify that these rules actually work.

Here the rules with the content type set

not working, I don't really know why, but I got only 1.4 req/s with that config while I was getting 3618.33 req/s using the attached one.

Hope I could help

Might have to do with the if block on the file_exists() part; should have done an elseif... Also looking at your code vs mine, did you change the path when you tested it? Do comments slow down lighttpd that much?

heres one that has full support for gzip; at least I hope it does...

@redox
try the lighttpd-old file, should work for your setup

Thanks for these rules - they look great! Sorry to be dense, but how do I implement them? I only need mod_magnet to do the rewrite for css & js, since I'm still using Apache to serve the actual files. What path should I attach the mod_magnet to so that it does redirects for css & js when necessary?

Ok, I was able to attach the rules using mod_magnet as follows:

$HTTP["url"] =~ "^/files/(css|js)" {
        magnet.attract-physical-path-to = ( "/home/[mysite]/public_html/rewrite.lua" )
}

(Note that if you are using Lighttpd w/fastcgi to serve all page requests you will probably want to use some form of $HTTP["host"] as the condition, since you'll need to attract *all* paths to mod_magnet.)

However, it looks like something is going wrong because all of the files in these directories are getting redirected to the wrong place. I'll come back with more comments as soon as I have some hard data for debugging.

A note for other would-be testers: you can use lighty.header["X-Lighty-Test"] = "Request Using Lua" to test whether mod_magnet is picking up the URLs & you can uncomment the print statements to get the info into your Lighttpd error log, which is typically at /var/log/lighttpd/error.log. (I just learned you can only have one server.errorlog defined (http://redmine.lighttpd.net/projects/1/wiki/Server.errorlogDetails), so make sure you weren't defining other errorlog paths in your vhost records, as I was. If you do that, your errorlog won't work at all.)

I'm looking at this Lua script and thinking it could use a bit of cleanup. A lot of the repetitiveness necessary due to the limited syntax in htaccess files seems to be unnecessarily duplicated in the Lua script. I'm going to give cleaning it up a try (and to integrate it in my Ultimate Lua script for Drupal since there seems to be some overlap in what they do), but though I've been using Lighttpd for a while, I'm new to Boost and still not entirely sure what sort of voodoo it uses. I'll give it a try though.

Okay, here's what I came up with. With all due respects to the previous folks who have posted code, I think my take on it will run a bit cleaner and be easier to maintain. There's still a bit too much code repetition for my taste… might see what I can do about that tomorrow. (Note that it can also take care of adding or removing the "www." prefix from domain names - read through the comments to find how to enable one or the other.)

Man, it's like every time I work in Lua, I forget how quirky it can be…

Some thoughts -

• What does boost-gzip-cookie-test.html do? What and how does it test? Can we safely forget about it on production sites? It would save a few lines of code. (Pardon me if it's obvious - again, I'm a newbie to Boost.)
• I haven't tested this with Drupal installations in a subdirectory yet. If someone has one handy, please give it a try (note that you'll need to modify the d_path value in this script).
• Unless I'm misreading something, it looks like it's possible that we'd send Gzip-compressed pages to browsers which don't support it if the browser sends "boost-gzip" in its cookie. Under what circumstances would that happen - why would we ever send the browser something it doesn't think it can handle? Is that something we can ignore on production sites too? (Again, please pardon the n00b.)

By the way, if you run Drupal on Lighty, please consider joining the Lighty group at g.d.o and sharing your questions and experiences.

boost-gzip-cookie-test.html is the way I do what I call "aggressive gzip"
http://actionable-stats.com/website-performance-activate-gzip
In order to deliver gzipped content independent of the header, this will test for gzip compression in a small iframe by sending it compressed content. This compressed content is javascript which creates a cookie with a note of gzip support. On the server side it checks for the cookie and then sends out gzipped content accordingly. To find out which percentage of your users sends mangled headers you’ have to analyze your logfile, usually about 10-15% are mangled, but it varies of course depending on your page. E.g. in Europe the problem is less common than in the middle east and the united states.

In short some firewalls/proxies mangle the gzip header; this gets around that. iframe is on non compressed version of the frontpage only. So the "aggressive gzip" setting adds some extra logic to the rewrite rules, but it allows for faster downloads, because it gets around bad proxies, ect...

I don't run lighttpd, so the rules I made where a "best effort"; thanks for taking the time to rewrite the rules :)

EDIT: Where does the redirect for the perm dir happen? (css/js) Also gz can be in a separate dir; thats why there are these 4 lines. It's overkill based off of 99% of the use cases; so you only need to 2 (normal & perm).

    -- Build filename based on URL for the boost cache
    local normal_boost_path_uribase = lighty.env["physical.doc-root"] .. prefix .. "/cache/normal/" .. lighty.request["Host"] .. "/" .. request_uri .. "_"
    local normal_gzip_boost_path_uribase = lighty.env["physical.doc-root"] .. prefix .. "/cache/normal/" .. lighty.request["Host"] .. "/" .. request_uri .. "_"
    local perm_boost_path_uribase = lighty.env["physical.doc-root"] .. prefix .. "/cache/perm/" .. lighty.request["Host"] .. "/" .. request_uri .. "_"
    local perm_gzip_boost_path_uribase = lighty.env["physical.doc-root"] .. prefix .. "/cache/perm/" .. lighty.request["Host"] .. "/" .. request_uri .. "_"

EDIT: Where does the redirect for the perm dir happen? (css/js) Also gz can be in a separate dir; thats why there are these 4 lines.

Hmm. Okay, guess I have some more tweaks to do.

EDIT: Reading back through this thread a bit, and came across this:

It seems like an ideal situation to serve the logged in users from Apache and the anonymous users from Lighthttpd. Has anybody got the 'ideal situation' working?

Too many people think Lighty and nginx are only good as static file servers. In actuality, they're both fully capable of running an entire Drupal installation top to bottom - no need to double up on web server daemons and tying them together with proxies. I'm adminning two FreeBSD VPSs (one for work, and one for my personal projects) each with LightTPD powering multi-site Drupal installations, and they run just fine with no bloated crufty Apache daemon in sight - fortunately, Drupal core is written to be as server-independent as sanely possible. Go Lighty full time and leave Apache in the dust!

I discovered that there was an error in the initial rules of #59: some of the if-statements were missing the "then". That was causing the rules to be unparsable by Lua. I don't have the corrected version, however, since I vastly stripped down the rules for my use case.

Here is the ruleset that I will use now in order to support the request of #54 to serve the logged in users from Apache and the rest from Lighttpd. Combining the ProxyPass directives from #10 with the Lua rules below to handle cached CSS & JS should enable that to work with no problems.

Garrett, I suspect you're right that it would be better simply to run Drupal off Lighty with fast-cgi enabled. But I don't have the resources to debug such a switch at the present time since my organization's site is receiving >2500 visits/day on average. Sometime in the next few months, I will probably give your rules a try though. Thanks!

EDIT: Sorry for the inconsistent tabbing on the rules. It looked fine in my editor b/c of the way I have it configured. An easy fix though. You should probably rename the file to something .lua when using it & then use the "attract-physical-path-to" setting in lighttpd.conf as I described in an earlier post.

Here's an update with some general clean-up, though it doesn't yet address the normal/perm issue you mentioned in #64 - not quite sure of the best way to go about that yet. But fortunately I'm in a lull between pressing projects, so I'm going to keep poking at this.

EDIT: Oops… Posting collision with EvanDonovan. Anyway, this script should work for you just fine even if you aren't using Lighty as your main server for CGI stuff.

Here's a new approach… an almost complete rewrite of the whole thing. This seems to work for the stuff in cache/perm.

mikey, since you don't have a Lighty server yourself, would it be useful if I set up a test subsite on my personal server and gave you the user 1 account? And possibly a shell account, if it would help?

Oops… yeah, the above will cause problems if the browser doesn't send a cookie. Here's the fix.

I now currently have Boost (plus #653928: Gzip encoding when not necessary) and this Lua script live on my latest personal web project, nocturban. And yes, that is spam, but it's also an example of Boost and Lighty in the wild. According to my Apache Bench before-and-after tests (ab -c 10 -n 500 http://nocturban.com/node/2), it only shaved 15 ms off the average download time (it obviously wasn't a busy site or taxed server to begin with), but that's a start.

If you where to turn the concurrency up, you would start to see a big difference; at least thats whats supposed to happen due to mysql getting hammered. Since it only shaved 15ms off, either A: your running an efficient php accelerator or B: the rules are very inefficient for some strange reason. Doing benchmarks of different http servers on the same box is the only way to tell how efficient the rules are. As an example, on nginx the rules are more efficient then its built in cache, at least according to this benchmark.

Thanks for the shell offer, but unless there is an issue where there is a really strange bug, I tend to stay away from accessing other peoples computers; keeps both of us sane that way.

Continue to update the rules; ideally I would pull in these rules and generate them from php like I do for apache, but we should wait until they are tested more. Seems like the Nginx rules have been throughly tested so I might do those when I get some free time.

Well, I suppose it would help if there were some standard site to test against. nocturban isn't using CCK or Views and anonymous visitors will only see one block (logged in visitors will only see two) - so hardly an intensive site to begin with. And I was benchmarking against a single node page instead of the front page or something else which would load more than one node. Perhaps those interested in Drupal performance should define a standard site (maybe an installation profile that builds a site pre-populated with content and such) and standard method (ab command line parameters, etc) to use in before/after benchmarking of this sort - not just for Drush, but for other performance enhancement tools.

One of the reasons I wasn't testing the front page to begin with is that my script wasn't properly working with the front page - oops. Also, it was checking for requested images in cache/perm thanks to a logic error - double oops. And here's some various other fixes.

subscribe

The last version is working great,
Thanks Garrett, Mikeytown2 and other guys

Hmm. I did some experimenting with the site http://aworldofpossibilities.org/ (warning: politically charged site - don't blame me, I'm just the web guy who doesn't necessarily agree or disagree with their message, etc, etc etc), and found I would actually get better results with Boost off and Aggressive core caching enabled (~430ms mean) than with Boost on (~485ms). This is using Lighty with my script with the Boost rewriting enabled, and with the most recent dev package of Boost. Needless to say, that result is rather surprising. I'm not sure where the fault lies, but I wouldn't be surprised if it's at the level of my script.

EDIT: Hmm. If I have ab send a header which triggers Lighty and/or Boost to gzip (-H 'Accept-Encoding: gzip'), my sans-Boost numbers are about the same (~430ms), but with Boost on I'm get numbers closer to ~255ms. Interesting. I'll keep experimenting.

set concurrency when doing the tests.
ab -c 25 -n 1000 -H 'Accept-Encoding: gzip'
If using an opcode cache with MySQL database on the same box, then due to Lua being a scripting language, the overhead of running your script could be greater then php being in an opcode cache. Seeing how having the files already gzipped gives such a dramatic speedup; it makes me think that lighttpd is not designed very well. Anyway with concurrency turned up it should start to tax MySQL more and hopefully make boost the winner. In any case, the gzip header is quite interesting.

Hi,

I haven't used Boost since I switched to Lighttpd, so I'm pretty new to the 6.x branch. That said: Shouldn't a cached page not have something like <!-- Page cached by Boost @ 2010-01-05 19:36:11, expires @ 2010-01-05 20:36:11 --> at the end of the page?

I'm not seeing this in my pages or at nocturban.com/node/2 (referenced in #69), but it's in the pages from aworldofpossibilities.org (referenced in #74) and in the Dogfish site. Currently my server is spiking with a load of 16 (dual core system, normal load with APC/Memcache and Drupal's built in caches was typically between 0.7 and 1.6), so something seems to go wrong. Ah, this issue is about boost 6.x-1.x-dev, I'm running 6.x-1.17 from 2009-Nov-28 (upgrading...)

Restarting Lighty (1.4.19-5 on Debian/Lenny with the Lua script from #71), now I'm getting a 500 - Internal Server Error. Restarting again, no 500er, but also no cacht tag at the page's bottom.

The Lua script is called like so:

$HTTP["host"] =~ "^(www\.)?mysite\.org" {
        server.document-root    = "/var/www/mysite"
        magnet.attract-physical-path-to = ( "/etc/lighttpd/boost-drupal.lua" )
        ...

Any ideas where I should start to look what's going wrong? (Deactivating Boost to get the server load back to normal).

Thanks & greetings, -asb

Shouldn't a cached page not have something like <!-- Page cached by Boost @ 2010-01-05 19:36:11, expires @ 2010-01-05 20:36:11 --> at the end of the page?

Cached pages should have something like that at the end of the page.

I disabled Boost at nocturban.com for reasons I can't recall at the moment.

Are you seeing cached files being created in your cache directory?

> Are you seeing cached files being created in your cache directory?

Yes, below /var/www/drupal/cache/normal/{mysite}.org. These files have the extension(s) _.html and _.html.gz, and include a footer like <!-- Page cached by Boost @ 2010-01-07 21:59:07, expires @ 2010-01-07 22:59:07 -->.

Greetings, -asb

So is your Drupal installation in a subdirectory of the web root? That might be confusing things…

> So is your Drupal installation in a subdirectory of the web root?

All sites are running from the web root. d_path = '/' in drupal.lua is set accordingly.

Greetings, -asb

undo spam bot

http://drupal.org/node/150909#comment-2353894
Is this (comment #71) the agreed upon way to do this in lighttpd?

Mikey, I've folded the stuff in #71 into my "[Ultimate Lua Script](http://groups.drupal.org/node/22787)" which also handles "www." addition/removal and clean URLs. It should be more up to date than what's in #71.

Can you confirm that the lue script from #71 might have issues if Drupal is not run from the web root? We have a client whose Drupal 6 site with Boost is installed under /drupal/... Boost creates the cache files correctly but lighty doesn't serve them.

This is using Boost 6.x-1.18... maybe we should try the -dev version? Lighty version is 1.4.19 (ssl) on Debian Lenny. Maybe I should build a newer verison?

theongrey, the latest version of the Lua script is over here - give that a try, and report any issues you have in this thread.

If lighty>1.4.24, you don't need mod_magnet anymore.

@veracium

How much of a performance boost does this bring to lighttpd 1.4.24+?

If so, what changes will be necessary to change in http://drupal.org/node/150909#comment-2817098 in order to take advantage of it?

I don't think anyone has any hard numbers as to what degree not using mod_magnet improves performance, but it stands to reason that it would to some degree. However, in my experiments, I've concluded that while the url.rewrite-if-not-file directive works for basic Drupal path rewriting, it's still limited in what it can do to the extent that the Lua/mod_magnet approach is still necessary for Boost support, which is what this issue is about.

Any chance to use Boost with Drupal 7 and Lighttpd?

Same rules should work

Any chance to use Boost with Drupal 7 and Lighttpd without lua script?

W32, according to my experiments, no, it can't be done using Lighty's configuration files alone. Things may have changed since I tested, but I doubt it - there's not a whole lot of changes going on with Lighty nowadays.

Why ? Modern releases of Lighttpd have rule "url.rewrite-if-not-file" which can be used for cache file existence. Outdated or invalidate state can be checked in cron task.

W32, as I mentioned in my write-up, it's because Boost's rules for Apache check for the existence of multiple files for a query, and to stop checking once a file is found. url.rewrite-if-not-file doesn't seem to want to work this way; it can check for the existence of a single file, but if that fails, it can't check for any others.

Incidentally, this is the third consecutive post by me in this thread in which I've linked to that write-up. I'd encourage you and others to actually read it before posting again; I'm not just linking to it out of my own vanity. And if you still think I'm wrong, then feel free to experiment for yourself and post the results; I'd actually be quite pleased if you found a way to get it to work.

No activity for 2 years, closing this 6.x-1.x issue.