Lighttpd rewrite rules wanted

As I don't myself need Lighttpd support presently, I have no plans for this right now. It will have to wait for somebody who is interested in running Drupal + Boost on Lighttpd and volunteers to figure it out.

I use this in my lighttpd.conf:

Hope it helps ;). You should enable mod_rewrite in the server.modules section. This should work with any version of Drupal, I tested on 5.
Cheers.

I forgot to add a rule for the home link, "^/$"=>"/index.php?q=node", so here it is:

Cheers.

Hi, has anyone made these rules? I have setup lighttpd using this tutorial:

http://joshhuckabee.com/drupal-lighttpd-and-xcache-debian-2

I would like to add the boost capability for it.

I am also looking for lighttpd rules for the boost rewrite rules. I tend to think it is possible since lighttpd does have mod_rewrite and using the mod_magnet we might be able to get to some logic in a lua extension.

ok here is some lines of code I added to my existing lua rules for lighty to work with boost.

Lighty Env Assumption:
==================
you are running lighty enabled with fast-cgi and with mod_magnet module enabled and using lua to process conditionals and/or clean URLs.

Need to patch:
===========
There is one additional patch you will need (unless the current version already has it) to get a handle on lighty.env["request.method"]. Due to a bug this table was rendered empty in mod_magnet. I used the patch from this link : http://trac.lighttpd.net/trac/changeset/2137.

So after patching to add the steps for MAGNET_ENV_REQUEST_METHOD and MAGNET_ENV_REQUEST_PROTOCOL don't forget to recompile lighty again with the lua path.

The LUA Conditionals and approach
===========================
Some of this code especially the part that hands control over to drupal to render physical path at index.php, I already had from an earlier source and you are probably using it too. The additions I made are to handle the boost rules. So far its working nicely for me after some hair pulling .. and I have tested for as many conditions I can think of .. essentially accomplishing the functionality similar to the .htaccess rules that boost comes with. Pls provide comments / opinion to improve this further or maybe even another better approach.

A Word on performance
==================
I have researched a bit on the string functions and how costly they are as opposed to the REGEXP rules processing .. lua should be much more efficient as long as its not processing/reading in a big files for input. Also in the rules conditions matching I tried to follow the least travelled path to get from point A to B. So first, I evaluate for static files like jpg, css etc. and if it finds it get lighty to serve it up. Next I apply the first combined filter "anonymous user" ? AND "not a query string" ? AND "GET" ? and in that order since I want to try to avoid any unnecessary evaluations and thus using the most restrictive filter criteria first (assumption is dependent on site type). Since this is an AND'ed evaluation I need to find the likelihood of any of the 3 criteria that will MOST likely give me false at the earliest point in time and help me avoid the other two evals .. for me that is the logged in user, since query strings will be quite prevalent and GET paths will be MOST prevalent. The next is the PATHS type filter for "/user" ? OR "/cache" ? OR "/admin" and in that order again based on the path most likely to be true first.. and then onwards..

I may consider the use of LPEG as another lua library for the expression matching , but given the simplicity of the patterns we are after here, the string functions should be quite efficient. In any case unless you are looking at trying to serve up > 1000 req/sec I dont think that lua performance will hold us back . If someone using lighttpd have a way to benchmark I'd love to hear more on this.

So here is the code attached .. If you see ways to improve for performance/functionality I would most appreciate it. I compacted the code as much as I could at this initial pass but there may be more iterations needed to get to an even smaller footprint. Also I am hoping that any bugs will come to surface as you try out different conditions. I can guarantee some bugs will be site dependent that is related to caching rules and some erratic behavior with caching (you have to test and tweak .. one thing I learnt about boost ) and may require individual tweaking for your own site but it would help to know if there any universal type bugs (fundamental logical flaw or any other type of performance improvement).

Attachment	Size
boost_lighttpd_luaRules_ver1.txt	2.58 KB

Hi drupdrips,

thank you for this great work, it's good to know that someone cares about Lighty ;)

I tried to get this to work on Debian GNU/Linux "Etch" for several hours; to save others the tedious work, some notes: "Etch" comes by default with Lighty 1.4.13-4 that does not include the Patch mentioned above. However drupdrips' modified "drupal.lua" won't harm Lighty, but Drupal doesn't use the Boost cache either.

Lighty version 1.4.19-5 - which should include the crucial patch - is available in Etch Backports; this package is broken at the time of this writing and includes several critical bugs; it can neither be easily fixed (dpkg --configure -a) nor overriden by the usual options (--force etc.). As far as I can see at the moment, running Drupal with Boost and Lighty on Etch requires currently to build it from source - if there's no fix for Lighty 1.4.13-4.

Happy hacking! -asb

If you can't get the patch to be able to do eval for lighty.env["request.method"] == "GET" or using another way to check the request method within the lua code environment (if you are using lua for conditional evals and clean URLs) , the boost rules will not work with lighty. From my observation, the patch did not seem to be Linux distro specific, although I could be wrong.

One thing I would say, nothing worked for me initially and I had to debug the hell out of it to get it to work. What I mentioned above is the packaged solution that is working for me, but you may have some other tweaks you have to invest some time in, but I do not think it should be unworkable.

You probably already tried debugging, but if not, try putting in a bunch of print statements at the beginning (under local prefix = '' line) to see if the variables values are as you expect them to be and at the end try to print the Final path also. Something like this is what I was using but also print statements in between wherever there is a transition or an important assignment taking place.

print (" ")
print ("********************************************")
print ("Initial Variable Values are: ")
print ("env.uri.query is : " .. (lighty.env["uri.query"] or "nil"))
print ("env.request.uri is : " .. (lighty.env["request.uri"] or "nil"))
print ("env.request.method is : " .. (lighty.env["request.method"] or "nil"))
print ("request.cookie is : " .. (lighty.request["Cookie"] or "nil"))
print ("request.Host is : " .. (lighty.request["Host"] or "nil"))
print ("uri path is " .. lighty.env["uri.path"] .. " -- physical path is " .. lighty.env["physical.path"])
print ("Referer is " .. (lighty.request["Referer"] or "nil"))
print ("Cache-Control is " .. (lighty.header["Cache-Control"] or "nil"))
print ("Expires is " .. (lighty.request["Expires"] or "nil"))

Without the mod_magnet patch the value for lighty.env["request.method"] will be empty and then boost rules will not work. So one way or the other you have to be able to patch it or use a version that already includes it. (I am using 1.4.19) and I still had to patch it. Not sure if the latest 1.4.20 has it or not.

BTW pls don't modify the Issue Title. I changed it back.

I'm using 1.4.20, drupal.lua works fine for me,
but when I add "boost_lighttpd_luaRules_ver1.txt" inside drupal.lua I've got "500 - Internal Server Error"
is it seem that I need to patch it?

To make it clear to me,.. how to use the patch?
is it used to patch the lighttpd tar ball source? so I should re compile and re install the lighttpd?

thanks in advanced

Note that if you want the power of Boost and you are willing to still run Apache for your dynamic pages, you can set up a ProxyPass that directs all requests for pages in the cache directory to Lighttpd. Just put something like the following in the VirtualHost record:

Check your memory limit in php.ini file. Is it at least 16mb ? The default 8mb is too small especially if you are enabling CCK and Views etc. Not having enough memory will be a cause for the 500 error.

you can try putting the print statements in the lua file and check for the debug output from lighttpd.error.log file to see what values you are getting.

If you do patch the lighttpd to include the patch for mod_magnet so you can get a handle on request_method, then yes you have to recompile lighty. I just ran sudu, make, make install after the patch is applied. Also make sure the config includes path to lua .. You can do the following :

./configure --with-lua PKG_CONFIG_PATH=/path/to/lua/
sudu
make
make install

Hi, does anyone confirm that http://drupal.org/node/150909#comment-695427 works for D6 version? Thanks.

If someone wants to write the rules for lighttpd, here's the bare minimum apache rules. I've been able to get everything down to this one rule, if your using the 6.x-1.x-beta1 or above.

If you want gzip support here are the rules for that

Looking at #6 and it's not supper clear on how to modify it to work with the new rules.

Just adding the lines suggested in #10 doesn't work (using 127.0.0.1:81, where lighttpd is listing my home files)
It gives me this error

but the URL at that page is still the correct one: http://localhost/family/white-eyes-zosteropidae
Removing these 4 lines and reloading results in apache serving the boosted html page.

It looks like I need to change the .htaccess file, not just the httpd.conf.

@ferrangil
your error is related to permissions. Try this patch #500944: internal umask overriding system defined umask & set your file permissions to something like 0644.

@ferrangil
Looks like your using 5.x so the above patch will not work in your case. Sorry about that.

Yes, I'm on 5.x. I'll try to look at the patch and adapt it to 5.x if possible. Thanks

After a file is written @chmod($filename, 0664); is all you need to do.

This would be nice to add to the release archive as something like boosted/lighttpd/boosted1.txt with a little documentation.

@andrewsuth
What's the standard way to do it?

@mikeytown2: Well, I actually didn't read the thread closely enough, I thought they'd already come up with a working and tested script for lighttpd.

I guess the stard procedure would be to make sure the community is happy with the code then add it to the CVS for future releases.

The solution in http://drupal.org/node/150909#comment-997460 is for those wanting to run webserver purely on lighttpd which is the original feature request of this thread. Running lighttpd for static files while apache serves up dynamic content is better to an extent, but if you are wanting to avoid apache altogether and go with lighttpd with fastcgi for smaller memory footprint and faster performance, you will still need to configure the rules that work with lighttpd.

BTW , the complaint by some of lighty having memory leaks has been patched in later versions. Also some people who complain of memory leaks do not understand how the php FCGI children work. When the php-FCGI initially spawns the children, not all all of them get touched or used by the php invoked by drupal, at first, maybe one or two, but over time and especially with heavy concurrent traffic, all of them will get touched at one point creating the appearance of memory leak as each children occupy the resident memory of an amount which is dependent on the number and type of drupal modules invoked for the authenticated user. (Some of this will be shared memory and some of it will be private). So that is matter of having correct expectations of your memory upper limit given the configs of the # of children * resident amount of memory that will be assumed for each thread over time .. .... if that is very close to your real RAM reduce # of children, or add more RAM or try reducing other components that might be eating server memory or reduce the # of drupal contrib modules (possibly with custom written module that does exactly what you want done). Make sure you are using an opcode cache like Xcache or APC. I have not seen lighty leak in my configuration even under heavy load and I know that because I do not see it crossing a threshold that is within my expectation (set right to begin with).

I have digressed in this thread into performance areas.. but what else is boost other than a performance solution.. when it comes to performance it is better, I think to take a holistic approach .. some things make a big difference, some things don't. .. for the ones that do make a big difference in my opinion are: boost / memcache(if you can do it), php op code cacher (I use xcache and happy with it), lighty for web server configured with php-fastcgi (unless you need features that are only available with apache web server), aggregation for JS and CSS and if you got too many modules in use, possibly look at opportunities to write a custom module that just applies your business/application rules to reduce the php memory footprint and to avoid unnecessary code execution.

I assume that this will also work for the 6.x branch of Boost too, right?

I will test it this week when I get a few hours. If others can also test the great work done by drupdrips, it would be really helpful. Then maybe we can try and get it submitted as part of the module archive, as I mentioned in #21.

@andrewsuth
5.x rules do not work for 6.x. The 6.x rules went from 3 down to 1, so it should make it simpler to do; if I could find detailed documentation.
Found this: http://redmine.lighttpd.net/projects/1/wiki/Docs:Configuration
Are there any more configuration pages out there? Finding documentation for lighttpd is almost impossible.

Subscribing. I'm using lighttpd 4.22 with drupal 6.13 and attempting to use the dev version of boost.

Could I get some comments on a proposal I have, as this could potentially effect Lighttpd integration.
#537186: Better prevention of URL collisions.

lighttpd rules attached.

this doesn't include everything as i don't need rss to be cached and stuff. This will work if you want to cache dynamic html content.

for some reason enabling gzip doesn't create file .gz extension. (Edit : it was created like gz/(host)). Fixed and updated the rewrite rule.

Attachment	Size
Boost Lighttpd Rewrite lua file	2.95 KB

couldn't update the attachement on the previous one. please check this one which handles gz

Attachment	Size
Boost Lighttpd Rewrite lua file	3.46 KB

@humble.techy Big Thanks!

Can I get someone to test his work on 6.x?

I tried Boost Lighttpd Rewrite lua file (via humble.techy-Aug. 10) on a local machine.
I renamed this boost.lua and moved it to /etc/lighttpd
In lighttpd.conf, I enabled mod_magnet and added the line:
magnet.attract-physical-path-to = ( "/etc/lighttpd/boost.lua" )

Using a different browser, I visited the local site. Boost is working in that the files are being created in cache/[site]

However, it does not appear that I am using the cache. The page source has no boost legend on the bottom.

Lighttpd=1.4.22, lua=5.1.4

Any suggestions?

the main problem with lighttpd mod_rewrite is that it doesn't check whether file exists or not

I've reviewed lighttpd bugtracker for such a feature request (file / directory existance check) and found that:

http://redmine.lighttpd.net/issues/985

after long debates (~1 year(!!)) the patch was written for 1.4.19. it was never (and it seems that it won't be) included to the official mod_rewrite.c. I don't know why. developers advice to use mod_magnet instead. пидоры блядорогие.
I use 1.4.24 now, but as far as I see there were no changes to mod_rewrite still 1.4.19, so it sutes for me too.

so I'll try it and if it will work fine I'll post the improved lighty rewrite rules for boost / drupal.

but!
there is still a problem with RewriteCond %{HTTP_COOKIE} !DRUPAL_UID
it seems that some guys, who want lighty to have apache rewrite syntax, should gather and cr8 something like mod_apache_rewrite

is there any way to avoid serverside drupal cookie detection?

cookies let the server know if your logged in or not. without that check, it would send cached pages to logged in users.

You could try enabling Turn off clean url's for logged in users and disabling Cache pages that contain URL Variables... that might work, but it's an ugly solution. This should allow you to skip the cookie test and have boost work most of the time.

cookies let the server know if your logged in or not. without that check, it would send cached pages to logged in users.

As I type this I suspect that I am setting myself up for a "Duh." Nevertheless, exactly what is wrong with serving cached pages to authenticated users?

depends on your site, some sites will look and behave normal if your logged in. but in general if you have users who login and anonymous users, the html that is served will be different from the anonymous; if the html is exactly the same, whats the point of being logged in?

what is wrong with serving cached pages to authenticated users?

it's the solution only if u display user-specific data via ajax. unfortunaly standart drupal core doesn't behave this way. u should install tonn of ajax modules then. maybe it'll be not bad.

mikeytown2, so there is no way to check cookie via drupal script and tell server which page to take... it's a pity.

http://drupal.org/project/ajaxify_regions

Explain in more detail what you mean with the cookie check

the check for presence in cookie of drupal user id

thnx 4 the link

The magic of boost is it doesn't invoke php. If you want to start to do php tests, to figure what what page to serve, then you should switch to
http://drupal.org/project/fastpath_fscache
or
http://drupal.org/project/authcache

See http://groups.drupal.org/node/21897 for more options.

ajax regions is very nice project
even better then cookie check

did u think to integrate this module into boost and forget about serving different (dynamic/static) content to registered/anonymous users? what we really need to be dynamically served are admin pages. am I right?

The magic of boost is it doesn't invoke php. If you want to start to do php tests, to figure what what page to serve, then you should switch to

No, I don't think it is good idea to check which page to serve by php, or any other script. It is not good for perfomance, cuz we should do this check every time the page is served. Then we'll have logic:

[ Server -> Script -> Server ]

on the other hand we don't invoke any script and therefore need less cpu and less memory. I have a site with over 120k hits / day. It'll be critical 4 me.

This is also the reason I don't want to use mod_magnet.

@tntclaus
#289674: Supporting cached pages for authenticated users - It's in the game plan.

I've checked patched mod_rewrite.c. It works just fine and there is no need for now in tonn of rewrite rules for each local file. So if one feel himself to serve static cache to logged in users - he's free to do this without mod_magnet.

The second way not to use mod_magnet is to disable clean url for logged in users.

The third way is to use mod_magnet ;) , but without checking file existence. We should check whether mod_rewrite modified url or not then. I think this case has almost no difference with the way lua is checking files. It's ugly, and there is no performance benefits too.

To make boost work with lighty in normal way we need to check cookies inside mod_rewrite. I won't have much time to try solving this task. Does anyone want to try?

btw, lua scripts above ( http://drupal.org/node/150909#comment-1907374 ) do not work properly

upd: sry, it doesn't work with mod_rewrite ;) w/o mod_rewrite it works fine
upd2: except / page

Good news - it looks like they finally added a -f equivalent to the Lighttpd mod_rewrite: http://redmine.lighttpd.net/wiki/lighttpd/Docs:ModRewrite. (This means a 2 year long feature request is closed - http://redmine.lighttpd.net/issues/show/985.)

I'll have to recompile Lighty and see if I can use this - here's the info on the new release: http://blog.lighttpd.net/articles/2009/10/17/pre-release-lighttpd-1-4-24.... I'll get back to you once I have my ruleset - I only need it for CSS & JS though so it might not be adequate for most people.

For people who still want to use mod_magnet, after much searching I also found this - http://redmine.lighttpd.net/projects/lighttpd/wiki/AbsoLUAtion. The external-static and Complex rewrites rulesets seem the most helpful to me.

Heads up, nginx+Boost beats varnish... would be interesting to test lighttpd as well. http://groups.drupal.org/node/26485#comment-101296

Glad to hear lighttpd is getting their act together in terms of rewrite support.

Good news - it looks like they finally added a -f equivalent to the Lighttpd mod_rewrite

It's good for drupal, but not for boost, cuz it requires more complex conditions (e.g. cookie check) for making decision about which rewrite rule to use.

The only way is to use nod_magnet and lua

The attached above drupal.lua is good enough to use with boost and drupal. After some slight nodifications I use it at my high traffic site. It differs from the one attached above is that it handles both files and directories.

After migrating from apache+mod_php to lighttpd + fscgi I've got about 50% less cpu usage. I have about 200-300 users online per second with peaks about 1k-4k at a time.

@tntclaus: which is the comment with the drupal.lua that you are recommending? And does that handle the js & css, because that's the part that I actually need.

I see your point that -f won't be enough for Boost as a whole, but since I only use Lighty to serve up JS, CSS & images, I think I can use the mod_rewrite.

My ineptitude is staggering and growing.

At this point, I confess that I am lost. There have been several threads on Lighty that originated many months ago. Is there now a Lighty-Boost recipe? Could someone with the erudition to do so, kindly summarize this somewhere?

@tntclaus: which is the comment with the drupal.lua that you are recommending? And does that handle the js & css, because that's the part that I actually need.

Check this one:
http://drupal.org/node/150909#comment-1907374

It doesn't handle directories though and has some minor bugs too ;) It doesn't handle css and js too.

I see your point that -f won't be enough for Boost as a whole, but since I only use Lighty to serve up JS, CSS & images, I think I can use the mod_rewrite.

If u use lighty only for images, css and js u doesn't need boost at all ;)

Drupal core caches js / css static files at "^/sites/((yoursite)|(default))/files/((js/js_.*\.js$)|(css/css_.*\.css$))".

You can enable it at admin/settings/performance page

@tntclaus:

I do use Boost. I use Boost to serve up the cached HTML of the PHP, as well as CSS and JS. It's just that, as I said earlier in the thread, I use the Apache ProxyPass directive to pass the /files directory to Lighty. But the Boost /cache directory is served by Apache.

I already have Drupal's native CSS & JS caching enabled. But I need to get rules that work with Lighty so it does the rewriting properly when the CSS/JS cache is cleared.

Thanks for the suggestion on the patch, but I think that I'll have to use a very different approach since my use case is so different.

I'm wondering if we can even settle on a standard set of Lighty rules since it seems like many of us who have commented here have different use cases.

So as a summery of this thread, using lighttpd to serve everything else besides dynamically generated text (html, json, xml, js, css) by the Apache ProxyPass directive is how to do it with 6.x

I remember reading that you can't use lighttpd to serve boosted files because you don't know if the file exists. I found this snippet of code; maybe this would be useful?

via: http://www.thingy-ma-jig.co.uk/blog/14-11-2009/drupal-imagecache-perform...

Boost 5.x is fully supported if following these rules http://drupal.org/node/150909#comment-997460

If someone wants to get 6.x fully supported that would be ideal; the rules are less complicated with 6.x.

It seems like an ideal situation to serve the logged in users from Apache and the anonymous users from Lighthttpd. Has anybody got the 'ideal situation' working?

Seems like this config would take Boost to the next level.