Use Varnish for cache storage instead of file system. [#701826]

From my understanding, varnish needs custom headers sent to it, thus the patch for core; Boost already sends custom headers. I can rip out the varnish control code for cache deletion from the varnish project. With cache creation and deletion, the basic parts of boost can work with varnish.
http://drupal.org/project/varnish

I originally thought these 2 projects where too different but in reality they are quite similar ATM. This should take varnish to the next level allowing almost all of the advanced features of Boost, but using Varnish instead of files for storage. This will be useful up until someone figures out ESI; but that can be used with Varnish & Nginx; and ajax loading is the alt to ESI, so I see this as a separate module that controls this.
http://en.wikipedia.org/wiki/Edge_Side_Includes

Comments

Comment #1

mikeytown2 CreditAttribution: mikeytown2 commented 3 February 2010 at 21:54

After talking with the people in the varnish IRC channel; it appears that I can't use any of the advanced features of varnish like page hits and other things like that; without some hassles. Thus I'll use Varnish just like I use the file system, a simple storage medium.

Option B From IRC:
"actually, you can do stuff like GET /$url?statsonly and then in vcl_hit do an error with the appropriate stats and ditto in vcl_miss. No backend request required."

Comment #2

joshk CreditAttribution: joshk commented 4 February 2010 at 20:03

I'm all in favor of bringing these together. However, I have some questions:

What are the advanced features of Boost are that we'd be including?

Also, given that Boost uses a write-based logic for caching pages, and Varnish does not, I don't know how much of the existing Boost code is directly applicable. Varnish's workflow is a bit different.

However, if this means unifying modules, and/or Drupal 6 users can take advantage of Varnish without having to patch their Core, it's a big win.

Comment #3

mikeytown2 CreditAttribution: mikeytown2 commented 5 February 2010 at 22:34

Very little logic in boost revolves around the actual file it's self; as long as the purge command in varnish works, there isn't a lot of works that needs to be done in order to make this happen. The fact that there is a filename stored in the database as the primary key doesn't mean a whole lot; I have the URL in there as well. Integrating varnish will help to make the boost code more generic; allowing for memcache storage in the future (Nginx).

All I need is cache creation and cache deletion; boost handles the rest.

Comment #4

mikeytown2 CreditAttribution: mikeytown2 commented 23 March 2010 at 02:06

Comment #5

aruna.kulatunga CreditAttribution: aruna.kulatunga commented 27 June 2010 at 23:08

will be following this. Boost+Varnish will make Drupal steam....

Comment #6

adrianmak CreditAttribution: adrianmak commented 3 January 2011 at 03:24

Comment #7

Fabianx CreditAttribution: Fabianx commented 26 March 2011 at 08:24

It is actually quite easy to combine varnish with boost if boost crawler is used (the records are duplicated though on disk and mem):

acl purge {
        "localhost";
        "127.0.0.1";
        "<ip boost is using>";
}

sub vcl_recv {

[...]

  if (client.ip ~ purge) {
    purge("req.url == " req.url " && req.http.host == " req.http.host);
  }

}

purging all records that are asked for from local ... (plus removing no-cache headers from cache/.htaccess)

Of course a PURGE request could also be sent via normal cron expire ...

Best Wishes,

Fabian

Comment #8

Fabianx CreditAttribution: Fabianx commented 23 May 2011 at 01:56

A followup to this:

To achieve full boostability with varnish and also making efficient and clever use of the boost crawler will all be detailed in my proposed session for Drupal Con London:

http://london2011.drupal.org/conference/sessions/boosted-varnish-how-inc...

I'll also write up the results here and link to a blog post explaining it in detail once I've cleaned up all the configuration.

Let me just tell you this:

Boost and Varnish are an excellent combination.

- Boost for the durability
- And Varnish for the "hot" pages and assets

Best Wishes,

Fabian

Comment #9

mikeytown2 CreditAttribution: mikeytown2 commented 23 May 2011 at 19:55

@Fabianx
Cool presentation. It would be pretty sweet if this got picked for DrupalCon!

Comment #10

ckreutz CreditAttribution: ckreutz commented 31 July 2011 at 17:48

@Fabianx
yes, I would also be interested to see your results. I made some good first experiences combining varnish and boost.

Comment #11

karljohann CreditAttribution: karljohann commented 26 October 2011 at 13:32

This idea is very appealing to me, any progress been made?

Comment #12

mealto CreditAttribution: mealto commented 26 October 2011 at 23:02

Varnish made our Drupal pages load pretty quickly. Unfortunately, with Varnish enabled, our writers were constantly getting logged out. Looks like when they open a new tab or window to browse our own website, the original browser that they have used to write an article would kick them out, thus, not saving the article. Because of this, we had to remove Varnish, what a pity. On a side note, is there a fix for this?

Comment #13

Fabianx CreditAttribution: Fabianx commented 27 October 2011 at 00:12

@karljohann:

Yes, see here:

http://www.trellon.com/content/blog/boosted-varnish-high-performance-cac...

I now also have a minimal Drupal 7 version working. (which is not using Boost at all, but just a boosted_varnish helper module).

Comment #14

karljohann CreditAttribution: karljohann commented 27 October 2011 at 14:08

Ah yes, thank you Fabian, that is very useful. I however cannot set a time limit on the cache flush because we're running a news site that often needs to be updated every other second. I'm currently trying out the Expire module but there are a lot of Boost options I'd like to see there.