Please note that this page contains advanced material. Both Drupal/Authcache as well as Varnish can be challenging to set up. The combination of these two pieces of software is a difficult undertaking. Therefore it is strongly recommended that people looking into taking this challenge first familiarize themselves with each component separately before trying to put them together. A good starting point is the Commerce Kickstart + Authcache tutorial and the Varnish Book.

Please do not use older versions of Safari had a bug in its cache implementation (see also #2147147: Stale content is served from the browser cache (Safari and/or Varnish)) but it is reported to be fixed in 9.0.3 and possibly other releases.

Also read / edit the Notes section at the bottom of this page. Hints for advanced configurations and non-standard setups are collected there.

Preparation

Before setting up Varnish, ensure that your site is working flawlessly with Authcache Builtin Storage Backend and Authcache Ajax. Visit Administration » Configuration » Development » Performance and make sure that Expiration of cached pages is set.

Set Expiration of cached pages on Administration » Configuration » Development » Performance

Also if Authcache Debug is installed, disable it for anonymous users. Instead use the developer tools of your browser and inspect the headers to verify that cached pages are served properly.

 public, max-age=600 is present

Install and test Varnish

Install and setup Varnish but do not customized the VCL yet. Just configure the backend section such that the drupal site is reachable through varnish.

We assume the following configuration throughout the rest of this page:

  • Web Server: 127.0.0.1:3000
  • Varnish HTTP: 127.0.0.1:6081
  • Varnish management interface: 127.0.0.1:6082

Then navigate to Administration » Configuration » Development » Varnish and set the Varnish version to 3.x. If everything is okay, a green check mark is displayed at the bottom of the page.

Now it is time to test whether the connection through varnish to the drupal site works. Open a browser and point it to your Varnish instance (e.g. hxxp://127.0.0.1:6081/). Varnish will not cache anything yet - it is still operating with the default policy.

Set up a minimal VCL for anonymous users

Varnish by default skips caching as soon as a cookie is on the request. Because Drupal always adds the has_js cookie indicating when a browser supports JavaScript, it is necessary to customize the VCL such that anonymous requests become cacheable. The easiest way to reach this is to simple remove any cookies in vcl_recv from the request unless the session-cookie is present. It is necessary to restore the has_js-cookie when a request is passed to the backend, otherwise Authcache Ajax will not work.

/**
 * Remove all cookies from a request unless there is a session open.
 */
sub vcl_recv {
    if (req.http.Cookie !~ "SESS") {
        unset req.http.Cookie;
    }
}

/**
 * If no cookies are on the request, add has_js before passing to the backend.
 */
sub vcl_pass {
    if (!req.http.Cookie) {
        set bereq.http.Cookie = "has_js=1";
    }
}

/**
 * If no cookies are on the request, add has_js before fetching a fresh copy
 * from the backend.
 */
sub vcl_miss {
    if (!req.http.Cookie) {
        set bereq.http.Cookie = "has_js=1";
    }
}

Verify the setup now by inspecting the response headers. When there are two figures in the X-Varnish header, the page was delivered from the Varnish Cache.

Inspect HTTP headers and make sure X-Varnish is present and shows two figures

Please make sure that you fully understand what is going on in this simple VCL snipped. If you don't, then first work through the VCL Basics chapter from the Varnish Book.

Level 2: Use Varnish Cache for all traffic

Before attempting to set up Varnish for authenticated users, ensure that your setup is caching anonymous requests flawlessly (see previous section).

Disable Authcache Builtin Storage Backend

Edit your settings.php and remove authcache.cache.inc and authcache_builtin.cache.inc from $conf['cache_backends']:

# On production systems make sure that reverse_proxy and reverse_proxy_adresses
# are configured properly, otherwise watchdog log entries will show the Varnish
# IP address instead of the one from the connected client.
# An exception to this rule is when using Nginx with the real_ip module, then
# setup $conf['authcache_varnish_passphrase'] here and also in vcl_miss().
# $conf['reverse_proxy'] = TRUE;
# $conf['reverse_proxy_addresses'] = array('127.0.0.1');
#
# For testing purposes it is easier to just configure a passphrase here and
# also in vcl_miss() of the VCL.
$conf['authcache_varnish_passphrase'] = 'correct horse battery staple';

$conf['cache_backends'][] = 'sites/all/modules/varnish/varnish.cache.inc';
$conf['cache_class_cache_page'] = 'VarnishCache';

# When not using the Authcache Builtin Storage Backend, the following lines should be removed:
# $conf['cache_backends'][] = 'sites/all/modules/authcache/authcache.cache.inc';
# $conf['cache_backends'][] = 'sites/all/modules/authcache/modules/authcache_builtin/authcache_builtin.cache.inc';

Then disable the Authcache Builtin Storage Backend module.

Enable Authcache Varnish Storage Backend

Enable Authcache Varnish Storage Backend module. Grab the example.vcl (Varnish 3) or the one for Varnish 4 from the module folder and use it as a template to setup Varnish.

The example.vcl is documented extensively. Please skim the file and especially read through the Example-sections. For an easy start, locate Example 4: Trigger key-retrieval for all users, including anonymous. in vcl_recv and uncomment this fragment. Also when using a passphrase locate Example 1: Use a passphrase to validate proxy requests. in vcl_miss and adapt it.

Restart Varnish and point a browser at it. Now verify that caching works for anonymous as well as authcache enabled authenticated users. Especially pay attention to the response headers on requests with session cookie.

Request with session cookie

Inspect HTTP headers and make sure X-Varnish is present and shows two figures even when a session cookie is present on the request

Troubleshooting

The changes outlined in this section are probably the most difficult ones. If things do not work, then take a look at the following things:

  • Administration » Reports » Status report: Make sure everything is green there.
  • Administration » Reports » Recent log messages

Especially look out for access denied entries to /authcache-varnish-get-key. Authcache Varnish Storage Backend will flat out refuse to operate when it fails to verify that a request came in via Varnish. This verification only works when $conf['reverse_proxy'] and $conf['reverse_proxy_addresses'] are set appropriately or alternatively a $conf['authcache_varnish_passphrase'] is used. See also the Notes section further down for hints on how to configure this on special setups.

Note that Authcache Debug can be configured to log some information instead of displaying it in the widget. This is especially helpful when dealing with Varnish related problems.

Nginx microcache

If you're using microcache in Nginx, you may need to disable it (comment out) as the microcache in Nginx can override the "Expires" header and prevent Varnish caching. You will see this issue as a response "Expires" header in the past, e.g 1970.

Level 3: Use ESI instead of Ajax for personalized content

Enable Authcache ESI

Enable the Authcache ESI module, then navigate to Administration » Configuration » System » Authcache » Personalization and ensure that the fragments use the ESI client.

Authcache Personalization settings, ensure that ESI is listed as client on personalized fragments

Then login with an authcache enabled user and ensure that the personalized fragments load via ESI. An easy way to test whether personalization works is to enable Authcache Block and add a custom block reporting the user name of the currently logged in user using the following PHP fragment:

Welcome

global $user;
echo format_username($user);

Then enable authcache with a per-user granularity on that block.

Please note: During testing it is common to switch between users with the same role combination (which is unlikely in production environments). Browsers may serve up stale content from the cache due to HTTP revalidation. Such responses are identifiable on their HTTP status (304 Not modified). Therefore it is recommended to disable the browser cache during this sort of testing.

Check with Authcache ESI Debug

The module Authcache ESI Debug provides a way to quickly verify whether the ESI setup is working. However please note that the status is only reported correctly for authcache enabled users (all of the roles of a user need to be authcache enabled).

Authcache ESI Debug

Notes

Nginx HTTP real ip module

When using the ngx_http_realip_module, the webserver validates whether the request came in via a trusted varnish instance. The client IP address is then replaced with the real one. As a result it is impossible for Authcache Varnish to check whether request came in via a trusted proxy. In this situation it is necessary to disable the validation by adding the following line to settings.php:

$conf['authcache_varnish_validate_reverse_proxy_address'] = FALSE;

In this case the $conf['reverse_proxy'] and $conf['reverse_proxy_addresses'] settings should be removed from settings.php.

If you decide to use this option you should make sure that Varnish is not bypassed (i.e. all users who access your site pass through Varnish).

The most common situation for Varnish being bypassed is if part of your site runs on SSL . In this case one possible solution is to enclose your validation statement in an if function which checks for SSL. In that case you would remove the following lines from your settings.php.

$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

And instead replace them with:

$is_https = isset($_SERVER['HTTPS']) && strtolower($_SERVER['HTTPS']) == 'on';
if (!$is_https) {
  $conf['authcache_varnish_validate_reverse_proxy_address'] = FALSE;
}

I'm not sure if I'm using real IP, how do I check?

You need to have both the module installed which you can check with:

nginx -V

And a piece of configuration in your nginx.conf file which will enable the real IP handling.

The configuration code will typically be in the http block and look something like this:(Following configuration taking from Perusio's Nginx config for Drupal.)

 ## Handling of IPs in proxied and load balancing situations.
    set_real_ip_from 127.0.0.1; # all addresses get a real IP.
    real_ip_header X-Forwarded-For; # the ip is forwarded from the load balancer/proxy

You can also look in your nginx access and error logs. If all your requests are internal (127.0.0.1), then real IP handling is not functioning and you don't need to alter settings.php.

See also #2289575: authcache-varnish-get-key denied for all requests

Comments

cthshabel’s picture

Wow. This is some really great material. I can't wait to integrate authcache into my LEMP stack with varnish proxy. I plan to read this documentation many times. Thank you so much.

dzy’s picture

i read this and installed varnish 3.x + authcache 2.0 + cache expire + Authcache ESI , no error show up.
checked Authcache ESI Debug, green!
after i edit a article i goto config/development/performance, clear page cache,
the anonymous user still see the old content,
if login he can see latest content , logout can not.
disable anonymous user authcache still not helping.

maybe i did it wrong? what are the key settings for clear cache? or it is a bug?

and, i got wrong ip address by using this
$conf['reverse_proxy'] = TRUE;
$conf['reverse_proxy_addresses'] = array('127.0.0.1');

or this:

$is_https = isset($_SERVER['HTTPS']) && strtolower($_SERVER['HTTPS']) == 'on';
if (!$is_https) {
$conf['authcache_varnish_validate_reverse_proxy_address'] = FALSE;
}

znerol’s picture

Please file a support request in the authcache issue queue.

candelas’s picture

You can find more actualized documentation on the module's examples, for Varnish 3.x and 4.x, inside the Authcache Varnish module.

//trying to answer one question for each one that i make.
//this way, drupal will be more friendly and strong

SocialNicheGuru’s picture

I added http authentication to my site and varnish stoped working. Well varnish will not cache in this situation.

http://SocialNicheGuru.com
Delivering inSITE(TM), we empower you to deliver the right product and the right message to the right NICHE at the right time across all product, marketing, and sales channels.

mpp’s picture

Has anyone tested authcache with Varnish 6?