Support for Varnish 3.x

@ attempt to get/apply this patch,

cd ./sites/all/modules/varnish
patch -p1 < /tmp/varnish3-compatibility.patch
patching file varnish.admin.inc
Hunk #1 FAILED at 31.
1 out of 1 hunk FAILED -- saving rejects to file varnish.admin.inc.rej
patching file varnish.module
patching file varnish.test

and,

cat varnish.admin.inc.rej

--- varnish.admin.inc
+++ varnish.admin.inc
@@ -31,14 +31,17 @@
     '#default_value' => variable_get('varnish_flush_cron', 0),
     '#description' => t('Internally Drupal will attempt to flush its page cache every time cron.php runs. This can mean too-frequent cache flushes if you have cron running frequently. NOTE: this cache flush is global!'),
   );
-
-  $form['varnish_legacy'] = array(
-    '#type' => 'checkbox',
-    '#title' => t('Varnish Legacy Mode'),
-    '#default_value' => variable_get('varnish_legacy', 0),
-    '#description' => t('Check this if you use a version of varnish lower than 2.1'),
+  $form['varnish_version'] = array(
+    '#type' => 'select',
+    '#title' => t('Varnish version'),
+    '#default_value' => variable_get('varnish_version', 2.1),
+    '#description' => t('Select your varnish version.'),
+    '#options' => array(
+      '2' => '2.0.x',
+      '2.1' => '2.1.x',
+      '3' => '3.x',
+    ),
   );
-      
   $form['varnish_control_terminal'] = array(
     '#type' => 'textfield',
     '#title' => t('Varnish Control Terminal'),

that's with,
drush @test pm-list | grep -i varnish
Caching Varnish (varnish) Module Enabled 6.x-1.x-dev

'crossed in the mail' ...

drush @test up
...
Code updates will be made to the following projects: Varnish HTTP Accelerator Integration [varnish-6.x-1.x-dev]
...

patch -p1 < /data/webapps/varnish3-compatibility.patch
patching file varnish.admin.inc
patching file varnish.module
patching file varnish.test

can't replicate

#1179672: error on 'drush up', "WD varnish: Recieved status code 101 running purge req.http.host ..."

on demand, so i'll have to watch for that.

as for,

#1171946: pressflow6 + varnish3b -- varnish working, but status report page is empty

with patch in place, after just-in-case system reboot,

varnishd -V
varnishd (varnish-3.0.0-beta1 revision edd0d11)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2011 Varnish Software AS

@ https://.../admin/reports/varnish is still empty, and

varnishstat -1
client_conn 92 0.81 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 203 1.80 Client requests received
...

is populated, as expected.

on nav to https://.../admin/reports/varnish, @ watchdog,

Jun 6 15:07:49 test6 loc-pf6: https://test6.loc|1307398069|varnish| 127.0.0.1|https://test6.loc/admin/reports/varnish||1||Recieved status code 101 running stats. Full response text: Unknown request.#012Type 'help' for more info.#012

hm. this make me a bit suspicious,

http://drupal.org/node/941788

'tho i don't see a 'secret' file anywhere.

EDIT: i think this is not relevant :-/

i'm able to reproducibly cause the reported error; unclear whether it's the ONLY way to cause it.

drush cc
	Enter a number to choose which cache to clear.
	 [0]  :  Cancel
	 [1]  :  all
	 [2]  :  theme-registry
	 [3]  :  menu
	 [4]  :  css-js
	 [5]  :  block
	 [6]  :  module-list
	 [7]  :  theme-list
	 [8]  :  nodeaccess
	 [9]  :  views
	...

drush cc theme-registry
	'theme-registry' cache was cleared                   [success]
drush cc menu
	'menu' cache was cleared                             [success]
drush cc css-js
	'css-js' cache was cleared                           [success]
drush cc block
	'block' cache was cleared                            [success]
drush cc module-list
	'module-list' cache was cleared                      [success]
drush cc theme-list
	'theme-list' cache was cleared                       [success]
drush cc nodeaccess
	'nodeaccess' cache was cleared                       [success]
	Content permissions have been rebuilt.               [status]
drush cc views
	'views' cache was cleared                            [success]

but,

drush cc all
	WD varnish: Recieved status code 101 running purge req.http.host ~ www.testsite.loc && req.url ~ ^/.    [error]
	Full response text: Unknown request.
	Type 'help' for more info.
	'all' cache was cleared                              [success]

iiuc, 'drush cc all' _should_ be the equivalent of running all of the other commands. apparently, there's a difference.

> Did you go to the settings page and set the Varnish Version setting to 3.x ? ... If you see a drop-down of varnish versions ... you can ignore the rest of the instructions.

yes, @ ".../admin/settings/varnish" i've set the dropdown to,

VARNISH VERSION: 3.x

that page also reports,

STATUS: Varnish running. Observer more detailed statistics .

where the link to,

.../admin/reports/varnish

works to redirect on click, but the report page's content, beneath the usual drupal site's headers, etc. is simply

...  
<div id='page' class='clear-block limiter page-content'>
  
  <div id='content'>
          </div>
</div>
...

blank :-/

i can verify that varnish is working -- pages are cached as expected, etc. reports at varnishstat/hist/etc at command line all are OK.

and, all other modules' reports etc are just fine. no additional errors reported in logs of my site; main status page is 'all systems go'.

minor progress?

upgrading to latest,

 varnishd -V
	varnishd (varnish-3.0.0-beta2 revision 939d229)

as well as latest drush, now

  drush cc all
  'all' cache was cleared                                                                                   [success]

varnish status page in drupal admin still blank, though, and in watchdog log,

Jun 12 09:57:55 test6 loc-pf6: https://test6.loc|1307897875|varnish| 127.0.0.1|https://test6.loc/admin/reports/varnish||1||Recieved status code 101 running stats. Full response text: Unknown request.#012Type 'help' for more info.#012

at cmd line, all's ok,

varnishstat -1
client_conn                593         0.00 Client connections accepted
client_drop                  0         0.00 Connection dropped, no sess/wrk
client_req                 783         0.00 Client requests received
cache_hit                   53         0.00 Cache hits                                                               
cache_hitpass                0         0.00 Cache hits for pass               
...

> you are not getting the errors when clearing the cache anymore?

not when doing 'drush cc all' ... which _was_ causing the error repeatedly before.

i've cleared & restarted everything, even rebooted. atm, no more error @ cache clear. at least none that I can initiate/repeat ...

the watchdog error _is_ repeatable at every visit to the varnish status page in drupal admin.

> too bad I don't have a way of reproducing this =/

that would make things far too easy ...

Silly me, I tried to apply this patch to the D7 version. Out of curiosity, does this apply to D7 or are these changes already committed/need to be committed to the D7 version?

Thanks!

just fyi, not that i expected it to make any difference, i switched to a build of official Varnish 3.0-RELEASE.

still no errors on cache clear -- good; but, stats page is still empty.

Do we use the CLI interface? The stats command has been removed from the CLI interface (see Changes from 3.0 beta 1 to 3.0 beta 2).

i'd wrongly presumed that 'varnishstat' was being used -- that's what i'd compared output to @ cmd line above.

i suspect you may be have found a (the?) issue -- perhaps this, in 'varnish.admin.inc',

...
/**
 * Menu callback for varnish admin settings.
 */
function varnish_admin_reports_page() {
  // connect to varnish and do a full status report
  $status = _varnish_terminal_run(array('stats')); 
...

?

but then, fabsor says WORKSFORHIM ... ?

> Using the varnishstats message directly from the module would require that the web user can run the varnishstat command, and I don't think we can support this for all environment, at least not in the stable 1.x release.

fwiw, an out-of-the-box src build of varnish3,

ls -al /usr/local/bin/varnish*
-rwxr-xr-x 1 root root 25401 Jun 18 20:08 /usr/local/bin/varnishadm
-rwxr-xr-x 1 root root 20236 Jun 18 20:08 /usr/local/bin/varnishhist
-rwxr-xr-x 1 root root 30486 Jun 18 20:08 /usr/local/bin/varnishlog
-rwxr-xr-x 1 root root 29437 Jun 18 20:08 /usr/local/bin/varnishncsa
-rwxr-xr-x 1 root root 30706 Jun 18 20:08 /usr/local/bin/varnishreplay
-rwxr-xr-x 1 root root 20237 Jun 18 20:08 /usr/local/bin/varnishsizes
-rwxr-xr-x 1 root root 20381 Jun 18 20:08 /usr/local/bin/varnishstat
-rwxr-xr-x 1 root root 97812 Jun 18 20:08 /usr/local/bin/varnishtest
-rwxr-xr-x 1 root root 20408 Jun 18 20:08 /usr/local/bin/varnishtop

and,

sudo -u wwwrun varnishstat -1

where 'wwwrun' is the web user, works fine.

perhaps, rather than removing for all varnish3 cases, adding on the varnish 'admin'/stats page, an admin-option to ID the path to varnishstat, AND ensure that it has the perms to exec as the web user?

additionally, CLI access can already be secured by 'secret file' specification.

IMHO trying to preserve the stats page is probably not worth the effort*. Don't forget that varnish is often on a different machine, so you can't just run varnishstat from a web server. (*we run a dedicated varnish box)

i'll disagree.

to the extent possible, providing the tools within Drupal & its modules to monitor what's going on in/with Drupal is a good idea. not all Drupal 'mgrs' will be given shell access @ the server ...
'stats' was a valued part of this module for v6 -- why would it be LESS so for D7?

as for bits-n-pieces running on other boxes -- that's true of, and possible for, just about everything. THAT config is in the vast minority, and IMHO, hardly a guideline for deciding mainstream applicability/value (inasmuch anyone using Varnish is in the mainstream in the 1st place ...)

that's just D6-only for now, right?

Do we care to add an update_N function to preserve the value of 'varnish_legacy' in the new 'varnish_version' variable?

sub

sub.

sub.

FYI: been using #25 on a dev site for about a week now with no problems to report :)

Subscribing.

sub

Patch #25 seems to be working fine for me, I had to manually apply the varnish.admin.inc changes so I'm attaching my patch as well. I was using the full 6.x.1.0 release, so may the one in #25 was applied to the dev version?

Definitely works, I actually made the mistake of coding up virtually the same patch upon learning of the ban vs. purge API change. The CLI banner still says "Varnish Cache CLI 1.0" which is silly, otherwise we could detect the varnish version from the banner. Hooray for meaningless API changes!

BTW: I used #25 which applied to master, #33 did not apply.

sub.

sorry about that wow fail

Subscribing...

Here's a patch for the D7 branch. Straight port of the existing patch at #25.

@joshk, this patch is empty.... There was some error with your upload, I guess...

Whoops! Blank Patch!

Hmm... that patch also looks wrong. Maybe I shouldn't try and chat with people while making patches.

Yes, my drupal-git-fu is weak. issue branches and pull requests would be so much easier. Here's a corrected patch.

Fixed my patch. README issue is here:

http://drupal.org/node/1260888

#44 is missing a line in the status check. Corrected here.

Um and also #46 is missing a comma and mis-labeled a variable.

This one passes test. Next time I'll test before patching. :P

I am using drupal 6.x and varnish 3.x. So I have fixed this issue by manually applying those patch in 6.x varnish module.

...please update the project's page. It still says:

We are working on support for the 3.x branch, see #1179964: Support for Varnish 3.x

Why isn't this in the stable branch for 6 yet?

I'm sorry to reopen this, but as the latest stable release doesnt fully support Varnish 3, I decided this was the cleanest way to get your attention to:
http://drupal.org/node/1179672#comment-6754244. Varnish 3 works fine in dev, but not in the latest recommended release. And people wont notice it unless they check varnishlog (which few do).

To fix this I would suggest that you either:
- Create a new recommended release
OR
- State on the module page that Varnish 3 is not fully supported in beta1 (for D7 at least).

I'm using the most recent dev release and it would appear that it's not working. I'm not getting any errors and it says that Varnish is running, but when I watch the varnishlog all I see is:

0 CLI - Rd ping
0 CLI - Wr 200 PONG 1273698726 1.0

And when I check varnishtop -i rxurl I just get:

list length 0

Is there another issue to help with this? The documentation seems to be lacking.

The issue that @bibo mentioned was closed in August of this year #1179672: error on 'drush up', "WD varnish: Recieved status code 101 running purge req.http.host ..." but @deggertsen was reporting a problem still in September.

Not sure if this issue can be closed.

I still never got Varnish working so I gave up for the time being. Maybe I just need better documentation as I am a total Varnish noob, but I did get drupal saying that varnish was running it just wasn't actually caching anything as I believe it explained with the "list length 0".

Hi deggertsen, I just started with a module and found the following solution (for everyone who has 101 error): To make the module to work with a Varnish daemon you have to teach Daemon what to do. But that's simple. :)

https://www.varnish-cache.org/docs/3.0/tutorial/purging.html

@guruslot, simple for you maybe ;). Could you paste an example of what you actually did? That documentation doesn't make much sense to me either.

Well deggertsen, it wasn't easy at all as I had spent a few hours.

Ok. Works Varnish v. 3.04 with following configuration:

This is not a 'clean' code so any ideas on how to improve it are welcome.

@guruslot: could you remove the code from your comment and upload a text file instead? That helps keeping the issue queues clean and pages load faster in slow/mobile devices. Thanx in advance.

Config file works out of a box. Tested with Varnish v. 3.04.
Feel free to improve the code.

I think current versions of the D7 branch work with varnish 3 I'll close this again, please check for other open issues if you can't get it working. So we can fix specific bugs rather than have a general support ticket.

This isn't standard, don't know if 405 is valid or not
error 405 "Your ip banned via NSA"

@guruslot thanks for contributing your default.vcl.

I do think we need a wiki or something in:
https://groups.drupal.org/high-performance

Would be good to consolidate some of these issues.

mgifford, no problem.

405 designed for a bad guys only. Actually two different options are useful to look into in conjunction with varnish:

1. Fail2ban. Add firewall rule to block remote hosts/bots (i.e. 24hrs) according rules in varnish and drupal logs.
2. Memcache + C++ + Captcha to prevent scraping.

But I suggest to use these tools only when you experience related problems and the 2nd for dedicated varnish servers.