Vanishing Varnish

I was recently saddled with a bunch of cryptic Varnish errors. For some reason, the varnish daemon just kept dying on me. I have previously used varnish in many places and I have never had to face such problems before. Varnish would actually start up and die about 5 seconds later, as shown in the log message below, which was very weird.


Jun 17 15:39:11 earth varnishd[2771]: child (2772) Started
Jun 17 15:39:16 earth varnishd[2771]: Pushing vcls failed: CLI communication error

The first thing I did was to start varnish on the command line using a -d -d parameter, which started varnish in debug mode. Everything worked normally in debug mode and nothing was amiss. So, it really vexed me for a while, not able to figure out why my cache was dying mysteriously.

Then, after lots of digging, turns out that on slower machines, varnish can kill the processes due to heavy load or timeout.

So, I had to add a startup parameter to the configuration file: cli_timeout.


DAEMON_OPTS="-a :8080 \
-T localhost:6082 \
-b localhost:80 \
-u varnish -g varnish \
-p cli_timeout=10 \
-s file,/var/lib/varnish/$INSTANCE/varnish_storage.bin,1G"

That fixed it! It seems that the default timeout was 3 seconds and it took about 7 seconds to start up Varnish and it would saturate the 100% of my processor. So, increasing the timeout to 10 seconds did the trick. I think that I would probably need to increase the number of VCPUs tied to this VM as well to cope with the increased load.

Advertisements

Published by

Shawn Tan

Chip Doctor, Chartered Engineer, Entrepreneur, Law Graduate.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s