Contact
Site: US UK AU |
Nexcess Blog

Installing Varnish for your Magento Store on CentOS 6

May 21, 2012 13 Comments RSS Feed

Installing Varnish for your Magento Store on CentOS 6

You’ve compressed and minified your JavaScript and CSS, optimized your database, and tweaked your caching config, and yet you still can’t seem to get that time-to-first-byte to drop below one second. Tweaking dynamic applications like Magento can be frustrating. Sometimes there’s so much behind-the-scenes processing that goes into generating that catalog page that even your developers don’t understand where the problem lies. This is where code profiling becomes crucial. But this post isn’t about profiling or tweaking algorithms. This is a post for those of you who have done all that. This is a post about making your dynamic site fly with a great piece of software called Varnish.

What is Varnish

Varnish calls itself a “web application accelerator”. It’s essentially a really fast caching HTTP reverse proxy that sits in front of your web or application server. It proxies web requests to the application and caches the result, essentially making your dynamic site as fast as one composed of static HTML, images and scripts.

Varnish performs really well. Our tests show up to 1000x improvements in transactions per second. It’s also very flexible with it’s domain-specific configuration language, VCL, that’s based on C and Perl. It can act as a load balancer for multiple back-end application servers and supports both IP and Name-based virtual hosts. And best of all, Varnish is free software, licensed under the FreeBSD license.

Varnish and Magento

Varnish works really well out-of-the-box for sites with minimal session-dependent content. However, with eCommerce sites like Magento, there needs to be tight integration with the application so that users can add products to their cart or wishlist and check out. Fortunately Nexcess has developed a free and open-source Magento-Varnish extension called Turpentine that has solved this problem.  Not only does this extension have great features like allowing custom exceptions for different routes and allowing users to purge the cache from the Magento Admin, it also comes with pre-configured VCL configs for Varnish 2.1 and 3.0, making implementation easy. There are other great Varnish integration extensions for Magento out there, but the rest of this post will focus on installing and configuring Nexcess Turpentine.

Caveats

Before we begin, I should mention a few caveats to consider before adding Varnish to your application stack. First, if you use any server-side analytic tools that sit behind Varnish, you need to understand that 90% of your traffic will not reach the application servers. Varnish will be serving the cached content for all non-exempt visitors (non-exempt meaning users who do not have an active cart or session). This means that if your traffic statistics depend on hits to the application or back-end web servers, then they will no longer reflect reality. Third-party tools that utilize JavaScript embedded in the page, like Google Analytics, should continue to work normally.

Second, the traffic that does pass to the backend servers will contain the IP address of the Varnish server instead of the client. This is because the client request is being proxied through Varnish. There are two ways to work around this. First, Varnish, like any proxy, can be configured to populate the X-Forwarded-For header with the “real” IP address of the client. If your back-end server is Apache, you can install mod_rpaf, which will rewrite the remote address with the IP contained in this header. The LiteSpeed web server can do this natively by toggling a configuration option, as I’m sure can other web servers.

With the above in mind, let’s move on to the installation process.

Step 1: Install Varnish

On CentOS 6 servers, varnish-2.1.5 is available in the EPEL repo. The Nexcess Turpentine extension can generate a VCL for Varnish versions 2.1 and 3.0. Varnish 3.0 has some great features like ESI support, but you need to rebuild the 3.0 RPMs from the Fedora package if you want to use it. For simplicity, we’ll stick with the 2.1 package so that a simple “yum install” will work:

$ yum install varnish

The varnish configuration files will be found under /etc/varnish, but we’ll skip that for now since we’ll be using the auto-generated configuration file that comes with the Turpentine extension.

Step 2: Install Nexcess Turpentine

You can use the Magento Admin to install this extension, which can be found on Magento Connect. I prefer to use the “mage” command line tool. Note: detailed installation docs can be found on our github site.

$ ./mage install connect20.magentocommerce.com/community Nexcessnet_Turpentine

The exension will be installed under “app/code/community/Nexcessnet/Turpentine/”. As mentioned, the configuration VCL is automatically generated by the Turpentine extension, so we don’t need to worry about editing it manually.

Step 3: Configure Varnish

Now we need to tell Varnish to sit in front of the web server and proxy requests to the web server. In this example, we’re using the Apache web server as the back-end, but the same concept applies to any web server you might be using. We’re also going to assume that all of this is being done on a single server, but keep in mind that this basic set-up can scale to any N-tier cluster.

First, open up the /etc/sysconfig/varnish file and and change the default port for Varnish to port 80.

VARNISH_LISTEN_PORT=80

Second, to take advantage of Varnish’s memory-backed storage, change the VARNISH_STORAGE variable to the following:

VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"

As you can see, we need to reconfigure the web server to listen on port 8080, since port 80 is the HTTP port and Varnish needs to listen there. We do this next.

Open up the httpd.conf (locations vary per distro, CentOS and RHEL is /etc/httpd/conf/httpd.conf). You want to find the line that says “Listen 80” and change it to “Listen 8080”.

#<br />
# Listen: Allows you to bind Apache to specific IP addresses and/or<br />
# ports, instead of the default. See also the<br />
# directive.<br />
#<br />
# Change this to Listen on specific IP addresses as shown below to<br />
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)<br />
#<br />
#Listen 12.34.56.78:80

Listen 8080

If you have VHOSTs or NameVirtualHosts, you need to update the ports in those directives as well.

NameVirtualHost 192.168.8.8:8080<br />
...<br />
&lt;VirtualHost 192.168.8.8:8080&gt;<br />
...<br />
etc.

Once you’ve updated your Apache configs, it’s always good to check your syntax.

$ httpd -t<br />
Syntax OK

Assuming all is good, you can now restart Apache and start up Varnish.

$ service httpd restart &amp;amp;&amp;amp; service varnish start

You can verify that the port changes are correct using netstat.

$ netstat -ntlp | grep -w 80<br />
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 29770/varnishd

Try loading your site in your browser. It should load without any problems. If the connection times out, then you’ve missed a step somewhere. Try connecting to your site on port 8080 to bypass varnish and make sure the backend web server is working (http://www.example.com:8080/). Also check to make sure you don’t have a firewall block in place.

Step 4: Enable VarnishCache in Magento

Log in to the Magento Admin and go to System->Configuration. Then under “Turpentine” click on “Varnish Options”.

VarnishCache Admin Settings

Change the “Backend Host” to your server’s IP address, and change the “Backend Port” to 80.  You will also need to provide the IP and port of the Varnish admin service. By default, this is listening on localhost, port 6082. Finally, you need to add the “Varnish authentication key” for accessing the admin port. This can be found on CentOS in the /etc/varnish/secret file. Just copy/paste the line in that file into the form.

To finish up, just click “Save Config”, then navigate to System -> Varnish Management. Click on “Apply Configuration” to save the auto-generated VCL to the default location.

That’s it! Load some pages on the front end a few times and watch Varnish in action. You should see considerable improvements in response time once Varnish has cached the page. Try adding some products to your cart to confirm that Varnish is not caching them. You can even try running a benchmark against a few URLs using Apache Bench, Siege, or magespeedtest.com.

Conclusion

Varnish is a terrific tool for improving the performance and scalability of dynamic websites like Magento. Its flexibility and options can’t all be covered in a short blog post, but with later versions, you can take advantage of more advanced features, like “hole punching” with Edge Side Includes or load balancing. As always, it’s important to make sure you’re not falling prey to what I call “Performance Tuning Sickness”. Your goal should be to have an accessible, dynamic, functional application first, and then try to optimize it for maximal performance. Varnish should not be used as a band-aid over bad code.

Until next time, cheers.

Posted in: Nexcess
  • sevenofmine

    This sounds very good, I would like to implement this on my Magento store which is hosted on Godaddy VPS. The only thing I’m unsure about is how will this affect other sites hosted on the same server? I have a few other WordPress, Drupal and Joomla based sites hosted on the same server as magento.

  • It is worth noting that if you run multiple sites and want those to be handled by apache as normal you can assign 1 IP to varnish and your other IPs to apache. Change your DNS to point to varnish for this site only. This will also run along side plesk or cpanel with out any conflict… all you need to do is to use apache Listen parameter (see above) and state all the IPs that you wish apache to server and leave out the 1 you have set aside for varnish. Also great for testing varnish on a production site. In the vlc file use ip:80 rather than 8080

  • bnugent

    As dx3webs mentioned, you can have varnish listen on a specific IP and assign the sites you don’t want to use varnish to different IPs. Alternately, you can set up the varnish VCL config to just proxy requests for your other sites without modifying or caching the responses.

  • Ryan Cheng

    Can varnish work on SIP-200? or does it require dedicated server?

  • bnugent

    Unfortunately, since varnish is a server-wide process that sits in front of the web server, it can only be implemented on a dedicated server or higher.

  • Ryan Cheng

    There’s really no way for us to justify upgrading to dedicated server for varnish. Does nexcess have any VPS plans that can utilize varnish for magento? Or maybe a varnish supported shared server SIP plan? I’d really hate to move our server as nexcess has been awesome.

  • GregSwaney

    We no longer offer VPS plans. In order to take advantage of Varnish, you would either need the SIP-400 plan, which includes all the bells and whistles for Magento, or one of our standard dedicated plans, which doesn’t include any of the bells and whistles by default. I did a quick look at your site and there are some other optimizations that you’re not currently taking advantage of. If you’d like to send me your Magento admin login info to gswaney@nexcess.net I’ll go through and audit your site to make sure all the optimizations available on the SIP-200 plan are in place. Thanks!

  • Brian

    Thanks so much, this helped a TON!

  • Nginx is a web-server, like Apache, but improves much better efficiency outcomes, and is much more ongoing when managing a lot of specifications, which outcomes in outstanding CPU utilization and can progressively down your whole web wide range web wide range web variety web host hosting server. Apache, for example, does not wide range silent so well without a lot of know-how.

  • Jon

    When I get to the Apply Configuration step I receive this error:

    Warning: fsockopen(): unable to connect to 192.168.72.35:6082 (Connection refused)

    Nexcessnet/Turpentine/Model/Varnish/Admin/Socket.php on line 315

    On Step 4 where it shows server list, do I keep the local address? I get the same error for both and port is allowed in FW. Does anyone know a solution to this?

  • When I used Varnish for the first time I didn’t thought that it will be that helpful, my Magento store was almost flying by using Varnish. Results were great, more than to what I expected from it.

  • vignesh

    If i am enabled the https 443 how i have to configure this module..?

  • Surjan

    Really nice and informative article .. Will implement this in my magento store ..