Magento Enterprise and Memcached

Magento Enterprise and Memcached

Feb 24

  • Created: Feb 24, 2012 12:15 PM

Magento Enterprise and Memcached

I recently worked with one of our Magento Enterprise Cluster clients who wanted to improve the performance of their site. This client was using our Content Delivery Network to offload the serving of static assets, and was following all of our recommended Magento Optimization best practices. They were also using Memcached as the fast-backend in their Magento configuration. Despite doing everything right in terms of optimization, we managed to shave considerable time off their time-to-first-byte by leveraging multiple instances of memcached and Magento’s terrific use of the Zend Framework for caching. Here’s how.

Two-Level Caching

Whenever you configure a shared-memory type of caching backend — e.g. memcached, xcache, or apc — Magento will enable the Zend Framework’s TwoLevel cache type. The basic idea behind the two level cache is that you have a fast, limited-sized cache for the initial lookup and a slower, larger-capacity cache for when the fast cache lookup misses. The benefit of this is that you get the speed of a memory-based caching backend plus the capacity of a disk- or database-based backend. That is, when the two-level cache works as it should. Fabrizio Branca has an excellent writeup about the problems with Magento / Zend TwoLevel caching, along with a patch that can be used to fix some of them. I have implemented this patch on a few sites now and have seen significant improvement in first-level cache hits.

Memcached and Sessions

The options for session storage on a single dedicated server are ‘File’, ‘Database’, ‘eAccelerator’ or ‘Memcache’. The default is ‘File’ and tends to work fine in this setup. However, once you move to a clustered system, the need arises to use a session storage that can be shared between the application nodes. Using ‘Database’ is almost always a bad idea because there is already enough DB traffic in Magento without compounding it with session traffic. ‘eAccelerator’ may not be available in every instance. That leaves ‘Memcache’, which is trivial to set up, uses TCP, and registers a session handler when the PECL extension is installed.

It is crucial to use a separate instance of memcached for session storage so that you don’t lose all of your sessions when you flush your Magento cache. You can configure the second instance to run on the same server, but on a different TCP port (e.g. 11212). As long as you don’t have unusually long session lifetimes (a bad idea), you can configure the cache size to be much smaller than that of your fast-backend in order to conserve memory. Once the second instance is up and running, configuring Magento to use it is just two lines in your local.xml file:

<session_save><![CDATA[memcache]]></session_save>
<session_save_path><![CDATA[tcp://10.0.0.1:11212?persistent=0&amp;weight=2&amp;timeout=10&amp;retry_interval=10]]></session_save_path>

Memcached and the Slow Backend

Remember how I said that when two-level caching is enabled, the second level is for the slower, larger capacity storage? Well, memory is cheap, my friend, and there’s nothing preventing you from setting up a third memcached instance and using that for the second level cache. The configuration is trivial:

<cache>
    <backend><![CDATA[memcached]]></backend>
    <slow_backend><![CDATA[Memcached]]></slow_backend>
    <auto_refresh_fast_cache>0</auto_refresh_fast_cache>
    <default_priority>10</default_priority>
    <memcached>
        <servers>
            <server>
                <host><![CDATA[10.0.0.1]]></host>
                <port><![CDATA[11211]]></port>
                <persistent><![CDATA[0]]></persistent>
            </server>
        </servers>
    </memcached>
    <slow_backend_options>
        <servers>
            <server>
                <host><![CDATA[10.0.0.1]]></host>
                <port><![CDATA[11213]]></port>
                <persistent><![CDATA[0]]></persistent>
            </server>
        </servers>
    </slow_backend_options>
</cache>

Here you can see that we set up a new instance of memcached to listen on port 11213 and set the ‘slow_backend’ to instantiate the Zend_Cache_Backend_Memcached class instead of the default Zend_Cache_Backend_File. The only thing to be aware of with this configuration is that when I set it up, I noticed that the hit percentage is much lower than that of the first level instance. I haven’t been able to determine whether that is typical or whether it has something to do with memcached’s lack of support for tags.

Edit: David Alger from Classy Llama Studios confirmed my suspicion about tags. It turns out that for the slow-backend to function properly, the backend cache type needs to support tagging. None of the shared memory backends support tags, which means that out of the box, the only slow-backends that work are files and database.

There are two extensions available that support tags and offer improved performance over the filesystem. One uses symlinking to impove lookup time. The other uses Redis.

Details here.

Memcached and the Full Page Cache

Finally, if you really want to see your website fly, you can take advantage of the Full Page Cache module that is found only in Magento Enterprise Edition. This took a bit of investigation on my part, since it isn’t documented anywhere, but it turns out that the Enterprise_PageCache module instantiates a Mage_Core_Model_Cache object to use for full page caching. This gives us full access to the range of cache backends available to the core cache module, including memcached. The configuration is similar to the one from the local.xml file, but it goes under the ‘full_page_cache’ node in the enterprise.xml file. This would be running on a fourth memcached instance listening on port 11214.

After implementing all four instances of memcached in Magento, the client’s website started flying. When I tested it using WebPageTest.org, something remarkable happened: straight A’s:

straight A's

Those of you that work with Magento regularly can attest to how rare that is. Hopefully this post has given you some hints about how you can leverage the great caching options provided in Magento and the Zend Framework to improve your website’s performance. And if you only take away one thing, let it be this: The key to performance on the web is good design and inserting caching wherever you can.

Posted in: Magento
  • http://paulvantuyl.com paulvantuyl

    Is Full Page Cache different from the External Full Page Cache that I found in CE 1.6.1.0?

  • http://www.nexcess.net/ Ben

    @paulvantuyl – Yes. The CE module named Mage_PageCache is different from the Enterprise_PageCache module found in EE. The Mage_PageCache enables you to use a feature of Zend Server for external full page caching. That is, if you’re running your store on Zend Server.

  • http://www.facebook.com/sean.villani Sean Villani

    I’m a little confused as to how to set up the full_page_cache is it the exact same except that you create a node in the enterprise.xml rather than a node? or do you add a node under the node in enterprise.xml?

  • Magentohostingxl

    Terrific work! I wonder what type of webserver was used. We’re using a combination of NginX with a PHP deamon and a CDN. NginX is in a good configuration over 8 times faster than Apache.

  • http://www.nexcess.net/ Ben

    @Magentohostingxl: This particular client is on a cluster running Apache. Nginx is great as well.

  • http://www.nexcess.net/ Ben

    @Sean: I didn’t post the enterprise.xml because it’s under the EE license, but essentially the ‘backend_options’ ‘memcached’, etc nodes go under the ‘full_page_cache’ node in enterprise.xml.

  • Brain Smith

    Good Work, keep going. I want help if you can, I am trying to cache some part of the page but its giving some error. Can you tell me how can I do same. Thanks in advance.

  • suzanac

    Ben, would you be able to contact me re; a possible consulting contract?
    Thanks,

  • http://twitter.com/Turnkeye Magento Developers

    Full Page Cache is Magento Enterprise feature.

    As for External Full Page Cache from CE, it will work only on Zend Server.

  • Chuck

    Great Post – I am just starting to wrap my head around the Magento cache.

    Do you think there is any benefit to ditching the two level cache and just using a very large “single level” memory-based cache? If 8GB of memory are available, basically set the Max Memory of Memcache (or Redis) to 8GB and have that be the only cache?

  • bnugent

    That’s essentially what Redis does. Since Redis doesn’t use the Zend TwoLevel cache, you can allocate as much memory as needed and it will serve as the sole memory-based cache. With Memcache, Magento defaults to using the TwoLevel cache, so some kind of programmatic override would be needed. Also, you would need to fix the lack of tag support.

  • http://www.facebook.com/vi.wickam Vi Wickam

    What is available in the world of full page caching on Community Edition?

  • TSD

    If we are using memcached, and for both backend and slow backend you write:

    ![CDATA[ memcached ]]
    ![CDATA[Memcached]]

    Why is it for session_save, while still using memcached, we instead write:

    ![CDATA[memcache]]

    and not:

    ![CDATA[memcached]]

    ****Notice the d****

    I have not found an explanation for this but I am incredibly curious. Thank you!