In this short series, Kevin Schroeder explains how to keep your website on the rails with proper load testing. Kevin owns consulting firm 10n Software, LLC, and has written several testing frameworks for Magento, Gmail, Twitter, and other applications.
Welcome back to Asking the Right Questions, my three-part series about all things load testing. Part 1 showed how to use entropy to make your load tests better predictors of performance.
Now, onto two types of load tests, I’m typically asked to perform for clients: sizing validation and concurrency validation.
Sizing validation usually comes in one of two flavors. The first attempts to learn if the infrastructure can handle what their analytics software says they should expect. The second checks the accuracy of your sizing recommendation.
I tend not to worry too much about sizing validation unless it’s way off. Most problems can be solved by re-working the parts of the application that are causing the problems. That’s not to say that you shouldn’t do sizing validation, only that concurrency validation tends to provide more useful results.
If you’ve sized your infrastructure improperly, you rack up a more powerful server. But sizing validation won’t help if your application uses the logic that is either not CPU-bound, or CPU-bound to compensate for sloppy SQL queries and database schemas.
I did one load test for a client because a table was getting locked during a request. Instead of resolving the issue through better logic, they explored upgrading to a larger database. This not only wastes time and money but utterly fails to solve the problem.
The other side of that is needless CPU usage that exists solely to shore up poor database management. Databases are notoriously difficult to size, and I’ve seen many situations where improperly executed SQL queries or poorly architected database schemas result in large table scans. Sometimes, the fix is simple: a simple index or a JOIN to a better table. (I see the latter in custom Magento code quite a bit.) Other times, the schema needs to be reworked.
So while sizing validation has its uses, it is the next type that usually yields more useful information.
Concurrency validation can also be called by its longer name, “let’s hit the site as hard as we can to find the breaking point.” In my opinion, this is the first test to run. Sizing a site before fixing its concurrency problems is like showering before you run. Most sites that have any significant write activity to the database will have concurrency problems. The only question is whether or not the solution is architected in a way that allows mitigation.
For example, relational databases like MySQL can have problems if you wrote your CUD SQL using transactions in a way that doesn’t worry about transactions. There are some INSERT INTO SELECT queries that result in table locks. And if you do that in a transaction, that table will remain locked until you issue a COMMIT or a ROLLBACK. Sometimes row locks may be pertinent across disparate requests. These are the kinds of things that you will often not see in your development environment.
Going to NoSQL isn’t the answer either, particularly if you have relational data that needs ACID compliance. If your application requires ACID compliance and you use a non-ACID database, you open up a can of problems just so you don’t have to think about the consequences of transactions.
Push Your Estimates
It is also important to push your system well beyond what you think it will be handling, perhaps even doubling your estimation. Once, I was working on a site when a costly testing company ran a full front-end load test on the site. They executed their test and gave the site a green light. My team then executed our test running at a lower level of code and found some fairly significant problems on the site. The team fixed the problem, we re-ran the test, and the site performed well.
This is not to argue against expensive load testing teams, even if they don’t know your software from the inside-out. They often provide insights that have escaped your notice. Just be aware they are often testing for front-end sizing validation, not concurrency validation.
Thank you for reading! Return here next week where I’ll provide some tips for building and running load tests, suggest software, and tie it all together with real-world examples.
Posted in: Web Hosting Basics