The first thing we need to do is clear the air as far as terminology. A stress test is a type of performance test that is thought to be synonymous with a load test. But making the distinction and gaining an understanding of how this type of test should be conducted can help mitigate one of the biggest limitations in testing pre-production systems with the goal of predicting performance. We already know that nine times out of ten we are not going to have a production system at our disposal to test against. Historically, this has been one of the best excuses for poor performance testing efforts or a unwillingness to stand on a predictive performance number. But looking at this as an opportunity instead of an obstacle can help us make more effective use of limited resources and actually achieve better test results more quickly.
So what you’re syaing is we can test better without a production system? Well I wouldn’t go so far as to say better, that may be an exaggeration, I mean after all, what trumps production? But the idea of a single-node of production might help come closer to meeting that requirement. A node expressed both in terms of the number of servers involved, or a clearly linear relationship in the computing power with a minimal amount variance in the rest of the environment specifications.
What does that mean? It’s a change in the scope of the questions you will be able to answer. A dual processor database server is going to give you a number crunching headache if you try to extrapolate out the results to a 64-way quad core box. Good luck with that, although BMC has some interesting products that propose to do just that, I’ll leave that investigation up to you. But what you can do is push that dual processor box to its operational limits and see what happens to your application servers or web servers as they wait for a response from the DB. The idea would be to come as close as possible to a representation of the planned or exisiting production infrastructure minus some bits and pieces. Ideally, in a multi-server environment you would have comparable web, app, load balancing, and network configurations. It’s a lot cheaper to obtain two production web servers and an application server with a load balancer than it is to get the accompanying database server being planned for, or in, production. This is a good thing because the volume, both in the amount of data needed and the testing time needed would mean having test windows as large as the operational production day or days. But when you have a single-node of your infrastructure with a single component or components that are sized smaller than what is expected or in place you are truly putting the environment under stress. The idea of stressing something not only means to inundate it with more transactions or users than it was designed to bear, but also to strip away some of the capability for dealing with that stress itself.
By taking away, versus adding to, performance under load can be observed with a “less is more” take on things. While this requires a shift in what can be predicted it still has a great deal of value and serves as a work-around of testing infrastructure limitations. Logistically this should be easier to do as production environment hardware purchases tend to be incremental, or at least they should be. Executed early enough in the deployment you can create a series of performance snapshots each step along the way helping in the cost justification of additional hardware being added either vertically or horizontally.
Stress Testing: A new definition? by Howard Clark
July 31st, 2007 · 2 Comments · Uncategorized
Tags:
I’m an aspiring performance tester and I really enjoyed reading this article. Prior to this article my readings lead me to believe that the only way to stress a system was to increase the load. However, now I see that removing components that are designed to enable the system to handle stress is in itself a form of stress testing. Since access to a production system is not always possible, I will probably encounter this scenario more often.
Thanks,
Bobby Washington
Addition by Subtraction to quote a popular saying in the sports world. This is a by-product of single-node testing, say for instance you were going into a configuration where they planned to scale out horizontally to 10 application servers that are round-robin load balanced(1st user to the 1st server, 2nd user to the 2nd server, etc..), you probably would not get all 10 servers for testing. Instead you would ask for a representative node(2 servers load balanced preferrably the same class as those intended for purchase) and because the scaling is horizontal you can assume to some degree that adding another set of nodes would double the capacity of the first. Do note that this would only speak to the components in your node and not necessarily the other tiers of the app(DB server and Web servers). But going back to the original point its a different spin on stress testing that fits and can be easily understood by project management. Just be careful when trying to quantify what will happen if the hardware you test on is not representative of what they intend to purchase, the waters get murky. If your node or stress test situation is on a single processor system and they intend to move to a multi-CPU multi-core system you know they will see an increase in performance, but its VERY difficult to say how much since your stress testing was on a platform vastly different from what they want to go live with. This new definition of stress testing really tries to put “lipstick on a pig” and make the best of a less than ideal situation by rationalizing it.