Testing is Rocket Science Not Brain Surgery

Before I begin let me share with you a a bit of wisdom from a great thinker.

“It is the mark of an instructed mind to rest satisfied with the degree of precision to which the nature of an object admits, and not to seek exactness when only an approximation of the truth is possible.” – Aristotle

I know this quote by heart because it acts as a sort of mantra helping me to ease my controlling and exacting tendencies.  It helps me stop myself from getting lost in the low-level details instead of being a big picture thinker.  At the onset of a performance testing engagement the ability to see the whole system and the broad array of usage and user profiles is critical.  Understanding the client’s business and discovering any possible impacts to the performance criteria before you begin the exploratory phase of testing for capacity via load and stress testing scenarios is important.

So let’s get a breakdown on the quote, “degree of precision to which the nature of an object admits” is the application itself, the known heavy hitting code blocks and SQL queries and or computations.  “Not to seek exactness when only an approximation of the truth is possible” is a warning to not drill-down to every possible functional pathway or attempt to precisely model what a user may or may not do.  This means that we don’t worry about the “happy”, and let’s not forget the “not-so-happy”, paths in our first conversations about how the user community will behave.

Despite the insistence of your client, until you actually have a day or more of real production usage (UAT notwithstanding) data you won’t know what your user community will do.  But, what you do know is what your application will allow the users to do and you should also know the conditions to create the complex and more demanding transactions.  Mapping the primary functional pathways that reflect how to get things accomplished in the system minus the use of ancillary features is a useful guideline.  The pathways chosen should be done so with a weighting leaning towards the hardware utilization cost of the transaction and not so much the perceived frequency of use.  This helps avoid situations where on a given site the most frequent pathway is a series of pages being browsed followed by an occasional subscription purchasing transaction for the content being browsed which ends up crippling the system because during testing the focus was weighted towards browsing content versus the subscription purchasing activity.  You want your universe of possible functional pathways to be diverse and expansive drawing from many different areas of the application.  Give yourself some latitude initially which also helps to avoid missing areas of the application’s infrastructure that may or may not be exercised often by focusing on what are perceived to be the known pathways.  Once this collection is built you can march down the path of refining the list of pathways or putting varying degrees of emphasis on different pathways and exploring those further using different testing types that embrace profiling test methods.

Obviously the frequency question has to be answered but it should be done with the understanding that any number you devise will be arbitrary.  The best course of action would be to lean towards future projected growth or worst case scenario numbers.  The hardware utilization measurements can guide you towards an acceptable range and rate of transactions.  Exploring through multiple testing iterations will begin to provide an idea of your system’s capacity for the application under test.  This type of testing should be done both singleton and in various combinations to begin painting the transaction mix picture and developing an understanding of how different transactions interplay with one another.

 So remember take an exploratory approach or theme, be open, and concentrate on your current reality not a fictional one born out of best guesses and assumptions.  It’s not to say that the requirements team’s best guess isn’t valuable, but unless those assumptions are backed up by some type of historical data or hard projections you should begin with what you do know.

Cheers!