Testing is Rocket Science Not Brain Surgery

They have a load testing tool purchased and paid for, albeit in a shrink wrapped box awaiting installation. A lab full of servers, left over from the last upgrade cycle two years ago. Monitoring software is available, just not for the newly christened performance environment. Infrastructure support is there, they just haven’t been prepped by management to address your needs. All of which may seem worst case scenario, but is actually one or two steps further along than usual unfortunately. So now the question, as always, is “What do I do now?”

Well given that this context isn’t what you might have hoped for, the types of testing you can do and it’s scope must change accordingly. Sometimes it just “Is what it is”, and you’ve got to roll with the punches. In doing this, it’s important to shape and/or re-shape your client’s expectations. This should be cooperative versus accusatory and negative, remember we’re rolling with the punches.

One of the biggest mistakes you might make as a novice performance tester is to tax your testing hardware resources inadvertently while trying to meet the target load. This can produce very inconsistent and therefore unreliable results. Understand that your testing sessions create application instances through various methods that basically come back to the instantiation of multiple threads or processes on the host or generator hardware. While having an exceptionally small footprint these sessions consume resources all their own along with the OS and anything else running on these load generation machines(it’s advisable that they be imaged with only the OS with minimal options). As such when the timer object begins in your session thread it is dependent on not only the response from the application under test but also the hardware on which the test is running to process the results. So its not hard to imagine the processor queue filling up when you try to run 1000 threads of your test application users which are all running a single application instance.

So it’s important to assess your hardware once you hit the ground. Now remember my mantra of not assuming anything, and high on that list is whether or not the software selected to do the testing is actually a good fit for the job. There are a number of reasons how this might happen, architecture changes between inception and the actual purchase, software reuse policies between projects, budget, general lack of understanding, misleading sales hype, etc. Whatever the reason, it happens, fortunately there are a number of ways to skin the proverbial cat, you can find a number of free performance tools available to achieve your testing goals Also be open to a hybrid approach where multiple packages are used in concert with each other. Just apply some critical and creative thinking, avoiding getting locked into the mindset of a “best practice” for a particular tool.

Lastly, infrastructure support, typically the group or party responsible for aiding in the monitoring and provision of access to the log files you’ll need more often than not hasn’t been informed of your planned activities. Building that bridge and raising their awareness is usually going to be your job. This is especially true of organizations that have not been through an iteration of performance testing. It goes a long way to have a strong working familiarty with the programs available natively to the OS your software runs on, skill at building log file aggregators and parsers, and conversion utilities to handle imports/exports and transformations to various formats. If this is a little beyond your reach make the client aware that a development resource will be needed and infrastructure guidance as well in advance. Like the application, the infrastructure SMEs will have or should have the best handle on the environment under test. It is always important to coordinate your activities and engage them in the planning process as early as possible. Helping them understand that they too are consumers of the test data will go a long way towards shifting their mindset from facilitator to participant, which from a psychological standpoint is a change of pace for them.

I bet you thought you would just come in, script, execute and report without getting dirty. Well, welcome to the playpen.