Testing is Rocket Science Not Brain Surgery

I can’t underscore the need to have experienced and knowledgeable interviewers when it comes time to hire people in positions that require not only technological prowseness but also critical thinking skills.  When it comes to software test automation and performance testing, resources need to have a mix of the art and science mindset, creative and critical thinking with logical and deductive reasoning skills.  These two test methods appear simple at first glance until you start to look at all of the areas you have to cover from planning to development to the actual execution and analysis.  So when I see sites like these,

 LoadRunner Interview Questions 

I begin to become worried about how people go about hiring for these positions.  I have on occassion suggested that clients have more experienced people interview me for a position.  While seemingly foolish or arrogant, this actualy serves a very beneficial purpose.  One, it helps increase the confidence in the hiring, and secondly it gives me a chance to see what the experience level is in the organization.  I’m of the opinion that every chance you have to sit in front of someone and discuss your craft is an opportunity to educate and engage in some intelligent conversation.  So its important that all parties involved are confident in their ability to hire for the duties and perform the duties.  When this doesn’t happen I can guarantee there will be issues.  Ignorance is not bliss when it come to investing in automation and performance testing efforts, things can get expensive quickly, not only in tool costs, but effort and time.  Compounding this is the perception of a lack of ROI when testing software as most companies fail to recognize the cost over time problem when bug discovery happens later in the SDLC beforehand.

So, what this means is everyone should be going into the acquisition of resources with eyes wide open, and not depending on some arbitrary list of tool specific questions that are easily answered by the help documentation in the tool.  Having been the recipient of certifications from two significant players in the software industry I know first hand that the path to certification is both expensive and time consuming.  Few of my colleagues are able to pursue certifications that require multiple weeks of missed work along with the reduced productivity around studying for the exam.  Both a lack of understanding by the interviewer and a dependence on certifications to validate hiring decisions can and often does lead to bringing people in that are so specialized that they fail to contribute in a meaningful way to the overall effort.  Even when all you need is a tools driver you’ll find that more often than not knowledge of the tool alone will not suffice, better to hire for an understanding of the technologies in play with some testing tool experience than the specific tools at the expense of not having real software domain knowledge.

Performance_Testing_Checklist_v2_0(1)

There are a lot of tasks that need to be executed before you press “Go”, here is a sample checklist to help you begin.  It utilizes my AMEA framework, where the work breakdown is separated into four groups of Assessment, Modeling, Execution, and Analysis.

It’s important to get everyone who has an action item to sign off on the checklist and develop SLAs around when and how they will meet the requirements to begin testing.  This is how you begin to normalize your test efforts between test runs.  Be an advocate for automation and taking on as many of the these tasks as possible to reduce the dependencies.  Performance testing should not be done ad-hoc without structure; the “a man is not an island” catch-phrase will serve you well in helping keep your sanity.  It forces you to engage others and in the process you are bound to gain insight into other aspects of the infrastructure you may or may not have been aware of.

This is inclusive of all the parties involved in the delivery of the system under test and your performance testing effort. Not only will you have dependencies on the application(s) being delivered and/or functionally sound but also on ancillary development to augment your ability to monitor and measure the application. In order to perform the best analysis on the application it’s often necessary to engage the development team as the SDLC rarely makes room for a tester who can perform development duties, if for no other reason than the risk of introducing defects through your coding efforts. So we mitigate this by engaging the development team as early as possible to ensure that the development necessary to facilitate testing occurs.

So recognizing this possible make or break dependency to your test effort is the first step, the second is being able to gauge with some level of accuracy what the time and resource requirements will be to make it happen. From a consultant’s perspective this means you better be pretty comfortable with the technologies at work and understand what that effort will be before you make your request. It means that your ally is going to be the technical architect, and when none is present the lead developer, and when he/she is available a driven developer, and when all else fails you need to be able to position yourself as being capable of doing the job. You won’t have the luxury of figuring this out at your leisure so look for the following signs and make a judgment call.

• You’ll need to assess where the project is in the timeline.
• Know the players on the technical and business side you’ll need an SLA and inputs about proposed or historical usage.
• What is the comfort level of the team with the technologies being used?
• How does the development team gauge the effort needed, if it’s a huge deal to add some logging, write some SQL and provide some metrics, you might be in for a challenge.
• If an environment build out takes longer than a few days absent the time to procure hardware you might be in store for some problems.
• If the performance testing IQ is low, hunker down and prepare to educate your consumers.

These are not obstacles but rather, impediments that with swift and decisive action can be rectified. The timing as far as delivery may not be in your control; ever, but you can be prepared when everything you need comes into place.

As colleagues can attest there have been times when it appeared as if I was on the payroll of certain industry leading commercial testing software providers. I have literally scoffed at open-source intiatives and languages such as Python and Ruby, since in my opinion they fell outside of the established norms of C/C++/C#, VB and all it’s flavors, and Java. Now a lot of that has to do with me being practical as there are only so many languages I can stay proficient in. But in the roles that I fill there are implications that reach beyond my own comfort level, decisions need to be made in an unbiased manner within the context of the client’s environment.

It is the understanding of context as a decision driver that sometimes gets missed because of the experiences and bias of the decision makers. It’s the context that should dictate the solution versus the solution dictating the context. By adopting this mindset we can avoid one of the pitfalls of tool selection, actually picking the wrong tool for the project. So how do you chose? Without specifics I can see the following axioms holding true:

• Performance testing a commercial packaged software solution will usually require a commercial solution. Big vendors prefer to play with other vendors like themselves in size and market reach. To unlock their software for use with non-intrusive technologies tends to require exceptionally large reverse engineering efforts, or direct vendor-to-vendor collaboration, which is usually outside the scope or capability of even the largest open-source communities. Hopefully this will change.
• If the system under test uses open standardized protocols like SOAP and HTTP/HTML then the number of options begins to include any number of commercial and open-source tools.
• Obviously budget can narrow or expand your choices, but its presence or lack thereof should not be the sole driver. There are commercial solutions that are hosted at considerable cost savings and open-source solutions that in some instances are just as robust, even more so depending on the application development environment.
• The skillsets needed to maintain the test scripts can be affected by the viability of the vendor. Is the platform proprietary, is training available, is the technology common in the marketplace?
• The extensibility of the solution should rank highly once we begin to develop around it, along with integration with various source code management tools.
• Experience in deploying centers of excellence around testing and the tools being used is also a consideration with buy-in from the strategic users to justify the investment.

Evaluate the context of the emvironment in it’s entirety and remain open to a variety of approaches, both from a testing approach and a tools perspective. Then the decision doesn’t become one of cliche open-source versus commercial tools battles. Fair and unbiased evaluations based on testing capability, sustainability and versatility will help make the decision a more well-rounded one.

They have a load testing tool purchased and paid for, albeit in a shrink wrapped box awaiting installation. A lab full of servers, left over from the last upgrade cycle two years ago. Monitoring software is available, just not for the newly christened performance environment. Infrastructure support is there, they just haven’t been prepped by management to address your needs. All of which may seem worst case scenario, but is actually one or two steps further along than usual unfortunately. So now the question, as always, is “What do I do now?”

Well given that this context isn’t what you might have hoped for, the types of testing you can do and it’s scope must change accordingly. Sometimes it just “Is what it is”, and you’ve got to roll with the punches. In doing this, it’s important to shape and/or re-shape your client’s expectations. This should be cooperative versus accusatory and negative, remember we’re rolling with the punches.

One of the biggest mistakes you might make as a novice performance tester is to tax your testing hardware resources inadvertently while trying to meet the target load. This can produce very inconsistent and therefore unreliable results. Understand that your testing sessions create application instances through various methods that basically come back to the instantiation of multiple threads or processes on the host or generator hardware. While having an exceptionally small footprint these sessions consume resources all their own along with the OS and anything else running on these load generation machines(it’s advisable that they be imaged with only the OS with minimal options). As such when the timer object begins in your session thread it is dependent on not only the response from the application under test but also the hardware on which the test is running to process the results. So its not hard to imagine the processor queue filling up when you try to run 1000 threads of your test application users which are all running a single application instance.

So it’s important to assess your hardware once you hit the ground. Now remember my mantra of not assuming anything, and high on that list is whether or not the software selected to do the testing is actually a good fit for the job. There are a number of reasons how this might happen, architecture changes between inception and the actual purchase, software reuse policies between projects, budget, general lack of understanding, misleading sales hype, etc. Whatever the reason, it happens, fortunately there are a number of ways to skin the proverbial cat, you can find a number of free performance tools available to achieve your testing goals Also be open to a hybrid approach where multiple packages are used in concert with each other. Just apply some critical and creative thinking, avoiding getting locked into the mindset of a “best practice” for a particular tool.

Lastly, infrastructure support, typically the group or party responsible for aiding in the monitoring and provision of access to the log files you’ll need more often than not hasn’t been informed of your planned activities. Building that bridge and raising their awareness is usually going to be your job. This is especially true of organizations that have not been through an iteration of performance testing. It goes a long way to have a strong working familiarty with the programs available natively to the OS your software runs on, skill at building log file aggregators and parsers, and conversion utilities to handle imports/exports and transformations to various formats. If this is a little beyond your reach make the client aware that a development resource will be needed and infrastructure guidance as well in advance. Like the application, the infrastructure SMEs will have or should have the best handle on the environment under test. It is always important to coordinate your activities and engage them in the planning process as early as possible. Helping them understand that they too are consumers of the test data will go a long way towards shifting their mindset from facilitator to participant, which from a psychological standpoint is a change of pace for them.

I bet you thought you would just come in, script, execute and report without getting dirty. Well, welcome to the playpen.