Testing is Rocket Science Not Brain Surgery

“Oh, what a tangled web we weave, when first we practice to deceive.”

The operative word here is practice. As testers credibility is just as important a skill to possess as any. However, anyone involved with software development for any length of time can agree that sometimes pressures from management can try to force us to alter or misconstrue our results, often met with grave consequences if we do not comply. Recently my wife applied for a part time job. The hiring company provides the contact information of executives and other high profile individuals of a particular organization to other companies that are interested in marketing to them. Needless to say this privileged information is not easy to come by. During the week of her training my wife noticed people essentially telling “little white lies” to garner this information. Nothing really damaging, but it would seem that professing to be a college student working on a terms paper was a tried and tested technique for acquiring the necessary contact information. My wife explained to the hiring manager that however innocent, this approach made her uncomfortable. Fortunately the hiring manager understood and still proceeded to hire my wife. Continuing on this theme, I just finished reading an article on www.stickyminds.com by Fiona Charles titled “No Compromise”. In this article she explains the fear testing professionals have with communicating less than desirable information and how to overcome them. Many times we read articles that detail different techniques and technical skills we can utilize to aid in performing more efficient, consistent, and robust testing. However once we have our results, how do we communicate this information in hostile environments? How do we communicate our findings to individuals with the professional title and influence to “spin” our findings in a way to protect themselves and/or further personal agendas by attempting to put lipstick on the pig?

Allow me to create a scenario for you. You have been brought in on a performance test engagement, while executing your scripts and reviewing results you begin to form an assessment of the overall performance of the system. You realize the performance of the system points to underling flaws in the design of the system and no amount of tuning will resolve the issues. You explain these findings to your manager who believes that the issues can be resolved with additional hardware, especially since he/she was a major proponent of hiring less qualified resources. Your experience however tells you that the performance of a system is non-linear and purchasing more hardware may only temporarily treat the symptom(s) of the problem and merely creates the illusion of a solution.

As a software testing professional it is our job to not intentionally condone or support deception. Our job is to free our clients of any misconceptions or preconceived notions, both positive and negative, as it relates to the overall quality of a system. As I stated earlier, credibility is just as important a skill for a software testing profession to possess as any. It can take a lifetime to acquire credibility. However, one falsification, white lie; call it what you want, when discovered can destroy what has taken you a career to build. Please understand that I know better than anyone the risk associated with communicating less than desirable information especially when management pressures you to “unofficially” alter your findings or else. By today’s standards I have what is considered a large family that I’m the sole provider for. This brings me back to the “how” to maintain my integrity, free my clients from their altered states of reality, and not get fired. I would like to explain 5 techniques that I use to help me meet these objectives.

1. Become the interviewer

At some point we have all been told that during a job interview you should ask good questions. A well-structured question with the supporting answer can really aid you in the future when determining how to assess the risk associated with communicating high profile issues in the wake of strong opposition. A question I like to ask is:
“Can you tell me about an experience when the testing department had to communicate a high profile issue in the face of opposition”? “How was it received, what was the outcome”?

While no one can predict the future, I can use the hiring manager’s past experience to gauge my potential outcome if found in a similar situation. Even if the hiring manger is lying about the entire experience, I’m hopeful that they will be less likely to deviate from what was communicated to me versus being viewed as an outright liar. Use your interview questions as a compass to navigate the uncertainty of communicating less than desirable information in a given organization.

2. Enlist support

No man is an island… There is strength in numbers… these sayings should remain at the forefront of your mind. Before presenting your results to management share your thoughts with your co-workers. Hopefully you have established a good working relationship with these individuals. Explain to them what you have found, why you believe it is an issue, the potential impact, and what could happen if the issue remained unresolved. By doing this not only do you get varying points of view and you solidify your own understanding. In the event you are challenged on your findings you increase your odds of having the support of other individuals to further your cause.

3. Keep your cool

My mentor once told me that you could react either before an intense professional confrontation or after but never during. We are emotional creatures by nature. Knowing this, we must make every effort maintain our professionalism. You are much more able to be objective in a conversation if you keep a level head. Ensure the other individual that you understand their position, however it may severely impact the teams ability to deliver a quality system. Continue to reiterate your concerns. Your interaction during this encounter is even more critical when other stakeholders are included in the conversation. If everyone sees that you remain cool in the midst of intense accusations and verbal attack you are viewed as a professional who is dedicated to the success of the project.

4. Keep your ear to the ground

In “Lessons Learned in Software Testing” the authors speak about how Cem Kaner knew that communicating less than desirable information was a part of his job. To help mitigate the risk of being fired he made sure he understood the climate of his industry. He kept a close eye on the different companies hiring and for what positions. In addition, I personally find that using tools like LinkedIn and recruiting firms are very helpful. I make an effort to keep in contact with different recruiters from various recruiting firms. This approach is extremely beneficial because I can have these guys shop my resume around right away if the unfortunate happens. While nothing is guarantee, I find having this information empowering in the event I have to take a stand. The age-old saying holds true it’s not always what you know but who you know…build your professional network.

5. Meet the Press

My mentor is great at arguing and debating issues. During our many conversations I asked him how did he become so effecting at arguing. He told me that he would study the political program “Meet the Press”. This show is full of people with differing views trying influence each other as well as the viewing audience. The conversations undoubtedly get heated at times but watching this program has helped me with reading body language, finding holes in a person’s argument and maintaining my composure in the midst of opposition…plus I find it entertaining.

In closing, just remember that timing and strategy should be a part of your thinking process when communicating less then desirable testing results. Unfortunately not everyone wants to be “unplugged” or desires to take the red pill. When all else fails sometimes the best way to maintain your credibility and employment is the trusty fallback CYA documentation.

Testrocket.org is very pleased to open up to Bobby Washington as it’s newest contributing author. Bobby brings a fresh perspective from the viewpoint of the intended reader of this site, the practitioner. He is a trusted colleague and valued contributor!

“Do more with less…”,less resources, less time, less money. If you have worked in IT for any length of time you will have experienced this mindset. In software development, testing is typically considered an afterthought. Typically, when there is a need to bring your project schedule in, cutting the time spent testing is the solution most often chosen. To which I say, “It sucks and I hate it.” Strong words I know, there I got the venting out of the way. There is always going to be less time and money to test to the real extent necessary. Now as career software testers what we must realize is that we ca not allow this mindset to force us to test with less passion, less creativity, and less skill. Since the odds of us receiving more time to test before a production release are small, it seems logical to find a way to conduct additional and/or continuous testing after the production release.

Just as IT management must justify the cost and value gained for a new change or system, we too must justify the cost and benefit of executing a test or suite of tests after a production release separate from any regression effort. When you think about it, developers embrace the concept of “Continuous Integration” .

According to Martin Fowler’s website:
“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.”

Essentially what we have is an ongoing process where all the changes to a code base are continually being verified to ensure that a particular code change does not have adverse effects on system functionality and/or the overall quality of the product. Testing may or may not be included depending on the nature and scope of the change.

Why not develop a similar ongoing process for testing called, oh let’s see… “Continual Testing.” Due to the inevitable reduction in test design and execution time we must explore new creative means to provide accurate and timely information with regards to the quality of our systems. Hopefully by doing this we are able increase our chances of discovering application defects before our customers do even after the official end date for testing and the start date of production.

I’m reminded of a situation where an application would have benefited from a “Continual Testing” concept. I know an individual whose wife is required to take maintenance medication. The company he works for requires that the medication be obtained via a mail order program. An order to refill the prescription was placed online with the option to overnight the medication. Later that week his wife was not happy with how long it was taking to receive the medication and decided to contact the mail order company to see what could be done. The representative assured her that the order should be received no later than Wednesday. Unfortunately Wednesday comes and goes and still no medication. After an additional follow up call the representative informed the husband that the system canceled out the order altogether as a result from the online request and the called in request. I would never just assume that the testing effort of this pharmaceutical system was substandard. I’m sure that the testing team covered as much critical functionality in the limited time allotted via executive management. We also know it is impossible for testers to identify ever defect in a system. So you combine this limitation with an ever shrinking timeline and it’s not surprising the major defects like this surface in production. What if the medication was life sustaining, who would be at fault.

A case like this solidifies my belief that a concept like “Continual Testing” is a viable approach in the wake of the “do more with less” mindset. Now this concept should not be treated as a silver bullet and must be used in context. You may not be able to justify the cost of this approach for a simple website. However if the class of your application is mission critical, public safety, large-scale enterprise, real-time, or financial “Continual Testing” may be an approach that can strengthen stakeholder confidence in the quality of your application. By the way the medication was finally received…on Friday.

LR-Citrix_Presentation

You need to understand that any project requiring a performance test of an application deployment via Citrix is not going to be your casual record and playback affair. So begin by digging up your old C reference manuals, actually they should be close by at all times when working with this tool, but for the neophyte a good reference is a must. Next get a lay of the land for what you can monitor over the network and what key Citrix performance counters you need. Then start to steel yourself for the onslaught of maintenance, you’re going to be using snapshots of defined screen areas to validate whether something has loaded on the screen or finished. What this means is any GUI changes or minor differences in the way the GUI renders (something that happens even when there are no screen changes, go figure) will require at least another run through of all your scripts and making the necessary bitmap hash code updates. LoadRunner provides these for you; it’s just a labor-intensive affair. Factor in that each load injector may render a little differently and you have the potential for a lot of debug pass throughs.

My colleague and friend Chetan Rahul of Fusion Alliance based in Indianapolis was nice enough to take me up on putting together a PowerPoint outlining some of the issues and lessons learned when we began our LoadRunner Citrix engagement. Enjoy!

NOTE: HP has provided some enhancements through a runtime setting that allows for the tool to accept some differential in what is rendered versus the snapshot originally taken, unfortunately despite patch 9.52 I’m still not seeing any benefit from changing the image tolerance.