Table of contents for A Side Note
- A Side Note: The failure to test consistently is inexcusable by Howard Clark
- A Side Note: Why I dislike XY Scatter Charts so much! by Howard Clark
- A Side Note:Get Ahead of the Testing Infrastucture Build Out by Howard Clark
- A Side Note: Job Posting for Aeronautical Engineer – Have experience as an airplane passenger? You’re hired! by Howard Clark
- A Side Note: They might be right by Howard Clark
- A Side Note: “Do More with Less…Can Someone Tell Me How?” by Bobby Washington
- A Side Note: “Fighting To Maintain A Tester’s Integrity” by Bobby Washington
- A Side Note: “To Pull the Plug or Not, Who Knew The Life And Death Of a Computer Would Depend On A Tester?” by Howard Clark
- A Side Note: “Faith in the Machinery” by Ed Cook
- A Side Note: Open-Source or Commercial Testing Solution by Howard Clark
Testrocket.org is very pleased to open up to Bobby Washington as it’s newest contributing author. Bobby brings a fresh perspective from the viewpoint of the intended reader of this site, the practitioner. He is a trusted colleague and valued contributor!
“Do more with less…”,less resources, less time, less money. If you have worked in IT for any length of time you will have experienced this mindset. In software development, testing is typically considered an afterthought. Typically, when there is a need to bring your project schedule in, cutting the time spent testing is the solution most often chosen. To which I say, “It sucks and I hate it.” Strong words I know, there I got the venting out of the way. There is always going to be less time and money to test to the real extent necessary. Now as career software testers what we must realize is that we ca not allow this mindset to force us to test with less passion, less creativity, and less skill. Since the odds of us receiving more time to test before a production release are small, it seems logical to find a way to conduct additional and/or continuous testing after the production release.
Just as IT management must justify the cost and value gained for a new change or system, we too must justify the cost and benefit of executing a test or suite of tests after a production release separate from any regression effort. When you think about it, developers embrace the concept of “Continuous Integration” .
According to Martin Fowler’s website:
“Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.”
Essentially what we have is an ongoing process where all the changes to a code base are continually being verified to ensure that a particular code change does not have adverse effects on system functionality and/or the overall quality of the product. Testing may or may not be included depending on the nature and scope of the change.
Why not develop a similar ongoing process for testing called, oh let’s see… “Continual Testing.” Due to the inevitable reduction in test design and execution time we must explore new creative means to provide accurate and timely information with regards to the quality of our systems. Hopefully by doing this we are able increase our chances of discovering application defects before our customers do even after the official end date for testing and the start date of production.
I’m reminded of a situation where an application would have benefited from a “Continual Testing” concept. I know an individual whose wife is required to take maintenance medication. The company he works for requires that the medication be obtained via a mail order program. An order to refill the prescription was placed online with the option to overnight the medication. Later that week his wife was not happy with how long it was taking to receive the medication and decided to contact the mail order company to see what could be done. The representative assured her that the order should be received no later than Wednesday. Unfortunately Wednesday comes and goes and still no medication. After an additional follow up call the representative informed the husband that the system canceled out the order altogether as a result from the online request and the called in request. I would never just assume that the testing effort of this pharmaceutical system was substandard. I’m sure that the testing team covered as much critical functionality in the limited time allotted via executive management. We also know it is impossible for testers to identify ever defect in a system. So you combine this limitation with an ever shrinking timeline and it’s not surprising the major defects like this surface in production. What if the medication was life sustaining, who would be at fault.
A case like this solidifies my belief that a concept like “Continual Testing” is a viable approach in the wake of the “do more with less” mindset. Now this concept should not be treated as a silver bullet and must be used in context. You may not be able to justify the cost of this approach for a simple website. However if the class of your application is mission critical, public safety, large-scale enterprise, real-time, or financial “Continual Testing” may be an approach that can strengthen stakeholder confidence in the quality of your application. By the way the medication was finally received…on Friday.
I like your way of thinking! I think every client I’ve worked for has cut the testing timeline and sometimes cut it drastically. However, I’d like to hear more of your ideas on continuous testing. What kinds of testing would you perform to discover to most critical bugs in such a short timeframe? Would you use a risk-based approach?
Josh, let me start off by saying thank you for your positive words. As to your question, yes I would adopt some sort of risk-based approach. Testing, in my opinion, should always have some risk-based component to it. Testers with the assistance of project stakeholders are always confronted with decisions of which tests to run, which not to run, and why. As you stated previously, it is common practice to cut testing time. I’ve found that some environments have tests that have already been constructed during the test design phase that simply could not be executed due to an aggressive release schedule. These tests can serve as a great starting point for the Continual Testing concept. Remember everything is up for negotiation. So upon a “pushed” release, we can request time to re-evaluate these scripts as a postproduction activity. It is simply too wasteful to not at least re-evaluate them to see if they do add value. If we let them become orphan scripts then we are essentially leaving money on the table. Brainstorming sessions can also be additional postproduction activities where we essentially discover new ways to challenge the system and determine if the risk of failure outweighs the cost of developing and executing the test. Essentially Continual Testing is our attempt to verify the quality of an application in the wake of an ever shrinking testing budget.
Just to reaffirm what was said concerning taking a risk-based approach, something that is absolutely critical when you look at the performance test effort, it also serves as a way to compress the time needed to test while still maintaining a pertinent amount of functional coverage. Performance test efforts typically get one shot at hitting the mark, while working under budget, resource and time constraints even more daunting than functional testing, a by product of it’s poor adoption rate and appreciation in the software development community except for the most mission critical of apps or sophisticated of companies. I typically grade risk in terms of complexity, concurrency and volume along with visibility with respect to performance testing. This serves to properly focus the effort so one doesn’t get too lost in the weeds.