Testing is Rocket Science Not Brain Surgery

As colleagues can attest there have been times when it appeared as if I was on the payroll of certain industry leading commercial testing software providers. I have literally scoffed at open-source intiatives and languages such as Python and Ruby, since in my opinion they fell outside of the established norms of C/C++/C#, VB and all it’s flavors, and Java. Now a lot of that has to do with me being practical as there are only so many languages I can stay proficient in. But in the roles that I fill there are implications that reach beyond my own comfort level, decisions need to be made in an unbiased manner within the context of the client’s environment.

It is the understanding of context as a decision driver that sometimes gets missed because of the experiences and bias of the decision makers. It’s the context that should dictate the solution versus the solution dictating the context. By adopting this mindset we can avoid one of the pitfalls of tool selection, actually picking the wrong tool for the project. So how do you chose? Without specifics I can see the following axioms holding true:

• Performance testing a commercial packaged software solution will usually require a commercial solution. Big vendors prefer to play with other vendors like themselves in size and market reach. To unlock their software for use with non-intrusive technologies tends to require exceptionally large reverse engineering efforts, or direct vendor-to-vendor collaboration, which is usually outside the scope or capability of even the largest open-source communities. Hopefully this will change.
• If the system under test uses open standardized protocols like SOAP and HTTP/HTML then the number of options begins to include any number of commercial and open-source tools.
• Obviously budget can narrow or expand your choices, but its presence or lack thereof should not be the sole driver. There are commercial solutions that are hosted at considerable cost savings and open-source solutions that in some instances are just as robust, even more so depending on the application development environment.
• The skillsets needed to maintain the test scripts can be affected by the viability of the vendor. Is the platform proprietary, is training available, is the technology common in the marketplace?
• The extensibility of the solution should rank highly once we begin to develop around it, along with integration with various source code management tools.
• Experience in deploying centers of excellence around testing and the tools being used is also a consideration with buy-in from the strategic users to justify the investment.

Evaluate the context of the emvironment in it’s entirety and remain open to a variety of approaches, both from a testing approach and a tools perspective. Then the decision doesn’t become one of cliche open-source versus commercial tools battles. Fair and unbiased evaluations based on testing capability, sustainability and versatility will help make the decision a more well-rounded one.