Testing is Rocket Science Not Brain Surgery

The business processes that need to be tested may be well understood and already decided on.  But how do you know if those processes really help mitigate the risk to the application?  Much like the issues with building the usage model in the absence of historical data (Building Usage Profiles Without Historical Data) there exists the potential for gaps to present themselves.  Obviously the risk grows in a custom application environment versus a COTS implementation, but that shouldn’t be taken as not having to worry about performance when you go COTS.  Vendors ship poorly performing code and/or poorly conceived sizing requirements all the time, so performance testing should be planned for in some capacity one way or another.  This is especially important when the implementation involves remote data sources or hosted applications with remote access methods. When the client has ownership of the code and/or the development process, code coverage analysis can take on a much more detailed and granular form with actual walkthroughs and re-compilations into instrumented code being possible.  In the case of a COTS implementation various logging levels inherent to the app and SQL database tracing can help fill in some of the blanks.  What’s important is that there is some type of effort around understanding how much of the application is being exercised.  The concern for functional coverage doesn’t begin and end with the functional test effort, as a performance tester you are not completely absolved from it.  It’s important that the story you tell is a complete and accurate one when the results are presented so that all involved have a greater sense of confidence.  Understand though that the point of this coverage analysis exercise is not to cover code blindly but to ensure that the pain points in the application are actually covered by the business processes that have been selected to script.  It is an effort to verify that what you think is going to cause a problem is actually being exercised. One especially useful tool out in the open-source community is EMMA for java-based apps.  Feel free to comment on your favorites!