Testing is Rocket Science Not Brain Surgery

In the first camp we have test efforts intended to be highly repeatable, obviously repetitive, and biased towards being regressive.  The performance testing infrastructure should be partitioned in such a way that these efforts are not compromised by the testing being conducted in the other camps.  These testing efforts should lend themselves to being highly predictable as far as scheduling, and a thorough planning session should precede each one where the schedule has taken testing concerns and code deployments into account from an enterprise point of view over a long-term time horizon.  This camp should be entirely devoid of the “mad scramble” for testing resources and free of the missteps that end up requiring follow-on or contingent test executions.  An effort to right-size this environment is especially important as these resources may not be utilized regularly and may suffer from the inclination to re-allocate and re-deploy them.  The idea of this camp in the testing infrastructure is that it has a known capacity and remains allocated for this purpose and only this purpose.

For example, in this camp this would consist of the following:
• Quarterly Baselines
• Volume Tests
• Regression Tests

In the second camp we have project-specific testing efforts that may or may not have a lifespan beyond the completion of the project.  If it is deemed that a particular effort will become a part of the regressive testing effort then those elements of that test should be merged into the proper regression test bed, or moved in it’s entirety into the first camp.  In order for this camp to run smoothly there must be a high degree of inter-team communication and cooperation.  This space can become crowded with scheduling efforts that shift and conflict with others.  A project hierarchy that denotes a project’s priority must be established and agreed on by the various parties involved.  This will require a great degree of care obviously, but is achievable provided the covenants are agreed on by all parties once they present their performance testing efforts. 

For example, in this camp this would consist of the following:
• Proof-of-Concept
• Short-Duration
• Project-Specific

In this third camp we would allocate resources defined as being “floating” based on the needs of the second camp’s testing requirements.  This environment will require more effort from an administrative standpoint.   The aim of this camp is to accommodate unplanned testing needs, or contingent test runs that have surpassed a scheduling threshold where these test runs cannot continue to affect other test efforts by having those test efforts continually accommodate it.  This pool of resources from a hardware perspective should be larger than that of the other two camps.  This camp serves as a safety blanket for the inevitable scheduling and planning shortfalls that can and do occur.  The supporting software configuration (Java SDKs, Windows Components, etc.) of these resources should allow for the execution of all of the testing efforts in process or that might occur going forward.

For example, in this camp this would consist of the following:
• Testing efforts dependent on application under test infrastructure changes.
• Follow-up testing efforts related to the validation of changes made per the recommendations of prior testing efforts or “tuning.”
• Ad-hoc testing

In each camp a certain priority is inherently built-in with the third camp being the most flexible.  While this creates some management issues, in an environment with multiple large-scale testing efforts going on simultaneously the benefits will present themselves quickly as inter-group dynamics avoid stress and miscommunication.  In most situations the second camp with project-specific work is the first one established, and what happens is that over time the hardware resources of this camp can be retired and put to work in establishing the first camp.  As new hardware is acquired and additional purchases are made in anticipation of greater testing resource needs, the second camp can be upgraded and the third camp can be created in anticipation of bolstering the second.  Done incrementally, this pattern works out well and keeps new and old hardware sufficiently utilized in the most effective way.

Cheers