Testing is Rocket Science Not Brain Surgery

Almost all of the World's Greatest Accomplishments Were The Result of Great Planning!

Testing is Rocket Science Not Brain Surgery header image 2

Test Planning the Marine Way: “Develop A Shared Situational Awareness” by Ed Cook

April 4th, 2011 · 1 Comment · Testing From The Field

“The process of planning itself should provide a common understanding of the nature of the problem and so support communication and cooperation. In other words, planning is a way of exploring the situation. Even if the understanding of that situation is incomplete or not entirely correct—and most attempts to attain situational awareness will be both—the common understanding provides a basis for unity of effort. In this respect, planning helps commanders both with formulating their intent and in conveying that intent to their subordinates.”
—Marine Corps Doctrinal Publication (MCDP) 5, Planning

For test efforts, “shared situational awareness” applies to both the test team’s exploring and understanding the application under test, as well as providing a framework to express that understanding to the rest of the project.

Outline team responsibilities
Because testing is often project based and also often such a large effort that no one can be an expert on everything (at least at first), one of the first orders of business is determining team responsibilities, and determining who will be the subject matter experts (SMEs) for each part of the project. For high risk areas, overlapping SME duties will allow for team test design and protect you from losing someone important at an inopportune time, however, most projects have more things needing testing than time to test them, so you have to be judicious about overlap.

Once team responsibilities are defined, enlist your SMEs to help you expand the test planning process to cover items specific to their subject matter. The Reporting SME may require different tools from the SME for Billing, for example.

Define expected metrics
One maxim of test management is that invariably, someone will ask for some metric you weren’t expecting, and won’t ask for it until some inconvenient time when test execution is underway. Defining your metrics beforehand and getting buy-in from the stakeholders will lessen this experience. Definition should include both a statement of what the metric means, along with the formula used to calculate it and where the inputs to the formula can be found.

If new metrics are requested after the plan is approved, include their definitions in the test summary report, so that future efforts can reuse your metrics.

Finally, don’t hoard your metrics! Explain to your team what metrics you are capturing and why – so that they understand the importance of whatever special fields you’ve added to their effort. They also often can suggest better ways of capturing the information.

Define what should be contained within a defect, to support expected and unexpected metrics
This works with the previous point – the defect template needs to be defined in your test plan before execution so that you ensure that everyone is getting the information they need. In addition to gathering metrics, your test team and the development team need to be in agreement about what information should be in a defect. In some cases, it may be beneficial to include fields to note whether design specs were referenced before writing the defect, whether a dev or BA was consulted, or what requirement the defect covers. In others, this is not important.

Outline the triage process, with a detailed guide to determining priority/severity
The triage process is important to ensure that defect severity is accurately determined so that development is working on the right defect at the right time. While triage meetings are often limited to management, the test team is the primary driver of severity since they have initial input, and human nature leads generally to initial severity not being questioned enough. Also, if there is a lot to triage, a defect marked low severity may not even be looked at.

The reality is, severity is subjective, and the test analyst is expected to rate defects based on the severity that the defect means to other people. This can be an area where test planning requires the test planner to act as a facilitator, and let the test team and stakeholders meet to hash this out, with the test planner taking the final agreed upon definition – because if the test team and stakeholders do not agree, it doesn’t matter what the test plan says.

Other ideas for developing situational awareness within your project and team:

  • Delineate points for when inside and outside parties (developers, business analysts) review the test suite and other deliverables. While reviews take up time, they help keep the project informed as to what is being tested, and help avoid missing key tests.
  • Involve the team in the test planning process. Don’t just show them the test plan – talk it over with them as a group so they understand why the document says what it does (as what is not said is as important as what is), and explain anything that is part of the planning process, but that is not included on the test plan document.
  • Tags: ····

    One Comment so far ↓

    Leave a Comment