Best Practices Plan & Design

Stakeholders may include: the program manager, the requirements representative, developmental and operational testers, operational users, and subject-matter experts in experimental design and analysis. This group of people is necessary to identify the key elements of test planning, including: goals, response variables, factors, analysis requirements, and resource constraints.

Rather, testing should aim to characterize operational performance across all feasible and operationally realistic conditions. For example, an unmanned intelligence aircraft might be required to detect 90 percent of pick-up trucks at a slant range of 10 miles. A fair test does not look at only one size truck at 10 miles. The requirement should consider the broader context of all potential targets of operational interest (cars, trucks, tanks, etc.). A broad spectrum of detection ranges should be used to fully understand the probability of detection as a function of range for a variety of potential targets. In cases where there has been previous testing across the environment, a more focused approach may be more appropriate.

Complex systems often require multiple dimensions to capture a successful outcome. For example, an unmanned intelligence aircraft should provide intelligence information that is both timely and accurate.  The system may not be effective if the provided intelligence is accurate but not timely, and vice versa. In this example, success requires both timeliness and accuracy.

Many DoD requirements are specified in terms of probabilities. These probabilities provide intuitive measures of success and often can be captured more easily than continuous measures.  However, they require more observations to provide precise estimates, and often they do not facilitate more nuanced differentiations. On the other hand, continuous metrics reduce test resource requirements to generate the same level of information and, if needed, they could be transformed into a pass/fail probability-based metric. For example, in the comparison of two new unmanned intelligence aircraft, both may accomplish a mission successfully, but one could outperform the other in terms of providing higher accuracy. Analysts should think hard about what underlying continuous variable results in success or failure to ensure testing captures as much information as possible.

When planning a test, align the goals with how the system is intended to be used, including established doctrine and CONOPS. Be mindful of the degree to which you may be asking operators to use a system or conduct a mission in a way that runs counter to their TTPs and training.  Consider testing a business system deployed at multiple locations with different roles at different locations. Avoid the temptation to visit only a single location.  Do not rely on a handful of users to test the complete system functionality, performing tasks that they otherwise would never perform.  To the extent possible, preserve operational realism and have users perform only those roles that they normally execute, using only those functions that they normally use.

Single hypothesis tests often incompletely characterize system performance.  Use the related factor-by-factor power calculations when sizing tests rather than a single “roll-up” power estimate. In addition to power, coverage of the space, prediction variance, and other metrics should be taken into account. Historically, test planners have used one-sample hypothesis calculations to compare results directly to a single requirement, even in cases where multiple conditions are evaluated. This practice fails to capture if the test design will characterize system capabilities across the operational environment. For example, when testing a new submarine sonar processing system, analyst take into account different factors that could affect the performance of the system. In order to do that, multiple hypothesis tests are established and used to determine the necessary number of runs in the design.

In the unmanned intelligence aircraft example, different factors may affect the timeliness and accuracy of information. For example, timeliness may be affected by the presence of decoy targets while accuracy is unaffected once the target of interest was identified. Here, the presence/absence of decoys is a factor in timeliness, but not accuracy. It is often more productive to develop the list of all factors that could affect both timeliness and accuracy and then use that list to design the experiment. This approach also results in experiments that tend to encompass the full operational environment instead of breaking the operational space into more narrowly defined experiments.

Select and ensure data is collected across all relevant test factors to guarantee that the test covers the conditions the system will encounter once fielded. Leverage information from previous tests to identify relevant factors and mitigate the risk of excluding important factors. Guard against omitting factors known to be important from the test designs in an effort to save test resources. Excluding relevant factors limits our knowledge of system capability. Suppose the initial design for assessing submarine sonar performance contained three factors: target type, version, and operator proficiency. If a subject-matter expert later determines that target noise also is important, then fold target noise into the design without deleting the already present factors. Designs often can add factors with little practical impact to statistical power.  More complete coverage of the operational space is almost always preferred to increased sensitivity in a small portion of the envelope.

Sequential approaches reduce required test resources in each test phase while developing a comprehensive view of system capability. Sequential testing can reduce overall resource requirements for the full campaign. Use sparse screening experiments early in the test process to identify the most important factors. Use follow-on phases of test designs to refine knowledge about the reduced set of factors. Smaller, strategically linked tests can reduce both the overall test program and the amount of time speculating and discussing unknowns, and thus reduce the resources for planning and unnecessarily “covering all our bases.” Consider testing a new type of radar.  Developmental testers might execute a sparse fractional factorial design with 10 different factors covering various targets, ranges, altitudes, angles, weather conditions, etc. Analysis could determine that maybe only two or three of those factors meaningfully affected the radar’s performance.  Operational testers could then create a denser, high-powered design to completely characterize and predict performance across the smaller design space. Leveraging results from earlier developmental test enables efficient use of resources to gain the most relevant information.

 It is often necessary to make hard decisions about reducing test size. Understand and acknowledge the implications of what information is lost when reducing the size and scope of a test.  Know if the changes reduce statistical power, obscure outliers and missing data, remove a portion of the tested operational envelope, or all of these. For example, a full factorial design with replicates might be infeasible because of time, budget, safety or other constraints. In these cases, options include reducing the number of replicates or using a different, sparser design.  Reducing replication preserves the operational space, but reduces statistical power.  Choosing a fractional factorial design causes certain interaction terms to become completely confounded, so testers should think carefully about which interactions are likely to occur and are important to estimate so that the chosen design de-aliases those terms. Optimal designs can be extremely fragile and one missed data point could cause the statistical model to become inestimable. If test points must be dropped, testers should strive to keep all points necessary to conduct the planned analysis.

In other words, pay attention to type III error. Throughout progression of the test, confirm that the variables measured are useful to evaluate the operational issues established at the beginning of test planning. If a variable cannot be measured and a proxy is measured instead, ensure that this alternative measurement is a good substitute of the original. For instance, a group of engineers identified six vehicle locations: left/back, left/middle, left/front, right/back, right/middle, and right/front. They wanted to test if there was a difference between two locations, but the resources were insufficient for the necessary design with high power. Therefore, the test planner re-parametrized the six-level factor into two factors (left/right and front/middle/back) and designed a test with high power based on this re-parametrization. By doing this, the main objective of the test cannot be evaluated. Now it is not possible to assess the difference between two of the original six locations (for example difference between left/back vs left/middle), but rather within the two new factors. That is, now it is only possible to assess the difference between left and right, or among the front/middle/back locations. If an interaction was present, then the difference among one factor within the levels of the other factor could be assessed. This, however, still does not address the main hypothesis. Therefore, a new design was created and it is correctly answering a different question.

Adhere as closely as possible to the planned order in which each test is conducted. Modify the randomization to fit operational realism only when this does not cause a systematic pattern (e.g., all day runs are conducted first, certain factor levels are not combined).