Introduction to Test Design
Kelly Avery, Rebecca Medlin

The best approach for collecting data from several factors is to conduct a statistically designed experiment. Design of Experiments (DOE) is an experimentation strategy in which testers vary factors in a coordinated, strategic way, instead of varying them one at a time, or arbitrarily choosing test cases. A test strategy that employs DOE will always provide the most powerful allocation of test resources for a given number of tests. This lesson introduces the test design process which includes carefully considering the test goals to ensure objectives can be met by the data collected; building the corresponding type of design; and evaluating the design across multiple criteria.

References

  1. Freeman, L. J., Johnson, T., Avery, M., Lillard, V. B., & Clutter, J. (2018). Testing Defense Systems. Analytic Methods in Systems and Software Testing, 441.
  2. Montgomery, D. C. (2017). Design and analysis of experiments (Ninth ed.): Hoboken, NJ: John Wiley & Sons, Inc.
  3. Director, Operational Test & Evaluation (2010). Guidance on the use of Design of Experiments (DOE) in Operational Test and Evaluation.
  4. Director, Operational Test & Evaluation (2013). Best Practices for Assessing the Statistical Adequacy of Experimental Designs Used in Operational Test and Evaluation.
  5. Office of the Director, Operational Test & Evaluation, Action Officer Training (2016). Experimental Designs. https://www.dote.osd.mil/Publications/Training/