Common test designs include factorial designs, response surface methodology, and optimal designs. The decision on which test design method is most appropriate depends on the question, metrics, types of factors (numeric or categorical), and available test resources. This lesson outlines how to build four frequently-used design types and what purpose they serve in test and evaluation. It also covers common challenges that may affect the test design, including restricted randomization and disallowed combinations.

References

1. Freeman, L. J., Johnson, T., Avery, M., Lillard, V. B., & Clutter, J. (2018). Testing Defense Systems. Analytic Methods in Systems and Software Testing, 441.
2. Montgomery, D. C. (2017). Design and analysis of experiments (Ninth ed.): Hoboken, NJ: John Wiley & Sons, Inc.
3. Director, Operational Test & Evaluation (2010). Guidance on the use of Design of Experiments (DOE) in Operational Test and Evaluation.
4. Director, Operational Test & Evaluation (2013). Best Practices for Assessing the Statistical Adequacy of Experimental Designs Used in Operational Test and Evaluation.
5. Office of the Director, Operational Test & Evaluation, Action Officer Training (2016). Experimental Designs. https://www.dote.osd.mil/Publications/Training/
6. Simpson, James, “Testing via Sequential Experiments,” Scientific Test and Analysis Techniques Center of Excellence (STAT COE), www.AFIT.edu/STAT/, Jan. 2014.
7. Willis A. Jensen (2018). Open problems and issues in optimal design, Quality Engineering, 30:4, 583-593.
8. Morgan-Wall, T. (2020). skpr R package. https://www.rdocumentation.org/packages/skpr/versions/0.64.2