Test Design

The best approach for collecting data from several factors is to conduct a statistically designed experiment. Design of Experiments (DOE) is an experimentation strategy in which testers vary factors in a coordinated, strategic way, instead of varying them one at a time, or arbitrarily choosing test cases. A test strategy that employs DOE will always provide the most powerful allocation of test resources for a given number of tests. This lesson introduces the test design process which includes carefully considering the test goals to ensure objectives can be met by the data collected; building the corresponding type of design; and evaluating the design across multiple criteria.

Design

---

Power in JMP Tutorial

Provides an overview of power calculations and detailed instructions for calculating power for Designed Experiments across a variety of software packages.Document-PDF

Contributed by: IDA Staff
Dec-03-2020

Power

Design

---

JEDIS JMP AddIn

JEDIS is an add-in for the JMP statistical software program that helps automate the Design of Experiments (DOE) process within JMP in a user-friendly manner. JEDIS V 3.2 updated for compatibility with JMP 16.JMP Add In

Contributed by: IDA Staff
Mar-19-2024

Power

Design, Analysis

---

One Sample t-test and Power

Estimates power and plots power by sample size for a one sample t-test.Interactive Shiny App

Contributed by: IDA Staff
Feb-27-2025

Power

Design

---

Two Sample t-test Power

Estimates power and plots power by sample size for a two sample t-test.Interactive Shiny App

Contributed by: IDA Staff
Feb-27-2025

Power

Design, Analysis

---

One Sample Proportion Test and Power

Estimates power and plots power by sample size for a one sample proportion test.Interactive Shiny App

Contributed by: IDA Staff
Feb-27-2025

Power

Design

---

Categorical Analysis Power

Estimates power and plots power by sample size for a multi-factor categorical test.Interactive Shiny App

Contributed by: IDA Staff
Feb-27-2025

Power

Design

---

GLM Power for Categorical Factors

Approximates power for effects in a generalized linear model for experiments with categorical factors. Currently supports Logit, Poisson, and linear regression.Interactive Shiny App

Contributed by: IDA Staff
Feb-27-2025

Power

References

1. Freeman, L. J., Johnson, T., Avery, M., Lillard, V. B., & Clutter, J. (2018). Testing Defense Systems. Analytic Methods in Systems and Software Testing, 441.
2. Montgomery, D. C. (2017). Design and analysis of experiments (Ninth ed.): Hoboken, NJ: John Wiley & Sons, Inc.
3. Director, Operational Test & Evaluation (2010). Guidance on the use of Design of Experiments (DOE) in Operational Test and Evaluation.
4. Director, Operational Test & Evaluation (2013). Best Practices for Assessing the Statistical Adequacy of Experimental Designs Used in Operational Test and Evaluation.
5. Office of the Director, Operational Test & Evaluation, Action Officer Training (2016). Experimental Designs. https://www.dote.osd.mil/Publications/Training/