Research & Emerging Directions

Technical papers and research articles on state-of-the-art design techniques

NameDescriptionAuthor(s)Type
Handbook on Statistical Design & Analysis Techniques for Modeling & Simulation ValidationThis handbook focuses on methods for data-driven validation to supplement the vast existing literature for Verification, Validation, and Accreditation (VV&A) and the emerging references on uncertainty quantification (UQ). The goal of this handbook is to aid the test and evaluation (T&E) community in developing test strategies that support model validation (both external validation and parametric analysis) and statistical UQ.Heather Wojton,
Kelly Avery,
Laura Freeman,
Samuel Parry,
Gregory Whittier,
Thomas Johnson,
Andrew Flack
Research Paper
Censored Data Analysis Methods for Performance Data: A TutorialBinomial metrics like probability-to-detect or probability-to-hit typically do not provide the maximum information from testing. Using continuous metrics such as time to detect provide more information, but do not account for non-detects. Censored data analysis allows us to account for both pieces of information simultaneously.V. Bram LillardTechnical Briefing
A First Step into the Bootstrap WorldBootstrapping is a powerful nonparametric tool for conducting statistical inference with many applications to data from operational testing. Bootstrapping is most useful when the population sampled from is unknown or complex or the sampling distribution of the desired statistic is difficult to derive. Careful use of bootstrapping can help address many challenges in analyzing operational test data.Matthew AveryTechnical Briefing
Bayesian Reliability: Combining InformationOne of the most powerful features of Bayesian analyses is the ability to combine multiple sources of information in a principled way to perform inference. This feature can be particularly valuable in assessing the reliability of systems where testing is limited. At their most basic, Bayesian methods for reliability develop informative prior distributions using expert judgment or similar systems. Appropriate models allow the incorporation of many other sources of information, including historical data, information from similar systems, and computer models. We introduce the Bayesian approach to reliability using several examples and point to open problems and areas for future work.Alyson Wilson
Kassandra Froncyzk
Research Paper
Designing experiments for nonlinear models—an introductionThis paper illustrates the construction of Bayesian D-optimal designs for nonlinear models and compares the relative efficiency of standard designs to these designs for several models and prior distributions on the parameters.Rachel T. Johnson
Douglas C. Montgomery
Research Paper
Choice of second-order response surface designs for logistic and Poisson regression modelsThis paper illustrates the construction of D-optimal second order designs for situations when the response is either binomial (pass/fail) or Poisson (count data).Rachel T. Johnson
Douglas C. Montgomery
Research Paper
Designed Experiments for the Defense CommunityThis paper presents the underlying tenets of design of experiments, as applied in the Department of Defense, focusing on factorial, fractional factorial and response surface design and analyses. The concepts of statistical modeling and sequential experimentation are also emphasized.Rachel T. Johnson, Gregory T. Hutto, James R. Simpson and Douglas C. MontgomeryResearch Paper
Comparing Computer Experiments for the Gaussian Process Model Using Integrated Prediction VarianceSpace filling designs are a common choice of experimental design strategy for computer experiments. This paper compares space filling design types based on their theoretical prediction variance properties with respect to the Gaussian Process model.Rachel T. Johnson
Douglas C. Montgomery
Bradley Jones
Chris Gotwalt
Research Paper
An Expository Paper on Optimal DesignThere are many situations where the requirements of a standard experimental design do not fit the research requirements of the problem. Three such situations occur when the problem requires unusual resource restrictions, when there are constraints on the design region, and when a non-standard model is expected to be required to adequately explain the response.Rachel T. Johnson, Douglas C. Montgomery, Bradley A. JonesResearch Paper
Examining Improved Experimental Designs for Wind Tunnel Testing Using Monte Carlo Sampling MethodsIn this paper we compare data from a fairly large legacy wind tunnel test campaign to smaller, statistically-motivated experimental design strategies. The comparison, using Monte Carlo sampling methodology, suggests a tremendous opportunity to reduce wind tunnel test efforts without losing test information.Raymond R. Hill
Derek A. Leggio
Shay R. Capehart
August G. Roesener
Research Paper
Hybrid Designs: Space Filling and Optimal Experimental Designs for Use in Studying Computer Simulation ModelsThis tutorial provides an overview of experimental design for modeling and simulation. Pros and cons of each design methodology are discussed.Rachel Johnson SilvestriniTechnical Briefing
Improving Reliability Estimates with Bayesian StatisticsThis paper shows how Bayesian methods are ideal for the assessment of complex system reliability assessments. Several examples illustrate the methodology.Kassandra Fronczyk, Laura J. FreemanResearch Paper
Power Analysis Tutorial for Experimental Design SoftwareThis guide provides both a general explanation of power analysis and specific guidance to successfully interface with two software packages, JMP and Design Expert (DX).James Simpson, Thomas Johnson, Laura J. FreemanHandbook
Regularization for Continuously Observed Ordinal Response Variables with Piecewise-Constant Functional PredictorsThis paper investigates regularization for continuously observed covariates that resemble step functions. Two approaches for regularizing these covariates are considered, including a thinning approach commonly used within the DoD to address autocorrelated time series data. Matthew Avery, Mark Orndorff, Timothy Robinson, Laura J. FreemanResearch Paper
Statistical Models for Combining Information Stryker Reliability Case StudyThis paper describes the benefits of using parametric statistical models to combine information across multiple testing events. Both frequentist and Bayesian inference techniques are employed, and they are compared and contrasted to illustrate different statistical methods for combining information.Rebecca Dickinson, Laura J. Freeman, Bruce Simpson, Alyson WilsonResearch Paper
A Comparison of Ballistic Resistance Testing Techniques in the Department of DefenseThis paper summarizes sensitivity test methods commonly employed in the Department of Defense. A comparison study shows that modern methods such as Neyer's method and Three-Phase Optimal Design are improvements over historical methods.Thomas Johnson, Laura J. Freeman, Janice Hester, Jonathan BellResearch Paper

Leave a Reply