Workshop Breakouts

Workshop Breakouts

Breakouts and tutorials for the Science of Test Workshop 2017 will be held on April 4th and 5th (Tuesday and Wednesday).
Filter Sessions: All Tuesday Sessions | All Wednesday Sessions | All Breakout Sessions | All Tutorials
[php snippet=1]

Title
(Theme)
Speaker(s)ScheduleMaterialsAbstract
Recent Advances in Measuring Display Clutter
(Evaluating System Performance with Human Interactions)
Dr. David Kaber
NCSU
Dr. Carl Pankok
Drexel
Tuesday, April 4th
Session I: 1030-1200
Display clutter has been defined as an unintended effect of display imagery obscuring or confusing other information or that may not be relevant to the task at hand. Negative effects of clutter on user performance have been documented; however, some work suggests differential effects with workload variations and measurement method. Existing measures of clutter either focus on physical display characteristics or user perceptions and they generally exhibit weak correlations with task performance, limiting utility for application in safety-critical domains. These observations have led to a new integrated measure of clutter accounting for display data, user knowledge and patterns of visual attention. Due to limited research on clutter effects in domains other than aviation, empirical studies have been conducted to evaluate the new measure in automobile driving. Data-driven measures and subjective perceptions of clutter were collected along with patterns of visual attention allocation when drivers searched ‘high’ and ‘low’ clutter navigation displays. The experimental paradigm was manipulated to include both presentation-based trials with static display images or use of a dynamic, driving simulator. The new integrated measure was more strongly correlated with driver performance than other, previously-developed measures of clutter. Results also revealed clutter to significantly alter attention and degrade performance with static displays but to have little to no effects in driving simulation. Findings corroborate trends in the literature that clutter has its greatest effects on behavior in domains requiring extended attention to displays, such as map-search, compared to use of displays to support secondary tasks, such as nav aids in driving. Integrating display data and user knowledge factors with patterns of attention shows promise for clutter measurement.
Trust in Automation
(Evaluating System Performance with Human Interactions)
Dr. Joseph Lyons
Air Force Research Lab
Tuesday, April 4th
Session I: 1030-1200
This brief talk will focus on the process of human-machine trust in context of automated intelligence tools. The trust process is multifaceted and this talk will define concepts such as trust, trustworthiness, trust behavior, and will examine how these constructs might be operationalized in user studies. The talk will walk through various aspects of what might make an automated intelligence tool more or less trustworthy. Further, the construct of transparency will be discussed as a mechanism to foster shared awareness and shared intent between humans and machines.
Dose-Response Model of Recent Sonic Boom Community Annoyance Data
(Evaluating System Performance with Human Interactions)
Dr. Jonathan Rathsam
NASA
Tuesday, April 4th
Session I: 1030-1200
To enable quiet supersonic passenger flight overland, NASA is providing national and international noise regulators with a low-noise sonic boom database. The database will consist of dose-response curves, which quantify the relationship between low-noise sonic boom exposure and community annoyance. The recently-updated international standard for environmental noise assessment, ISO 1996-1:2016, references multiple fitting methods for dose-response analysis. One of these fitting methods, Fidell’s community tolerance level method, is based on theoretical assumptions that fix the slope of the curve, allowing only the intercept to vary. This fitting method is applied to an existing pilot sonic boom community annoyance data set from 2011 with a small sample size. The purpose of this exercise is to develop data collection and analysis recommendations for future sonic boom community annoyance surveys.
Overview of Statistical Validation Tools
(Rigorous Validation of Modeling and Simulation)
Dr. Kelly McGinnity
Institute for Defense Analyses
Tuesday, April 4th
Session II: 1030-1200
When Modeling and Simulation (M&S) is used as part of operational evaluations of effectiveness, suitability, survivability, or lethality, the M&S capability should first be rigorously validated to ensure it is representing the real world accurately enough for the intended use. Specifically, we need to understand and characterize the usefulness and limitations of the M&S, especially in terms of uncertainty! Many statistical techniques are available to compare M&S output with live test data. This presentation will describe and present results from a simulation study conducted to determine which techniques provide the highest statistical power to detect differences in mean and variance between live and sim for a variety of data types and sizes.
Improving the Rigor of Navy M&S VV&A through the application of Design of Experiments Methodologies and related Statistical Techniques
(Rigorous Validation of Modeling and Simulation)
Dr. Stargel Doane
COTF
Tuesday, April 4th
Session II: 1030-1200
In the coming years, modeling and simulation is expected to play an increasing role in operational test and evaluation. Although this expectation is driven by many factors, the primary rationale supporting the use of M&S in OT is the ability to mitigate safety, security, and budgetary constraints. However, the use of M&S in OT comes with significant challenges, the most significant from a T&E perspective being Verification, Validation, and Accreditation (VV&A). Adding to this challenge, current VV&A processes vary widely in scope, resourcing, and overall level of statistical/technical rigor. In alignment with March 2016 and January 2017 DOT&E memos, COMOPTEVFOR is currently updating VV&A guidance that will affect all M&S efforts supporting Navy operational test and evaluation. This updated guidance relies on basic V&V concepts and Design of Experiments methodologies to establish a data-driven foundation supporting current and future VV&A efforts. This presentation highlights the rationale, methodology, and expected impacts of the updated M&S VV&A guidance.
Validation of AIM-9X Modeling and Simulation
(Rigorous Validation of Modeling and Simulation)
Dr. Rebecca Dickinson
Institute for Defense Analyses
Tuesday, April 4th
Session II: 1030-1200
One use for Modeling and Simulation (M&S) in Test and Evaluation (T&E) is to produce weapon miss distances to evaluate the effectiveness of a weapons. This is true for the Air Intercept Missile-9X (AIM-9X) T&E community. Since flight testing is expensive, the test program uses relatively few flight tests at critical conditions, and supplements those data with large numbers of miss distances from simulated tests across the weapons operational space. However, before the model and simulation is used to predict performance it must first be validated. Validation is an especially daunting task when working with a limited number of live test data. In this presentation we shown that even with a limited number of live test points (e.g. 16 missile fires), we can still perform a statistical analysis for the validation. Specifically, we introduce a validation technique known as Fisher’s Combined Probability Test and we show how to apply Fisher’s test to validate the AIM-9X model and simulation.
Software Reliability Modeling
(Reliability Research)
Dr. Lance Fiondella
UMASS
Tuesday, April 4th
Session III: 1030-1200
Many software reliability models characterize the number of faults detected during the testing process as a function of testing time, which is performed over multiple stages. Typically, the later stages are progressively more expensive because of the increased number of personnel and equipment required to support testing as the system nears completion. Such transitions from one stage of testing to the next change in the operational environment. One statistical approach to combine software reliability growth models in a manner capable of characterizing multi-stage testing is the concept of a change-point process, where the intensity of a process experiences a distinct change at one or more discrete times during testing. Thus, change-point processes can be used to model change in the failure rate of software due to changes in the testing strategy and environment, integration testing, and resource allocation as it proceeds through multiple stages of testing.

This presentation generalizes change-point models to the heterogeneous case, where fault detection before and after a change-point can be characterized by distinct nonhomogeneous Poisson processes (NHPP). Experimental results suggest that heterogeneous change-point models better characterize some failure data sets, which can improve the applicability of software reliability models to large-scale software systems that are tested over multiple stages.
Reliability Growth Modeling
(Reliability Research)
Dr. Kelly Sullivan
University of Arkansas
Tuesday, April 4th
Session III: 1030-1200
Several optimization models are described for allocating resources to different testing activities in a system’s reliability growth program. These models assume availability of an underlying reliability growth model for the system, and capture the tradeoffs associated with focusing testing resources at various levels (e.g., system, subsystem, component) and/or how to divide resources within a given level. In order to demonstrate insights generated by solving the model, we apply the optimization models to an example series-parallel system in which reliability growth is assumed to follow the Crow/AMSAA reliability growth model. We then demonstrate how the optimization models can be extended to incorporate uncertainty in Crow/AMSAA parameters.
Updating R and Reliability Training with Bill Meeker
(Reliability Research)
Dr. Jason Freels
AFIT
Tuesday, April 4th
Session III: 1030-1200
Since its publication, Statistical Methods for Reliability Data by W. Q. Meeker and L. A. Escobar has been recognized as a foundational resource in analyzing failure time to and survival data. Along with the text, the authors provided an S-Plus software package, called SPLIDA, to help readers utilize the methods presented in the text. Today, R is the most popular statistical computing language in the world, largely supplanting S-Plus. The SMRD package is the result of a multi-year effort to completely rebuild SPLIDA, to take advantage of the improved graphics and workflow capabilities available in R. This presentation introduces the SMRD package, outlines the improvements and shows how the package works seamlessly with the rmarkdown and shiny packages to dramatically speed up your workflow. The presentation concludes with a discussion on what improvements still need to be made prior to publishing the package on the CRAN.
Introduction to Bayesian Statistics
(Tutorial)
Dr. Alyson Wilson
North Carolina State University
Tuesday, April 4th
Tutorial I: 1030-1200
One of the most powerful features of Bayesian analyses is the ability to combine multiple sources of information in a principled way to perform inference. For example, this feature can be particularly valuable in assessing the reliability of systems where testing is limited for some reason (e.g., expense, treaty). At their most basic, Bayesian methods for reliability develop informative prior distributions using expert judgment or similar systems. Appropriate models allow the incorporation of many other sources of information, including historical data, information from similar systems, and computer models. I will introduce the approach and then consider examples from defense acquisition and lifecycle extension, focusing on the strengths and weaknesses of the Bayesian analyses.
How do the Framework and Design of Experiments Fundamentally Help?
(Statistical Engineering for Decision Making)
Dr. Luis A. Cortes and Mr. Mike Sheeha
The MITRE
Tuesday, April 4th
Session I: 1345-1515
The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals.
The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface.
When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help.
This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.
Allocating Information Gathering Efforts for Selection Decisions
(Statistical Engineering for Decision Making)
Dr. Dennis Leber,
NIST and Dr. Jeffrey Herrmann,
University of Maryland
Tuesday, April 4th
Session I: 1345-1515
Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.
Statistical Methods for Programmatic Assessment and Anomaly Detection
(Statistical Engineering for Decision Making)
Mr. Douglas Brown and Dr. Ray McCollum
BAH
Tuesday, April 4th
Session I: 1345-1515


Increasing data base size and scope have enabled a high degree of multidimensional data. As a result multidimensional data is the new norm in data analysis. This presentation will focus on two uses of high dimensional data, Risk Analysis (MRISK) and outlier detection, multivariate analysis of stealth quantities (MASQ).
MRISK (Multivariate Risk Analysis)
Informed decision making rests on the assessment of potential risks which allow projects and programs to focus their efforts on effective risk management. Since the 1990 risk consequences have contained multiple dimensions. This presentation focuses on risks with multidimensional consequences and the methods available for accurately analyzing risk in the presence of multiple dimensions. Multidimensional Risk (MRISK) is an approach based in Statistical science that accounts for the complexity of relationships in multi-dimensional space.
MASQ (Multidimensional Analysis of Stealth Quantities)
Outlier detection in highly dimensional data promises to solve many problems in areas such as fraud detection, non-destructive evaluation, cybersecurity, etc. Outliers exist in multidimensional space. The pitfalls of Simpsons Paradox and similar fallacies can disguise and hide outliers when looking at one variable subspaces. A new approach is proposed to employ K-Means clustering, linear regression, and expert knowledge in a self-learning outlier detection scheme to identify outliers within large amounts data based solely on their current characteristics, rather than any previously defined business rules. This approach has been tested on highly dimensional data related to malicious traffic detection, financial audit readiness, and non-destructive evaluation applications for U.S. government clients.
Flight Test and Evaluation of Airborne Spacing Application
(Experimental Design Applications in Flight and Classification)
Dr. Sara Wilson
NASA
Tuesday, April 4th
Session II: 1345-1515
NASA’s Airspace Technology Demonstration (ATD) project was developed to facilitate the transition of mature air traffic management technologies from the laboratory to operational use. The first ATD focused on an integrated set of advanced NASA technologies to enable efficient arrival operations in high-density terminal airspace. This integrated arrival solution was validated and verified in laboratories and transitioned to a field prototype for an operational demonstration. Within NASA, this was a collaborative effort between Ames and Langley Research Centers involving a multi-year iterative experimentation process consisting of a series of sequential batch computer simulations and human-in-the-loop experiments, culminating in a flight test. Designing and analyzing the flight test involved a number of statistical challenges. There were several variables which are known to impact the performance of the system, but which could not be controlled in an operational environment. Changes in the schedule due to weather and the dynamic positioning of the aircraft on the arrival routes resulted in the need for a design that could be modified in real-time. This presentation describes a case study from a recent NASA flight test, highlights statistical challenges, and discusses lessons learned.
Testing and Estimation in Sequential High-Dimension Data
(Experimental Design Applications in Flight and Classification)
Dr. Eric Chicken
Florida State University
Tuesday, April 4th
Session II: 1345-1515
Many modern processes generate complex data records not readily analyzed by traditional techniques. For example, a single observation from a process might be a radar signal consisting of n pairs of bivariate data described via some functional relation between reflection and direction. Methods are examined here for detecting changes in such sequences from some known or estimated nominal state. Additionally, estimates of the degree of change (scale, location, frequency, etc.) are desirable and discussed. The proposed methods are designed to take advantage of all available data in a sequence. This can become unwieldy for long sequences of large-sized observations, so dimension reduction techniques are needed. In order for these methods to be as widely applicable as possible, we make limited distributional assumptions and so we propose new nonparametric and Bayesian tools to implement these estimators.
Blast Noise Event Classification from a Spectrogram
(Experimental Design Applications in Flight and Classification)
Dr. Edward Nykaza
Army Engineering Research and Development Center, Construction Engineering Research Laboratory
Tuesday, April 4th
Session II: 1345-1515
Spectrograms (i.e., squared magnitude of short-time Fourier transform) are commonly used as features to classify audio signals in the same way that social media companies (e.g., Google, Facebook, Yahoo) use images to classify or automatically tag people in photos. However, a serious problem arises when using spectrograms to classify acoustic signals, in that the user must choose the input parameters (hyperparameters), and such choices can have a drastic effect on the accuracy of the resulting classifier. Further, considering all possible combinations of the hyperparameters is a computationally intractable problem. In this study, we simplify the problem making it computationally tractable, explore the utility of response surface methods for sampling the hyperparameter space, and find that response surface methods are a computationally efficient means of identifying the hyperparameter combinations that are likely to give the best classification results.
Automated Software Testing Best Practices and Framework: A STAT COE Project
(Software Testing)
Dr. Jim Simpson
JK Analytics
Dr. Jim Wisnowski
Adsurgo
Tuesday, April 4th
Session III: 1345-1515
The process for testing military systems which are largely software intensive involves techniques and procedures often different from those for hardware-based systems. Much of the testing can be performed in laboratories at many of the acquisition stages, up to operational testing. Testing software systems is not different from testing hardware-based systems in that testing earlier and more intensively benefits the acquisition program in the long run. Automated testing of software systems enables more frequent and more extensive testing, allowing for earlier discovery of errors and faults in the code. Automated testing is beneficial for unit, integrated, functional and performance testing, but there are costs associated with automation tool license fees, specialized manpower, and the time to prepare and maintain the automation scripts. This presentation discusses some of the features unique to automated software testing and offers a framework organizations can implement to make the business case for, to organize for, and to execute and benefit from automating the right aspects of their testing needs. Automation has many benefits in saving time and money, but is most valuable in freeing test resources to perform higher value tasks.
Software Test Techniques
(Software Testing)
Dr. Jose Calderon
Galois
Tuesday, April 4th
Session III: 1345-1515
In recent years, software testing techniques based on formal methods have made their way into industrial practice as a supplement to system and unit testing. I will discuss three core techniques that have proven particularly amenable to transition: 1) Concolic execution, which enables the automatic generation of high-coverage test suites; 2) Property-based randomized testing, which automatically checks sequences of API calls to ensure that expected high-level behavior occurs; and 3) Bounded model checking, which enables systematic exploration of both concrete systems and high-level models to check temporal properties, including ordering of events and timing requirements.
Combinational Testing
(Software Testing)
Dr. Raghu Kacker and Dr. Rick Kuhn
NIST
Tuesday, April 4th
Session III: 1345-1515
Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced.
Introduction to Human Measurement
(Tutorial)
Dr. Cynthia Null
NASA
Tuesday, April 4th
Tutorial II: 1345-1515
Abstract: Coming Soon!
The (Empirical) Case for Analyzing Likert-Type Data with Parametric Tests
(Human Factors in Test)
Dr. Heather Wojton
Institute for Defense Analyses
Tuesday, April 4th
Session I: 1545-1645
Surveys are commonly used to evaluate the quality of human-system interactions during the operational testing of military systems. Testers use Likert-type response options to measure the intensity of operators’ subjective experiences (e.g., usability, workload) while operating the system. Recently, appropriate methods for analyzing Likert data have become a point of contention within the operational test community. Some argue that Likert data can be analyzed with parametric techniques whereas others argue that only non-parametric techniques should be used. However, the reasons stated for holding a particular view are rarely tied to findings in the empirical literature. This presentation sheds light on this debate by reviewing existing research on how parametric statistics affect the conclusions drawn from Likert data and debunk common myths and misunderstandings about the nature of Likert data within the operational test community and academia.
The System Usability Scale: A measurement Instrument Should Suit the Measurement Needs
(Human Factors in Test)
Mr. Keith Kidder
(Det 4), AFOTEC
Tuesday, April 4th
Session I: 1545-1645
The System Usability Scale (SUS) was developed by John Brooke in 1986 “to take a quick measurement of how people perceived the usability of (office) computer systems on which they were working.” The SUS is a 10-item, generic usability scale that is assumed to be system agnostic, and it results in a numerical score that ranges from 0-100. It has been widely employed and researched with non-military systems. More recently, it has been strongly recommended for use with military systems in operational test and evaluation, in part because of its widespread commercial use, but largely because it produces a numerical score that makes it amendable to statistical operations.
Recent lessons learned with SUS in operational test and evaluation strongly question its use with military systems, most of which differ radically from non-military systems. More specifically, (1) usability measurement attributes need to be tailored to the specific system under test and meet the information needs of system users, and (2) a SUS numerical cutoff score of 70—a common benchmark with non-military systems—does not accurately reflect “system usability” from an operator or test team perspective. These findings will be discussed in a psychological and human factors measurement context, and an example of system-specific usability attributes will be provided as a viable way forward. In the event that the SUS is used in operational test and evaluation, some recommendations for interpreting the outcomes will be provided.
Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation
(Panel Session)
Mr. Scott West, NESC JSC; Mr. Martin Annett, NASA and Dr. James Womack,The Aerospace CorporationTuesday, April 4th
Session II: 1545-1645
Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.
Machine Learning: Overview and Applications to Test
(Testing Autonomy)
Lt Takayuki Iguchi and Lt Megan Lewis
(Det 5), AFOTEC
Tuesday, April 4th
Session III: 1545-1645
“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning.

Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”
Range Adversarial Planning Tool for Autonomy Test and Evaluation
(Testing Autonomy)
Dr. Chad Hawthorne
JHU/APL
Tuesday, April 4th
Session III: 1545-1645
Abstract: Coming Soon!
Response Surface Designs: A Mars Rover Vibration Case Study and Panel Discussion
(Practitioners Session)

Tuesday, April 4th
Tutorial III: 1545-1645
NASA Jet Propulsion Lab engineers had to select a design strategy to characterize the performance of the Rover using multiple responses and 6 factors of interest. Engineering principles and previous testing suggested there would likely be significant main, two-factor interaction, and quadratic effects, albeit in a relatively noisy system. This interactive panel session will first introduce the vibration testing problem and offer possible solutions that include classical sequential response surface designs and the more recent definitive screening designs. Panel experts from DoD, NASA, and industry will provide insights, lessons learned, and recommendations that balance theoretical and practical considerations for test design, execution, and analysis. Topics will include power, sample size, prediction quality, testability, aliasing, test and design constraints, testability, and other important metrics to consider when weighing design choices. This session will encourage audience participation and questions.
Search for Extended Test Design Methods for Complex Systems of Systems
(Closing Leadership Perspective)
Dr. Alex Alaniz
AFOTEC
Tuesday, April 4th
Closing Leadership Perspective: 1700-1730
Abstract: Coming Soon!
The Future of Engineering at NASA Langley
(Closing Leadership Perspective)
Mr. Joe Gasbarre
NASA
Tuesday, April 4th
Closing Leadership Perspective: 1700-1730
In May 2016, the NASA Langley Research Center’s Engineering Director stood up a group consisting of employees within the directorate to assess the current state of engineering being done by the organization. The group was chartered to develop ideas, through investigation and benchmarking of other organizations within and outside of NASA, for how engineering should look in the future. This effort would include brainstorming, development of recommendations, and some detailed implementation plans which could be acted upon by the directorate leadership as part of an enduring activity. The group made slow and sporadic progress in several specific, self-selected areas including: training and development; incorporation of non-traditional engineering disciplines; capturing and leveraging historical data and knowledge; revolutionizing project documentation; and more effective use of design reviews.
The design review investigations have made significant progress by leveraging lessons learned and techniques gained by collaboration with operations research analysts within the local Lockheed Martin Center for Innovation (the “Lighthouse”) and pairing those techniques with advanced data analysis tools available through the IBM Watson Content Analytics environment. Trials with these new techniques are underway but show promising results for the future of providing objective, quantifiable data from the design review environment – an environment which to this point has remained essentially unchanged for the past 50 years.
Combinatorial Testing for Link-16 Developmental Test and Evaluation
(Innovative Statistical Solutions for Common Test Challenges)
Mr. Tim McLean
MCTSSA
Wednesday, April 5th
Session I: 1030-1200
Due to small Tactical Data Link testing windows, only commonly used messages are tested resulting in the evaluation of only a small subset of all possible Link 16 messages. To increase the confidence that software design and implementation issues are discovered in the earliest phases of government acceptance testing, Marine Corps Tactical Systems Support Activity (MCTSSA) Instrumentation and Data Management Section (IDMS) successfully implemented an extension of the traditional form of Design of Experiments (DOE), called Combinatorial Testing (CT). CT was utilized to reduce the human bias and inconsistencies involved in Link 16 testing and replace them with a thorough test that can validate a system's ability to properly consume all of the possible valid combinations of Link 16 message field values. MCTSSA's unique team of subject matter experts was able to bring together the tenants of virtualization, automation, C4I Air systems testing, tactical data link testing, and Design of Experiments methodology, to invent a testing paradigm that will exhaustively evaluate tactical Air systems. This presentation will give an overview of how CT was implemented for the test.
Censored Data Analysis for Performance Data
(Innovative Statistical Solutions for Common Test Challenges)
Dr. Bram Lillard, Institute for Defense AnalysesWednesday, April 5th
Session I: 1030-1200
Binomial metrics like probability-to-detect or probability-to-hit typically provide operationally meaningful and easy to interpret test outcomes. However, they are information poor metrics and extremely expensive to test. The standard power calculations to size a test employ hypothesis tests, which typically result in many tens to hundreds of runs. In addition to being expensive, the test is most likely inadequate for characterizing performance over a variety of conditions due to the inherently large statistical uncertainties associated with binomial metrics. A solution is to convert to a continuous variable, such as miss distance or time-to-detect. The common objection to switching to a continuous variable is that the hit/miss or detect/non-detect binomial information is lost, when the fraction of misses/no-detects is often the most important aspect of characterizing system performance. Furthermore, the new continuous metric appears to no longer be connected to the requirements document, which was stated in terms of a probability. These difficulties can be overcome with the use of censored data analysis. This presentation will illustrate the concepts and benefits of this approach, and will illustrate a simple analysis with data, including power calculations to show the cost savings for employing the methodology.
Estimating the Distribution of an Extremum using a Peaks-Over-Threshold Model and Monte Carlo Simulation
(Innovative Statistical Solutions for Common Test Challenges)
Dr. Adam Pintar
NIST
Wednesday, April 5th
Session I: 1030-1200
Estimating the probability distribution of an extremum (maximum or minimum), for some fixed amount of time, using a single time series typically recorded for a shorter amount of time, is important in many application areas, e.g., structural design, reliability, quality, and insurance. When designing structural members, engineers are concerned with maximum wind effects, which are functions of wind speed. With respect to reliability and quality, extremes experienced during storage or transport, e.g., extreme temperatures, may substantially impact product quality, lifetime, or both. Insurance companies are of course concerned about very large claims.

In this presentation, a method to estimate the distribution of an extremum using a well-known peaks-over-threshold (POT) model and Monte Carlo simulation is presented. Since extreme values have long been a subject of study, some brief history is first discussed. The POT model that underlies the approach is then laid out. A description of the algorithm follows. It leverages pressure data collected on scale models of buildings in a wind tunnel for context. Essentially, the POT model is fitted to the observed data and then used to simulate many times series of the desired length. The empirical distribution of the extrema is obtained from the simulated series. Uncertainty in the estimated distribution is quantified by a bootstrap algorithm. Finally, an R package implementing the computations is discussed.
Design & Analysis of a Computer Experiment for an Aerospace Conformance Simulation Study
(Advancing Experimental Design Research)
Dr. David Edwards
Virginia Commonwealth University
Wednesday, April 5th
Session II: 1030-1200
Within NASA's Air Traffic Management Technology Demonstration # 1 (ATD-1), Interval Management (IM) is a flight deck tool that enables pilots to achieve or maintain a precise in-trail spacing behind a target aircraft. Previous research has shown that violations of aircraft spacing requirements can occur between an IM aircraft and its surrounding non-IM aircraft when it is following a target on a separate route. This talk focuses on the experimental design and analysis of a computer experiment which models the airspace configuration of interest in order to determine airspace/aircraft conditions leading to spacing violations during IM operation. We refer to multi-layered nested continuous factors as those that are continuous and ordered in their selection; they can only be selected sequentially with a level selected for one factor affecting the range of possible values for each subsequently nested factor. While each factor is nested within another factor, the exact nesting relationships have no closed form solution. In this talk, we describe our process of engineering an appropriate space-filling design for this situation. Using this space-filling design and Gaussian process modeling, we found that aircraft delay assignments and wind profiles significantly impact the likelihood of spacing violations and the interruption of IM operations.
Optimal Multi-Response Designs
(Advancing Experimental Design Research)
Dr. Sarah Burke
STAT COE
Wednesday, April 5th
Session II: 1030-1200
The problem of constructing a design for an experiment when multiple responses are of interest does not have a clear answer, particularly when the response variables are of different types. Planning an experiment for an air-to-air missile simulation, for example, might have the following responses simultaneously: hit or miss the target (a binary response) and the time to acquire the target (a continuous response). With limited time and resources and only one experiment possible, the question of selecting an appropriate design to model both responses is important. In this presentation, we discuss a method for creating designs when two responses, each with a different distribution (normal, binomial, or Poisson), are of interest. We demonstrate the proposed method using various weighting schemes for the two models to show how the designs change as the weighting scheme changes. In addition, we explore the effect of the specified priors for the nonlinear models on these designs.
Augmenting Definitive Screening Designs
(Advancing Experimental Design Research)
Ms. Abby Nachtsheim
ASU
Wednesday, April 5th
Session II: 1030-1200
Jones and Nachtsheim (2011) introduced a class of three-level screening designs called definitive screening designs (DSDs). The structure of these designs results in the statistical independence of main effects and two-factor interactions; the absence of complete confounding among two-factor interactions; and the ability to estimate all quadratic effects. Because quadratic effects can be estimated, DSDs can allow for the screening and optimization of a system to be performed in one step, but only when the number of terms found to be active during the screening phase of analysis is less than about half the number or runs required by the DSD (Errore, et al., 2016). Otherwise, estimation of second-order models requires augmentation of the DSD. In this paper we explore the construction of series of augmented designs, moving from the starting DSD to designs capable of estimating the full second-order model. We use power calculations, model-robustness criteria, and model-discrimination criteria to determine the number of runs by which to augment in order to identify the active second-order effects with high probability.
Integrated Uncertainty Quantification for Risk and Resource Management: Building Confidence in Design
(CFD Vision 2030: Modeling and Simulation)
Dr. Eric Walker
NASA
Wednesday, April 5th
Session III: 1030-1200
Abstract: Coming Soon!
Background of NASA’s Juncture Flow Validation Test
(CFD Vision 2030: Modeling and Simulation)
Mr. Joseph Morrison
NASA
Wednesday, April 5th
Session III: 1030-1200
Abstract: Coming Soon!
A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime
(CFD Vision 2030: Modeling and Simulation)
Dr. Eric Walker and Mr. Joseph Morrison
NASA
Wednesday, April 5th
Session III: 1030-1200
Abstract: Coming Soon!
Communicating Complex Statistical Methodologies to Leadership
(Tutorial)
Dr. Jane Pinelis
Johns Hopkins University Applied Physics Lab or JHU / APL Mr. Paul Johnson
MCOTEA
Wednesday, April 5th
Tutorial I: 1030-1200
More often than not, the data we analyze for the military is plagued with statistical issues. Multicollinearity, small sample sizes, quasi-experimental designs, and convenience samples are some examples of what we commonly see in military data. Many of these complications can be resolved either in the design or analysis stage with appropriate statistical procedures. But, to keep our work useful, usable, and transparent to the military leadership who sponsors it, we must strike the elusive balance between explaining and justifying our design and analysis techniques and not inundating our audience with unnecessary details. It can be even more difficult to get military leadership to understand the statistical problems and solutions so well that they are enthused and supportive of our approaches. Using literature written on the subject as well as a variety of experiences, we will showcase several examples, as well as present ideas for keeping our clients actively engaged in statistical methodology discussions.
DOE Case Studies in Aerospace Research and Development
(Aerospace Experimental Design in Practice)
Dr. Drew Landman
Old Dominion University
Wednesday, April 5th
Session I: 1345-1515
This presentation will provide a high level view of recent DOE applications to aerospace research. Two broad categories are defined, aerodynamic force measurement system calibrations and aircraft model wind tunnel aerodynamic characterization. Each case study will outline the application of DOE principles including design choices, accommodations for deviations from classical DOE approaches, discoveries, and practical lessons learned.

Case Studies

Aerodynamic Force Measurement System Calibrations

Large External Wind Tunnel Balance Calibration
- Fractional factorial
- Working with non-ideal factor settings
- Customer driven uncertainty assessment

Internal Balance Calibration Including Temperature
- Restrictions to randomization – split plot design requirements
- Constraints to basic force model
- Crossed design approach

Aircraft Model Wind Tunnel Aerodynamic Characterization

The NASA/Boeing X-48B Blended Wing Body Low-Speed Wind Tunnel Test
- Overcoming a culture of OFAT
- General approach to investigating a new aircraft configuration
- Use of automated wind tunnel models and randomization
- Power of residual analysis in detecting problems

NASA GL–10 UAV Aerodynamic Characterization
- Use of the Nested-Face Centered Design for aerodynamic characterization
- Issues working with over 20 factors
- Discoveries
Sample Size and Considerations for Statistical Power
(Aerospace Experimental Design in Practice)
Mr. Vance Oas and Mr. Nick Garcia
(HQ A-2/9), AFOTEC-
Wednesday, April 5th
Session I: 1345-1515
Sample size drives the resources and supports the conclusions of operational test. Power analysis is a common statistical methodology used in planning efforts to justify the number of samples. Power analysis is sensitive to extreme performance (e.g. 0.1% correct responses or 99.999% correct responses) relative to a threshold value, extremes in response variable variability, numbers of factors and levels, system complexity, and a myriad of other design- and system-specific criteria. This discussion will describe considerations (correlation/aliasing, operational significance, thresholds, etc.) and relationships (design, difference to detect, noise, etc.) associated with power. The contribution of power to design selection or adequacy must often be tempered when significant uncertainty or test resources constraints exist. In these situations, other measures of merit and alternative analytical approaches become at least as important as power in the development of designs that achieve the desired technical adequacy. In conclusion, one must understand what power is, what factors influence the calculation, and when to leverage alternative measures of merit.
Experimental Design for Composite Pressure Vessel Life Prediction
(Aerospace Experimental Design in Practice)
Dr. Anne Driscoll
Virginia Tech
Wednesday, April 5th
Session I: 1345-1515
One of the major pillars of experimental design is sequential learning. The experimental design should not be viewed as a “one-shot” effort, but rather as a series of experiments where each stage builds upon information learned from the previous study. It is within this realm of sequential learning that experimentation soundly supports the application of the scientific method.
This presentation illustrates the value of sequential experimentation and also the connection between the scientific method and experimentation through a discussion of a multi-stage project supported by NASA’s Engineering Safety Center (NESC) where the objective was to assess the safety of composite overwrapped pressure vessels (COPVs). The analytical team was tasked with devising a test plan to model stress rupture failure risk in carbon fiber strands that encase the COPVs with the goal of understanding the reliability of the strands at use conditions for the expected mission life. This presentation highlights the recommended experimental design for the strand tests and then discusses the benefits that resulted from the suggested sequential testing protocol.
Improving Sensitivity Experiments
(Sensitivity and Reliability)
Mr. Douglas Ray
US Army RDECOM ARDEC and Mr. Kevin Singer
US Army
Wednesday, April 5th
Session II: 1345-1515
This presentation will provide a brief overview of sensitivity testing, and emphasize applications to several products and system of importance to the Defense as well as private industry, including Insensitive Energetics, Ballistic testing of protective armor, testing of munition fuzes and Microelectromechanical Systems (MEMS) components, and safety testing of high-pressure test ammunition, and packaging for high-value materials.
Sequential Experimentation for a Binary Response - The Break Separation Method
(Sensitivity and Reliability)
Mr. Darsh Thakkar and Dr. Rachel Silvestrini
RIT- S
Wednesday, April 5th
Session II: 1345-1515
Binary response experiments are common in epidemiology, biostatistics as well as in military applications. The Up and Down method, Langlie’s Method, Neyer’s method, K in a Row method and 3 Phase Optimal Design are methods used for sequential experimental design when there is a single continuous variable and a binary response. During this talk, we will discuss a new sequential experimental design approach called the Break Separation Method (BSM). BSM provides an algorithm for determining sequential experimental trials that will be used to find a median quantile and fit a logistic regression model using Maximum Likelihood estimation. BSM results in a small sample size and is designed to efficiently compute the median quantile.
Carrier Reliability Model Validation
(Sensitivity and Reliability)
Dr. Dean Thomas
Institute for Defense Analyses
Wednesday, April 5th
Session II: 1345-1515
Model Validation for Simulations of CVN-78 Sortie Generation

As part of the test planning process, IDA is examining flight operations on the Navy’s newest carrier, CVN-78. The analysis uses a model, the IDA Virtual Carrier Model (IVCM), to examine sortie generation rates and whether aircraft can complete missions on time. Before using IVCM, it must be validated. However, CVN-78 has not been delivered to the Navy, and data from actual operations are not available to validate the model. Consequently, we will validate IVCM by comparing it to another model. This is a reasonable approach when a model is used in general analyses such as test planning, but is not acceptable when a model is used in the assessment of system effectiveness and suitability. The presentation examines the use of various statistical tools – Wilcoxon Rank Sum Test, Kolmogorov-Smirnov Test, and lognormal regression – to examine whether the results from two models provide similar results and to quantify the magnitude of any differences. From the analysis, IDA concluded that locations and distribution shapes are consistent, and that the differences between the models are less than 15 percent, which is acceptable for test planning.
Model Based Systems Engineering Panel Discussion
Mr. Jon Holladay
NASA
Panel Members:
Ms. Kim Simpson, NASA;
Dr. David Richardson, NASA; Ms. Philomena "Phil" Zimmerman, ODASD(SE)/SA; Mr. Scott Lucero, DoD, Systems Engineering
Wednesday, April 5th
Session III: 1345-1515
This panel will share status, experiences and expectations within DoD and NASA for transitioning Systems Engineering to a more integrated digital engineering domain. A wide range of perspectives will be provided, covering the implementation waterfront of practitioner, management, research and strategy. Panelist will also be prepared to discuss more focused areas of digital systems engineering, such as test and evaluation, and engineering statistics.
Resampling Methods
(Tutorial)
Dr. David Ruth
United States Naval Academy
Wednesday, April 5th
Tutorial II: 1345-1515
Resampling Methods: This tutorial presents widely used resampling methods to include bootstrapping, cross-validation, and permutation tests. Underlying theories will be presented briefly, but the primary focus will be on applications. A new graph-theoretic approach to change detection will be discussed as a specific application of permutation testing. Examples will be demonstrated in R; participants are encouraged to bring their own portable computers to follow along using datasets provided by the instructor.
Structured Decision Making
(Communicating Statistical Analysis to Decision Makers)
Dr. Christine Anderson-Cook
LANL
Wednesday, April 5th
Session I: 1545-1715
Difficult choices are often required in a decision-making process where resources and budgets are increasingly constrained. This talk demonstrates a structured decision-making approach using layered Pareto fronts to prioritize the allocation of funds between munitions stockpiles based on their estimated reliability, the urgency of needing available units, and the consequences if adequate numbers of units are not available. This case study illustrates the process of first identifying appropriate metrics that summarize important dimensions of the decision, and then eliminating non-contenders from further consideration in an objective stage. The final subjective stage incorporates subject matter expert priorities to select the four stockpiles to receive additional maintenance and surveillance funds based on understanding the trade-offs and robustness to various user priorities.
Do Asymmetries in Nuclear Arsenals Matter?
(Communicating Statistical Analysis to Decision Makers)
Dr. Jane Pinelis, JHU/APL, Co-Author- Dr. James Scouras, JHU- Wednesday, April 5th
Session I: 1545-1715
The importance of the nuclear balance vis-a-vis our principal adversary has been the subject of intense but unresolved debate in the international security community for almost seven decades. Perspectives on this question underlie national security policies regarding potential unilateral reductions in strategic nuclear forces, the imbalance of nonstrategic nuclear weapons in Europe, nuclear crisis management, nuclear proliferation, and nuclear doctrine.

The overwhelming majority of past studies of the role of the nuclear balance in nuclear crisis evolution and outcome have been qualitative and focused on the relative importance of the nuclear balance and national resolve. Some recent analyses have invoked statistical methods, however, these quantitative studies have generated intense controversy because of concerns with analytic rigor. We apply a multi-disciplinary approach that combines historical case study, international relations theory, and appropriate statistical analysis. This approach results in defensible findings on causal mechanisms that regulate nuclear crisis resolution. Such findings should inform national security policy choices facing the Trump administration.
Communication in Statistics & the Five Hardest Concepts
(Communicating Statistical Analysis to Decision Makers)
Dr. Jennifer Van-Mellekom
Virginia Tech
Wednesday, April 5th
Session I: 1545-1715
Abstract: Coming Soon!
Uncertainty Quantification: What is it and Why it is Important to Test, Evaluation, and Modeling and Simulation in Defense and Aerospace
(Uncertainty Quantification)
Dr. Peter Qian
University of Wisconsin and SmartUQ
Wednesday, April 5th
Session II: 1545-1715
Uncertainty appears in many aspects of systems design including stochastic design parameters, simulation inputs, and forcing functions. Uncertainty Quantification (UQ) has emerged as the science of quantitative characterization and reduction of uncertainties in both simulation and test results. UQ is a multidisciplinary field with a broad base of methods including sensitivity analysis, statistical calibration, uncertainty propagation, and inverse analysis. Because of their ability to bring greater degrees of confidence to decisions, uncertainty quantification methods are playing a greater role in test, evaluation, and modeling and simulation in defense and aerospace. The value of UQ comes with better understanding of risk from assessing the uncertainty in test and modeling and simulation results.
The presentation will provide an overview of UQ and then discuss the use of some advanced statistical methods, including DOEs and emulation for multiple simulation solvers and statistical calibration, for efficiently quantifying uncertainties. These statistical methods effectively link test, evaluation and modeling and simulation by coordinating the valuation of uncertainties, simplifying verification and validation activities.
Model Uncertainty and its Inclusion in Testing Results
(Uncertainty Quantification)
Dr. Steve Lund
NIST
Wednesday, April 5th
Session II: 1545-1715
Answers to real world questions are often based on the use of judiciously chosen mathematical/statistical/physical models. In particular, assessment of failure probabilities of physical systems rely heavily on such models. Since no model describes the real world exactly, sensitivity analyses are conducted to examine influences of (small) perturbations of an assumed model. In this talk we present a structured approach, using an "Assumptions Lattice" and corresponding "Uncertainty Pyramid", for transparently conveying the influence of various assumptions on analysis conclusions. We illustrate this process in the context of a simple multicomponent system.
VV&UQ - Uncertainty Quantification for Model-Based Engineering of DoD Systems
(Uncertainty Quantification)
Mr. Douglas Ray
US Army RDECOM ARDEC and Ms. Melissa Jablonski
US Army
Wednesday, April 5th
Session II: 1545-1715
The US Army ARDEC has recently established an initiative to integrate statistical and probabilistic techniques into engineering modeling and simulation (M&S) analytics typically used early in the design lifecycle to guide technology development. DOE-driven Uncertainty Quantification techniques, including statistically rigorous model verification and validation (V&V) approaches, enable engineering teams to identify, quantify, and account for sources of variation and uncertainties in design parameters, and identify opportunities to make technologies more robust, reliable, and resilient earlier in the product’s lifecycle. Several recent armament engineering case studies - each with unique considerations and challenges - will be discussed.
Data Visualization
(Data Synthesis and Visualization with Big Data)
Ms. Lori Perkins
NASA
Wednesday, April 5th
Session III: 1545-1715
Teams of people with many different talents and skills work together at NASA to
improve our understanding of our planet Earth, our Sun and solar system, and the
Universe. The Earth System is made up of complex interactions and dependencies
of the solar, oceanic, terrestrial, atmospheric, and living components. Solar storms
have been recognized as a cause of technological problems on Earth since the invention
of the telegraph in the 19th century. Solar flares, coronal holes, and coronal mass
ejections (CME's) can emit large bursts of radiation, high speed electrons and
protons, and other highly energetic particles that are released from the sun, and are
sometimes directed at Earth. These particles and radiation can damage satellites in space,
shutdown power grids on earth, cause GPS outages, and have serious health concerns to humans
flying at high altitudes on earth, as well as astronauts in space. NASA builds and operates a fleet
of satellites to study the sun and a fleet of satellites and aircraft to observe the Earth system.
NASA’s Computer Models combine the observations with numerical models, to understand
how these systems work. Using satellite observations alongside computer models we can combine
many pieces of information to form a coherent view of Earth and the Sun. NASA research helps us
understand how processes combine to affect life on Earth: this includes severe weather, health,
changes in climate, and space weather. The Scientific Visualization Studio wants you to learn
about NASA programs through visualization. The SVS works closely with scientists
in the creation of data visualizations, animations, and images in order to promote
a greater understanding of Earth and Space Science research activities at NASA
and within the academic research community supported by NASA.
Big Data, Big Think
(Data Synthesis and Visualization with Big Data)
Mr. Robert Beil
NASA
Wednesday, April 5th
Session III: 1545-1715
The NASA Big Data, Big Think team jump-starts coordination, strategy, and progress for NASA applications of Big Data Analytics techniques, fosters collaboration and teamwork among centers and improves agency-wide understanding of Big Data research techniques & technologies and their application to NASA mission domains. The effort brings the Agency’s Big Data community together and helps define near term projects and leverages expertise throughout the agency. This presentation will share examples of Big Data activities from the Agency and discuss knowledge areas and experiences, including data management, data analytics and visualization.
Project Data Flow Is an Engineered System
(Data Synthesis and Visualization with Big Data)
Mr. Ken Johnson
NASA
Wednesday, April 5th
Session III: 1545-1715
Data within a project, investigation or test series are often seen as a bunch of numbers that were produced. While this is part of the story, it forgets the most important part: the data’s users. This more powerful process begins with early focus on planning, executing and managing data flow within a test or project as a system, treating each handoff between internal and external stakeholders a system interface. This presentation will persuade you why data production should be replaced by the idea of a data supply chain focused on goals and customers. The presenter will outline how this could be achieved in your team. The talk is aimed at not only project and data managers, but also team members who produce or use data. Retooling team thinking and processes along these lines will help communication, facilitate availability, display and understanding of data by any stakeholder, make data verification, validation and analysis easier, and help keep team members focused on what is necessary and important: solving the problem at hand.
Comparing Experimental Designs
(Tutorial)
Dr. Tom Donnelly
JMP
Wednesday, April 5th
Tutorial III: 1545-1715
This tutorial will show how to compare and choose experimental designs based on multiple criteria. Answers to questions like "Which Design of Experiments (DOE) is better/best?" will be answered by looking at both data and graphics that show the relative performance of the designs based on multiple criteria, including; power of the designs for different model terms, how well the designs minimize predictive variance across the design space, to what level are model terms confounded or correlated, what are the relative efficiencies that measure how well coefficients are estimated or how well predictive variance is minimized. Many different case studies of screening, response surface, and screening augmented to response surface designs will be compared. Designs with both continuous and categorical factors, and with constraints on the experimental region will also be compared.

See All Breakouts and Tutorials