Session Title | Speaker | Type | Recording | Materials | Year |
---|---|---|---|---|---|
Breakout “High Velocity Analytics for NASA JPL Mars Rover Experimental Design” (Abstract)
Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used. |
Jim Wisnowski Co-founder/Principle Adsurgo (bio)
James Wisnowski provides training and consulting services in Design of Experiments, Predictive Analytics, Reliability Engineering, Quality Engineering, Text Mining, Data Visualization, and Forecasting to government and industry. Previously, he spent a career in analytics for the government. He retired from the Air Force having had leadership positions at the Pentagon, Air Force Academy, Air Force Operational Test and Evaluation Center, and units across the Air Force. He has published numerous papers in technical journals and presented several invited conference presentations. He was co-author of the Design and Analysis of Experiments by Douglas Montgomery: A Supplement for using JMP. |
Breakout | Materials | 2016 |
|
Breakout “High Velocity Analytics for NASA JPL Mars Rover Experimental Design” (Abstract)
Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used. |
Heath Rushing Co-founder/Principle Adsurgo (bio)
Heath Rushing is the cofounder of Adsurgo and author of the book Design and Analysis of Experiments by Douglas Montgomery: A Supplement for using JMP. Previously, he was the JMP Training Manager at SAS, a quality engineer at Amgen, an assistant professor at the Air Force Academy, and a scientific analyst for OT&E in the Air Force. In addition, over the last six years, he has taught Science of Tests (SOT) courses to T&E organizations throughout the DoD. |
Breakout | Materials | 2016 |
|
Breakout 3D Mapping, Plotting, and Printing in R with Rayshader (Abstract)
Is there ever a place for the third dimension in visualizing data? Is the use of 3D inherently bad, or can a 3D visualization be used as an effective tool to communicate results? In this talk, I will show you how you can create beautiful 2D and 3D maps and visualizations in R using the rayshader package. Additionally, I will talk about the value of 3D plotting and how good aesthetic choices can more clearly communicate results to stakeholders. Rayshader is a free and open source package for transforming geospatial data into engaging visualizations using a simple, scriptable workflow. It provides utilities to interactively map, plot, and 3D print data from within R. It was nominated by Hadley Wickham to be one of 2018’s Data Visualizations of the Year for the online magazine Quartz. |
Tyler Morgan-Wall | Breakout | 2019 |
||
Breakout A 2nd-Order Uncertainty Quantification Framework Applied to a Turbulence Model Validation Effort (Abstract)
Computational fluid dynamics is now considered to be an indispensable tool for the design and development of scramjet engine components. Unfortunately, the quantification of uncertainties is rarely addressed with anything other than sensitivity studies, so the degree of confidence associated with the numerical results remains exclusively with the subject matter expert that generated them. This practice must be replaced with a formal uncertainty quantification process for computational fluid dynamics to play an expanded role in the system design, development, and flight certification process. Given the limitations of current hypersonic ground test facilities, this expanded role is believed to be a requirement by some in the hypersonics community if scramjet engines are to be given serious consideration as a viable propulsion system. The present effort describes a simple, relatively low cost, nonintrusive approach to uncertainty quantification that includes the basic ingredients required to handle both aleatoric (random) and epistemic (lack of knowledge) sources of uncertainty. The nonintrusive nature of the approach allows the computational fluid dynamicist to perform the uncertainty quantification with the flow solver treated as a “black box”. Moreover, a large fraction of the process can be automated, allowing the uncertainty assessment to be readily adapted into the engineering design and development workflow. In the present work, the approach is applied to a model scramjet isolator problem where the desire is to validate turbulence closure models in the presence of uncertainty. In this context, the relevant uncertainty sources are determined and accounted for to allow the analyst to delineate turbulence model-form errors from other sources of uncertainty associated with the simulation of the facility flow. |
Robert Baurle | Breakout |
![]() | 2019 |
|
Breakout A Causal Perspective on Reliability Assessment (Abstract)
Causality in an engineered system pertains to how a system output changes due to a controlled change or intervention on the system or system environment. Engineered systems designs reflect a causal theory regarding how a system will work, and predicting the reliability of such systems typically requires knowledge of this underlying causal structure. The aim of this work is to introduce causal modeling tools that inform reliability predictions based on biased data sources and illustrate how these tools can inform data integration in practice. We present a novel application of the popular structural causal modeling framework to reliability estimation in an engineering application, illustrating how this framework can inform whether reliability is estimable and how to estimate reliability using data integration given a set of assumptions about the subject matter and data generating mechanism. When data are insufficient for estimation, sensitivity studies based on problem-specific knowledge can inform how much reliability estimates can change due to biases in the data and what information should be collected next to provide the most additional information. We apply the approach to a pedagogical example related to a real, but proprietary, engineering application, considering how two types of biases in data can influence a reliability calculation. |
Lauren Hund | Breakout |
![]() | 2019 |
|
Breakout A Decision-Theoretic Framework for Adaptive Simulation Experiments (Abstract)
We describe a model-based framework for increasing effectiveness of simulation experiments in the presence of uncertainty. Unlike conventionally designed simulation experiments, it adaptively chooses where to sample, based on the value of information obtained. A Bayesian perspective is taken to formulate and update the framework’s four models. A simulation experiment is conducted to answer some question. In order to define precisely how informative a run is for answering the question, the answer must be defined as a random variable. This random variable is called a query and has the general form of p(theta | y), where theta is the query parameter and y is the available data. Examples of each of the four models employed in the framework are briefly described below: 1. The continuous correlated beta process model (CCBP) estimates the proportions of successes and failures using beta-distributed uncertainty at every point in the input space. It combines results using an exponentially decaying correlation function. The output of the CCBP is used to estimate value of a candidate run. 2. The mutual information model quantifies uncertainty in one random variable that is reduced by observing the other one. The model quantifies the mutual information between any candidate runs and the query, thereby scoring the value of running each candidate. 3. The cost model estimates how long future runs will take, based upon past runs using, e.g., a generalized linear model. A given simulation might have multiple fidelity options that require different run times. It may be desirable to balance information with the cost of a mixture of runs using these multi-fidelity options. 4. The grid state model, together with the mutual information model, are used to select the next collection of runs for optimal information per cost, accounting for current grid load. The framework has been applied to several use cases, including model verification and validation with uncertainty quantification (VVUQ). Given a mathematically precise query, an 80 percent reduction in total runs has been observed. |
Terril Hurst Senior Engineering Fellow Raytheon Technologies ![]() (bio)
Terril N Hurst is a Senior Engineering Fellow at Raytheon Technologies, where he works to ensure that model-based engineering is based upon credible models and protocols that allow uncertainty quantification. Prior to coming to Raytheon in 2005, Dr. Hurst worked at Hewlett-Packard Laboratories, including a post-doctoral appointment in Stanford University’s Logic-Based Artificial Intelligence Group under the leadership of Nils Nilsson. |
Breakout | Session Recording |
![]() Recording | 2022 |
Breakout A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge (Abstract)
This session will focus on the novel application of a design of experiments approach to optimize a propulsion charge configuration for a U.S. Army artillery round. The interdisciplinary design effort included contributions from subject matter experts in statistics, propulsion charge design, computational physics and experimentation. The process, which we will present in this session, consisted of an initial, low fidelity modeling and simulation study to reduce the parametric space by eliminating inactive variables and reducing the ranges of active variables for the final design. The final design used a multi-tiered approach that consolidated data from multiple sources including low fidelity modeling and simulation, high fidelity modeling and simulation and live test data from firings in a ballistic simulator. Specific challenges of the effort that will be addressed include: integrating data from multiple sources, a highly constrained design space, functional response data, multiple competing design objectives and real-world test constraints. The result of the effort is a final, optimized propulsion charge design that will be fabricated for live gun firing. |
Sarah Longo Data Scientist US Army CCDC Armaments Center ![]() (bio)
Sarah Longo is a data scientist in the US Army CCDC Armaments Center’s Systems Analysis Division. She has a background in Chemical and Mechanical Engineering and ten years experience in gun propulsion and armament engineering. Ms. Longo’s gun-propulsion expertise has played a part in enabling the successful implementation of Design of Experiments, Empirical Modeling, Data Visualization and Data Mining for mission-critical artillery armament and weapon system design efforts. |
Breakout |
![]() | 2021 |
|
Breakout A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge (Abstract)
This session will focus on the novel application of a design of experiments approach to optimize a propulsion charge configuration for a U.S. Army artillery round. The interdisciplinary design effort included contributions from subject matter experts in statistics, propulsion charge design, computational physics, and experimentation. The process, which we will present in this session, consisted of an initial, low fidelity modeling and simulation study to reduce the parametric space by eliminating inactive variables and reducing the ranges of active variables for the final design. The final design used a multi-tiered approach that consolidated data from multiple sources including low fidelity modeling and simulation, high fidelity modeling and simulation and live test data from firings in a ballistic simulator. Specific challenges of the effort that will be addressed include: integrating data from multiple sources, a highly constrained design space, functional response data, multiple competing design objectives, and real-world test constraints. The result of the effort is a final, optimized propulsion charge design that will be fabricated for live gun firing. |
Melissa Jablonski Statistician US Army CCDC Armaments Center ![]() (bio)
Melissa Jablonski is a statistician at the US Army Combat Capabilities Development Command Armaments Center. She graduated from Stevens Institute of Technology with a Bachelor’s and Master’s degree in Mechanical Engineering and started her career in the area of finite element analysis. She now works as a statistical consultant focusing in the areas of Design and Analysis of Computer Experiments (DACE) and Uncertainty Quantification (UQ). She also acts as a technical expert and consultant in Design of Experiments (DOE), Probabilistic System Optimization, Data Mining/Machine Learning, and other statistical analysis areas for munition and weapon systems. She is currently pursuing a Master’s degree in Applied Statistics from Pennsylvania State University. |
Breakout |
![]() | 2021 |
|
Breakout A Framework for Efficient Operational Testing through Bayesian Adaptive Design (Abstract)
When developing a system, it is important to consider system performance from a user perspective. This can be done through operational testing—assessing the ability of representative users to satisfactorily accomplish tasks or missions with the system in operationally-representative environments. This process can be expensive and time-consuming, but is critical for evaluating a system. We show how an existing design of experiments (DOE) process for operational testing can be leveraged to construct a Bayesian adaptive design. This method, nested within the larger design created by the DOE process, allows interim analyses using predictive probabilities to stop testing early for success or futility. Furthermore, operational environments with varying probabilities of encountering are directly used in product evaluation. Representative simulations demonstrate how these interim analyses can be used in an operational test setting, and reductions in necessary test events are shown. The method allows for using either weakly informative priors when data from previous testing is not available, or for priors built using developmental testing data when it is available. The proposed method for creating priors using developmental testing data allows for more flexibility in which data can be incorporated into analysis than the current process does, and demonstrates that it is possible to get more precise parameter estimates. This method will allow future testing to be conducted in less time and at less expense, on average, without compromising the ability of the existing process to verify the system meets the user’s needs. |
Victoria Sieck Student / Operations Research Analyst University of New Mexico / Air Force Institute of Technology ![]() (bio)
Victoria R.C. Sieck is a PhD Candidate in Statistics at the University of New Mexico. She is also an Operations Research Analyst in the US Air Force (USAF), with experiences in the USAF testing community as a weapons and tactics analyst and an operational test analyst. Her research interests include design of experiments and improving operational testing through the use of Bayesian methods. |
Breakout |
![]() | 2021 |
|
A Framework for Using Priors in a Continuum of Testing (Abstract)
A strength of the Bayesian paradigm is that it allows for the explicit use of all available information—to include subject matter expert (SME) opinion and previous (possibly dissimilar) data. While frequentists are constrained to only including data in an analysis (that is to say, only including information that can be observed), Bayesians can easily consider both data and SME opinion, or any other related information that could be constructed. This can be accomplished through the development and use of priors. When prior development is done well, a Bayesian analysis will not only lead to more direct probabilistic statements about system performance, but can result in smaller standard errors around fitted values when compared to a frequentist approach. Furthermore, by quantifying the uncertainty surrounding a model parameter, through the construct of a prior, Bayesians are able to capture the uncertainty across a test space of consideration. This presentation develops a framework for thinking about how different priors can be used throughout the continuum of testing. In addition to types of priors, how priors can change or evolve across the continuum of testing—especially when a system changes (e.g., is modified or adjusted) during phases of testing—will be addressed. Priors that strive to provide no information (reference priors) will be discussed, and will build up to priors that contain available information (informative priors). Informative priors—both those based on institutional knowledge or summaries from databases, as well as those developed based on previous testing data—will be discussed, with a focus on how to consider previous data that is dissimilar in some way, relative to the current test event. What priors might be more common in various phases of testing, types of information that can be used in priors, and how priors evolve as information accumulates will all be discussed. |
Victoria Sieck Deputy Director / Assistant Professor of Statistics Scientific Test & Analysis Techniques Center of Excellence (STAT COE) / Air Force Institute of Technology (AFIT) ![]() (bio)
Dr. Victoria R. C. Sieck is the Deputy Director of the Scientific Test & Analysis (STAT) Center of Excellence (COE), where she works with major acquisition programs within the Department of Defense (DoD) to apply rigor and efficiency to current and emerging test and evaluation methodologies through the application of the STAT process. Additionally, she is an Assistant Professor of Statistics at the Air Force Institute of Technology (AFIT), where her research interests include design of experiments, and developing innovate Bayesian approaches to DoD testing. As an Operations Research Analyst in the US Air Force (USAF), her experiences in the USAF testing community include being a weapons and tactics analyst and an operational test analyst. Dr. Sieck has a M.S. in Statistics from Texas A&M University, and a Ph.D. in Statistics from the University of New Mexico. Her Ph.D. research was on improving operational testing through the use of Bayesian adaptive testing methods. |
Session Recording |
![]() Recording | 2022 |
|
Breakout A Great Test Requires a Great Plan (Abstract)
The Scientific Test and Analysis Techniques (STAT) process is designed to provide structure for a test team to progress from a requirement to decision quality information. The four phases of the STAT process are Plan, Design, Execute, and Analyze. Within the Test and Evaluation (T&E) community we tend to focus on the quantifiable metrics and the hard science of testing, which are the Design and the Analyze phases. At the STAT Center of Excellence (COE) we have emphasized an increased focus on the planning phase and in this presentation we focus on the elements necessary for a comprehensive planning session. In order to efficiently and effectively test a system it is vital that the test team understand the requirements, the System Under Test (SUT) to include any subsystems that will be tested, and the test facility. To accomplish this the right team members with the necessary knowledge must be in the room and prepared to present their information and have an educated discussion to arrive at a comprehensive agreement about the desired end stated of the test. Our recommendations for the initial planning meeting are based on a thorough study of the STAT process and on lessons learned from actual planning meetings. |
Aaron Ramert STAT Analyst Scientific Test and Analytics Techniques Center of Excellence (STAT COE) ![]() (bio)
Mr. Ramert is a graduate of the US Naval Academy and the Naval Postgraduate School and a 20 year veteran of the Marine Corps. During his career in the Marines he served tours in operational air and ground units as well as academic assignments. He joined the Scientific Test and Analysis Techniques (STAT) Center of Excellence (COE) in 2016 where he works with major acquisition programs with the Department of Defense to apply rigor and efficiency to their test and evaluation methodology through the application of the STAT process. |
Breakout | Session Recording |
![]() Recording | 2021 |
Webinar A HellerVVA Problem: The Catch-22 for Simulated Testing of Fully Autonomous Systems (Abstract)
In order to verify, validate, and accredit (VV&A) a simulation environment for testing the performance of an autonomous system, testers must examine more than just sensor physics—they must also provide evidence that the environmental features which drives system decision making are represented at all. When systems are black boxes though, these features are fundamentally unknown, necessitating that we first test to discover these features. An umbrella known as “model induction” provides approaches for demystifying black boxes and obtaining models of their decision making, but the current state of the art assumes testers can input large quantities of operationally relevant data. When systems only make passive perceptual decisions or operate in purely virtual environments, these assumptions are typically met. However, this will not be the case for black-box, fully autonomous systems. These systems can make decisions about the information they acquire—which cannot be changed in pre-recorded passive inputs—and a major reason to obtain a decision model is to VV&A the simulation environment—preventing the valid use of a virtual environment to obtain a model. Furthermore, the current consensus is that simulation will be used to get limited safety releases for live testing. This creates a catch-22 of needing data to obtain the decision-model, but needing the decision-model to validly obtain the data. In this talk, we provide a brief overview of this challenge and possible solutions. |
Daniel Porter Research Staff Member IDA ![]() |
Webinar | Session Recording |
![]() Recording | 2020 |
Contributed A Metrics-based Software Tool to Guide Test Activity Allocation (Abstract)
Existing software reliability growth models are limited to parametric models that characterize the number of defects detected as a function of testing time or the number of vulnerabilities discovered with security testing. However, the amount and types of testing effort applied are rarely considered. This lack of detail regarding specific testing activities limits the application of software reliability growth models to general inferences such as the additional amount of testing required to achieve a desired failure intensity, mean time to failure, or reliability (period of failure free operation). This presentation provides an overview of an open source software reliability tool implementing covariate software reliability models [1] to aid DoD organizations and their contractors who desire to quantitatively measure and predict the reliability and security improvement of software. Unlike traditional software reliability growth models, the models implemented in the tool can accept multiple discrete time series corresponding to the amount of each type of test activity performed as well as dynamic metrics computed in each interval. When applied in the context of software failure or vulnerability discovery data, the parameters of each activity can be interpreted as the effectiveness of that activity to expose reliability defects or security vulnerabilities. Thus, these enhanced models provide the structure to assess existing and emerging techniques in an objective framework that promotes thorough testing and process improvement, motivating the collection of relevant metrics and precise measurements of the time spent performing various testing activities. References [1] Vidhyashree Nagaraju, Chathuri Jayasinghe, Lance Fiondella, Optimal test activity allocation for covariate software reliability and security models, Journal of Systems and Software, Volume 168, 2020, 110643. |
Jacob Aubertine Graduate Research Assistant University of Massachusetts Dartmouth ![]() (bio)
Jacob Aubertine is pursuing a MS degree in the Department of Electrical and Computer Engineering at the University of Massachusetts Dartmouth, where he also received his BS (2020) in Computer Engineering. His research interests include software reliability, performance engineering, and statistical modeling. |
Contributed |
![]() | 2021 |
|
Breakout A Multi-method, Triangulation Approach to Operational Testing (Abstract)
Humans are not produced in quality-controlled assembly lines, and we typically are much more variable than the mechanical systems we employ. This mismatch means that when characterizing the effectiveness of a system, the system must be considered in the context of its users. Accurate measurement is critical to this endeavor, yet while human variability is large, effort to reduce measurement error of those humans is relatively small. The following talk discusses the importance of using multiple measurement methods—triangulation—to reduce error and increase confidence when characterizing the quality of HSI. A case study from an operational test of an attack helicopter demonstrates how triangulation enables more actionable recommendations. |
Daniel Porter Research Staff Member IDA |
Breakout | Materials | 2018 |
|
Breakout A New Method for Planning Full-Up System-Level (FUSL) Live Fire Tests (Abstract)
Planning Full-Up System-Level (FUSL) Live Fire tests is a complex process that has historically relied solely on subject matter expertise. In particular, there is no established method to determine the appropriate number of FUSL tests necessary for a given program. We developed a novel method that is analogous to the Design of Experiments process that is used to determine the scope of Operational Test events. Our proposed methodology first requires subject matter experts (SMEs) to define all potential FUSL shots. For each potential shot, SMEs estimate the severity of that shot, the uncertainty of that severity estimate, and the similarity of that shot to all other potential shots. We developed a numerical optimization algorithm that uses the SME inputs to generate a prioritized list of FUSL events and a corresponding plot of the total information gained with each successive shot. Together, these outputs can help analysts determine the adequate number of FUSL tests for a given program. We illustrate this process with an example on a notional ground vehicle. Future work is necessary prior to implementation on a program of record. |
Lindsey Butler Research Staff Member IDA (bio)
Dr. Lindsey Butler holds a B.S. in Chemical Engineering from Virginia Tech and a Ph.D. in Biomedical Engineering from the University of South Carolina. She has worked at the Institute for Defense Analyses for 5 years where she supports the Director of Operational Test and Evaluation. Dr. Butler is the Deputy Live Fire lead at the Institute for Defense Analyses. Her primary projects focus on assessing the survivability of body armor and armored vehicles against operationally realistic threats. She also has an expertise evaluating casualty assessments to personnel after live fire tests. |
Breakout | Session Recording |
![]() Recording | 2022 |
Webinar A Practical Introduction To Gaussian Process Regression (Abstract)
Abstract: Gaussian process regression is ubiquitous in spatial statistics, machine learning, and the surrogate modeling of computer simulation experiments. Fortunately their prowess as accurate predictors, along with an appropriate quantification of uncertainty, does not derive from difficult-to-understand methodology and cumbersome implementation. We will cover the basics, and provide a practical tool-set ready to be put to work in diverse applications. The presentation will involve accessible slides authored in Rmarkdown, with reproducible examples spanning bespoke implementation to add-on packages. Instructor Bio: Robert Gramacy is a Professor of Statistics in the College of Science at Virginia Polytechnic and State University (Virginia Tech). Previously he was an Associate Professor of Econometrics and Statistics at the Booth School of Business, and a fellow of the Computation Institute at The University of Chicago. His research interests include Bayesian modeling methodology, statistical computing, Monte Carlo inference, nonparametric regression, sequential design, and optimization under uncertainty. Professor Gramacy is a computational statistician. He specializes in areas of real-data analysis where the ideal modeling apparatus is impractical, or where the current solutions are inefficient and thus skimp on fidelity. Such endeavors often require new models, new methods, and new algorithms. His goal is to be impactful in all three areas while remaining grounded in the needs of a motivating application. His aim is to release general purpose software for consumption by the scientific community at large, not only other statisticians. Professor Gramacy is the primary author on six R packages available on CRAN, two of which (tgp, and monomvn) have won awards from statistical and practitioner communities. |
Robert “Bobby” Gramacy Virginia Tech ![]() (bio)
Robert Gramacy is a Professor of Statistics in the College of Science at Virginia Polytechnic and State University (Virginia Tech). Previously he was an Associate Professor of Econometrics and Statistics at the Booth School of Business, and a fellow of the Computation Institute at The University of Chicago. His research interests include Bayesian modeling methodology, statistical computing, Monte Carlo inference, nonparametric regression, sequential design, and optimization under uncertainty. Professor Gramacy is a computational statistician. He specializes in areas of real-data analysis where the ideal modeling apparatus is impractical, or where the current solutions are inefficient and thus skimp on fidelity. Such endeavors often require new models, new methods, and new algorithms. His goal is to be impactful in all three areas while remaining grounded in the needs of a motivating application. His aim is to release general purpose software for consumption by the scientific community at large, not only other statisticians. Professor Gramacy is the primary author on six R packages available on CRAN, two of which (tgp, and monomvn) have won awards from statistical and practitioner communities. |
Webinar |
![]() | 2020 |
|
Short Course A Practitioner’s Guide to Advanced Topics in DOE (Abstract)
Having completed a first course in DOE and begun to apply these concepts, engineers and scientists quickly learn that test and evaluation often demands knowledge beyond the use of classical designs. This one-day short course, taught by an engineer from a practitioner’s perspective, targets this problem. Three broad areas are covered:
The course format is to introduce relevant background material, discuss case studies, and provide software demonstrations. Case studies and demonstrations are derived from a variety of sources, including aerospace testing and DOD T&E. Learn design approaches, design comparison metrics, best practices, and lessons learned from the instructor’s experience. A first course in Design of Experiments is a prerequisite. |
Drew Landman Professor Old Dominion University ![]() (bio)
Drew Landman has 34 years of experience in engineering education as a professor at Old Dominion University. Dr. Landman’s career highlights include13 years (1996-2009) as chief engineer at the NASA Langley Full-Scale Wind Tunnel in Hampton, VA. Landman was responsible for program management, test design, instrument design and calibration and served as the lead project engineer for many automotive, heavy truck, aircraft, and unmanned aerial vehicle wind tunnel tests including the Centennial Wright Flyer and the Boeing X-48B and C. His research interests and sponsored programs are focused on wind tunnel force measurement systems and statistically defensible experiment design primarily to support wind tunnel testing. Dr. Landman has served as a consultant and trainer in the area of statistical engineering to test and evaluation engineers and scientists at AIAA, NASA, Aerovironment, Airbus, Aerion, ATI, USAF, US Navy, US Marines and the Institute for Defense Analysis. Landman founded a graduate course sequence in statistical engineering within the ODU Department of Mechanical and Aerospace Engineering. He is currently co-authoring a text on wind tunnel test techniques. |
Short Course | Materials | 2022 |
|
Breakout A Quantitative Assessment of the Science Robustness of the Europa Clipper Mission (Abstract)
Existing characterization of Europa’s environment is enabled by the Europa Clipper mission’s successful predecessors: Pioneer, Voyager, Galileo, and most recently, Juno. These missions reveal high intensity energetic particle fluxes at Europa’s orbit, requiring a multidimensional design challenge to ensure mission success (i.e. meeting Level 1 science requirements). Risk averse JPL Design Principles and the Europa Environment Requirement Document (ERD) dictate practices and policy, which if masterfully followed, are designed to protect Clipper from failure or degradation due to radiation. However, even if workmanship is flawless and no waivers are assessed, modeling errors, shielding uncertainty, and natural variation in the Europa environment are cause for residual concern. While failure and part degradation are of paramount concern, the occurrence of temporary outages, causing loss or degradation of science observations, is also a critical mission risk, left largely unmanaged by documents like the ERD. The referenced risk is monitored and assessed through a Project Systems Engineering-led mission robustness effort, which attempts balance the risk of science data loss with potential design cost and increased mission complexity required to mitigate such risk. The Science Sensitivity Model (SSM) was developed to assess mission and science robustness, with its primary goal being to ensure a high probability of achieving Level 1 (L1) science objectives by informing the design of a robust spacecraft, instruments, and mission design. This discussion will provide an overview of the problem, the model, and solution strategies. Subsequent presentations discuss the experimental design used to understand the problem space and the graphics and visualization used to reveal important conclusions. |
Kelli McCoy | Breakout |
![]() | 2019 |
|
Breakout A Statistical Approach for Uncertainty Quantification with Missing Data (Abstract)
Uncertainty quantification (UQ) has emerged as the science of quantitative characterization and reduction of uncertainties in simulation and testing. Stretching across applied mathematics, statistics, and engineering, UQ is a multidisciplinary field with broad applications. A popular UQ method to analyze the effects of input variability and uncertainty on the system responses is generalized Polynomial Chaos Expansion (gPCE). This method was developed using applied mathematics and does not require knowledge of a simulation’s physics. Thus, gPCE may be used across disparate industries and is applicable to both individual component and system level simulations. The gPCE method can encounter problems when any of the input configurations fail to produce valid simulation results. gPCE requires that results be collected on a sparse grid Design of Experiment (DOE), which is generated based on probability distributions of the input variables. A failure to run the simulation at any one input configuration can result in a large decrease in the accuracy of a gPCE. In practice, simulation data sets with missing values are common because simulations regularly yield invalid results due to physical restrictions or numerical instability. We propose a statistical approach to mitigating the cost of missing values. This approach yields accurate UQ results if simulation failure makes gPCE methods unreliable. The proposed approach addresses this missing data problem by introducing an iterative machine learning algorithm. This methodology allows gPCE modelling to handle missing values in the sparse grid DOE. The study will demonstrate the convergence characteristics of the methodology to reach steady state values for the missing points using a series of simulations and numerical results. Remarks about the convergence rate and the advantages and feasibility of the proposed methodology will be provided. Several examples are used to demonstrate the proposed framework and its utility including a secondary air system example from the jet engine industry and several non-linear test functions. This is based on joint work with Dr. Mark Andrews at SmartUQ. |
Mark Andrews | Breakout | 2019 |
||
Tutorial A Statistical Tool for Efficient and information-Rich Testing (Abstract)
Binomial metrics like probability-to-detect or probability-to-hit typically provide operationally meaningful and easy to interpret test outcomes. However, they are informationpoor metrics and extremely expensive to test. The standard power calculations to size a test employ hypothesis tests, which typically result in many tens to hundreds of runs. In addition to being expensive, the test is most likely inadequate for characterizing performance over a variety of conditions due to the inherently large statistical uncertainties associated with binomial metrics. A solution is to convert to a continuous variable, such as miss distance or time-todetect. The common objection to switching to a continuous variable is that the hit/miss or detect/non-detect binomial information is lost, when the fraction of misses/no-detects is often the most important aspect of characterizing system performance. Furthermore, the new continuous metric appears to no longer be connected to the requirements document, which was stated in terms of a probability. These difficulties can be overcome with the use of censored data analysis. This presentation will illustrate the concepts and benefits of this approach, and will illustrate a simple analysis with data, including power calculations to show the cost savings for employing the methodology. |
Bram Lillard Research Staff Member IDA |
Tutorial | Materials | 2016 |
|
Breakout A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime |
Eric Walker NASA |
Breakout | Materials | 2017 |
|
Breakout A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime |
Joseph Morrison NASA |
Breakout | Materials | 2017 |
|
Breakout A Survey of Statistical Methods in Aeronautical Ground Testing |
Drew Landman | Breakout |
![]() | 2019 |
|
Breakout A Systems Perspective on Bringing Reliability and Prognostics to Machine Learning (Abstract)
Machine learning is being deployed into the real-world, yet the body of knowledge on testing, evaluating, and maintaining machine learning models is overwhelmingly centered on component-level analysis. But, machine learning and engineered systems are tightly coupled. This is evidenced by extreme sensitivity to of ML to changes in system structure and behavior. Thus, reliability, prognostics, and other efforts related to test and evaluation for ML cannot be divorced from the system. That is, machine learning and its system go hand-in-hand. Any other way makes an unjustified assumption about the existence of an independent variable. This talk explores foundational reasons for this phenomena, and the foundational challenges it poses to existing practice. Cases in machine health monitoring and in cyber defense are used to motivate the position that machine learning is not independent of physical changes to the system with which it interacts, and ML is not independent of the adversaries it defends against. By acknowledging these couplings, systems and mission engineers can better align test and evaluation practices with the fundamental character of ML. |
Tyler Cody Research Assistant Professor Virginia Tech National Security Institute ![]() (bio)
Tyler Cody is an Assistant Research Professor at the Virginia Tech National Security Institute. His research interest is in developing principles and best practices for the systems engineering of machine learning and artificial intelligence. His research has been applied to machine learning for engineered systems broadly, including hydraulic actuators, industrial compressors, rotorcraft, telecommunication systems, and computer networks. He received his Ph.D. in systems engineering from the University of Virginia in May 2021 for his work on a systems theory of transfer learning. |
Breakout | Session Recording |
![]() Recording | 2022 |
Breakout A User-Centered Design Approach to Military Software Development (Abstract)
This case study highlights activities performed during the front-end process of a software development effort undertaken by the Fire Support Command and Control Program Office. This program office provides the U.S. Army, Joint and coalition commanders with the capability to plan, execute and deliver both lethal and non-lethal fires. Recently, the program office has undertaken modernization of its primary field artillery command and control system that has been in use for over 30 years. The focus of this case study is on the user-centered design process and activities taken prior to and immediately following contract award. A modified waterfall model comprised of three cyclic, yet overlapping phases (observation, visualization, and evaluation) provided structure for the iterative, user-centered design process. Gathering and analyzing data collected during focus groups, observational studies, and workflow process mapping, enabled the design team to identify 1) design patterns across the role/duty, unit and echelon matrix (a hierarchical organization structure), 2) opportunities to automate manual processes, 3) opportunities to increase efficiencies for fire mission processing, 4) bottlenecks and workarounds to be eliminated through design of the modernized system, 5) shortcuts that can be leveraged in design, 6) relevant and irrelevant content for each user population for streamlining access to functionality, 7) a usability baseline for later comparison (e.g., the number of steps and time taken to perform a task as captured in workflows for comparison to the same task in the modernized system), and provided the basis for creating visualizations using wireframes. Heuristic evaluations were conducted early to obtain initial feedback from users. In the next few months, usability studies will enable users to provide feedback based on actual interaction with the newly designed software. Included in this case study are descriptions of the methods used to collect user-centered design data, how results were visualized/documented for use by the development team, and lessons learned from applying user-centered design techniques during software development of a military field artillery command and control system. |
Pam Savage-Knepshield | Breakout |
![]() | 2019 |
Session Title | Speaker | Type | Recording | Materials | Year |
---|---|---|---|---|---|
Breakout “High Velocity Analytics for NASA JPL Mars Rover Experimental Design” |
Jim Wisnowski Co-founder/Principle Adsurgo |
Breakout | Materials | 2016 |
|
Breakout “High Velocity Analytics for NASA JPL Mars Rover Experimental Design” |
Heath Rushing Co-founder/Principle Adsurgo |
Breakout | Materials | 2016 |
|
Breakout 3D Mapping, Plotting, and Printing in R with Rayshader |
Tyler Morgan-Wall | Breakout | 2019 |
||
Breakout A 2nd-Order Uncertainty Quantification Framework Applied to a Turbulence Model Validation Effort |
Robert Baurle | Breakout |
![]() | 2019 |
|
Breakout A Causal Perspective on Reliability Assessment |
Lauren Hund | Breakout |
![]() | 2019 |
|
Breakout A Decision-Theoretic Framework for Adaptive Simulation Experiments |
Terril Hurst Senior Engineering Fellow Raytheon Technologies ![]() |
Breakout | Session Recording |
![]() Recording | 2022 |
Breakout A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge |
Sarah Longo Data Scientist US Army CCDC Armaments Center ![]() |
Breakout |
![]() | 2021 |
|
Breakout A DOE Case Study: Multidisciplinary Approach to Design an Army Gun Propulsion Charge |
Melissa Jablonski Statistician US Army CCDC Armaments Center ![]() |
Breakout |
![]() | 2021 |
|
Breakout A Framework for Efficient Operational Testing through Bayesian Adaptive Design |
Victoria Sieck Student / Operations Research Analyst University of New Mexico / Air Force Institute of Technology ![]() |
Breakout |
![]() | 2021 |
|
A Framework for Using Priors in a Continuum of Testing |
Victoria Sieck Deputy Director / Assistant Professor of Statistics Scientific Test & Analysis Techniques Center of Excellence (STAT COE) / Air Force Institute of Technology (AFIT) ![]() |
Session Recording |
![]() Recording | 2022 |
|
Breakout A Great Test Requires a Great Plan |
Aaron Ramert STAT Analyst Scientific Test and Analytics Techniques Center of Excellence (STAT COE) ![]() |
Breakout | Session Recording |
![]() Recording | 2021 |
Webinar A HellerVVA Problem: The Catch-22 for Simulated Testing of Fully Autonomous Systems |
Daniel Porter Research Staff Member IDA ![]() |
Webinar | Session Recording |
![]() Recording | 2020 |
Contributed A Metrics-based Software Tool to Guide Test Activity Allocation |
Jacob Aubertine Graduate Research Assistant University of Massachusetts Dartmouth ![]() |
Contributed |
![]() | 2021 |
|
Breakout A Multi-method, Triangulation Approach to Operational Testing |
Daniel Porter Research Staff Member IDA |
Breakout | Materials | 2018 |
|
Breakout A New Method for Planning Full-Up System-Level (FUSL) Live Fire Tests |
Lindsey Butler Research Staff Member IDA |
Breakout | Session Recording |
![]() Recording | 2022 |
Webinar A Practical Introduction To Gaussian Process Regression |
Robert “Bobby” Gramacy Virginia Tech ![]() |
Webinar |
![]() | 2020 |
|
Short Course A Practitioner’s Guide to Advanced Topics in DOE |
Drew Landman Professor Old Dominion University ![]() |
Short Course | Materials | 2022 |
|
Breakout A Quantitative Assessment of the Science Robustness of the Europa Clipper Mission |
Kelli McCoy | Breakout |
![]() | 2019 |
|
Breakout A Statistical Approach for Uncertainty Quantification with Missing Data |
Mark Andrews | Breakout | 2019 |
||
Tutorial A Statistical Tool for Efficient and information-Rich Testing |
Bram Lillard Research Staff Member IDA |
Tutorial | Materials | 2016 |
|
Breakout A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime |
Eric Walker NASA |
Breakout | Materials | 2017 |
|
Breakout A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime |
Joseph Morrison NASA |
Breakout | Materials | 2017 |
|
Breakout A Survey of Statistical Methods in Aeronautical Ground Testing |
Drew Landman | Breakout |
![]() | 2019 |
|
Breakout A Systems Perspective on Bringing Reliability and Prognostics to Machine Learning |
Tyler Cody Research Assistant Professor Virginia Tech National Security Institute ![]() |
Breakout | Session Recording |
![]() Recording | 2022 |
Breakout A User-Centered Design Approach to Military Software Development |
Pam Savage-Knepshield | Breakout |
![]() | 2019 |