DATAWorks Sessions Archive

This briefing discusses the practical advantages of using the probabilistic programming language (PPL) Stan to answer statistical questions, especially those related to the quantification of uncertainty. Stan is a relatively new statistical tool that allows users to specify probability models and reason about the processes that generate the data they encounter. Stan has quickly become a popular language for writing statistical models because it allows one to specify rich (or sparse) Bayesian models using high level language. Further, Stan is fast, memory efficient, and robust. Stan requires users be explicit about the model they wish to evaluate, which makes the process of statistical modeling more transparent to users and decision makers. This is valuable because it forces practitioners to consider assumptions at the beginning of the model building procedure, rather than at the end (or not at all). In this sense, Stan is the opposite of a “black box” modeling approach. This approach may be tedious and labor intensive at first, but the pay-offs are large. For example, once a model is set-up inferential tasks all essentially automatic, as changing the model does not change the how one analyzes the data. This is a generic approach to inference. To illustrate these points, we use Stan to study a ballistic miss distance problem. In ballistic missile testing, the p-content circular error probable (CEP) in the circle that contains p percent of future shots fired, on average. Statistically, CEP is a bivariate prediction region, constrained by the model to be circular. In Frequentist statistics, the determination of CEP is highly dependent on the model fit, and a different calculation of CEP must be produced for each plausible model. However, with Stan, we can approach the CEP calculation invariant of the model we use to fit the data. We show how to use Stan to calculate CEP and uncertainty intervals for the parameters using summary statistics. Statistical practitioners can access Stan from several programming languages, including R and Python.

Sensors abound in aerospace testing and while many scientists look at the data from a physics perspective, the comparative statistics information is what drives decisions. A multi-company project was comparing launch data from the 1980’s to a current set of data that included 30 sensors. Each sensor was designed to gather 3000 data points during the 3-second launch event. The data included temperature, acceleration, and pressure information. This talk will compare the data analysis methods developed for this project as well as the use of the new Functional Data Analysis tool within JMP for its ability to discern in-family launch performances.

The goal of the Crew State Monitoring (CSM) project is to use machine learning models trained with physiological data to predict unsafe cognitive states in pilots such as Channelized Attention (CA) and Startle/Surprise (SS). These models will be used in a real-time system that predicts a pilot’s mental state every second, a tool that can be used to help pilots recognize and recover from these mental states. Pilots wore different sensors that collected physiological data such as a 20-channel electroencephalography (EEG), respiration, and galvanic skin response (GSR). Pilots performed non-flight benchmark tasks designed to induce these states, and a flight simulation with “surprising” or “channelizing” events. The team created a pipeline to generate pilot-dependent models that trains on benchmark data, tune on a portion of a flight task, and be deployed onto the remaining flight task. The model is a series of anomaly-detection based ensembles, where each ensemble focuses on predicting a single state. Ensembles were comprised of several anomaly detectors such as One Class SVMs, each focusing on a different subset of sensor data. We will discuss the performance of these models, as well as the ongoing research generalizing models across pilots and improving accuracy.

Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.

The zoonotic emergence of the coronavirus SARS-CoV-2 at the beginning of 2020 and the subsequent global pandemic of COVID-19 has caused massive disruptions to economies and health care systems, particularly in the United States. Using the results of serology testing, we have developed true prevalence estimates for COVID-19 case counts in the U.S. over time, which allows for more clear estimates of infection and case fatality rates throughout the course of the pandemic. In order to elucidate policy, demographic, weather, and behavioral factors that contribute to or inhibit the spread of COVID-19, IDA compiled panel data sets of empirically derived, publicly available COVID-19 data and analyzed which factors were most highly correlated with increased and decreased spread within U.S. states and counties. These analyses lead to several recommendations for future pandemic response preparedness.

There are many testing situations that historically involve a large number of runs. The use of experimental design methods can reduce the number of runs required to obtain the information desired. Example applications include wind tunnel test campaigns, computational experiments and live fire tests. In this work we present three case studies conducted under the auspices of the Science of Test Research Consortium comparing the information obtained via a historical experimental approach with the information obtained via an experimental design approach. The first case study involves a large scale wind tunnel experimental campaign. The second involves a computational fluid dynamics model of a missile through various speeds and angles of attack. The third case involves ongoing live fire testing involve hot surface testing. In each case, results suggest a tremendous opportunity to reduce experimental test efforts without losing test information.

Statistical engineering has been defined as: “The study of how to best utilize statistical concepts, methods, and tool, and integrate them with IT and other relevant sciences to generate improved results.” A key principle is that significant untapped benefits are often achievable by integrating multiple methods in novel ways to address a problem, without having to invent new statistical techniques. In this presentation, we discuss the application of statistical engineering to the problem of design and analysis of mixture experiments when process variables are also involved. In such cases, models incorporating interaction between the mixture and process variables have been developed, but tend to require large designs and models. By considering models nonlinear in the parameters, also well known in the literature, we demonstrate how experimenters can utilize an alternative, iterative strategy in attacking such problems. We show that this strategy potentially saves considerable experimental time and effort, while producing models that are nearly as accurate as much larger linear models. These results are illustrated using two published data sets and one new data set, all involving interactive mixture and process variable problems.

Problems faced in defense and aerospace often go well beyond textbook problems presented in academic settings. Textbook problems are typically very well defined, and can be solved through application of one “correct” tool. Conversely, many defense and aerospace problems are ill-defined, at least initially, and require an overall strategy to attack, one involving multiple tools and often multiple disciplines. Statistical engineering is an approach recently developed for addressing large, complex, unstructured problems, particularly those for which data can be effectively utilized. This session will present a brief overview of statistical engineering, and how it can be applied to engineer solutions to complex problems. Following this introduction, two case studies of statistical engineering will be presented, to illustrate the concepts.

This panel will share status, experiences and expectations within DoD and NASA for transitioning Systems Engineering to a more integrated digital engineering domain. A wide range of perspectives will be provided, covering the implementation waterfront of practitioner, management, research and strategy. Panelist will also be prepared to discuss more focused areas of digital systems engineering, such as test and evaluation, and engineering statistics.

The key to acquiring a cybersecure system is the ability to drive considerations about security from the operational level into the tactics and procedures. For T&E to support development and acquisition decisions, we must also adopt the perspective that a cyberattack is an attack on the mission using technology. A well-defined process model linking tools, tasks, and operators to mission performance supports this perspective. We will discuss an approach based on best practices learned from various DHS programs.

Causality in an engineered system pertains to how a system output changes due to a controlled change or intervention on the system or system environment. Engineered systems designs reflect a causal theory regarding how a system will work, and predicting the reliability of such systems typically requires knowledge of this underlying causal structure. The aim of this work is to introduce causal modeling tools that inform reliability predictions based on biased data sources and illustrate how these tools can inform data integration in practice. We present a novel application of the popular structural causal modeling framework to reliability estimation in an engineering application, illustrating how this framework can inform whether reliability is estimable and how to estimate reliability using data integration given a set of assumptions about the subject matter and data generating mechanism. When data are insufficient for estimation, sensitivity studies based on problem-specific knowledge can inform how much reliability estimates can change due to biases in the data and what information should be collected next to provide the most additional information. We apply the approach to a pedagogical example related to a real, but proprietary, engineering application, considering how two types of biases in data can influence a reliability calculation.

Using Combinatorial Test Design methods to select software test scenarios has repeatedly delivered large efficiency and thoroughness gains – which begs the questions: • Why are these proven methods not used everywhere? • Why do some efforts to promote adoption of new approaches stagnate? • What steps can leaders take to introduce successfully introduce and spread new test design methods? For more than a decade, Justin Hunter has helped large global organizations across six continents adopt new test design techniques at scale. Working in some environments, he has felt like Sisyphus, forever condemned to roll a boulder uphill only to watch it roll back down again. In other situations, things clicked; teams smoothly adopted new tools and techniques, and impressive results were quickly achieved. In this presentation, Justin will discuss several common challenges faced by large organizations, explain why adopting test design tools is more challenging than adopting other types of development and testing tools, and share actionable recommendations to consider when you roll out new test design approaches.

For the past 7 years, USAF DT&E has been exploring ways to adapt the principles of experimental design to rapidly evolving developmental test articles and test facilities – often with great success. This paper discusses three case studies that span the range of USAF DT&E activities from EW to Ground Test to Flight Test and shows the truly revolutionary impact Fisher’s DOE can have on development. The Advanced Strategic and Tactical Expendable (ASTE) testing began in 1990 to develop, enhance, and test new IR flares, flare patterns, and dispense tactics. More than 60 aircraft & flare types have been tested. The typical output is a “Pancake Plot” of 200+ “cases” of flare, aspect angle, range, elevation, and flare effectiveness using a stop-light chart (red-yellow-green) approach. In usual testing – ~3000 flare engagements costing $1M to participate. The response, troublingly enough, is binary – 15-30 binomial response trials measuring P(Decoy). Binary responses are information-poor. Legacy testing does not assess present statistical power in reporting P(decoy) results. Analysts investigated replacing P(Decoy) w/ continuous metrics – e.g. time to decoy. This research is ongoing. We found we could spread the replicates out to examine 3x to 5x more test conditions without affecting power materially. Analysis with the Generalized Linear Model (GLZ) replaced legacy “cases” analysis with 75% improvement to confidence intervals with same data. We are seeking to build a Monte-Carlo simulation to estimate how many runs are required in a logistics regression model to achieve adequate power. We hope to reduce customer expenditures for flare information by as much as 50%. Co-authors J. Higdon, B, Knight. AEDC completed a major upgrade with new nozzle hardware to vary Mach. The Arnold Transonic Wind Tunnel 4T spans the range of flows from subsonic to approximately M9. The new wind tunnel was to be computer controlled. A number of key instrumentation improvements were made at the same time. The desire was to calibrate the resulting modified tunnel. The last calibration of 4T was 25 years ago in 1990. The calibration ranged across the full range of Mach and pressure capabilities, spanning a four-D space: pressure, Mach, wall angle, and wall porosity. Both the traditional OFAT effort – vary one factor at a time – and a parallel DOE effort were run to compare design, execution, modeling, and prediction capabilities against cost and time to run. The robust embedded face-centered CCD DOE design (J. Simpson and D. Landman ’05) employed 75 vs. 176 OFAT runs. The RSM design achieved 57% run savings. Due to an admirable discipline in randomization during the DOE trials, the smaller design required longer to run. As a result of using the DOE approach, engineers found it easier predict offcondition tunnel operating characteristics using RSM models, optimize facility flow quality for any given test condition. In addition, the RSM regression models support future “spot check” calibration in future by comparing predictions to measured values. If the measurement falls within the prediction interval, the existing calibration still appropriate. AEDC is using split-plot style designs a for current wind tunnel probe calibration. Co-author: Dr Dough Garrard.

“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning. Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”

The US Army ARDEC has recently established an initiative to integrate statistical and probabilistic techniques into engineering modeling and simulation (M&S) analytics typically used early in the design lifecycle to guide technology development. DOE-driven Uncertainty Quantification techniques, including statistically rigorous model verification and validation (V&V) approaches, enable engineering teams to identify, quantify, and account for sources of variation and uncertainties in design parameters, and identify opportunities to make technologies more robust, reliable, and resilient earlier in the product’s lifecycle. Several recent armament engineering case studies – each with unique considerations and challenges – will be discussed.