DATAWorks Sessions Archive

In order to verify, validate, and accredit (VV&A) a simulation environment for testing the performance of an autonomous system, testers must examine more than just sensor physics—they must also provide evidence that the environmental features which drives system decision making are represented at all. When systems are black boxes though, these features are fundamentally unknown, necessitating that we first test to discover these features. An umbrella known as “model induction” provides approaches for demystifying black boxes and obtaining models of their decision making, but the current state of the art assumes testers can input large quantities of operationally relevant data. When systems only make passive perceptual decisions or operate in purely virtual environments, these assumptions are typically met. However, this will not be the case for black-box, fully autonomous systems. These systems can make decisions about the information they acquire—which cannot be changed in pre-recorded passive inputs—and a major reason to obtain a decision model is to VV&A the simulation environment—preventing the valid use of a virtual environment to obtain a model. Furthermore, the current consensus is that simulation will be used to get limited safety releases for live testing. This creates a catch-22 of needing data to obtain the decision-model, but needing the decision-model to validly obtain the data. In this talk, we provide a brief overview of this challenge and possible solutions.

A vulnerability assessment, which evaluates the ability of an armored combat vehicle and its crew to withstand the damaging effects of an anti-armor weapon, presents a unique challenge because vehicles are expensive and testing is destructive. This limits the number of full-up-system-level tests to quantities that generally do not support meaningful statistical inference. The prevailing solution to this problem is to obtain test data that is more affordable from sources which include component- and subsystem-level testing. This creates a new challenge that forms the premise of this paper: how can lower-level data sources be connected to provide a credible system-level prediction of vehicle vulnerability? This paper presents a case study that demonstrates an approach to this problem that emphasizes the use of fundamental statistical techniques — design of experiments, statistical modeling, and propagation of uncertainty — in the context of a combat scenario that depicts a ground vehicle being engaged by indirect artillery.

Challenges in adapting the system design process, including test and evaluation, to autonomous systems have been well documented. These include the inability to verify autonomous behaviors, designing systems for assurance, and dealing with adaptive systems post-fielding. This talk will briefly recap those challenges, and then propose a path toward overcoming these challenges in the case of Army ground autonomous systems. The path begins with unmanned systems designed for test. Training and experimentation within a safe infrastructure can then be used to specify system characteristics and inform decisions as to what behaviors should be autonomous. Assurance comes in the form of assurance arguments updated as the system technology and understanding increases.

Reliable modeling and simulation (M&S) allows the undersea warfare community to understand torpedo performance in scenarios that could never be created in live testing, and do so for a fraction of the cost of an in-water test. The Navy hopes to use the Environment Centric Weapons Analysis Facility (ECWAF), a hardware-in-the-loop simulation, to predict torpedo effectiveness and supplement live operational testing. In order to trust the model’s results, the T&E community has applied rigorous statistical design of experiments techniques to both live and simulation testing. As part of ECWAF’s two-phased validation approach, we ran the M&S experiment with the legacy torpedo and developed an empirical emulator of the ECWAF using logistic regression. Comparing the emulator’s predictions to actual outcomes from live test events supported the test design for the upgraded torpedo. This talk overviews the ECWAF’s validation strategy, decisions that have put the ECWAF on a promising path, and the metrics used to quantify uncertainty.

Recent developments in artificial intelligence have captured the popular imagination and the attention of governments and businesses across the world. From digital assistants, to smart homes, to self-driving cars, AI appears to be on the verge of taking over many parts of our daily lives. Meanwhile, as the world becomes more networked, industry, governments, and individuals are facing a growing array of cybersecurity threats. In this talk, we will discuss the intersection of artificial intelligence, machine learning, and cybersecurity. In particular, we will look at how some popular machine learning methods will and will not change how we do cybersecurity, we will separate what in the AI and ML space is realistic from what is science fiction, and we will attempt to identify the true potential of ML to positively impact cybersecurity.

Nonregular designs are a preferable alternative to regular resolution IV designs because they avoid confounding two-factor interactions. As a result nonregular designs can estimate and identify a few active two-factor interactions. However, due to the sometimes complex alias structure of nonregular designs, standard screening strategies can fail to identify all active effects. In this talk, two-level nonregular screening designs with orthogonal main effects will be discussed. By utilizing knowledge of the alias structure, we propose a design based model selection process for analyzing nonregular designs. Our Aliased Informed Model Selection (AIMS) strategy is a design specific approach that is compared to three generic model selection methods; stepwise regression, Lasso, and the Dantzig selector. The AIMS approach substantially increases the power to detect active main effects and two-factor interactions versus the aforementioned generic methodologies.

Sterility Assurance Level (SAL) is the probability that a product, after being exposed to a given sterilization process, contains one or more viable organisms. The SAL is a standard way of defining cleanliness requirements on a product or acceptability of a sterilization procedure in industry and regulatory agencies. Since the SAL acknowledges the inherent probabilistic nature of detecting sterility – that we cannot be absolutely sure that sterility is achieved – a probabilistic approach to its assessment is required that considers the actual end-to-end process involved with demonstrating sterility. Provided here is one such approach. We assume the process of demonstrating sterility is based on the scientific method, and therefore starts with a scientific hypothesis of the model that generates the life/death outcomes of the organisms that will be observed in experiment. Experiments are then designed (e.g. environmental conditions determined, reference/indicator organisms selected, number of samples/replicates, instrumentation) and performed, and an initial conclusion regarding the appropriate model is drawn from observed numbers of organisms remaining once exposed to the sterilization process. Ideally, this is then validated by future experiment by independent scientific inquiry, or the results are used to develop new hypotheses and the process repeats. Ultimately, a decision is made regarding the validity of the sterilization process by means of comparing with a SAL. Bayesian statistics naturally lends itself to develop a probability distribution from this process to compare against a given SAL, which is the approach taken in this paper. We provide an example application to provide a simple demonstration of this from actual experiments performed, and discuss its relevance to the future NASA mission, Mars Sample Return.

Because of the formidable challenge of observing the time-evolving full-depth global ocean circulation, numerical simulations play an essential role in quantifying the ocean’s role in climate variability and long-term change. For the same reason, predictive capabilities are confounded by the high-dimensional space of uncertain variables (initial conditions, internal parameters, external forcings, and model inadequacy). Bayesian inverse methods (loosely known in approximate form as data assimilation) that optimally extract and merge information from sparse, heterogeneous observations and models are powerful tools to enable rigorously calibrated and initialized predictive models to optimally learn from the sparse data. A key enabling computational approach is the use of derivative (adjoint and Hessian) methods for solving a deterministic nonlinear least-squares optimization problem. Such a parameter and state estimation system is practiced by the NASA-supported Estimating the Circulation and Climate of the Ocean (ECCO) consortium. An end-to-end system that propagates uncertainties from observational data to relevant oceanographic metrics or quantities of interest within an inverse modeling framework should address, within a joint approach, all sources of uncertainties, including those in (1) observations (measurement, sampling and representation errors), (2) the model (parametric and structural model error), (3) the data assimilation method (algorithmic approximations and data ingestion), (4) initial and boundary conditions (external forcings, bathymetry), and (5) the prior knowledge (error covariances). Here we lay out a vision for such an end-to-end framework. Recent developments in computational science and engineering are beginning to render theoretical concepts practical in real-world applications.

Pivotal to current US military operations is the quick identification of buildings on Gridded Reference Graphics (GRGs), gridded satellite images of an objective. At present, the United States Special Operations Command (SOCOM) identifies these buildings by hand through the work of individual intelligence officers. Recent advances in Convolutional Neural Networks (CNNs), however, allow the possibility for this process to be streamlined through the use of object detection algorithms. In this presentation, we describe an object detection algorithm designed to quickly identify and highlight every building present on a GRG. Our work leverages both the U-Net and the Mask R-CNN architectures as well as a four-city dataset to produce an algorithm that accurately identifies a large breadth of buildings. Our model reports an accuracy of 87% and is capable of detecting buildings on a diverse set of test images.

In this presentation we will discuss a methodology to automatically extract road networks from aerial images using machine learning algorithms. There are many civilian applications for this technology, such as maintaining GPS maps, real-estate monitoring and disaster relief, but this presentation is aimed a military applications for analyzing remote sensing data. The algorithm we propose identifies road networks and classifies each road as either improved or unimproved. Implementing this system in a military context will require our model to work across a wide range of environments. We will discuss the effectiveness of the model in a military context.

According to Forbes, merchants in the United States lose approximately $190 billion annually to credit card fraud (Shaughnessy 2012). Nordstrom, a leading retailer in the fashion industry, alone incurs losses upwards of $100 million every year. While these losses hurt the company financially, the more pressing concern for Nordstrom is the negative impact on the customer. If fraud is incorrectly flagged and an account is unnecessarily frozen, a customer will be dissatisfied with their experience. Conversely, if legitimate fraud goes undetected, valued customers can experience monetary loss and lose faith in the company. To minimize losses while maximizing customer satisfaction, Nordstrom Card Services created a fraud detection machine that analyzes every transaction and will take one of four actions based on the transaction’s characteristics: approve, approve and send to a queue, decline and send to a queue, or decline. Once a transaction is sent to the queue, it is assessed by either a human analyst or a virtual analyst. Those analysts will determine if any further action must be taken within the customer’s account. This project focuses on how to appropriately assign human and virtual decision-makers to maximize accuracy when determining whether or not a credit card transaction is fraud.

The problem of augmenting an existing sensor network can be solved by considering how best to answer the question of interest. In nuclear explosion monitoring, this could include where best to place a new seismic sensor to most accurately and precisely predict the unknown location of a seismic event. In this talk, we will solve this problem using a Bayesian Design of Experiments (DoE) approach where each design consists of the existing sensor network coupled with a new, possible location. We will incorporate complex computer simulation and experimental data to rank the possible sensor locations with respect to their ability to provide accurate and precise event location estimates. This will result in a map of desirability for new locations, so that if the most highly ranked location is not available or possible, regions of similar predictability can be identified. The Bayesian DoE approach also allows for incorporating denied access locations due to either geographical or political constraints.

When analyzing binary responses, logistic regression is sometimes difficult when some scenarios tested result in only successes or failures; this is called separation in the data. Using typical frequentist methods, logistic regression often fails because of separation. However, Bayesian logistic regression does not suffer from this limitation. This talk will walk through a Bayesian logistic regression of data with separation using the R brms package.

Co-authors: Benjamin Ashwell, Edward Beall, and V. Bram Lillard Bottom-up emulations of real sustainment systems that explicitly model spares, personnel, operations, and maintenance are a powerful way to tie funding decisions to their impact on readiness, but they are not widely used. The simulations require extensive data to properly model the complex and variable processes involved in a sustainment system, and the raw data used to populate the simulation are often scattered across multiple organizations. The Navy has encountered challenges in keeping sustaining the desired number of F/A-18 Super Hornets in mission capable states. IDA was asked to build an end-to-end model of the Super Hornet sustainment system using the OPUS/SIMLOX suite of tools to investigate the strategic levers that drive readiness. IDA built an R package (“honeybee”) that aggregates and interprets Navy sustainment data using statistical techniques to create component-level metrics, and a second R package (“stinger”) that uses these metrics to create a high-fidelity representation of the Navy’s operational tempo for OPUS/SIMLOX.

The calibration of complex models to data is both a challenge and an opportunity. It can be posed as an Inverse Problem. This work focuses on the interface of Ensemble Kalman algorithms used for inversion or posterior sampling (EKI/EKS) , Gaussian process emulation (GPE) and Markov Chain Monte Carlo (MCMC) for the calibration of, and quantification of uncertainty in, parameters learned from data. The goal is to perform uncertainty quantification in predictions made from complex models, reflecting uncertainty in these parameters, with relatively few computationally expensive forward model evaluations. This is achieved by propagating approximate posterior samples obtained by judicious combination of ideas from EKI/EKS, GPE and MCMC. The strategy will be illustrated with idealized models related to climate modeling.

Given the rapid increase of novel machine learning applications in cybersecurity and people analytics, there is significant evidence that these tools can give meaningful and actionable insights. Even so, great care must be taken to ensure that automated decision making tools are deployed in such a way as to mitigate bias in predictions and promote security of user data. In this talk, Dr. Burns will take a deep dive into an open source data set in the area of people analytics, demonstrating the application of basic machine learning techniques, while discussing limitations and potential pitfalls in using an algorithm to predict human behavior. In the end, Dustin will draw a comparison between the potential to predict human behavioral propensity to things such as becoming an insider threat to how assisted diagnosis tools are used in medicine to predict development or reoccurrence of illnesses.

With the rise of machine learning and artificial intelligence, there has been a huge surge in data-driven approaches to solve computational science and engineering problems. In the context of uncertainty quantification (UQ), a common use case for machine learning is in the construction of efficient surrogate models (i.e., response surfaces) to replace expensive, physics-based simulations. However, relying solely on data-driven models for UQ without any further recourse to the original high-fidelity simulation will generally produce biased estimators and can yield unreliable or non-physical results, especially when training data is sparse or predictions are required outside of the training data domain.