DATAWorks Sessions Archive

Principal Component Analysis is a standard tool in an analyst’s toolbox. The standard practice of rescaling each column can be reframed as a copula-based decomposition in which the marginal distributions are fit with a univariate Gaussian distribution and the joint distribution is modeled with a Gaussian copula. In this light, we present an alternative to traditional PCA we call XPCA by relaxing the marginal Gaussian assumption and instead fit each marginal distribution with the empirical distribution function. Interval-censoring methods are used to account for the discrete nature of the empirical distribution function when fitting the Gaussian copula model. In this talk, we derive the XPCA estimator and inspect the differences in fits on both simulated and real data applications.

The Europa Clipper Science Sensitivity Model (SSM) can be thought of as a graph in which the nodes are mission requirements at ten levels in a hierarchy, and edges represent how requirements at one level of the hierarchy depend on those at lower levels. At the top of the hierarchy, there are ten nodes representing ten, Level 1 science requirements for the mission. At the bottom of the hierarchy, there are 100 or so nodes representing instrument-specific science requirements. In between, nodes represent intermediate science requirements with complex interdependencies. Meeting, or failing to meet, bottom-level requirements depends on the frequency of faults and the lengths of recovery times on the nine Europa Clipper instruments and the spacecraft. Our task was to design and analyze the results of a Monte Carlo experiment to estimate the probabilities of meeting the Level 1 science requirements based on parameters of the distributions of time between failures and of recovery times. We simulated an ensemble of synthetic missions in which failures and recoveries were random realizations from those distributions. The pass-fail status of the bottom-level instrument-specific requirements were propagated up the graph for each of the synthetic missions. Aggregating over the collection of synthetic missions produced estimates of the pass-fail probabilities for the Level 1 requirements. We constructed a definitive screening design and supplemented it with additional space-filing runs, using JMP 14 software. Finally, we used the vectors of failure and recovery parameters as predictors, and the pass-fail probabilities of the high-level requirements as responses, and built statistical models to predict the latter from the former. In this talk, we will describe the design considerations and review the fitted models and their implications for mission success.

Biomechanical experiments investigating the failure modes of biological tissue require a significant investment of time and money due to the complexity of procuring, preparing, and testing tissue. Furthermore, the potentially destructive nature of these tests makes repeated testing infeasible. This leads to experiments with notably small sample sizes in light of the high variance common to biological material. When the goal is to estimate parameters for an analytic artifact such as an injury risk curve (IRC), which relates an input quantity to a probability of injury, small sample sizes result in undesirable uncertainty. One way to ameliorate this effect is through a Bayesian approach, incorporating expert opinion and previous experimental data into a prior distribution. This has the advantage of leveraging the information contained in expert opinion and related experimental data to obtain faster convergence to an appropriate parameter estimation with a desired certainty threshold. We explore several ways of implementing Bayesian methods in a biomechanical setting, including permutations on the use of expert knowledge and prior experimental data. Specifically, we begin with a set of experimental data from which we generate a reference IRC. We then elicit expert predictions of the 10th and 90th quantiles of injury, and use them to formulate both uniform and normal prior distributions. We also generate priors from qualitatively similar experimental data, both directly on the IRC parameters and on the injury quantiles, and explore the use of weighting schemes to assign more influence to better datasets. By adjusting the standard deviation and shifting the mean, we can create priors of variable quality. Using a subset of the experimental data in conjunction with our derived priors, we then re-fit the IRC and compare it to the reference curve. For all methods we will measure the certainty, speed of convergence, and accuracy relative to the reference IRC, with the aim of recommending a best practices approach for the application of Bayesian methods in this setting. Ultimately an optimized approach for handling small samples sizes with Bayesian methods has the potential to increase the information content of individual biomechanical experiments by integrating them into the context of expert knowledge and prior experimentation.

Co-authors: Dr. Darryl Ahner, Director, STAT COE Dr. Lenny Truett, STAT COE Mr. Scott Wacker, 96 TG/OL-ACS. This presentation will present the benefits of sequential testing and demonstrate how sequential testing can be used to address complex test conditions by developing well controlled early experiments to explore basic questions before proceeding to full-scale testing. This approach can result in increased knowledge and decreased cost. As of FY13 the Air Force had spent an estimated $47M on dry bay fire testing making fire the largest cost contributor for Live Fire Test and Evaluation (LFT&E) programs. There is currently an estimated 60% uncertainty in total platform vulnerable area (Av) driven by probability of kill (PK) due to ballistically ignited fires. A large part of this uncertain comes from the fact that current spurt modeling does not predict fuel spurt delay with reasonable accuracy despite a large amount of test data. A low-cost sequential approach was developed to improve spurts models. Initial testing used a spherical projectile to test 10 different factors in a definitive screening design. Once the list of factors was refined, a second phase of testing determined if a suitable methodology could be developed to scaled results using water as a surrogate for JP-8 fuel. Finally testing was performed with cubical projectiles to evaluate the effect of fragment orientation. The entire cost for this effort was less than one or two typical full-scale live fire tests.

Autonomous robotic systems (hereafter referred to simply as autonomous systems) have attracted interest in recent years as capabilities improve to operate in unstructured, dynamic environments without continuous human guidance. Acquisition of autonomous systems potentially decrease personnel costs and provide a capability to operate in dirty, dull, or dangerous mission segments or achieve greater operational performance. Autonomy enables a particular action of a system to be automatic or, within programmed boundaries, self-governing. For our purposes, autonomy is defined as the system having a set of intelligence-based capabilities (i.e., learned behaviors) that allows it to respond to situations that were not pre-programmed or anticipated (i.e. learning-based responses) prior to system deployment. Autonomous systems have a degree of self-governance and self-directed behavior, possibly with a human’s proxy for decisions. Because of these intelligence-based capabilities, autonomous systems pose new challenges in conducting test and evaluation that assures adequate performance, safety, and cybersecurity outcomes. We propose an autonomous systems architecture concept and map the elements of a decision theoretic view of a generic decision problem to the components of this architecture. These models offer a foundation for developing a decision-based, common framework for autonomous systems. We also identify some of the various challenges faced by the Department of Defense (DoD) test and evaluation community in assuring the behavior of autonomous systems as well as test and evaluation requirements, processes, and methods needed to address these challenges.

Abstract: Data scientists spend approximately 80% of their time preparing, cleaning, and feature engineering data sets. In this talk I will share use cases that show why this is important and why we need to do it. I will also describe the Earth System Grid Federation (ESGF) which is an open source effort providing a robust, distributed data and computation platform, enabling world wide access to Peta/Exa-scale scientific data. ESGF will help reduce the amount of effort needed for climate data preprocessing by integrating the necessary analysis and data sharing tools.

The Scientific Test and Analysis Techniques (STAT) process is designed to provide structure for a test team to progress from a requirement to decision quality information. The four phases of the STAT process are Plan, Design, Execute, and Analyze. Within the Test and Evaluation (T&E) community we tend to focus on the quantifiable metrics and the hard science of testing, which are the Design and the Analyze phases. At the STAT Center of Excellence (COE) we have emphasized an increased focus on the planning phase and in this presentation we focus on the elements necessary for a comprehensive planning session. In order to efficiently and effectively test a system it is vital that the test team understand the requirements, the System Under Test (SUT) to include any subsystems that will be tested, and the test facility. To accomplish this the right team members with the necessary knowledge must be in the room and prepared to present their information and have an educated discussion to arrive at a comprehensive agreement about the desired end stated of the test. Our recommendations for the initial planning meeting are based on a thorough study of the STAT process and on lessons learned from actual planning meetings.