DATAWorks Archive

We created the Open Architectures Tradeoff (OAT) tool, a simple, computational game engine for rapidly exploring hypotheses about mission effectiveness in Battle Management Command and Control (BMC2). Each run of an OAT game simulates a military mission in contested airspace. Game objects represent U.S., adversary, and allied assets, each of which moves through the simulated airspace. Each U.S. asset has a Command and Control (C2) package the controls its actions—currently, neural networks form the basis of each U.S. asset’s C2 package. The weights of the neural network are randomized at the beginning of each game and are updated over the course of the game as the U.S. asset learns which of its actions lead to rewards, e.g., intercepting an adversary. Weights are updated via a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) altered to accommodate a Reinforcement Learning paradigm. OAT allows a user to winnow down the trade space that should be considered when setting up more expensive and time-consuming campaign models. OAT could be used to weed out bad ideas for “fast failure”, thus avoiding waste of campaign modeling resources. Questions can be explored via OAT such as: Which combination of system capabilities is likely to be more or less effective in a particular military mission? For example, in an early analysis, OAT was used to test the hypothesis that increases in U.S. assets’ sensor range always lead to increases in mission effectiveness, quantified as the percent of adversaries intercepted. We ran over 2500 OAT games, each time varying the sensor range of U.S. assets and the density of adversary assets. Results show that increasing sensor range did lead to an increase in military effectiveness—but only up to a certain point. Once the sensor range surpassed approximately 10-15% of the simulated airspace size, no further gains were made in the percent of adversaries intercepted. Thus, campaign modelers should hesitate to devote resources to exploring sensor range in isolation. More recent OAT analyses are exploring more complex hypotheses regarding the trade space between sensor range and communications range.

Goodness-of-fit (GOF) testing is used in many applications, including statistical hypothesis testing to determine if a set of data come from a hypothesized distribution. In addition, combined probability tests are extensively used in meta-analysis to combine results from several independent tests to asses an overall null hypothesis. This paper summarizes a study conducted to determine which GOF and/or combined probability test(s) can be used to determine if a set of data with relative small sample size comes from the standard uniform distribution, U(0,1). The power against different alternative hypothesis of several GOF tests and combined probability methods were examined. The GOF methods included: Anderson-Darling, Chi-Square, Kolmogorov-Smirnov, Cramér-Von Mises, Neyman-Barton, Dudewicz-van der Meulen, Sherman, Quesenberry-Miller, Frosini, and Hegazy-Green; while thecombined probability test methods included: Fisher's Combined Probability Test, Mean Z, Mean P, Maximum P, Minimum P, Logit P, and Sum Z. While no one method was determined to provide the best power in all situations, several useful methods to support model validation were identified.

Technology advances are accelerating at a rapid pace, with the potential to enable greater capability and power to the Warfighter. However, if human capabilities and limitations are not central to concepts, requirements, design, and development then new/upgraded weapons and systems will be difficult to train, operate, and maintain, may not result in the skills, job, grade, and manpower mix as projected, and may result in serious human error, injury or Soldier loss. The Army Human Systems Integration (HSI) program seeks to overcome these challenges by ensuring appropriate consideration and integration of seven technical domains: Human Factors Engineering (e.g., usability), Manpower, Personnel, Training, Safety and Occupational Health, Habitability, Force Protection and Survivability. The tradeoffs, constraints, and limitations occurring among and between these technical domains allows HSI to execute a coordinated, systematic process for putting the warfighter at the center of the design process – equipping the warfighter rather than manning equipment. To that end, the Army HSI Headquarters, currently as a directorate within the Army Headquarters Deputy Chief of Staff (DCS), G-1 develops strategies and ensures human systems factors are early key drivers in concepts, strategy, and requirements, and are fully integrated throughout system design, development, testing and evaluation, and sustainment The need to consider HSI factors early in the development cycle is critical. Too often, man-machine interface issues are not addressed until late in the development cycle (i.e. production and deployment phase) after the configuration of a particular weapon or system has been set. What results is a degraded combat capability, suboptimal system and system-of-systems integration, increased training and sustainment requirements, or fielded systems not in use. Acquisition test data are also good sources to glean HSI return on investment (ROI) metrics. Defense acquisition reports such as test and evaluation operational assessments identifies HSI factors as root causes when Army programs experience increase cost, schedule overruns, or low performance. This is identifiable by the number and type of systems that require follow-on test and evaluation (FOT&E), over reliance on field service representatives (FSRs), costly and time consuming engineering change requests (ECRs), or failures in achieving reliability, availability, and maintainability (RAM) key performance parameters (KPPs) and key system attributes (KSAs). In this presentation, we will present these data and submit several return on investment (ROI) metrics, closely aligned to the defense acquisition process, to emphasize and illustrate the value of HSI. Optimizing Warfighter-System performance and reducing human errors, minimizing risk of Soldier loss or injury, and reducing personnel and materiel life cycle costs produces data that are inextricably linked to early, iterative, and measurable HSI processes within the defense acquisition system.

The Air Force (AF) Human Systems Integration (HSI) program is led by the 711th Human Performance Wing’s Human Systems Integration Directorate (711 HPW/HP). 711 HPW HP provides direct support to system program offices and AF Major Commands (MAJCOMs) across the acquisition lifecycle from requirements development to fielding and sustainment in addition to providing home office support. With an ever-increasing demand signal for support, HSI practitioners within 711 HPW/HP assess HSI domain areas for human-centered risks and strive to ensure systems are designed and developed to safely, effectively, and affordably integrate with human capabilities and limitations. In addition to system program offices and MAJCOMs, 711 HPW/HP provides HSI support to AF Centers (e.g., AF Sustainment Center, AF Test Center), the AF Medical Service, and special cases as needed. The AF Global Strike Command (AFGSC) is the largest MAJCOM with several Programs of Record (POR), such as the B-1, B-2, and B-52 bombers, Intercontinental Ballistic Missiles (ICBM), Ground-Based Strategic Deterrent (GBSD), Airborne Launch Control System (ALCS), and other support programs/vehicles like the UH-1N. Mr. Anthony Thomas (711 HPW/HP), the AFGSC HSI representative, will discuss how 711 HPW/HP supports these programs at the MAJCOM headquarters level and in the system program offices.

Materials scientists, computer scientists and statisticians at LANL have teamed up to investigate how to make near real time decisions during fast-paced experiments. For instance, a materials scientist at a beamline typically has a short window in which to perform a number of experiments, after which they analyze the experimental data, determine interesting new experiments and repeat. In typical circumstances, that cycle could take a year. The goal of this research and development project is to accelerate that cycle so that interesting leads are followed during the short window for experiments, rather than in years to come. We detail some of our UQ work in materials science, including emulation, sensitivity analysis, and solving inverse problems, with an eye toward real-time decision making in experimental settings.

Model validation is a process for determining how accurate a model is when compared to a true value. The methodology uses uncertainty analysis in order to assess the discrepancy between a measured and predicted value. In the literature, there have been several area metrics introduced to handle these type of discrepancies. These area metrics were applied to problems that include aleatory uncertainty, epistemic uncertainty, and mixed uncertainty. However, these methodologies lack the ability to fully characterize the true dierences between the experimental and prediction data when mixed un- certainty exists in the measurements and/or in the predictions. This work will introduce a new area metric validation approach which aims to com- pensate for the shortcomings in current techniques. The approach will be described in detail and comparisons between existing metrics will be shown. To demonstrate its applicability the new area metric will be applied to a stagnation point calibration probe's surface predictions for a low-enthalpy conditions. For this application, testing was preformed in the Hypersonic Materials Environmental Test System (HYMETS) facility located at NASA Langley Research Center.

Computational fluid dynamics is now considered to be an indispensable tool for the design and development of scramjet engine components. Unfortunately, the quantification of uncertainties is rarely addressed with anything other than sensitivity studies, so the degree of confidence associated with the numerical results remains exclusively with the subject matter expert that generated them. This practice must be replaced with a formal uncertainty quantification process for computational fluid dynamics to play an expanded role in the system design, development, and flight certification process. Given the limitations of current hypersonic ground test facilities, this expanded role is believed to be a requirement by some in the hypersonics community if scramjet engines are to be given serious consideration as a viable propulsion system. The present effort describes a simple, relatively low cost, nonintrusive approach to uncertainty quantification that includes the basic ingredients required to handle both aleatoric (random) and epistemic (lack of knowledge) sources of uncertainty. The nonintrusive nature of the approach allows the computational fluid dynamicist to perform the uncertainty quantification with the flow solver treated as a ``black box''. Moreover, a large fraction of the process can be automated, allowing the uncertainty assessment to be readily adapted into the engineering design and development workflow. In the present work, the approach is applied to a model scramjet isolator problem where the desire is to validate turbulence closure models in the presence of uncertainty. In this context, the relevant uncertainty sources are determined and accounted for to allow the analyst to delineate turbulence model-form errors from other sources of uncertainty associated with the simulation of the facility flow.

No set of experimental data is perfect and researchers are aware that data from experimental studies invariably contain some margin of error. This is particularly true of studies with human subjects since human behavior is vulnerable to a range of intrinsic and extrinsic influences beyond the variables being manipulated in a controlled experimental setting. Potential sources of error may lead to wide variations in the interpretation of results and the formulation of subsequent implications. This talk will discuss specific sources of error and bias in the design of experiments and present systematic ways to overcome these effects. First, some of the basic errors in general experimental design will be discussed, including human errors, systematic errors and random errors. Second, we will explore specific types of experimental error that appear in human subjects research. Lastly, we will discuss the role of bias in experiments with human subjects. Bias is a type of systematic error that is introduced into the sampling or testing phase and encourages one outcome over another. Often, bias is the result of the intentional or unintentional influence that an experimenter may exert on the outcomes of a study. We will discuss some common sources of bias in research with human subjects, including biases in sampling, selection, response, performance execution, and measurement. The talk will conclude with a discussion of how errors and bias influence the validity of human subjects research and will explore some strategies for controlling these errors and biases

An overview of Deep Reinforcement Learning and it's recent successes in creating high performing agents. Covering it's application in “easy” environments up to massively complex multi-agent strategic environments. Will analyze the behaviors learned, discuss research challenges, and imagine future possibilities.

Collaborative autonomous sensor networks have recently been used in many applications including inspection, law enforcement, search and rescue, and national security. They offer scalable, low cost solutions which are robust to the loss of multiple sensors in hostile or dangerous environments. While often comprised of less capable sensors, the performance of a large network can approach the performance of far more capable and expensive platforms if nodes are effectively coordinating their sensing actions and data processing. This talk will summarize work to date at LLNL on distributed signal processing and decentralized optimization algorithms for collaborative autonomous sensor networks, focusing on ADMM-based solutions for detection/estimation problems and sequential greedy optimization solutions which maximize submodular functions, e.g. mutual information.

Cyber system defenders face the challenging task of continually protecting critical assets and information from a variety of malicious attackers. Defenders typically function within resource constraints, while attackers operate at relatively low costs. As a result, design and development of resilient cyber systems that support mission goals under attack, while accounting for the dynamics between attackers and defenders, is an important research problem. This talk will highlight decision support modeling challenges under uncertainty within non-cooperative cybersecurity settings. Multiple attacker-defender game formulations under uncertainty are discussed with steps for further research.

Software reliability models enable several quantitative predictions such as the number of faults remaining, failure rate, and reliability (probability of failure free operation for a specified period of time in a specified environment). This talk will describe recent efforts in collaboration with NASA, including (1) the development of an automated script for the SFRAT (Software Failure and Reliability Assessment Tool) to streamline application of software reliability methods to ongoing programs, (2) application to a NASA program, (3) lessons learned, (4) and future directions for model and tool development to support the practical needs of the software reliability and security assessment frameworks.

The U.S. Army Research Laboratory's (ARL) Essential Research Program (ERP) on Artificial Intelligence & Machine Learning (AI & ML) seeks to research, develop and employ a suite of AI-inspired and ML techniques and systems to assist teams of soldiers and autonomous agents in dynamic, uncertain, complex operational conditions. Systems will be robust, scalable, and capable of learning and acting with varying levels of autonomy, to become integral components of networked sensors, knowledge bases, autonomous agents, and human teams. Three specific research gaps will be examined: (i) Learning in Complex Data Environments, (ii) Resource-constrained AI Processing at the Point-of-Need and (iii) Generalizable & Predictable AI. The talk will highlight ARL's internal research efforts over the next 3-5 years that are connected, cumulative and converging to produce tactically-sensible AI-enabled capabilities for decision making at the tactical edge, specifically addressing topics in: (1) adversarial distributed machine learning, (2) robust inference & machine learning over heterogeneous sources, (3) adversarial reasoning integrating learned information, (4) adaptive online learning and (5) resource-constrained adaptive computing. The talk will also highlight collaborative research opportunities in AI & ML via ARL’s Army AI Innovation Institute (A2I2) which will harness the distributed research enterprise via the ARL Open Campus & Regional Campus initiatives.

"The wise use of one's resources will keep one from poverty." This is the definition of the proverbial saying "waste not, want not" according to www.dictionary.com. Indeed, one of the most common resources analysts encounter is text in free-form. This text might come from survey comments, feedback, websites, transcriptions of interviews, videos, etcetera. Notably, researchers have used wisely the information conveyed in text for many years. However, in many instances, the qualitative methods employed require numerous hours of reading, training, coding, and validating, among others. As technology continues to evolve, simple access to text data is blooming. For example, analysts conducting online studies can have thousands of text entries from participants' comments. Even without recent advances in technology analysts have had access to text in books, letters, and other archival data for centuries. One important challenge, however, is figuring out how to make sense of text data without investing a large number of resources, time, and the effort involved in qualitative methodology or "old-school" quantitative approaches (such as reading a collection of 200 books and counting the occurrence of important terms in the text). This challenge has been solved in the information retrieval field –a branch of computer science—with the implementation of a technique called latent semantic analysis (LSA; Manning, Raghavan, & Schütze, 2008) and a closely related technique called topic analysis (TA; SAS Institute Inc., 2018). Undoubtedly, other quantitative methods for text analysis, such as latent Dirichlet analysis (Blei, Ng, & Jordan, 2003), are also apt for the task of unveiling knowledge from text data, but we restrict the discussion in this presentation to LSA and TA because these exclusively deal with the underlying structure of the text rather than identifying clusters. In this presentation, we aim to make quantitative text analysis --specifically LSA and TA-- accessible to researchers and analysts from a variety of disciplines. We do this by leveraging understanding of a popular multivariate technique: principal components analysis (PCA). We start by describing LSA and TA by drawing comparisons and equivalencies to PCA. We make these comparisons in an intuitive, user-friendly manner and then through a technical description of mathematical statements, which rely on the singular value decomposition of a document-term matrix. Moreover, we explain the implementation of LSA and TA using statistical software to enable simple application of these techniques. Finally, we show a practical application of LSA and TA with empirical data of aircraft incidents.

Sensors that record sequences of measurements are now embedded in many products from wearable exercise watches to chemical and semiconductor manufacturing equipment. There is information in the shapes of the sensor stream curves that is highly predictive of a variety of outcomes such as the likelihood of a product failure event or batch yield. Despite this data now being common and readily available, it is often being used either inefficiently or not at all due to lack of knowledge and tools for how to properly leverage it. In this presentation, we will propose fitting splines to sensor streams and extracting features called functional principal component scores that offer a highly efficient low dimensional compression of the signal data. Then, we use these features as inputs into machine learning models like neural networks and LASSO regression models. Once one sees sensor data in this light, answering a wide variety of applied questions becomes a straightforward two stage process of data cleanup/functional feature extraction followed by modeling using those features as inputs.

Abstract: In applications where modeling and simulation runs are quick and cheap, space filling designs will give the tester all the information they need to make decisions about their system. In some applications however, this luxury does not exist, and each M&S run can be time consuming and expensive. In these scenarios, a sequential test approach provides an efficient solution where an initial screening is conducted, followed by an augmentation to fit specified models of interest. Until this point, no dedicated screening designs for UQ applications in resource constrained situations existed. Due to the Army’s frequent exposure to this type of situation, the need sparked a collaboration between Picatinny’s Statistical Methods and Analysis group and Professor V. Roshan Joseph of Georgie Tech, where a new type of UQ screening design was created. This paper provides a brief introduction to the design, its intended use, and a case study in which this new methodology was applied.

Whether it was in a Design of Experiments course or through your own work, you’ve no doubt seen and become well acquainted with the standard experimental design. You know the features: they’re “orthogonal” (no messy correlations to deal with), their correlation matrices are nice pretty diagonals, and they can only happen with run sizes of 4, 8, 12, 16, and so on. Well what if I told you that there existed optimal designs that defied convention. What if I told you that, yes, you can run an optimal design with, say, 5 factors in 9 runs. Or 10. Or even 11 runs! Join me as I show you a strange new world of optimal designs that are the best at what they do, even though they might not look very nice.

During the past several years, the DoD has begun applying rapid prototyping and fielding authorities—granted by Congress in the FY2016-FY2018 National Defense Authorization Acts (NDAA)—to many acquisition programs. Other programs have implemented an agile acquisition strategy where incremental capability is delivered in iterative cycles. As a result, Operational Test Agencies (OTA) have had to adjust their test processes to accommodate shorter test timelines and periodic delivery of capability to the warfighter. In this session, representatives from the Service OTAs will brief examples where they have implemented new practices and processes for conducting Operational Test on acquisition programs categorized as agile, DevOps, and/or Section 804 rapid-acquisition efforts. During the final 30 minutes of the session, a panel of OTA representatives will field questions from the audience concerning the challenges and opportunities related to test design, data collection, and analysis, that rapid-acquisition programs present.

NASA is investigating the dose-response relationship between quiet sonic boom exposure and community noise perceptions. This relationship is the key to possible future regulations that would replace the ban on commercial supersonic flights with a noise limit. We have built several Bayesian statistical models using pilot community study data. Using goodness of fit measures, we downselected to a subset of models which are the most appropriate for the data. From this subset of models we demonstrate how to calculate sample size requirements for a simplified example without any missing data. We also suggest how to modify the sample size calculation to account for missing data.

Swarms of small, fast speedboats can challenge even the most capable modern warships, especially when they operate in or near crowded shipping lanes. As part of the Navy’s operational testing of new ships and systems, at-sea live-fire tests against remote-controlled targets allow us to test our capability against these threats. To ensure operational realism, these events are minimally scripted and allow the crew to respond in accordance with their training. This is a trade-off against designed experiments, which ensure statistically optimal sampling of data from across the factor space, but introduce many artificialities. A recent test provided data on the effectiveness of naval gunnery. However, standard binomial (hit/miss) analyses fell short, as the number of misses was much larger than the number of hits. This prevented us from fitting more than a few factors and resulted in error bars so large as to be almost useless. In short, binomial analysis taught us nothing we did not already know. Recasting gunfire data from binomial (hit/miss) to continuous (time-to-kill) allowed us to draw statistical conclusions with tactical implications from these free-play, live-fire surface gunnery events. Using a censored-data analysis approach enabled us to make this switch and avoid the shortcomings of other statistical methods. Ultimately, our analysis provided the Navy with suggestions for improvements to its tactics and the employment of its weapons.

Recent data from Boeing’s Statistical Summary of Commercial Jet Airplane Accidents shows that Loss of Control - In Flight (LOC-I) is the leading cause of fatalities in commercial aviation accidents worldwide. The Commercial Aviation Safety Team (CAST), a joint government and industry effort tasked with reducing the rate of fatal accidents, requested that the National Aeronautics and Space Administration (NASA) conduct research on virtual day-visual meteorological conditions displays, such as synthetic vision, in order to combat LOC-I. NASA recently concluded a series of experiments using commercial pilots from various backgrounds to evaluate synthetic vision displays. This presentation will focus on the two most recent experiments: one conducted with the Navy’s Disorientation Research Device and one completed at NASA Langley Research Center that utilized the Microsoft HoloLens to display synthetic vision. Statistical analysis was done on aircraft performance data, pilot inputs, and a range of subjective questionnaires to assess the efficacy of the displays.

This presentation will use a notional armament system case-study to illustrate the use of M&S DOE, surrogate modeling, sensitivity analysis, multi-objective optimization and model calibration during early lifecycle development and design activities in the context of a new armament system. In addition to focusing on the statistician’s, data scientist’s, or analyst’s role and the key statistical techniques in engineering DoD systems, this presentation will also emphasize the non-statistical / engineering domain-specific aspects in a multidisciplinary design and development process which make uses of these statistical approaches at the subcomponent and subsystem-level as well as the end-to-end system modeling. A statistical engineering methodology which emphasizes the use of ‘virtual’ DOE-based model emulators developed at the subsystem-level and integrated using a systems-engineering architecture framework can yield a more tractable engineering problem compared to traditional ‘design-build-test-fix’ cycles or direct simulation of computationally expensive models. This supports a more informed prototype design for physical experimentation while providing a greater variety of materiel solutions thereby reducing development and testing cycles and time to field complex systems.

Combining physical measurements with computational models is key to many investigations involving validation and uncertainty quantification (UQ). This talk surveys some of the many approaches taken for validation and UQ, with large-scale computational models. Experience with such applications suggests classifications of different types of problems with common features (e.g. data size, amount of empiricism in the model, computational demands, availability of data, extent of extrapolation required, etc.). More recently, social and social-technical systems are being considered for similar analyses, bringing new challenges to this area. This talk will approaches for such problems and will highlight what might be new research directions for application and methodological development in UQ.

This case study highlights activities performed during the front-end process of a software development effort undertaken by the Fire Support Command and Control Program Office. This program office provides the U.S. Army, Joint and coalition commanders with the capability to plan, execute and deliver both lethal and non-lethal fires. Recently, the program office has undertaken modernization of its primary field artillery command and control system that has been in use for over 30 years. The focus of this case study is on the user-centered design process and activities taken prior to and immediately following contract award. A modified waterfall model comprised of three cyclic, yet overlapping phases (observation, visualization, and evaluation) provided structure for the iterative, user-centered design process. Gathering and analyzing data collected during focus groups, observational studies, and workflow process mapping, enabled the design team to identify 1) design patterns across the role/duty, unit and echelon matrix (a hierarchical organization structure), 2) opportunities to automate manual processes, 3) opportunities to increase efficiencies for fire mission processing, 4) bottlenecks and workarounds to be eliminated through design of the modernized system, 5) shortcuts that can be leveraged in design, 6) relevant and irrelevant content for each user population for streamlining access to functionality, 7) a usability baseline for later comparison (e.g., the number of steps and time taken to perform a task as captured in workflows for comparison to the same task in the modernized system), and provided the basis for creating visualizations using wireframes. Heuristic evaluations were conducted early to obtain initial feedback from users. In the next few months, usability studies will enable users to provide feedback based on actual interaction with the newly designed software. Included in this case study are descriptions of the methods used to collect user-centered design data, how results were visualized/documented for use by the development team, and lessons learned from applying user-centered design techniques during software development of a military field artillery command and control system.

Due to the immense potential use cases, configurations, and threat behaviors, thorough and efficient cyber testing is a significant challenge for the defense community. In this presentation, Phadke will present case studies where STO was successfully deployed for cyber testing, resulting in higher assurance, reduced schedule, and reduced testing cost. Phadke will also discuss importance first focusing on the engineering and science analysis, and only after that is complete, implementing statistical methods.

Advances in high performance computing have enabled detailed simulations of real-world physical processes, and these simulations produce large datasets. Even as detailed as they are, these simulations are only approximations of imperfect mathematical models, and furthermore, their outputs depend on inputs that are themselves uncertain. The main goal of a validation and uncertainty quantication methodology is to determine the uncertainty, that is, the relationship between the true value of a quantity of interest and its prediction by the simulation. The value of the computational results is limited unless the uncertainty can be quantied or bounded. Bayesian calibration is a common method for estimating model parameters and quantifying their associated uncertainties; however, calibration becomes more complicated when the data arise from dierent types of experiments. On an example from material science we will employ two types of data and demonstrate how one can obtain a set of material strength models that agree with both data sources.

Communication is critical both for analysts and decision-makers who rely on analysis to inform their choices. The Service Academies are responsible for educating men and women who may serve in both roles over the course of their careers. Analysts must be able to summarize their results concisely and communicate them to the decision-maker in a way that is relevant and actionable. Decision-makers understand that analytical results may carry with them uncertainty and be able to incorporate this uncertainty properly when evaluating different options. This panel explores the role of the US Service Academies in preparing their students for both roles. Featuring representatives from the US Air Force Academy, the US Naval Academy, and the US Military Academy, this panel will cover how future US Officers are taught to use and communicate with data. Topics include developing and motivating numerical literacy, understandings of uncertainty, how analysts should frame uncertainty to decision-makers, and how decision-makers should understand information presented with uncertainty. Panelists will discuss what they think the academies do well and areas that are ripe for improvement.

This talk focuses on a means to characterize the variability in multivariate data. This characterization, given in terms of both probability distributions and closed sets, is instrumental in assessing and improving the robustness/reliability properties of system designs. To this end, we propose the Sliced-Normal (SN) class of distributions. The versatility of SNs enables modeling complex multivariate dependencies with minimal modeling effort. A polynomial mapping is defined which injects the physical space into a higher dimensional (so-called) feature space on which a suitable normal distribution is defined. Optimization-based strategies for the estimation of SNs from data in both physical and feature space are proposed. The formulations in physical space yield non-convex optimization programs whose solutions often outperform the solutions in feature space. However, the formulations in feature space yield either an analytical solution or a convex program thereby facilitating their application to problems in high dimensions. The superlevel sets of a SN density have a closed semi-algebraic form making them amenable to rigorous uncertainty quantification methods. Furthermore, we propose a chance-constrained optimization framework for identifying and eliminating the effects of outliers in the prescription of such regions. These strategies can be used to mitigate the conservatism intrinsic to many methods in system identification, fault detection, robustness/reliability analysis, and robust design caused by assuming parameter independence and by including outliers in the dataset.

Uncertainty quantification (UQ) has emerged as the science of quantitative characterization and reduction of uncertainties in simulation and testing. Stretching across applied mathematics, statistics, and engineering, UQ is a multidisciplinary field with broad applications. A popular UQ method to analyze the effects of input variability and uncertainty on the system responses is generalized Polynomial Chaos Expansion (gPCE). This method was developed using applied mathematics and does not require knowledge of a simulation’s physics. Thus, gPCE may be used across disparate industries and is applicable to both individual component and system level simulations. The gPCE method can encounter problems when any of the input configurations fail to produce valid simulation results. gPCE requires that results be collected on a sparse grid Design of Experiment (DOE), which is generated based on probability distributions of the input variables. A failure to run the simulation at any one input configuration can result in a large decrease in the accuracy of a gPCE. In practice, simulation data sets with missing values are common because simulations regularly yield invalid results due to physical restrictions or numerical instability. We propose a statistical approach to mitigating the cost of missing values. This approach yields accurate UQ results if simulation failure makes gPCE methods unreliable. The proposed approach addresses this missing data problem by introducing an iterative machine learning algorithm. This methodology allows gPCE modelling to handle missing values in the sparse grid DOE. The study will demonstrate the convergence characteristics of the methodology to reach steady state values for the missing points using a series of simulations and numerical results. Remarks about the convergence rate and the advantages and feasibility of the proposed methodology will be provided. Several examples are used to demonstrate the proposed framework and its utility including a secondary air system example from the jet engine industry and several non-linear test functions. This is based on joint work with Dr. Mark Andrews at SmartUQ.

One of the primary roles of navy surface combatants is defending high-value units against attack by anti-ship cruise missiles (ASCMs). They accomplish this either by launching their interceptor missiles and shooting the ASCMs down with rapid-firing guns (hard kill), or through the use of deceptive jamming, decoys, or other non-kinetic means (soft kill) to defeat the threat. The wide range of hostile ASCM capabilities and the different properties of friendly defenses, combined with the short time-scale for defeating these ASCMs, makes this a difficult problem to study. IDA recently completed a study focusing on the extent to which friendly forces were vulnerable to massed ASCM attacks, and possible avenues for improvement. To do this we created a pair of complementary models with the combined flexibility to explore a wide range of questions. The first model employed a set of closed-form equations, and the second a time-dependent Monte Carlo simulation. This presentation discusses the thought processes behind the models and their relative strengths and weaknesses.

The concept of sequential testing has many disparate meanings. Often, for statisticians it takes on a purely mathematical context while possibly meaning multiple disconnected test events for some practitioners. Here we present a pedagogical approach to creating test designs involving constrained factors using JMP software. Recent experiences testing one of the U.S. military’s fast jet life support systems (LSS) serves as a case study and back drop to support the presentation. The case study discusses several lessons learned during LSS testing, applicable to all practitioners of scientific test and analysis techniques (STAT) and design of experiments (DOE). We conduct a short analysis to specifically determine a test region with a set of factors pertinent to modeling human breathing and the use of breathing machines as part of the laboratory setup. A comparison of several government and industry laboratory test points and regions with governing documentation is made, along with the our proposal for determining a necessary and sufficient test region for tests involving human breathing as a factor.

Steven Thorsen, Sarah Burke &

A challenging aspect of a system reliability assessment is integrating multiple sources of information, including component, subsystem, and full-system data, previous test data, or subject matter expert opinion. A powerful feature of Bayesian analyses is the ability to combine these multiple sources of data and variability in an informed way to perform statistical inference. This feature is particularly valuable in assessing system reliability where testing is limited and only a small number (or no failures at all) are observed. The F-35 is DoD's largest program; approximately one-third of the operations and sustainment cost is attributed to the cost of spare parts and the removal, replacement, and repair of components. The failure rate of those components is the driving parameter for a significant portion of the sustainment cost, and yet for many of these components, poor estimates of the failure rate exist. For many programs, the contractor produces estimates of component failure rates, based on engineering analysis and legacy systems with similar parts. While these are useful, the actual removal rates can provide a more accurate estimate of the removal and replacement rates the program anticipates to experience in future years. In this presentation, we show how we applied a Bayesian analysis to combine the engineering reliability estimates with the actual failure data to overcome the problems of cases where few data exist. Our technique is broadly applicable to any program where multiple sources of reliability information need be combined for the best estimation of component failure rates and ultimately sustainment costs.

Sensors abound in aerospace testing and while many scientists look at the data from a physics perspective, the comparative statistics information is what drives decisions. A multi-company project was comparing launch data from the 1980’s to a current set of data that included 30 sensors. Each sensor was designed to gather 3000 data points during the 3-second launch event. The data included temperature, acceleration, and pressure information. This talk will compare the data analysis methods developed for this project as well as the use of the new Functional Data Analysis tool within JMP for its ability to discern in-family launch performances.

Using Combinatorial Test Design methods to select software test scenarios has repeatedly delivered large efficiency and thoroughness gains - which begs the questions: • Why are these proven methods not used everywhere? • Why do some efforts to promote adoption of new approaches stagnate? • What steps can leaders take to introduce successfully introduce and spread new test design methods? For more than a decade, Justin Hunter has helped large global organizations across six continents adopt new test design techniques at scale. Working in some environments, he has felt like Sisyphus, forever condemned to roll a boulder uphill only to watch it roll back down again. In other situations, things clicked; teams smoothly adopted new tools and techniques, and impressive results were quickly achieved. In this presentation, Justin will discuss several common challenges faced by large organizations, explain why adopting test design tools is more challenging than adopting other types of development and testing tools, and share actionable recommendations to consider when you roll out new test design approaches.

Existing characterization of Europa’s environment is enabled by the Europa Clipper mission’s successful predecessors: Pioneer, Voyager, Galileo, and most recently, Juno. These missions reveal high intensity energetic particle fluxes at Europa’s orbit, requiring a multidimensional design challenge to ensure mission success (i.e. meeting Level 1 science requirements). Risk averse JPL Design Principles and the Europa Environment Requirement Document (ERD) dictate practices and policy, which if masterfully followed, are designed to protect Clipper from failure or degradation due to radiation. However, even if workmanship is flawless and no waivers are assessed, modeling errors, shielding uncertainty, and natural variation in the Europa environment are cause for residual concern. While failure and part degradation are of paramount concern, the occurrence of temporary outages, causing loss or degradation of science observations, is also a critical mission risk, left largely unmanaged by documents like the ERD. The referenced risk is monitored and assessed through a Project Systems Engineering-led mission robustness effort, which attempts balance the risk of science data loss with potential design cost and increased mission complexity required to mitigate such risk. The Science Sensitivity Model (SSM) was developed to assess mission and science robustness, with its primary goal being to ensure a high probability of achieving Level 1 (L1) science objectives by informing the design of a robust spacecraft, instruments, and mission design. This discussion will provide an overview of the problem, the model, and solution strategies. Subsequent presentations discuss the experimental design used to understand the problem space and the graphics and visualization used to reveal important conclusions.

Using the right visualizations as part of broad system and statistical Monte Carlo analysis supports interpretation of key drivers and relationships between variables, provides context about the full system, and communicates to non-statistician stakeholders. An experimental design was used to understand the relationships between instrument and spacecraft fault rate and recovery time in relation to the probability of achieving Europa Clipper science objectives during the Europa Clipper tour. Given spacecraft and instrument outages, requirement achievement checks were performed to determine the probability of meeting scientific objectives. Visualizations of the experimental design output enabled analysis of the full parameter set. Correlation between individual instruments and specific scientific objectives is not straight forward; some scientific objectives require a single instrument to be on at certain times and during varying conditions across the trajectory, while other science objectives require multiple instruments to function concurrently. By examining the input conditions that meet scientific objectives with the highest probability, and comparing those to trials with the lowest probability of meeting scientific objectives, key relationships could be visualized, enabling valuable mission and engineering design insights. Key system drivers of scientific success were identified, such as fault rate tolerance and recovery time required for each instrument and the spacecraft. Key steps, methodologies, difficulties and result-highlights are presented, along with a discussion of next steps and options for refinement and future analysis.

The Europa Clipper Science Sensitivity Model (SSM) can be thought of as a graph in which the nodes are mission requirements at ten levels in a hierarchy, and edges represent how requirements at one level of the hierarchy depend on those at lower levels. At the top of the hierarchy, there are ten nodes representing ten, Level 1 science requirements for the mission. At the bottom of the hierarchy, there are 100 or so nodes representing instrument-specific science requirements. In between, nodes represent intermediate science requirements with complex interdependencies. Meeting, or failing to meet, bottom-level requirements depends on the frequency of faults and the lengths of recovery times on the nine Europa Clipper instruments and the spacecraft. Our task was to design and analyze the results of a Monte Carlo experiment to estimate the probabilities of meeting the Level 1 science requirements based on parameters of the distributions of time between failures and of recovery times. We simulated an ensemble of synthetic missions in which failures and recoveries were random realizations from those distributions. The pass-fail status of the bottom-level instrument-specific requirements were propagated up the graph for each of the synthetic missions. Aggregating over the collection of synthetic missions produced estimates of the pass-fail probabilities for the Level 1 requirements. We constructed a definitive screening design and supplemented it with additional space-filing runs, using JMP 14 software. Finally, we used the vectors of failure and recovery parameters as predictors, and the pass-fail probabilities of the high-level requirements as responses, and built statistical models to predict the latter from the former. In this talk, we will describe the design considerations and review the fitted models and their implications for mission success.

A crucial step in the design and development of a fight vehicle, such as NASA's Space Launch System (SLS), is understanding its vibration behavior while in fight. Vehicle designers rely on low-cost finite element analysis (FEA) to predict the vibration behavior of the vehicle. During ground and flight tests, sensors are strategically placed at predefined locations that contribute the most vibration information under the assumption that FEA is accurate, producing points to validate the FEA models. This collaborative work focused on developing optimal sensor placement algorithms to validate FEA models against test data, and to characterize the vehicles vibration characteristics.

Abstract: Data scientists spend approximately 80% of their time preparing, cleaning, and feature engineering data sets. In this talk I will share use cases that show why this is important and why we need to do it. I will also describe the Earth System Grid Federation (ESGF) which is an open source effort providing a robust, distributed data and computation platform, enabling world wide access to Peta/Exa-scale scientific data. ESGF will help reduce the amount of effort needed for climate data preprocessing by integrating the necessary analysis and data sharing tools.

In support of the Navy’s effort to obtain improved outcomes through data-driven decision making, The Center for Naval Analyses’ Data Science Program (CNA/DSP) supports the performance-to plan(P2P) forum, which is co chaired by the Vice Chief of Naval Operations and the Assistant Secretary of the Navy (RD&A). The P2P forum provides senior Navy leadership forward looking performance forecasts, which are foundational to articulating Navy progress toward readiness and capability goals. While providing analytical support for this forum, CNA/DSP leveraged machine learning techniques, including Random Forests and Artificial Neural Networks, to develop improved estimates of future maintenance durations for the Navy. When maintenance durations exceed their estimated timelines, these delays can affect training, manning, and deployments in support of operational commanders. Currently, the Navy creates maintenance estimates during numerous timeframes including the program objective memorandum (POM) process, the Presidential Budget (PB), and at contract award leading to evolving estimates over time. The limited historical accuracy for these estimates, especially with the POM and PB estimates, have persisted over the last decade. These errors have led to a gap between planned funding and actual costs in addition to changes in the assets available for operational commanders each year. The CNA/DSP prediction model reduces the average error in forecasted maintenance duration days from 128 days to 31 days for POM estimates. Improvements in duration accuracy for the PB and contract award time frames were also achieved using similar ML processes. The data curation for these models involved numerous data sources of varying quality and required significant feature engineering to provide usable model inputs that could allow for forecasts over the Future Years Defense Program (FYDP) in order to support improved resource allocation and scheduling in support of the optimized fleet response training plan (OFRTP).

The application opportunities for behavioral analytics in the cybersecurity space are based upon simple realities. 1. The great majority of breaches across all cybersecurity venues is due to human choices and human error. 2. With communication and information technologies making for rapid availability of data, as well as behavioral strategies of bad actors getting cleverer, there is need for expanded perspectives in cybersecurity prevention. 3. Internally-focused paradigms must now be explored that place endogenous protection from security threats as an important focus and integral dimension of cybersecurity prevention. The development of cybersecurity monitoring metrics and tools as well as the creation of intrusion prevention standards and policies should always include an understanding of the underlying drivers of human behavior. As temptation follows available paths, cyber-attacks follow technology, business models, and behavioral habits. The human element will always be the most significant part in the anatomy of any final decision. Choice options – from input, to judgement, to prediction, to action – need to be better understood for their relevance to cybersecurity work. Behavioral Performance Indexes harness data about aggregate human participation in an active system, helping to capture some of the detail and nuances of this critically important dimension of cybersecurity.

Is there ever a place for the third dimension in visualizing data? Is the use of 3D inherently bad, or can a 3D visualization be used as an effective tool to communicate results? In this talk, I will show you how you can create beautiful 2D and 3D maps and visualizations in R using the rayshader package. Additionally, I will talk about the value of 3D plotting and how good aesthetic choices can more clearly communicate results to stakeholders. Rayshader is a free and open source package for transforming geospatial data into engaging visualizations using a simple, scriptable workflow. It provides utilities to interactively map, plot, and 3D print data from within R. It was nominated by Hadley Wickham to be one of 2018’s Data Visualizations of the Year for the online magazine Quartz.

With nearly continuous recording of sensor values now common, a new type of data called “functional data” has emerged. Rather than the individual readings being modeled, the shape of the stream of data over time is being modeled. As an example, one might model many historical vibration-over-time streams of a machine at start-up to identify functional data shapes associated with the onset of system failure. Functional Principal Components (FPC) analysis is a new and increasingly popular method for reducing the dimensionality of functional data so that only a few FPCs are needed to closely approximate any of a set of unique data streams. When combined with Design of Experiments (DoE) methods the response to be modeled in as fewest tests as possible is now the shape of a stream of data over time. Example analyses will be shown where the form of the curve is modeled as the function of several input variables allowing one to determine the input settings associated with shapes indicative of good or poor system performance. This allows the analyst to predict the shape of the curve as a function of the input variables.

The Algorithmic Warfare Cross Functional Team (AWCFT or Project Maven) organizes DoD stakeholders to enhance intelligence support to the warfighter through the use of automation and artificial intelligence. The AWCFT’s objective is to turn the enormous volume of data available to DoD into actionable intelligence and insights at speed. This requires consolidating and adapting existing algorithm-based technologies as well as overseeing the development of new solutions. This brief will describe some of the methodological challenges in test and evaluation that the Maven team is working through to facilitate speedy and agile acquisition of reliable and effective AI / ML capabilities.

Systems with autonomy are beginning to permeate civilian, industrial, and military sectors. Though these technologies have the potential to revolutionize our world, they also bring a host of new challenges in evaluating whether these tools are safe, effective, and reliable. The Institute for Defense Analyses is developing methodologies to enable testing systems that can, to some extent, think for themselves. In this talk, we share how we think about this problem and how this framing can help you develop a test strategy for your own domain.

The Satellite Affordability in LEO (SAL) model identifies the cheapest constellation capable of providing a desired level of performance within certain constraints. SAL achieves this using a combination of analytical models, statistical emulators, and geometric relationships. SAL is flexible and modular, allowing users to customize certain components while retaining default behavior in other cases. This is desirable if users wish to consider an alternative cost formulation or different types of payload. Uses for SAL include examining cost tradeoffs with respect to factors like constellation size and desired performance level, evaluating the sensitivity of constellation costs to different assumptions about cost behavior, and providing a first-pass look at what proliferated smallsats might be capable of. At this point, SAL is limited to Walker constellations with sun-synchronous, polar orbits.

The Icing Research Tunnel (IRT) at NASA Glenn Research Center follows the recommended practice for icing tunnel calibration outlined in SAE’s ARP5905 document. The calibration team has followed the schedule of a full calibration every five years with a check calibration done every six months following. The liquid water content of the IRT has maintained stability within in the specifications presented to customers that the variation is within +/- 10% of the calibrated, target measurement. With recent measurements and instrumentation errors, a more thorough assessment of error source was desired. By constructing statistical process control charts, the ability to determine how the instrument varies in the short term, mid term, and long term was gained. The control charts offer a view of instrument error, facility error, or installation changes. It was discovered that there was a shift from target to mean baseline thus leading to the study of the overall capability indices of the liquid water content measuring instrument to perform within specifications defined in the IRT. This presentation describes data processing procedures for the Multi-Element Sensor in the IRT, including collision efficiency corrections, canonical correlation analysis, Chauvenet’s Criterion for rejection of data, distribution check of data, and mean, median and mode for construction of control charts. Further data is presented to describe the repeatability of the IRT with the Multi-Element Sensor and the ability to maintain a stable process for the defined calibration schedule.

This briefing discusses the practical advantages of using the probabilistic programming language (PPL) Stan to answer statistical questions, especially those related to the quantification of uncertainty. Stan is a relatively new statistical tool that allows users to specify probability models and reason about the processes that generate the data they encounter. Stan has quickly become a popular language for writing statistical models because it allows one to specify rich (or sparse) Bayesian models using high level language. Further, Stan is fast, memory efficient, and robust. Stan requires users be explicit about the model they wish to evaluate, which makes the process of statistical modeling more transparent to users and decision makers. This is valuable because it forces practitioners to consider assumptions at the beginning of the model building procedure, rather than at the end (or not at all). In this sense, Stan is the opposite of a “black box” modeling approach. This approach may be tedious and labor intensive at first, but the pay-offs are large. For example, once a model is set-up inferential tasks all essentially automatic, as changing the model does not change the how one analyzes the data. This is a generic approach to inference. To illustrate these points, we use Stan to study a ballistic miss distance problem. In ballistic missile testing, the p-content circular error probable (CEP) in the circle that contains p percent of future shots fired, on average. Statistically, CEP is a bivariate prediction region, constrained by the model to be circular. In Frequentist statistics, the determination of CEP is highly dependent on the model fit, and a different calculation of CEP must be produced for each plausible model. However, with Stan, we can approach the CEP calculation invariant of the model we use to fit the data. We show how to use Stan to calculate CEP and uncertainty intervals for the parameters using summary statistics. Statistical practitioners can access Stan from several programming languages, including R and Python.

Hardly a week goes by without news of a cybersecurity breach or an attack by cyber adversaries against a nation’s infrastructure. These incidents have wide-ranging effects, including reputational damage and lawsuits against corporations with poor data handling practices. Further, these attacks do not require the direction, support, or funding of technologically advanced nations; instead, significant damage can be – and has been – done with small teams, limited budgets, modest hardware, and open source software. Due to the significance of these threats, it is critical to analyze past events to predict trends and emerging threats. In this document, we present an implementation of a cybersecurity taxonomy and a methodology to characterize and analyze all stages of a cyberattack. The chosen taxonomy, MITRE ATT&CK™, allows for detailed definitions of aggressor actions which can be communicated, referenced, and shared uniformly throughout the cybersecurity community. We translate several open source cyberattack descriptions into the analysis framework, thereby constructing cyberattack data sets. These data sets (supplemented with notional defensive actions) illustrate example Red Team activities. The data collection procedure, when used during penetration testing and Red Teaming, provides valuable insights about the security posture of an organization, as well as the strengths and shortcomings of the network defenders. Further, these records can support past trends and future outlooks of the changing defensive capabilities of organizations. From these data, we are able to gather statistics on the timing of actions, detection rates, and cyberattack tool usage. Through analysis, we are able to identify trends in the results and compare the findings to prior events, different organizations, and various adversaries.

The average and standard deviation of, say, strength or dimensional test data are basic engineering math, simple to calculate. What those resulting values actually mean, however, may not be simple, and can be surprisingly different from what a researcher wants to calculate and communicate. Mistakes can lead to overlarge estimates of spread, structures that are over- or under-designed and other challenges to understanding or communicating what your data is really telling you. This talk will discuss some common errors and missed opportunities seen in engineering and scientific analyses along with mitigations that can be applied through smart and efficient test planning and analysis. It will cover when - and when not - to report a simple mean of a dataset based on the way the data was taken; why ignoring this often either hides or overstates risk; and a standard method for planning tests and analyses to avoid this problem. And it will cover what investigators can correctly (or incorrectly) say about means and standard deviations of data, including how and why to describe uncertainty and assumptions depending on what a value will be used for. The presentation is geared toward the engineer, scientist or project manager charged with test planning, data analysis or understanding findings from tests and other analyses. Attenders' basic understanding of quantitative data analysis is recommended; more-experienced participants will grasp correspondingly more nuance from the pitch. Some knowledge of statistics is helpful, but not required. Participants will be challenged to think about an average as not just “the average”, but a valuable number that can and must relate to the engineering problem to be solved, and must be firmly based in the data. Attenders will leave the talk with a more sophisticated understanding of this basic, ubiquitous but surprisingly nuanced statistic and greater appreciation of its power as an engineering tool.

Causality in an engineered system pertains to how a system output changes due to a controlled change or intervention on the system or system environment. Engineered systems designs reflect a causal theory regarding how a system will work, and predicting the reliability of such systems typically requires knowledge of this underlying causal structure. The aim of this work is to introduce causal modeling tools that inform reliability predictions based on biased data sources and illustrate how these tools can inform data integration in practice. We present a novel application of the popular structural causal modeling framework to reliability estimation in an engineering application, illustrating how this framework can inform whether reliability is estimable and how to estimate reliability using data integration given a set of assumptions about the subject matter and data generating mechanism. When data are insufficient for estimation, sensitivity studies based on problem-specific knowledge can inform how much reliability estimates can change due to biases in the data and what information should be collected next to provide the most additional information. We apply the approach to a pedagogical example related to a real, but proprietary, engineering application, considering how two types of biases in data can influence a reliability calculation.

The identification of the Stuxnet worm in 2010 provided a highly publicized example of a cyber attack on an industrial control system. This raised public awareness about the possibility of similar attacks against other industrial targets – including critical infrastructure. Here, we use hypergames to analyze how adversarial perturbations can be used to manipulate a system using optimal control. Hypergames form an extension of game theory that models strategic interactions between players with significantly different perceptions of the game(s) they are playing. Previous work on hypergames has been limited to simple interactions, where a small set of discrete choices are available to each player. However, we apply hypergames to larger systems with continuous variables. Our results highlight that manipulating constraints can be a more effective attacker strategy than directly manipulating objective function parameters. Moreover, the attacker need not influence the underlying system to carry out a successful attack – it may be sufficient to deceive the defender controlling the system. Finally, we identify several characteristics that will make our analysis amenable to higher-dimensional control systems.

Military grade energetics are, by design, required to operate under extreme conditions. As such, warheads in a munition must demonstrate a high level of structural integrity in order to ensure safe and reliable operation by the Warfighter. In this example which involved an artillery munition, a systematic analytics-driven approach was executed which synthesized physical test data results with probabilistic analysis, non-destructive evaluation, modeling and simulation, and comprehensive risk analysis tools in order to determine the probability of a catastrophic event. Once the severity, probability of detection, occurrence, were synthesized, a model was built to determine the risk of a catastrophic event during firing which then accounts for defect growth occurring as a result of rough-handling. This comprehensive analysis provided a defensible, credible, and dynamic snapshot of risk while allowing for a transparent assessment of contribution to risk of the various inputs through sensitivity analyses. This paper will illustrate intersection of product safety, reliability, systems-safety policy, and analytics, and highlight the impact of a holistic multidisciplinary approach. The benefits of this rigorous assessment included quantifying risk to the user, supporting effective decision-making, improving resultant safety and reliability of the munition, and supporting triage and prioritization of future Non-Destructive Evaluation (NDE) screening efforts by identifying at-risk subpopulations.

Target detection in hyperspectral images (HSI) has broad value in defense applications, and neural networks have recently begun to be applied for this problem. A common criticism of neural networks is they give a point estimate with no uncertainty quantification (UQ). In defense applications, UQ is imperative because the cost of a false positive or negative is high. Users desire high confidence in either “target” or “not target” predictions, and if high confidence cannot be achieved, more inspection is warranted. One possible solution is Bayesian neural networks (BNN). Compared to traditional neural networks which are constructed by choosing a loss function, BNN take a probabilistic approach and place a likelihood function on the data and prior distributions for all parameters (weights and biases), which in turn implies a loss function. Training results in posterior predictive distributions, from which prediction intervals can be computed, rather than only point estimates. Heatmaps show where and how much uncertainty there is at any location and give insight into the physical area being imaged as well as possible improvements to the model. Using pytorch and pyro software, we test BNN on a simulated HSI scene produced using the Rochester Institute of Technology (RIT) Digital Imaging and Remote Sensing Image Generation (DIRSIG) model. The scene geometry used is also developed by RIT and is a detailed representation of a suburban neighborhood near Rochester, NY, named “MegaScene.” Target panels were inserted for this effort, using paint reflectance and bi-directional reflectance distribution function (BRDF) data acquired from the Nonconventional Exploitation Factors Database System (NEFDS). The target panels range in size from large to subpixel, with some targets only partially visible. Multiple renderings of this scene are created under different times of day and with different atmospheric conditions to assess model generalization. We explore the uncertainty heatmap for different times and environments on MegaScene as well as individual target predictive distributions to gain insight into the power of BNN.

Abstract. While fault testing a system with many factors each appearing at some number of levels, it may not be possible to test all combinations of factor levels. Most faults are caused by interactions of only a few factors, so testing interactions up to size t will often find all faults in the system without executing an exhaustive test suite. Call an assignment of levels to t of the factors a t-way interaction. A covering array is a collection of tests that ensures that every t-way interaction is covered by at least one test in the test suite. Locating arrays extend covering arrays with the additional feature that they not only indicate the presence of faults but locate the faulty interactions when there are no more than d faults in the system. If an array is (d, t)-locating, for every pair of sets of t-way interactions of size d, the interactions do not appear in exactly the same tests. This ensures that the faulty interactions can be differentiated from non-faulty interactions by the results of some test in which interactions from one set or the other but not both are tested. When the property holds for t-way interaction sets of size up to d, the notation (d, t ¯ ) is used. In addition to fault location, locating arrays have also been used to identify significant effects in screening experiments. Locating arrays are fairly new and few techniques have been explored for their construction. Most of the available work is limited to finding only one fault (d = 1). Known general methods require a covering array of strength t + d and produce many more tests than are needed. In this talk, we present Partitioned Search with Column Resampling (PSCR), a computational search algorithm to verify if an array is (d, t ¯ )-locating by partitioning the search space to decrease the number of comparisons. If a candidate array is not locating, random resampling is performed until a locating array is constructed or an iteration limit is reached. Algorithmic parameters determine which factor columns to resample and when to add additional tests to the candidate array. We use a 5 × 5 × 3 × 2 × 2 full factorial design to analyze the performance of the algorithmic parameters and provide guidance on how to tune parameters to prioritize speed, accuracy, or a combination of both. Last, we compare our results to the number of tests in locating arrays constructed for the factors and levels of real-world systems produced by other methods.

Scientific computing has undergone extraordinary growth in sophistication in recent years, enabling the simulation of a wide range of complex multiphysics and multiscale phenomena. Along with this increase in computational capability is the growing recognition that uncertainty quantification (UQ) must go hand-in-hand with numerical simulation in order to generate meaningful and reliable predictions for engineering applications. If not rigorously considered, uncertainties due to manufacturing defects, material variability, modeling assumptions, etc. can cause a substantial disconnect between simulation and reality. Packaging these complex computational models within an UQ framework, however, can be a significant challenge due to the need to repeatedly evaluate the model when even a single evaluation is time-consuming. This talk discusses efforts at NASA Langley Research Center (LaRC) to enable rapid UQ for problems with expensive computational models. Under the High Performance Computing Incubator (HPCI) program at LaRC, several open-source software libraries are being developed and released to provide access to general-purpose, state-of-the-art UQ algorithms. The common denominator of these methods is that they all expose parallelism among the model evaluations needed for UQ and, as such, are implemented to leverage HPC resources when available to achieve tremendous computational speedup. While the methods and software presented are broadly applicable, they will be demonstrated in the context of applications that have particular interest at NASA, including structural health management and trajectory simulation.

For chemical/biological testing, test chambers are sometimes designed with a vapor or aerosol homogeneity requirement. For example, a test community may require that the difference in concentration between any two test locations in a chamber be no greater than 20 percent. To validate the chamber, testers must demonstrate that such a requirement is met with a specified amount of certainty, such as 80 percent. With a validated chamber, multiple systems can be simultaneously tested at different test locations with the assurance that each system is exposed to nearly the same concentration. In some cases, however, homogeneity requirements are difficult to achieve. This presentation demonstrates a valid Bayesian method for testing probability of detection as a function of concentration in a chamber that fails to meet a homogeneity requirement. The demonstrated method of analysis is based on recent experience with an actual test chamber. Multiple systems are tested simultaneously at different locations in the chamber. Because systems tested in the chamber are exposed to different concentrations depending on these locations, the differences must be quantified to the greatest extent possible. To this end, data from the failed validation efforts are used to specify informative prior distributions for probability-of-detection modeling. Because these priors quantify and incorporate uncertainty in model parameters, they ensure that the final probability-of-detection model constitutes a valid comparison of the performance of the different systems.

There are many testing situations that historically involve a large number of runs. The use of experimental design methods can reduce the number of runs required to obtain the information desired. Example applications include wind tunnel test campaigns, computational experiments and live fire tests. In this work we present three case studies conducted under the auspices of the Science of Test Research Consortium comparing the information obtained via a historical experimental approach with the information obtained via an experimental design approach. The first case study involves a large scale wind tunnel experimental campaign. The second involves a computational fluid dynamics model of a missile through various speeds and angles of attack. The third case involves ongoing live fire testing involve hot surface testing. In each case, results suggest a tremendous opportunity to reduce experimental test efforts without losing test information.

An important property of any experimental design is its ability to detect active factors. For supersaturated designs, in which model parameters outnumber experimental runs, power is even more critical. In this talk, we review several popular supersaturated design construction criteria and analysis methods. We then demonstrate how simulation studies can be useful for practitioners in selecting a supersaturated design with regards to power to detect active factors. One of our findings based on an extensive simulation study is that although differences clearly exist among analysis methods, most supersaturated design construction methods are indistinguishable in terms of power. This conclusion can be reassuring for practitioners as supersaturated designs can then be sensibly chosen based upon convenience. For instance, the Bayesian D-optimal supersaturated designs can be easily constructed in JMP and SAS for any run size and number of factors. On the other hand, software for constructing E(s2)-optimal supersaturated designs is not as accessible.

https://s3.amazonaws.com/workshop-archives-2016/IDA+Workshop+2016/testsciencemeeting.ida.org/pdfs/1b-ExperimentalDesignMethodsandApplications.pdf

NASA has instituted requirements for establishing Agency-level safety thresholds and goals that define “long-term targeted and maximum tolerable levels of risk to the crew as guidance to developers in evaluating ‘how safe is safe enough’ for a given type of mission.” With the adoption of this policy for human space flight and with ongoing Agency efforts to increase formality in the development and review of the basis for risk acceptance, the decision-support demands placed on risk models are becoming more stringent at NASA. While these models play vital roles in informing risk acceptance decisions, they are vulnerable to incompleteness of risk identification, as well as to incomplete understanding of probabilities of occurrence, potentially leaving a substantial portion of the actual risk unaccounted for, especially for new systems. This presentation argues that management of uncertainty about the “actual” safety performance of a system must take into account the contribution of unknown and/or underappreciated (UU) risk. Correspondingly, responsible risk-acceptance decision-making requires the decision-maker to think beyond the model and address factors (e.g., organizational and management factors) that live outside traditional engineering risk models. This presentation advocates the use of a safety-case approach to risk acceptance decision-making.

The American National Standards Institute (ANSI) maintains a set of test standards that provide methods to characterize and determine the acceptability of radiation detection systems for use in Homeland security. With a focus on the environmental, electromagnetic, and mechanical functionality tests, we describe the test formulation and discuss challenges faced in administering the standard to include the assurance of comparable evaluations across multiple test facilities and the handling of systems that provide a non-standard, unit-less response. We present proposed solutions to these difficulties that are currently being considered in updated versions of the ANSI standards. We briefly describe a decision analytic approach that could allow for the removal of minimum performance requirements from the standards and enable the end user to determine system suitability based on operation-specific requirements.

This paper presents the results of two hypervelocity impact failure probabilistic risk assessments for critical wire bundles exposed aboard the Joint Polar Satellite System (JPSS-1) to an increased orbital debris environment at its 824 km, 98.8 deg inclination orbit. The first “generic” approach predicted the number of wires broken by orbital debris ejecta emerging from normal impact with multi-layer insulation (MLI) covering 36-, 18-, and 6-strand wire bundles at a 5 cm standoff using the Smooth Particle Hydrodynamic (SPH) code. This approach also included a mathematical approach for computing the probability that redundant wires were impacted then severed within the bundle. Based in part on the high computed risk of a critical wire bundle failure from the generic approach, an enhanced orbital debris protection design was examined, consisting of betacloth-reinforced MLI suspended at a 5 cm standoff over a seven layer betacloth and Kevlar blanket, draped over the exposed wire bundles. A second SPH-based risk assessment was conducted that also included the beneficial effects from the high (75 degree) obliquity of orbital debris impact and shadowing by other spacecraft components, and resulted in a considerably reduced likelihood of critical wire bundle failure compared to the original baseline design.

Statistical engineering has been defined as: “The study of how to best utilize statistical concepts, methods, and tool, and integrate them with IT and other relevant sciences to generate improved results.” A key principle is that significant untapped benefits are often achievable by integrating multiple methods in novel ways to address a problem, without having to invent new statistical techniques. In this presentation, we discuss the application of statistical engineering to the problem of design and analysis of mixture experiments when process variables are also involved. In such cases, models incorporating interaction between the mixture and process variables have been developed, but tend to require large designs and models. By considering models nonlinear in the parameters, also well known in the literature, we demonstrate how experimenters can utilize an alternative, iterative strategy in attacking such problems. We show that this strategy potentially saves considerable experimental time and effort, while producing models that are nearly as accurate as much larger linear models. These results are illustrated using two published data sets and one new data set, all involving interactive mixture and process variable problems.

Two responses to an expensive, time consuming test on a final product will be referred to as “adhesion” and “strength”. A screening test was performed on compounds that comprise the final product. These screening tests are multivariate profile measurements. Previous models to predict the expensive, time consuming test lacked accuracy and precision. Data visualization was used to guide a statistical engineering model that makes use of multiple statistical techniques. The modeling approach raised some interesting statistical questions for partial least square models regarding over-fitting and cross validation. Ultimately, the model interpretation and the visualization both make engineering sense and led to interesting insights regarding the product development program and screening compounds.

Co-Authors: Sean A. Commo, Ph.D., P.E. and Peter A. Parker, Ph.D., P.E. NASA Langley Research Center, Hampton, Virginia, USA Austin D. Overmeyer, Philip E. Tanner, and Preston B. Martin, Ph.D. U.S. Army Research, Development, and Engineering Command, Hampton, Virginia, USA. The application of statistical engineering to helicopter wind-tunnel testing was explored during two powered rotor entries. The U.S. Army Aviation Development Directorate Joint Research Program Office and the NASA Revolutionary Vertical Lift Project performed these tests jointly at the NASA Langley Research Center. Both entries were conducted in the 14- by 22-Foot Subsonic Tunnel with a small segment of the overall tests devoted to developing case studies of a statistical engineering approach. Data collected during each entry were used to estimate response surface models characterizing vehicle performance, a novel contribution of statistical engineering applied to powered rotor-wing testing. Additionally, a 16- to 47-times reduction in the number of data points required was estimated when comparing a statistically-engineered approach to a conventional one-factor-at-a-time approach.

Co-authors: Adam L. Pintar, Blaza Toman, and Dennis Leber. A task of the Domestic Nuclear Detection Office (DNDO) is the evaluation of radiation and nuclear (rad/nuc) detection systems used to detect and identify illicit rad/nuc materials. To obtain estimated system performance measures, such as probability of detection, and to determine system acceptability, the DNDO sometimes conduct large scale field tests of these systems at great cost. Typically, non adaptive designs are employed where each rad/nuc test source is presented to each system under test a predetermined and fixed number of times. This approach can lead to unnecessary cost if the system is clearly acceptable or unacceptable. In this presentation, an adaptive design with Bayesian decision theoretic foundations is discussed as an alternative to, and contrasted with, the more common single stage design. Although the basis of the method is Bayesian decision theory, designs may be tuned to have desirable type I and II error rates. While the focus of the presentation is a specific DNDO example, the method is applicable widely. Further, since constructing the designs is somewhat compute intensive, software in the form of an R package will be shown and is available upon request.

Over the past 10 years, a number of activities have incorporated statistical methods for the purpose of database development and associated uncertainty modeling. This presentation will highlight approaches taken in aerodynamic database development for space vehicle projects, specifically the Orion spacecraft and abort vehicle, and the Space Launch System (SLS) launch vehicle. Additionally, statistical methods have been incorporated into the Commercial Supersonic Transport Project for test technique development and optimization as well as a certification prediction methodology, which is planned to be verified with the Low-Boom Flight Demonstrator data. Discussion will conclude with the use of statistical methods for quality control and assurance in the NASA Langley Research Center ground testing facilities related to our Check Standard Project and characterization and calibration testing.

Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used.

James Wisnowski provides training and consulting services in Design of Experiments, Predictive Analytics, Reliability Engineering, Quality Engineering, Text Mining, Data Visualization, and Forecasting to government and industry. Previously, he spent a career in analytics for the government. He retired from the Air Force having had leadership positions at the Pentagon, Air Force Academy, Air Force Operational Test and Evaluation Center, and units across the Air Force. He has published numerous papers in technical journals and presented several invited conference presentations. He was co-author of the Design and Analysis of Experiments by Douglas Montgomery: A Supplement for using JMP.

Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used.

Heath Rushing is the cofounder of Adsurgo and author of the book Design and Analysis of Experiments by Douglas Montgomery: A Supplement for using JMP. Previously, he was the JMP Training Manager at SAS, a quality engineer at Amgen, an assistant professor at the Air Force Academy, and a scientific analyst for OT&E in the Air Force. In addition, over the last six years, he has taught Science of Tests (SOT) courses to T&E organizations throughout the DoD.

A question about whether how post-processing affects the flammability of a given metal an oxygen enriched environment could be answered with the sensitivity of a standard test that is typically used to measure differences in flammability between different metals. The principal investigator was familiar with design of experiments (DOE) and wanted to optimize the value of the information to be gained. This talk will focus on the interchange between the engineer and a statistician. Their negotiations will illustrate the process of clarifying the problem, planning the test including choosing factors and responses, running the experiment and analyzing and reporting on the data. It will focus on the choices made to help squeeze extra knowledge from each test run and leverage statistics to result in increased engineering insight.

For the past 7 years, USAF DT&E has been exploring ways to adapt the principles of experimental design to rapidly evolving developmental test articles and test facilities – often with great success. This paper discusses three case studies that span the range of USAF DT&E activities from EW to Ground Test to Flight Test and shows the truly revolutionary impact Fisher’s DOE can have on development. The Advanced Strategic and Tactical Expendable (ASTE) testing began in 1990 to develop, enhance, and test new IR flares, flare patterns, and dispense tactics. More than 60 aircraft & flare types have been tested. The typical output is a “Pancake Plot” of 200+ “cases” of flare, aspect angle, range, elevation, and flare effectiveness using a stop-light chart (red-yellow-green) approach. In usual testing – ~3000 flare engagements costing $1M to participate. The response, troublingly enough, is binary – 15-30 binomial response trials measuring P(Decoy). Binary responses are information-poor. Legacy testing does not assess present statistical power in reporting P(decoy) results. Analysts investigated replacing P(Decoy) w/ continuous metrics – e.g. time to decoy. This research is ongoing. We found we could spread the replicates out to examine 3x to 5x more test conditions without affecting power materially. Analysis with the Generalized Linear Model (GLZ) replaced legacy “cases” analysis with 75% improvement to confidence intervals with same data. We are seeking to build a Monte-Carlo simulation to estimate how many runs are required in a logistics regression model to achieve adequate power. We hope to reduce customer expenditures for flare information by as much as 50%. Co-authors J. Higdon, B, Knight. AEDC completed a major upgrade with new nozzle hardware to vary Mach. The Arnold Transonic Wind Tunnel 4T spans the range of flows from subsonic to approximately M9. The new wind tunnel was to be computer controlled. A number of key instrumentation improvements were made at the same time. The desire was to calibrate the resulting modified tunnel. The last calibration of 4T was 25 years ago in 1990. The calibration ranged across the full range of Mach and pressure capabilities, spanning a four-D space: pressure, Mach, wall angle, and wall porosity. Both the traditional OFAT effort – vary one factor at a time – and a parallel DOE effort were run to compare design, execution, modeling, and prediction capabilities against cost and time to run. The robust embedded face-centered CCD DOE design (J. Simpson and D. Landman ’05) employed 75 vs. 176 OFAT runs. The RSM design achieved 57% run savings. Due to an admirable discipline in randomization during the DOE trials, the smaller design required longer to run. As a result of using the DOE approach, engineers found it easier predict offcondition tunnel operating characteristics using RSM models, optimize facility flow quality for any given test condition. In addition, the RSM regression models support future “spot check” calibration in future by comparing predictions to measured values. If the measurement falls within the prediction interval, the existing calibration still appropriate. AEDC is using split-plot style designs a for current wind tunnel probe calibration. Co-author: Dr Dough Garrard.

Co-authors: Dr. Darryl Ahner, Director, STAT COE Dr. Lenny Truett, STAT COE Mr. Scott Wacker, 96 TG/OL-ACS. This presentation will present the benefits of sequential testing and demonstrate how sequential testing can be used to address complex test conditions by developing well controlled early experiments to explore basic questions before proceeding to full-scale testing. This approach can result in increased knowledge and decreased cost. As of FY13 the Air Force had spent an estimated $47M on dry bay fire testing making fire the largest cost contributor for Live Fire Test and Evaluation (LFT&E) programs. There is currently an estimated 60% uncertainty in total platform vulnerable area (Av) driven by probability of kill (PK) due to ballistically ignited fires. A large part of this uncertain comes from the fact that current spurt modeling does not predict fuel spurt delay with reasonable accuracy despite a large amount of test data. A low-cost sequential approach was developed to improve spurts models. Initial testing used a spherical projectile to test 10 different factors in a definitive screening design. Once the list of factors was refined, a second phase of testing determined if a suitable methodology could be developed to scaled results using water as a surrogate for JP-8 fuel. Finally testing was performed with cubical projectiles to evaluate the effect of fragment orientation. The entire cost for this effort was less than one or two typical full-scale live fire tests.

Co-Authors: Sean A. Commo, Ph.D., P.E. and Peter A. Parker, Ph.D., P.E. NASA Langley Research Center, Hampton, Virginia, USA Austin D. Overmeyer, Philip E. Tanner, and Preston B. Martin, Ph.D. U.S. Army Research, Development, and Engineering Command, Hampton, Virginia, USA. The application of statistical engineering to helicopter wind-tunnel testing was explored during two powered rotor entries. The U.S. Army Aviation Development Directorate Joint Research Program Office and the NASA Revolutionary Vertical Lift Project performed these tests jointly at the NASA Langley Research Center. Both entries were conducted in the 14- by 22-Foot Subsonic Tunnel with a small segment of the overall tests devoted to developing case studies of a statistical engineering approach. Data collected during each entry were used to estimate response surface models characterizing vehicle performance, a novel contribution of statistical engineering applied to powered rotor-wing testing. Additionally, a 16- to 47-times reduction in the number of data points required was estimated when comparing a statistically-engineered approach to a conventional one-factor-at-a-time approach.

In this case study we demonstrate an application of Pratt & Whitney’s “Design for Variation” discipline applied to the task of a roller bearing design. The ultimate goal, in this application, was to utilize test data from the “real world” to calibrate a computer model used for design and ensure that roller bearing designs obtained from this model were optimized for maximum robustness to major sources of variation in bearing manufacture and operation. The “Design for Variation” process provides engineers with many useful analysis results even before real world data is applied: high fidelity sensitivity analysis, uncertainty analysis (quantifying a baseline risk of failing to meet design intent) and model verification. The combining of real world data and Bayesian statistical methods that Design for Variation employs to calibrate models, however, goes a step further, validating the accuracy of the model’s outputs and quantifying any bias between the model and the real world. As a result of this application, the designers were able to identify the sources of the bias and correct the model’s physics-based aspects to more accurately model reality. The improved model is now integrated into all successive bearing design activities. The benefits of this method, which only required a small amount of high quality test data, are now available to all present and future roller bearing designs.

Modeling and simulation (M&S) is often an important element of operational evaluations of effectiveness, suitability, survivability, and lethality. For example, the testing of new systems designed to operate against advanced foreign threats, as well as the testing of systems of systems, will involve the use of M&S to examine scenarios that cannot be created using live testing. In order to have an adequate understanding of, and confidence in, the results obtained from M&S, statistically rigorous techniques should be applied to the validation process wherever possible. Design of experiments methodologies should be employed to determine what live and simulation data are needed to support rigorous validation, and formal statistical tests should be used to compare live and simulated data. This talk will discuss the importance of M&S in operational testing through a few examples, and outline several statistically rigorous techniques for validation.

The key to acquiring a cybersecure system is the ability to drive considerations about security from the operational level into the tactics and procedures. For T&E to support development and acquisition decisions, we must also adopt the perspective that a cyberattack is an attack on the mission using technology. A well-defined process model linking tools, tasks, and operators to mission performance supports this perspective. We will discuss an approach based on best practices learned from various DHS programs.

Display clutter has been defined as an unintended effect of display imagery obscuring or confusing other information or that may not be relevant to the task at hand. Negative effects of clutter on user performance have been documented; however, some work suggests differential effects with workload variations and measurement method. Existing measures of clutter either focus on physical display characteristics or user perceptions and they generally exhibit weak correlations with task performance, limiting utility for application in safety-critical domains. These observations have led to a new integrated measure of clutter accounting for display data, user knowledge and patterns of visual attention. Due to limited research on clutter effects in domains other than aviation, empirical studies have been conducted to evaluate the new measure in automobile driving. Data-driven measures and subjective perceptions of clutter were collected along with patterns of visual attention allocation when drivers searched ‘high’ and ‘low’ clutter navigation displays. The experimental paradigm was manipulated to include both presentation-based trials with static display images or use of a dynamic, driving simulator. The new integrated measure was more strongly correlated with driver performance than other, previously-developed measures of clutter. Results also revealed clutter to significantly alter attention and degrade performance with static displays but to have little to no effects in driving simulation. Findings corroborate trends in the literature that clutter has its greatest effects on behavior in domains requiring extended attention to displays, such as map-search, compared to use of displays to support secondary tasks, such as nav aids in driving. Integrating display data and user knowledge factors with patterns of attention shows promise for clutter measurement

Display clutter has been defined as an unintended effect of display imagery obscuring or confusing other information or that may not be relevant to the task at hand. Negative effects of clutter on user performance have been documented; however, some work suggests differential effects with workload variations and measurement method. Existing measures of clutter either focus on physical display characteristics or user perceptions and they generally exhibit weak correlations with task performance, limiting utility for application in safety-critical domains. These observations have led to a new integrated measure of clutter accounting for display data, user knowledge and patterns of visual attention. Due to limited research on clutter effects in domains other than aviation, empirical studies have been conducted to evaluate the new measure in automobile driving. Data-driven measures and subjective perceptions of clutter were collected along with patterns of visual attention allocation when drivers searched ‘high’ and ‘low’ clutter navigation displays. The experimental paradigm was manipulated to include both presentation-based trials with static display images or use of a dynamic, driving simulator. The new integrated measure was more strongly correlated with driver performance than other, previously-developed measures of clutter. Results also revealed clutter to significantly alter attention and degrade performance with static displays but to have little to no effects in driving simulation. Findings corroborate trends in the literature that clutter has its greatest effects on behavior in domains requiring extended attention to displays, such as map-search, compared to use of displays to support secondary tasks, such as nav aids in driving. Integrating display data and user knowledge factors with patterns of attention shows promise for clutter measurement

This brief talk will focus on the process of human-machine trust in context of automated intelligence tools. The trust process is multifaceted and this talk will define concepts such as trust, trustworthiness, trust behavior, and will examine how these constructs might be operationalized in user studies. The talk will walk through various aspects of what might make an automated intelligence tool more or less trustworthy. Further, the construct of transparency will be discussed as a mechanism to foster shared awareness and shared intent between humans and machines.

To enable quiet supersonic passenger flight overland, NASA is providing national and international noise regulators with a low-noise sonic boom database. The database will consist of dose-response curves, which quantify the relationship between low-noise sonic boom exposure and community annoyance. The recently-updated international standard for environmental noise assessment, ISO 1996-1:2016, references multiple fitting methods for dose-response analysis. One of these fitting methods, Fidell’s community tolerance level method, is based on theoretical assumptions that fix the slope of the curve, allowing only the intercept to vary. This fitting method is applied to an existing pilot sonic boom community annoyance data set from 2011 with a small sample size. The purpose of this exercise is to develop data collection and analysis recommendations for future sonic boom community annoyance surveys.

When Modeling and Simulation (M&S) is used as part of operational evaluations of effectiveness, suitability, survivability, or lethality, the M&S capability should first be rigorously validated to ensure it is representing the real world accurately enough for the intended use. Specifically, we need to understand and characterize the usefulness and limitations of the M&S, especially in terms of uncertainty! Many statistical techniques are available to compare M&S output with live test data. This presentation will describe and present results from a simulation study conducted to determine which techniques provide the highest statistical power to detect differences in mean and variance between live and sim for a variety of data types and sizes.

One use for Modeling and Simulation (M&S) in Test and Evaluation (T&E) is to produce weapon miss distances to evaluate the effectiveness of a weapons. This is true for the Air Intercept Missile-9X (AIM-9X) T&E community. Since flight testing is expensive, the test program uses relatively few flight tests at critical conditions, and supplements those data with large numbers of miss distances from simulated tests across the weapons operational space. However, before the model and simulation is used to predict performance it must first be validated. Validation is an especially daunting task when working with a limited number of live test data. In this presentation we shown that even with a limited number of live test points (e.g. 16 missile fires), we can still perform a statistical analysis for the validation. Specifically, we introduce a validation technique known as Fisher’s Combined Probability Test and we show how to apply Fisher’s test to validate the AIM-9X model and simulation.

Many software reliability models characterize the number of faults detected during the testing process as a function of testing time, which is performed over multiple stages. Typically, the later stages are progressively more expensive because of the increased number of personnel and equipment required to support testing as the system nears completion. Such transitions from one stage of testing to the next change in the operational environment. One statistical approach to combine software reliability growth models in a manner capable of characterizing multi-stage testing is the concept of a change-point process, where the intensity of a process experiences a distinct change at one or more discrete times during testing. Thus, change-point processes can be used to model change in the failure rate of software due to changes in the testing strategy and environment, integration testing, and resource allocation as it proceeds through multiple stages of testing. This presentation generalizes change-point models to the heterogeneous case, where fault detection before and after a change-point can be characterized by distinct nonhomogeneous Poisson processes (NHPP). Experimental results suggest that heterogeneous change-point models better characterize some failure data sets, which can improve the applicability of software reliability models to large-scale software systems that are tested over multiple stages.

Several optimization models are described for allocating resources to different testing activities in a system’s reliability growth program. These models assume availability of an underlying reliability growth model for the system, and capture the tradeoffs associated with focusing testing resources at various levels (e.g., system, subsystem, component) and/or how to divide resources within a given level. In order to demonstrate insights generated by solving the model, we apply the optimization models to an example series-parallel system in which reliability growth is assumed to follow the Crow/AMSAA reliability growth model. We then demonstrate how the optimization models can be extended to incorporate uncertainty in Crow/AMSAA parameters.

Since its publication, Statistical Methods for Reliability Data by W. Q. Meeker and L. A. Escobar has been recognized as a foundational resource in analyzing failure time to and survival data. Along with the text, the authors provided an S-Plus software package, called SPLIDA, to help readers utilize the methods presented in the text. Today, R is the most popular statistical computing language in the world, largely supplanting S-Plus. The SMRD package is the result of a multi-year effort to completely rebuild SPLIDA, to take advantage of the improved graphics and workflow capabilities available in R. This presentation introduces the SMRD package, outlines the improvements and shows how the package works seamlessly with the rmarkdown and shiny packages to dramatically speed up your workflow. The presentation concludes with a discussion on what improvements still need to be made prior to publishing the package on the CRAN.

One of the most powerful features of Bayesian analyses is the ability to combine multiple sources of information in a principled way to perform inference. For example, this feature can be particularly valuable in assessing the reliability of systems where testing is limited for some reason (e.g., expense, treaty). At their most basic, Bayesian methods for reliability develop informative prior distributions using expert judgment or similar systems. Appropriate models allow the incorporation of many other sources of information, including historical data, information from similar systems, and computer models. I will introduce the approach and then consider examples from defense acquisition and lifecycle extension, focusing on the strengths and weaknesses of the Bayesian analyses.

The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals. The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface. When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help. This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.

The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals. The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface. When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help. This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.

Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.

Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.

NASA’s Airspace Technology Demonstration (ATD) project was developed to facilitate the transition of mature air traffic management technologies from the laboratory to operational use. The first ATD focused on an integrated set of advanced NASA technologies to enable efficient arrival operations in high-density terminal airspace. This integrated arrival solution was validated and verified in laboratories and transitioned to a field prototype for an operational demonstration. Within NASA, this was a collaborative effort between Ames and Langley Research Centers involving a multi-year iterative experimentation process consisting of a series of sequential batch computer simulations and human-in-the-loop experiments, culminating in a flight test. Designing and analyzing the flight test involved a number of statistical challenges. There were several variables which are known to impact the performance of the system, but which could not be controlled in an operational environment. Changes in the schedule due to weather and the dynamic positioning of the aircraft on the arrival routes resulted in the need for a design that could be modified in real-time. This presentation describes a case study from a recent NASA flight test, highlights statistical challenges, and discusses lessons learned.

Many modern processes generate complex data records not readily analyzed by traditional techniques. For example, a single observation from a process might be a radar signal consisting of n pairs of bivariate data described via some functional relation between reflection and direction. Methods are examined here for detecting changes in such sequences from some known or estimated nominal state. Additionally, estimates of the degree of change (scale, location, frequency, etc.) are desirable and discussed. The proposed methods are designed to take advantage of all available data in a sequence. This can become unwieldy for long sequences of large-sized observations, so dimension reduction techniques are needed. In order for these methods to be as widely applicable as possible, we make limited distributional assumptions and so we propose new nonparametric and Bayesian tools to implement these estimators.

Spectrograms (i.e., squared magnitude of short-time Fourier transform) are commonly used as features to classify audio signals in the same way that social media companies (e.g., Google, Facebook, Yahoo) use images to classify or automatically tag people in photos. However, a serious problem arises when using spectrograms to classify acoustic signals, in that the user must choose the input parameters (hyperparameters), and such choices can have a drastic effect on the accuracy of the resulting classifier. Further, considering all possible combinations of the hyperparameters is a computationally intractable problem. In this study, we simplify the problem making it computationally tractable, explore the utility of response surface methods for sampling the hyperparameter space, and find that response surface methods are a computationally efficient means of identifying the hyperparameter combinations that are likely to give the best classification results.

The process for testing military systems which are largely software intensive involves techniques and procedures often different from those for hardware-based systems. Much of the testing can be performed in laboratories at many of the acquisition stages, up to operational testing. Testing software systems is not different from testing hardware-based systems in that testing earlier and more intensively benefits the acquisition program in the long run. Automated testing of software systems enables more frequent and more extensive testing, allowing for earlier discovery of errors and faults in the code. Automated testing is beneficial for unit, integrated, functional and performance testing, but there are costs associated with automation tool license fees, specialized manpower, and the time to prepare and maintain the automation scripts. This presentation discusses some of the features unique to automated software testing and offers a framework organizations can implement to make the business case for, to organize for, and to execute and benefit from automating the right aspects of their testing needs. Automation has many benefits in saving time and money, but is most valuable in freeing test resources to perform higher value tasks.

The process for testing military systems which are largely software intensive involves techniques and procedures often different from those for hardware-based systems. Much of the testing can be performed in laboratories at many of the acquisition stages, up to operational testing. Testing software systems is not different from testing hardware-based systems in that testing earlier and more intensively benefits the acquisition program in the long run. Automated testing of software systems enables more frequent and more extensive testing, allowing for earlier discovery of errors and faults in the code. Automated testing is beneficial for unit, integrated, functional and performance testing, but there are costs associated with automation tool license fees, specialized manpower, and the time to prepare and maintain the automation scripts. This presentation discusses some of the features unique to automated software testing and offers a framework organizations can implement to make the business case for, to organize for, and to execute and benefit from automating the right aspects of their testing needs. Automation has many benefits in saving time and money, but is most valuable in freeing test resources to perform higher value tasks.

In recent years, software testing techniques based on formal methods have made their way into industrial practice as a supplement to system and unit testing. I will discuss three core techniques that have proven particularly amenable to transition: 1) Concolic execution, which enables the automatic generation of high-coverage test suites; 2) Property-based randomized testing, which automatically checks sequences of API calls to ensure that expected high-level behavior occurs; and 3) Bounded model checking, which enables systematic exploration of both concrete systems and high-level models to check temporal properties, including ordering of events and timing requirements.

Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced

Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced

Surveys are commonly used to evaluate the quality of human-system interactions during the operational testing of military systems. Testers use Likert-type response options to measure the intensity of operators’ subjective experiences (e.g., usability, workload) while operating the system. Recently, appropriate methods for analyzing Likert data have become a point of contention within the operational test community. Some argue that Likert data can be analyzed with parametric techniques whereas others argue that only non-parametric techniques should be used. However, the reasons stated for holding a particular view are rarely tied to findings in the empirical literature. This presentation sheds light on this debate by reviewing existing research on how parametric statistics affect the conclusions drawn from Likert data and debunk common myths and misunderstandings about the nature of Likert data within the operational test community and academia.

The System Usability Scale (SUS) was developed by John Brooke in 1986 “to take a quick measurement of how people perceived the usability of (office) computer systems on which they were working.” The SUS is a 10-item, generic usability scale that is assumed to be system agnostic, and it results in a numerical score that ranges from 0-100. It has been widely employed and researched with non-military systems. More recently, it has been strongly recommended for use with military systems in operational test and evaluation, in part because of its widespread commercial use, but largely because it produces a numerical score that makes it amendable to statistical operations. Recent lessons learned with SUS in operational test and evaluation strongly question its use with military systems, most of which differ radically from non-military systems. More specifically, (1) usability measurement attributes need to be tailored to the specific system under test and meet the information needs of system users, and (2) a SUS numerical cutoff score of 70—a common benchmark with non-military systems—does not accurately reflect “system usability” from an operator or test team perspective. These findings will be discussed in a psychological and human factors measurement context, and an example of system-specific usability attributes will be provided as a viable way forward. In the event that the SUS is used in operational test and evaluation, some recommendations for interpreting the outcomes will be provided.

Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.

Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.

Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.

“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning. Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”

“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning. Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”

In May 2016, the NASA Langley Research Center’s Engineering Director stood up a group consisting of employees within the directorate to assess the current state of engineering being done by the organization. The group was chartered to develop ideas, through investigation and benchmarking of other organizations within and outside of NASA, for how engineering should look in the future. This effort would include brainstorming, development of recommendations, and some detailed implementation plans which could be acted upon by the directorate leadership as part of an enduring activity. The group made slow and sporadic progress in several specific, self-selected areas including: training and development; incorporation of non-traditional engineering disciplines; capturing and leveraging historical data and knowledge; revolutionizing project documentation; and more effective use of design reviews. The design review investigations have made significant progress by leveraging lessons learned and techniques gained by collaboration with operations research analysts within the local Lockheed Martin Center for Innovation (the “Lighthouse”) and pairing those techniques with advanced data analysis tools available through the IBM Watson Content Analytics environment. Trials with these new techniques are underway but show promising results for the future of providing objective, quantifiable data from the design review environment – an environment which to this point has remained essentially unchanged for the past 50 years.

Due to small Tactical Data Link testing windows, only commonly used messages are tested resulting in the evaluation of only a small subset of all possible Link 16 messages. To increase the confidence that software design and implementation issues are discovered in the earliest phases of government acceptance testing, Marine Corps Tactical Systems Support Activity (MCTSSA) Instrumentation and Data Management Section (IDMS) successfully implemented an extension of the traditional form of Design of Experiments (DOE), called Combinatorial Testing (CT). CT was utilized to reduce the human bias and inconsistencies involved in Link 16 testing and replace them with a thorough test that can validate a system's ability to properly consume all of the possible valid combinations of Link 16 message field values. MCTSSA's unique team of subject matter experts was able to bring together the tenants of virtualization, automation, C4I Air systems testing, tactical data link testing, and Design of Experiments methodology, to invent a testing paradigm that will exhaustively evaluate tactical Air systems. This presentation will give an overview of how CT was implemented for the test.

Binomial metrics like probability-to-detect or probability-to-hit typically provide operationally meaningful and easy to interpret test outcomes. However, they are information poor metrics and extremely expensive to test. The standard power calculations to size a test employ hypothesis tests, which typically result in many tens to hundreds of runs. In addition to being expensive, the test is most likely inadequate for characterizing performance over a variety of conditions due to the inherently large statistical uncertainties associated with binomial metrics. A solution is to convert to a continuous variable, such as miss distance or time-to-detect. The common objection to switching to a continuous variable is that the hit/miss or detect/non-detect binomial information is lost, when the fraction of misses/no-detects is often the most important aspect of characterizing system performance. Furthermore, the new continuous metric appears to no longer be connected to the requirements document, which was stated in terms of a probability. These difficulties can be overcome with the use of censored data analysis. This presentation will illustrate the concepts and benefits of this approach, and will illustrate a simple analysis with data, including power calculations to show the cost savings for employing the methodology.

Estimating the probability distribution of an extremum (maximum or minimum), for some fixed amount of time, using a single time series typically recorded for a shorter amount of time, is important in many application areas, e.g., structural design, reliability, quality, and insurance. When designing structural members, engineers are concerned with maximum wind effects, which are functions of wind speed. With respect to reliability and quality, extremes experienced during storage or transport, e.g., extreme temperatures, may substantially impact product quality, lifetime, or both. Insurance companies are of course concerned about very large claims. In this presentation, a method to estimate the distribution of an extremum using a well-known peaks-over-threshold (POT) model and Monte Carlo simulation is presented. Since extreme values have long been a subject of study, some brief history is first discussed. The POT model that underlies the approach is then laid out. A description of the algorithm follows. It leverages pressure data collected on scale models of buildings in a wind tunnel for context. Essentially, the POT model is fitted to the observed data and then used to simulate many times series of the desired length. The empirical distribution of the extrema is obtained from the simulated series. Uncertainty in the estimated distribution is quantified by a bootstrap algorithm. Finally, an R package implementing the computations is discussed.

Within NASA's Air Traffic Management Technology Demonstration # 1 (ATD-1), Interval Management (IM) is a flight deck tool that enables pilots to achieve or maintain a precise in-trail spacing behind a target aircraft. Previous research has shown that violations of aircraft spacing requirements can occur between an IM aircraft and its surrounding non-IM aircraft when it is following a target on a separate route. This talk focuses on the experimental design and analysis of a computer experiment which models the airspace configuration of interest in order to determine airspace/aircraft conditions leading to spacing violations during IM operation. We refer to multi-layered nested continuous factors as those that are continuous and ordered in their selection; they can only be selected sequentially with a level selected for one factor affecting the range of possible values for each subsequently nested factor. While each factor is nested within another factor, the exact nesting relationships have no closed form solution. In this talk, we describe our process of engineering an appropriate space-filling design for this situation. Using this space-filling design and Gaussian process modeling, we found that aircraft delay assignments and wind profiles significantly impact the likelihood of spacing violations and the interruption of IM operations.

The problem of constructing a design for an experiment when multiple responses are of interest does not have a clear answer, particularly when the response variables are of different types. Planning an experiment for an air-to-air missile simulation, for example, might have the following responses simultaneously: hit or miss the target (a binary response) and the time to acquire the target (a continuous response). With limited time and resources and only one experiment possible, the question of selecting an appropriate design to model both responses is important. In this presentation, we discuss a method for creating designs when two responses, each with a different distribution (normal, binomial, or Poisson), are of interest. We demonstrate the proposed method using various weighting schemes for the two models to show how the designs change as the weighting scheme changes. In addition, we explore the effect of the specified priors for the nonlinear models on these designs.

Jones and Nachtsheim (2011) introduced a class of three-level screening designs called definitive screening designs (DSDs). The structure of these designs results in the statistical independence of main effects and two-factor interactions; the absence of complete confounding among two-factor interactions; and the ability to estimate all quadratic effects. Because quadratic effects can be estimated, DSDs can allow for the screening and optimization of a system to be performed in one step, but only when the number of terms found to be active during the screening phase of analysis is less than about half the number or runs required by the DSD (Errore, et al., 2016). Otherwise, estimation of second-order models requires augmentation of the DSD. In this paper we explore the construction of series of augmented designs, moving from the starting DSD to designs capable of estimating the full second-order model. We use power calculations, model-robustness criteria, and model-discrimination criteria to determine the number of runs by which to augment in order to identify the active second-order effects with high probability.

More often than not, the data we analyze for the military is plagued with statistical issues. Multicollinearity, small sample sizes, quasi-experimental designs, and convenience samples are some examples of what we commonly see in military data. Many of these complications can be resolved either in the design or analysis stage with appropriate statistical procedures. But, to keep our work useful, usable, and transparent to the military leadership who sponsors it, we must strike the elusive balance between explaining and justifying our design and analysis techniques and not inundating our audience with unnecessary details. It can be even more difficult to get military leadership to understand the statistical problems and solutions so well that they are enthused and supportive of our approaches. Using literature written on the subject as well as a variety of experiences, we will showcase several examples, as well as present ideas for keeping our clients actively engaged in statistical methodology discussions.

More often than not, the data we analyze for the military is plagued with statistical issues. Multicollinearity, small sample sizes, quasi-experimental designs, and convenience samples are some examples of what we commonly see in military data. Many of these complications can be resolved either in the design or analysis stage with appropriate statistical procedures. But, to keep our work useful, usable, and transparent to the military leadership who sponsors it, we must strike the elusive balance between explaining and justifying our design and analysis techniques and not inundating our audience with unnecessary details. It can be even more difficult to get military leadership to understand the statistical problems and solutions so well that they are enthused and supportive of our approaches. Using literature written on the subject as well as a variety of experiences, we will showcase several examples, as well as present ideas for keeping our clients actively engaged in statistical methodology discussions.

This presentation will provide a high level view of recent DOE applications to aerospace research. Two broad categories are defined, aerodynamic force measurement system calibrations and aircraft model wind tunnel aerodynamic characterization. Each case study will outline the application of DOE principles including design choices, accommodations for deviations from classical DOE approaches, discoveries, and practical lessons learned. Case Studies Aerodynamic Force Measurement System Calibrations Large External Wind Tunnel Balance Calibration - Fractional factorial - Working with non-ideal factor settings - Customer driven uncertainty assessment Internal Balance Calibration Including Temperature - Restrictions to randomization – split plot design requirements - Constraints to basic force model - Crossed design approach Aircraft Model Wind Tunnel Aerodynamic Characterization The NASA/Boeing X-48B Blended Wing Body Low-Speed Wind Tunnel Test - Overcoming a culture of OFAT - General approach to investigating a new aircraft configuration - Use of automated wind tunnel models and randomization - Power of residual analysis in detecting problems NASA GL–10 UAV Aerodynamic Characterization - Use of the Nested-Face Centered Design for aerodynamic characterization - Issues working with over 20 factors - Discoveries

Sample size drives the resources and supports the conclusions of operational test. Power analysis is a common statistical methodology used in planning efforts to justify the number of samples. Power analysis is sensitive to extreme performance (e.g. 0.1% correct responses or 99.999% correct responses) relative to a threshold value, extremes in response variable variability, numbers of factors and levels, system complexity, and a myriad of other design- and system-specific criteria. This discussion will describe considerations (correlation/aliasing, operational significance, thresholds, etc.) and relationships (design, difference to detect, noise, etc.) associated with power. The contribution of power to design selection or adequacy must often be tempered when significant uncertainty or test resources constraints exist. In these situations, other measures of merit and alternative analytical approaches become at least as important as power in the development of designs that achieve the desired technical adequacy. In conclusion, one must understand what power is, what factors influence the calculation, and when to leverage alternative measures of merit.

Sample size drives the resources and supports the conclusions of operational test. Power analysis is a common statistical methodology used in planning efforts to justify the number of samples. Power analysis is sensitive to extreme performance (e.g. 0.1% correct responses or 99.999% correct responses) relative to a threshold value, extremes in response variable variability, numbers of factors and levels, system complexity, and a myriad of other design- and system-specific criteria. This discussion will describe considerations (correlation/aliasing, operational significance, thresholds, etc.) and relationships (design, difference to detect, noise, etc.) associated with power. The contribution of power to design selection or adequacy must often be tempered when significant uncertainty or test resources constraints exist. In these situations, other measures of merit and alternative analytical approaches become at least as important as power in the development of designs that achieve the desired technical adequacy. In conclusion, one must understand what power is, what factors influence the calculation, and when to leverage alternative measures of merit.

One of the major pillars of experimental design is sequential learning. The experimental design should not be viewed as a “one-shot” effort, but rather as a series of experiments where each stage builds upon information learned from the previous study. It is within this realm of sequential learning that experimentation soundly supports the application of the scientific method. This presentation illustrates the value of sequential experimentation and also the connection between the scientific method and experimentation through a discussion of a multi-stage project supported by NASA’s Engineering Safety Center (NESC) where the objective was to assess the safety of composite overwrapped pressure vessels (COPVs). The analytical team was tasked with devising a test plan to model stress rupture failure risk in carbon fiber strands that encase the COPVs with the goal of understanding the reliability of the strands at use conditions for the expected mission life. This presentation highlights the recommended experimental design for the strand tests and then discusses the benefits that resulted from the suggested sequential testing protocol.

This presentation will provide a brief overview of sensitivity testing, and emphasize applications to several products and system of importance to the Defense as well as private industry, including Insensitive Energetics, Ballistic testing of protective armor, testing of munition fuzes and Microelectromechanical Systems (MEMS) components, and safety testing of high-pressure test ammunition, and packaging for high-value materials.

This presentation will provide a brief overview of sensitivity testing, and emphasize applications to several products and system of importance to the Defense as well as private industry, including Insensitive Energetics, Ballistic testing of protective armor, testing of munition fuzes and Microelectromechanical Systems (MEMS) components, and safety testing of high-pressure test ammunition, and packaging for high-value materials.

Binary response experiments are common in epidemiology, biostatistics as well as in military applications. The Up and Down method, Langlie’s Method, Neyer’s method, K in a Row method and 3 Phase Optimal Design are methods used for sequential experimental design when there is a single continuous variable and a binary response. During this talk, we will discuss a new sequential experimental design approach called the Break Separation Method (BSM). BSM provides an algorithm for determining sequential experimental trials that will be used to find a median quantile and fit a logistic regression model using Maximum Likelihood estimation. BSM results in a small sample size and is designed to efficiently compute the median quantile.

Binary response experiments are common in epidemiology, biostatistics as well as in military applications. The Up and Down method, Langlie’s Method, Neyer’s method, K in a Row method and 3 Phase Optimal Design are methods used for sequential experimental design when there is a single continuous variable and a binary response. During this talk, we will discuss a new sequential experimental design approach called the Break Separation Method (BSM). BSM provides an algorithm for determining sequential experimental trials that will be used to find a median quantile and fit a logistic regression model using Maximum Likelihood estimation. BSM results in a small sample size and is designed to efficiently compute the median quantile.

Model Validation for Simulations of CVN-78 Sortie Generation As part of the test planning process, IDA is examining flight operations on the Navy’s newest carrier, CVN-78. The analysis uses a model, the IDA Virtual Carrier Model (IVCM), to examine sortie generation rates and whether aircraft can complete missions on time. Before using IVCM, it must be validated. However, CVN-78 has not been delivered to the Navy, and data from actual operations are to validate the model. Consequently, we will validate IVCM by comparing it to another model. This is a reasonable approach when a model is used in general analyses such as test planning, but is not acceptable when a model is used in the assessment of system effectiveness and suitability. The presentation examines the use of various statistical tools – Wilcoxon Rank Sum Test, Kolmogorov-Smirnov Test, and lognormal regression – to examine whether the results from two models provide similar results and to quantify the magnitude of any differences. From the analysis, IDA concluded that locations and distribution shapes are consistent, and that the differences between the models are less than 15 percent, which is acceptable for test planning.

This panel will share status, experiences and expectations within DoD and NASA for transitioning Systems Engineering to a more integrated digital engineering domain. A wide range of perspectives will be provided, covering the implementation waterfront of practitioner, management, research and strategy. Panelist will also be prepared to discuss more focused areas of digital systems engineering, such as test and evaluation, and engineering statistics.

Resampling Methods: This tutorial presents widely used resampling methods to include bootstrapping, cross-validation, and permutation tests. Underlying theories will be presented briefly, but the primary focus will be on applications. A new graph-theoretic approach to change detection will be discussed as a specific application of permutation testing. Examples will be demonstrated in R; participants are encouraged to bring their own portable computers to follow along using datasets provided by the instructor.

Difficult choices are often required in a decision-making process where resources and budgets are increasingly constrained. This talk demonstrates a structured decision-making approach using layered Pareto fronts to prioritize the allocation of funds between munitions stockpiles based on their estimated reliability, the urgency of needing available units, and the consequences if adequate numbers of units are . This case study illustrates the process of first identifying appropriate metrics that summarize important dimensions of the decision, and then eliminating non-contenders from further consideration in an objective stage. The final subjective stage incorporates subject matter expert priorities to select the four stockpiles to receive additional maintenance and surveillance funds based on understanding the trade-offs and robustness to various user priorities.

The importance of the nuclear balance vis-a-vis our principal adversary has been the subject of intense but unresolved debate in the international security community for almost seven decades. Perspectives on this question underlie national security policies regarding potential unilateral reductions in strategic nuclear forces, the imbalance of nonstrategic nuclear weapons in Europe, nuclear crisis management, nuclear proliferation, and nuclear doctrine. The overwhelming majority of past studies of the role of the nuclear balance in nuclear crisis evolution and outcome have been qualitative and focused on the relative importance of the nuclear balance and national resolve. Some recent analyses have invoked statistical methods, however, these quantitative studies have generated intense controversy because of concerns with analytic rigor. We apply a multi-disciplinary approach that combines historical case study, international relations theory, and appropriate statistical analysis. This approach results in defensible findings on causal mechanisms that regulate nuclear crisis resolution. Such findings should inform national security policy choices facing the Trump administration.

Uncertainty appears in many aspects of systems design including stochastic design parameters, simulation inputs, and forcing functions. Uncertainty Quantification (UQ) has emerged as the science of quantitative characterization and reduction of uncertainties in both simulation and test results. UQ is a multidisciplinary field with a broad base of methods including sensitivity analysis, statistical calibration, uncertainty propagation, and inverse analysis. Because of their ability to bring greater degrees of confidence to decisions, uncertainty quantification methods are playing a greater role in test, evaluation, and modeling and simulation in defense and aerospace. The value of UQ comes with better understanding of risk from assessing the uncertainty in test and modeling and simulation results. The presentation will provide an overview of UQ and then discuss the use of some advanced statistical methods, including DOEs and emulation for multiple simulation solvers and statistical calibration, for efficiently quantifying uncertainties. These statistical methods effectively link test, evaluation and modeling and simulation by coordinating the valuation of uncertainties, simplifying verification and validation activities.

Answers to real world questions are often based on the use of judiciously chosen mathematical/statistical/physical models. In particular, assessment of failure probabilities of physical systems rely heavily on such models. Since no model describes the real world exactly, sensitivity analyses are conducted to examine influences of (small) perturbations of an assumed model. In this talk we present a structured approach, using an "Assumptions Lattice" and corresponding "Uncertainty Pyramid", for transparently conveying the influence of various assumptions on analysis conclusions. We illustrate this process in the context of a simple multicomponent system.

The US Army ARDEC has recently established an initiative to integrate statistical and probabilistic techniques into engineering modeling and simulation (M&S) analytics typically used early in the design lifecycle to guide technology development. DOE-driven Uncertainty Quantification techniques, including statistically rigorous model verification and validation (V&V) approaches, enable engineering teams to identify, quantify, and account for sources of variation and uncertainties in design parameters, and identify opportunities to make technologies more robust, reliable, and resilient earlier in the product’s lifecycle. Several recent armament engineering case studies - each with unique considerations and challenges - will be discussed.

The US Army ARDEC has recently established an initiative to integrate statistical and probabilistic techniques into engineering modeling and simulation (M&S) analytics typically used early in the design lifecycle to guide technology development. DOE-driven Uncertainty Quantification techniques, including statistically rigorous model verification and validation (V&V) approaches, enable engineering teams to identify, quantify, and account for sources of variation and uncertainties in design parameters, and identify opportunities to make technologies more robust, reliable, and resilient earlier in the product’s lifecycle. Several recent armament engineering case studies - each with unique considerations and challenges - will be discussed.

Teams of people with many different talents and skills work together at NASA to improve our understanding of our planet Earth, our Sun and solar system, and the Universe. The Earth System is made up of complex interactions and dependencies of the solar, oceanic, terrestrial, atmospheric, and living components. Solar storms have been recognized as a cause of technological problems on Earth since the invention of the telegraph in the 19th century. Solar flares, coronal holes, and coronal mass ejections (CME's) can emit large bursts of radiation, high speed electrons and protons, and other highly energetic particles that are released from the sun, and are sometimes directed at Earth. These particles and radiation can damage satellites in space, shutdown power grids on earth, cause GPS outages, and have serious health concerns to humans flying at high altitudes on earth, as well as astronauts in space. NASA builds and operates a fleet of satellites to study the sun and a fleet of satellites and aircraft to observe the Earth system. NASA’s Computer Models combine the observations with numerical models, to understand how these systems work. Using satellite observations alongside computer models we can combine many pieces of information to form a coherent view of Earth and the Sun. NASA research helps us understand how processes combine to affect life on Earth: this includes severe weather, health, changes in climate, and space weather. The Scientific Visualization Studio wants you to learn about NASA programs through visualization. The SVS works closely with scientists in the creation of data visualizations, animations, and images in order to promote a greater understanding of Earth and Space Science research activities at NASA and within the academic research community supported by NASA.

The NASA Big Data, Big Think team jump-starts coordination, strategy, and progress for NASA applications of Big Data Analytics techniques, fosters collaboration and teamwork among centers and improves agency-wide understanding of Big Data research techniques & technologies and their application to NASA mission domains. The effort brings the Agency’s Big Data community together and helps define near term projects and leverages expertise throughout the agency. This presentation will share examples of Big Data activities from the Agency and discuss knowledge areas and experiences, including data management, data analytics and visualization.

Data within a project, investigation or test series are often seen as a bunch of numbers that were produced. While this is part of the story, it forgets the most important part: the data’s users. This more powerful process begins with early focus on planning, executing and managing data flow within a test or project as a system, treating each handoff between internal and external stakeholders a system interface. This presentation will persuade you why data production should be replaced by the idea of a data supply chain focused on goals and customers. The presenter will outline how this could be achieved in your team. The talk is aimed at not only project and data managers, but also team members who produce or use data. Retooling team thinking and processes along these lines will help communication, facilitate availability, display and understanding of data by any stakeholder, make data verification, validation and analysis easier, and help keep team members focused on what is necessary and important: solving the problem at hand.

This tutorial will show how to compare and choose experimental designs based on multiple criteria. Answers to questions like "Which Design of Experiments (DOE) is better/best?" will be answered by looking at both data and graphics that show the relative performance of the designs based on multiple criteria, including; power of the designs for different model terms, how well the designs minimize predictive variance across the design space, to what level are model terms confounded or correlated, what are the relative efficiencies that measure how well coefficients are estimated or how well predictive variance is minimized. Many different case studies of screening, response surface, and screening augmented to response surface designs will be compared. Designs with both continuous and categorical factors, and with constraints on the experimental region will also be compared.

Autonomous robotic systems (hereafter referred to simply as autonomous systems) have attracted interest in recent years as capabilities improve to operate in unstructured, dynamic environments without continuous human guidance. Acquisition of autonomous systems potentially decrease personnel costs and provide a capability to operate in dirty, dull, or dangerous mission segments or achieve greater operational performance. Autonomy enables a particular action of a system to be automatic or, within programmed boundaries, self-governing. For our purposes, autonomy is defined as the system having a set of intelligence-based capabilities (i.e., learned behaviors) that allows it to respond to situations that were not pre-programmed or anticipated (i.e. learning-based responses) prior to system deployment. Autonomous systems have a degree of self-governance and self-directed behavior, possibly with a human’s proxy for decisions. Because of these intelligence-based capabilities, autonomous systems pose new challenges in conducting test and evaluation that assures adequate performance, safety, and cybersecurity outcomes. We propose an autonomous systems architecture concept and map the elements of a decision theoretic view of a generic decision problem to the components of this architecture. These models offer a foundation for developing a decision-based, common framework for autonomous systems. We also identify some of the various challenges faced by the Department of Defense (DoD) test and evaluation community in assuring the behavior of autonomous systems as well as test and evaluation requirements, processes, and methods needed to address these challenges.

Small screening designs are frequently used in the initial stages of experimentation with the goal of identifying important main effects as well as to gain insight on potentially important two-factor interactions. Commonly utilized experimental designs for screening (e.g., resolution III or IV two-level fractional factorials, Plackett-Burman designs, etc.) are unreplicated and as such, provide no unbiased estimate of experimental error. However, if statistical inference is considered an integral part of the experimental analysis, one view is that inferential procedures should be performed using the unbiased pure error estimate. As full replication of an experiment may be quite costly, partial replication offers an alternative for obtaining a model independent error estimate. Gilmour and Trinca (2012, Applied Statistics) introduce criteria for the design of optimal experiments for statistical inference (providing for the optimal selection of replicated design points). We begin with an extension of their work by proposing a Bayesian criterion for the construction of partially replicated screening designs with less dependence on an assumed model. We then consider the use of the proposed criterion within the context of multi-criteria design selection where estimation and protection against model misspecification are considered. Insights for analysis and model selection in light of partial replication will be provided.

In an experiment with a single factor with three levels, treatments A, B, and C, a single treatment is to be applied to each of several experimental units selected from some set of units. The response variable is continuous, and differences in its value show the relative effectiveness of the treatments. An experimental design will dictate which treatment is applied to what units. Since differences in the response variable are used to judge differences between treatments, the most important goal of the design is to prevent the treatment effect being masked by some unrelated property of the experimental units. Second important function of the design is to ensure power, that is, that if the treatments are not equally effective, the differences in the response variable are likely to be larger than background noise. Classical experimental design theory uses three principles: replication, randomization, and blocking, to produce an experimental design. Replication refers to how many units are used, blocking is a possible grouping of the units to reduce between unit heterogeneity, and randomization governs the assignment of units to treatment. Classical experimental designs are balanced as much as possible, that is, the three treatments are applied the same number of times, in each potential block of units. Bayesian experimental design aims to make use of additional related information, often called prior information, to produce a design. The information may be in the form of related experimental results, for example, treatments A and B may have been previously studied. It could be additional information about the experimental units, or about the response variable. This additional information could be used to change the usual blocking, to reduce the number of units assigned to treatments A and B compared to C, and/or reduce the total number of units needed to ensure power. This talk aims to explain Bayesian design concepts and illustrate them on realistic examples.

Rigorous, efficient and effective test science techniques are individually taking hold in many software centric DoD acquisition programs, both in developmental and operational test regimes. These techniques include agile software development, cybersecurity test and evaluation (T&E), design and analysis of experiments and automated software testing. Many software centric programs must also be tested together with other systems to demonstrate they can be successfully integrated into a more complex systems of systems. This presentation focuses on the two test science disciplines of designed experiments (DOE) and automated software testing (AST) and describes how they can be used effectively and leverage one another in planning for and executing a system of systems test strategy. We use the Navy’s Distributed Common Ground System as an example.

Helicopters serve a number of useful roles within the community; however, community acceptance of helicopter operations is often limited by the resulting noise. Because the noise characteristics of helicopters depend strongly on the operating condition of the vehicle, effective noise abatement procedures can be developed for a particular helicopter type, but only when the noisy regions of the operating envelope are identified. NASA Langley Research Center—often in collaboration with other US Government agencies, industry, and academia—has conducted noise measurements for a wide variety of helicopter types, from light commercial helicopters to heavy military utility helicopters. While this database is expansive, it covers only a fraction of helicopter types in current commercial and military service and was measured under a limited set of ambient conditions and vehicle configurations. This talk will describe a new “universal” helicopter noise model suitable for planning helicopter noise abatement procedures. Modern machine learning techniques will be combined with the principle of nondimensionalization and applied to NASA’s helicopter noise data in order to develop a model capable of estimating the noisy operating states of any conventional helicopter under any specific ambient conditions and vehicle configurations.

The goal of the Crew State Monitoring (CSM) project is to use machine learning models trained with physiological data to predict unsafe cognitive states in pilots such as Channelized Attention (CA) and Startle/Surprise (SS). These models will be used in a real-time system that predicts a pilot’s mental state every second, a tool that can be used to help pilots recognize and recover from these mental states. Pilots wore different sensors that collected physiological data such as a 20-channel electroencephalography (EEG), respiration, and galvanic skin response (GSR). Pilots performed non-flight benchmark tasks designed to induce these states, and a flight simulation with “surprising” or “channelizing” events. The team created a pipeline to generate pilot-dependent models that trains on benchmark data, tune on a portion of a flight task, and be deployed onto the remaining flight task. The model is a series of anomaly-detection based ensembles, where each ensemble focuses on predicting a single state. Ensembles were comprised of several anomaly detectors such as One Class SVMs, each focusing on a different subset of sensor data. We will discuss the performance of these models, as well as the ongoing research generalizing models across pilots and improving accuracy.

Reconnaissance, footprinting, and enumeration are critical steps in the CYBER penetration testing process because if these steps are not fully and extensively executed, the information available for finding a system’s vulnerabilities may be limited. During the CYBER testing process, penetration testers often find themselves doing the same initial enumeration scans over and over for each system under test. Because of this, automated scripts have been developed that take these mundane and repetitive manual steps and perform them automatically with little user input. Once automation is present in the penetration testing process, Scientific Test and Analysis Techniques (STAT) can be incorporated. By combining automation and STAT in the CYBER penetration testing process, Mr. Tim McLean at Marine Corps Tactical Systems Support Activity (MCTSSA) coined a new term called CYBERSTAT. CYBERSTAT is applying scientific test and analysis techniques to offensive CYBER penetration testing tools with an important realization that CYBERSTAT assumes the system under test is the offensive penetration test tool itself. By applying combinatorial testing techniques to the CYBER tool, the CYBER tool’s scope is expanded beyond “one at a time” uses as the combinations of the CYBER tool’s capabilities and options are explored and executed as test cases against the target system. In CYBERSTAT, the additional test cases produced by STAT can be run automatically using scripts. This talk will show how MCTSSA is preparing to use CYBERSTAT in the Developmental Test and Evaluation process of USMC Command and Control systems.

This presentation discusses a methodology to use knowledge of cyber attacks and defender responses from operational assessments to gain insights into the defensive posture and to inform a strategy for improvement. The concept is to use the attack thread as the instrument to probe and measure the detection capability of the cyber defenses. The data enable a logistic regression approach to provide a quantitative basis for the analysis and recommendations.

Over the past decade, uncertainty quantification has become an integral part of engineering design and analysis. Both NASA and the DoD are making significant investments to advance the science of uncertainty quantification, increase the knowledge base, and strategically expanding its use. This increased use of uncertainty based results improves investment strategies and decision making. However, in complex systems, many challenges still exist when dealing with uncertainty in cases that have sparse, unreliable, poorly understood, and/or conflicting data. Often times, assumptions are made regarding the statistical nature of data that may not be well grounded and the impact of those assumptions is not well understood, which can lead to ill-informed decision making. This talk will focus on the quantification of uncertainty when both well characterized, aleatory, and not well known, epistemic, uncertainty sources exist. Particular focus is given to the treatment and management of epistemic uncertainty. A summary of non-probabilistic methods will be presented along with the propagation of mixed uncertainty and optimization under uncertainty. A discussion of decision making under uncertainty is also included to illustrate the use of uncertainty quantification.

Over the past decade, uncertainty quantification has become an integral part of engineering design and analysis. Both NASA and the DoD are making significant investments to advance the science of uncertainty quantification, increase the knowledge base, and strategically expanding its use. This increased use of uncertainty based results improves investment strategies and decision making. However, in complex systems, many challenges still exist when dealing with uncertainty in cases that have sparse, unreliable, poorly understood, and/or conflicting data. Often times, assumptions are made regarding the statistical nature of data that may not be well grounded and the impact of those assumptions is not well understood, which can lead to ill-informed decision making. This talk will focus on the quantification of uncertainty when both well characterized, aleatory, and not well known, epistemic, uncertainty sources exist. Particular focus is given to the treatment and management of epistemic uncertainty. A summary of non-probabilistic methods will be presented along with the propagation of mixed uncertainty and optimization under uncertainty. A discussion of decision making under uncertainty is also included to illustrate the use of uncertainty quantification.

Under current FAA regulations, civilian aircraft may not operate at supersonic speeds over land. However, over the past few decades, there have been renewed efforts to invest in technologies to mitigate sonic boom from supersonic aircraft through advances in both vehicle design and sonic boom prediction. NASA has heavily invested in tools and technologies to enable commercial supersonic flight and currently has several technical challenges related to sonic boom reduction. One specific technical challenge relates to the development of tools and methods to predict, under uncertainty, the noise on the ground generated by an aircraft flying at supersonic speeds. In attempting to predict ground noise, many factors from multiple disciplines must be considered. Further, classification and treatment of uncertainties in coupled systems, mutlifidelity simulations, experimental data, and community responses are all concerns in system level analysis of sonic boom prediction. This presentation will introduce the various methodologies and techniques utilized for uncertainty quantification with a focus on the build up to system level analysis. An overview of recent research activities and case studies investigating the impact of various disciplines and factors on variance in ground noise will be discussed.

The Boeing Company is assessing uncertainty quantification methodologies across many phases of aircraft design in order to establish confidence in computational fluid dynamics-based simulations of aircraft performance. This presentation provides an overview of several of these efforts. First, the uncertainty in aerodynamic performance metrics of a commercial aircraft at transonic cruise due to turbulence model and flight condition variability is assessed using 3D CFD with non-intrusive polynomial chaos and second order probability. Second, a sample computation of uncertainty in increments is performed for an engineering trade study, leading to the development of a new method for propagating input-uncontrolled uncertainties as well as input-controlled uncertainties. This type of consideration is necessary to account for variability associated with grid convergence on different configurations, for example. Finally, progress toward applying the computed uncertainties in forces and moments into an aerodynamic database used for flight simulation will be discussed. This approach uses a combination of Gaussian processes and multiple-fidelity Kriging meta-modeling to synthesize the required data.

Increasingly humans will rely on the outputs of our computational partners to make sense of the complex systems in our world. To be employed, the statistical and algorithmic analysis tools that are deployed to analysts’ toolboxes must afford their proper use and interpretation. Interface design for these tool users should provide decision support appropriate for the current stage of sensemaking. Understanding how users build, test, and elaborate their mental models of complex systems can guide the development of robust interfaces.

During the summer of 2001, Microsoft launched Windows XP, which was lauded by many users as the most reliable and usable operating system at the time. Miami Herald columnist, Dave Berry, responded to this praise by stating that “this is like saying asparagus is the most articulate vegetable ever.” Whether you agree or disagree with Dave Berry, these users’ reactions are relative (to other operating systems and to other past and future versions). This is due to an array of technological factors that have facilitated human-system improvements. Automation is often cited as improving human-system performance across many domains. It is true that when the human and automation are aligned, performance improves. But, what about the times that this is not the case? This presentation will describe the myths and facts about human-system performance and increasing levels of automation through examples of human-system R&D conducted on a satellite ground system. Factors that affect human-system performance and a method to characterize mission performance as it relates to increasing levels of automation will also be discussed.

Loss of control in flight has been a leading cause of accidents and incidents in commercial aviation worldwide. The Commercial Aviation Safety Team (CAST) requested studies on virtual day-visual meteorological conditions displays, such as synthetic vision, in order to combat loss of control. Over the last four years NASA has conducted a series of experiments evaluating the efficacy of synthetic vision displays for increased spatial awareness. Commercial pilots with various levels of experience from both domestic and international airlines were used as subjects. This presentation describes the synthetic vision research and how pilot subjects affected experiment design and statistical analyses.

The Stratospheric Aerosol and Gas Experiment (SAGE III) aboard the International Space Station (ISS) was experiencing a series of anomalies called Single Event Upsets (SEUs). Booz Allen Hamilton was tasked with conducting a statistical analysis to model the incidence of SEUs in the SAGE III equipment aboard the ISS. The team identified factors correlated with SEU incidences, set up a model to track degradation of Sage III, and showed current and past probabilities as a function of the space environment. The space environment of SAGE III was studied to identify possible causes of SEUs. The analysis revealed variables most correlated with the anomalies, including solar wind strength, solar and geomagnetic field behavior, and location/orientation of the ISS, sun, and moon. The data was gathered from a variety of sources including US government agencies, foreign and domestic academic centers, and state-of-the-art simulation algorithms and programs. Logistic regression was used to analyze SEUs and gain preliminary results. The data was divided into small time intervals to approximate independence and allow logistic regression. Due to the rarity of events the initial model results were based on few SEUs. The team set up a Graphical User Interface (GUI) program to automatically analyze new data as it became available to the SAGE III team. A GUI was built to allow the addition of more data over the life of the SAGE III mission. As more SEU incidents occur and are entered into the model, its predictive power will grow significantly. The GUI enables the user to easily rerun the regression analysis and visualize its results to inform operational decision making.

Automated systems are technologies that actively select data, transform information, make decisions, and control processes. The U.S. military uses automated systems to perform search and rescue and reconnaissance missions, and to assume control of aircraft to avoid ground collision. Facilitating appropriate trust in automated systems is essential to improving the safety and performance of human-system interactions. In two studies, we developed and validated an instrument to measure trust in automated systems. In study 1, we demonstrated that the scale has a 2-factor structure and demonstrates concurrent validity. We replicated these results using an independent sample in study 2.

Traffic Aware Strategic Aircrew Requests (TASAR) is a NASA-developed operational concept for flight efficiency and route optimization for the near-term airline flight deck. TASAR provides the aircrew with a cockpit automation tool that leverages a growing number of information sources on the flight deck to make fuel- and time-saving route optimization recommendations while in route. In partnership with a commercial airline, a research prototype software that implements TASAR has been installed on three aircraft to enable the evaluation of this software in operational use. During the flight trials, data are being collected to quantify operational performance, which will enable NASA to improve algorithms and enhance functionality in the software based on real-world user experience. This presentation highlights statistical challenges and discusses lessons learned during the initial stages of the operational evaluation.

In the operational testing of DoD weapons systems, modeling and simulation (M&S) is often used to supplement live test data in order to support a more complete and rigorous evaluation. Before the output of the M&S is included in reports to decision makers, it must first be thoroughly verified and validated to show that it adequately represents the real world for the purposes of the intended use. Part of the validation process should include a statistical comparison of live data to M&S output. This presentation includes an example of one such validation analysis for a tactical missile system. In this case, the goal is to validate a lethality model that predicts the likelihood of destroying a particular enemy target. Using design of experiments, along with basic analysis techniques such as the Kolmogorov-Smirnov test and Poisson regression, we can explore differences between the M&S and live data across multiple operational conditions and quantify the associated uncertainties.

Shelley Cazares served as a crewmember of the 14th mission of NASA’s Human Exploration Research Analog (HERA). In August 2017, Dr. Cazares and her three crewmates were enclosed in an approximately 600-sq. ft. simulated spacecraft for an anticipated 45 days of confined isolation at Johnson Space Center, Houston, TX. In preparation for long-duration missions to Mars in the 2030s and beyond, NASA seeks to understand what types of diets, habitats, and activities can keep astronauts healthy and happy on deep space voyages. To collect this information, NASA is conducting several analog missions simulating the conditions astronauts face in space. HERA is a set of experiments to investigate the effects of isolation, confinement, and remote conditions in space exploration scenarios. Dr. Cazares will discuss the application procedure, the pre-mission training process, the life and times inside the habitat during the mission, and her crew’s emergency evacuation from the habitat due to the risk of rising floodwaters in Hurricane Harvey.

Although reliability analysis is a part of Operational Test and Evaluation, it is uncommon for analysts to have a background in reliability theory or experience applying it. This presentation highlights some lessons learned from reliability analysis conducted on several AFOTEC test programs. Topics include issues related to censored data, limitations and alternatives to using the exponential distribution, and failure rate analysis using test data.

The Mars Rover 2020 team commonly faces nonlinear behavior across the test program that is often closely related to the underlying physics. Classical and newer response surface designs do well with quadratic approximations while space filling designs have proven useful for modeling & simulation of complex surfaces. This talk specifically covers fitting nonlinear equations based on engineering functional forms as well as sigmoid and exponential decay curves. We demonstrate best practices on how to design and augment nonlinear designs using the Bayesian D-Optimal Criteria. Several examples, to include drill bit degradation, illustrate the relative ease of implementation with popular software and the utility of these methods.

The fundamental principles of experiment design are factorization, replication, randomization, and local control of error. In many industries, however, departure from these principles is commonplace. Often in our experiments complete randomization is not feasible because the factor level settings are hard, impractical, or inconvenient to change or the resources available to execute under homogeneous conditions are limited. These restrictions in randomization lead to split-plot experiments. We are also often interested in fitting second-order models leading to second-order split-plot experiments. Although response surface methodology has grown tremendously since 1951, the lack of alternatives for second-order split-plots remains largely unexplored. The literature and textbooks offer limited examples and provide guidelines that often are too general. This deficit of information leaves practitioners ill prepared to face the many roadblocks associated with these types of designs. This presentation provides practical strategies to help practitioners in dealing with second-order split-plot and by extension, split-split-plot experiments, including an innovative approach for the construction of a response surface design referred to as second-order sub-array Cartesian product split-plot design. This new type of design, which is an alternative to other classes of split-plot designs that are currently in use in defense and industrial applications, is economical, has a low prediction variance of the regression coefficients, and low aliasing between model terms. Based on an assessment using well accepted key design evaluation criterion, second-order sub-array Cartesian product split-plot designs perform as well as historical designs that have been considered standards up to this point.

Randomization is one of the basic principles of experimental design and the associated statistical methods. In Navy operational testing, complete randomization is often not possible due to scheduling or execution constraints. Given these constraints, operational test designers often utilize split-plot designs to accommodate the hard-to-change nature of various factors of interest. Several case studies will be presented to provide insight into the challenges associated with Navy operational test design and execution.

Inherent system processes, restrictions on collection, or cost may impact the practical execution of an operational test. This study presents the use of blocking and split-plot designs when complete randomization is not feasible in operational test. Specifically, the USAF B-52 Radar Modernization Program test design is used to present tradeoffs of different design choices and the impacts of those choices on cost, operational relevance, and analytical rigor.

Inherent system processes, restrictions on collection, or cost may impact the practical execution of an operational test. This study presents the use of blocking and split-plot designs when complete randomization is not feasible in operational test. Specifically, the USAF B-52 Radar Modernization Program test design is used to present tradeoffs of different design choices and the impacts of those choices on cost, operational relevance, and analytical rigor.

Aerodynamic databases for space flight vehicles rely on wind-tunnel tests utilizing precision force measurement systems (FMS). Recently, NASA’s Space Launch System (SLS) program has conducted numerous wind-tunnel testing. This presentation will focus on the calibration of a unique booster FMS through the use of design of experiments (DoE) and regression modeling. Utilization of DoE resulted in a sparse, time-efficient, design with results exceeding researcher’s expectations.

Recent work at the National Transonic Facility (NTF) at the NASA Langley Research Center has shown that a substantial reduction in freestream pressure fluctuations can be achieved by positioning the moveable model support walls and plenum re-entry flaps to choke the flow just downstream of the test section. This choked condition reduces the upstream propagation of disturbances from the diffuser into the test section, resulting in improved Mach number control and reduced freestream variability. The choked conditions also affect the Mach number gradient and distribution in the test section, so a calibration experiment was undertaken to quantify the effects of the model support wall and re-entry flap movements on the facility freestream flow using a centerline static pipe. A design of experiments (DOE) approach was used to develop restricted-randomization experiments to determine the effects of total pressure, reference Mach number, model support wall angle, re-entry flap gap height, and test section longitudinal location on the centerline static pressure and local Mach number distributions for a reference Mach number range from 0.7 to 0.9. Tests were conducted using air as the test medium at a total temperature of 120 °F as well as for gaseous nitrogen at cryogenic total temperatures of -50, -150, and -250 °F. The resulting data were used to construct quadratic polynomial regression models for these factors using a Restricted Maximum Likelihood (REML) estimator approach. Independent validation data were acquired at off-design conditions to check the accuracy of the regression models. Additional experiments were designed and executed over the full Mach number range of the facility (0.2 £ Mref £ 1.1) at each of the four total temperature conditions, but with the model support walls and re-entry flaps set to their nominal positions, in order to provide calibration regression models for operational experiments where a choked condition downstream of the test section is either not feasible or not required. This presentation focuses on the design, execution, analysis, and results for the two experiments performed using air at a total temperature of 120 °F. Comparisons are made between the regression model output and validation data, as well as the legacy NTF calibration results, and future work is discussed.

Recent work at the National Transonic Facility (NTF) at the NASA Langley Research Center has shown that a substantial reduction in freestream pressure fluctuations can be achieved by positioning the moveable model support walls and plenum re-entry flaps to choke the flow just downstream of the test section. This choked condition reduces the upstream propagation of disturbances from the diffuser into the test section, resulting in improved Mach number control and reduced freestream variability. The choked conditions also affect the Mach number gradient and distribution in the test section, so a calibration experiment was undertaken to quantify the effects of the model support wall and re-entry flap movements on the facility freestream flow using a centerline static pipe. A design of experiments (DOE) approach was used to develop restricted-randomization experiments to determine the effects of total pressure, reference Mach number, model support wall angle, re-entry flap gap height, and test section longitudinal location on the centerline static pressure and local Mach number distributions for a reference Mach number range from 0.7 to 0.9. Tests were conducted using air as the test medium at a total temperature of 120 °F as well as for gaseous nitrogen at cryogenic total temperatures of -50, -150, and -250 °F. The resulting data were used to construct quadratic polynomial regression models for these factors using a Restricted Maximum Likelihood (REML) estimator approach. Independent validation data were acquired at off-design conditions to check the accuracy of the regression models. Additional experiments were designed and executed over the full Mach number range of the facility (0.2 £ Mref £ 1.1) at each of the four total temperature conditions, but with the model support walls and re-entry flaps set to their nominal positions, in order to provide calibration regression models for operational experiments where a choked condition downstream of the test section is either not feasible or not required. This presentation focuses on the design, execution, analysis, and results for the two experiments performed using air at a total temperature of 120 °F. Comparisons are made between the regression model output and validation data, as well as the legacy NTF calibration results, and future work is discussed.

The use of statistically rigorous methods to support testing at Arnold Engineering Development Complex (AEDC) has been an area of focus in recent years. As part of this effort, the use of Design of Experiments (DOE) has been introduced for calibration of AEDC wind tunnels. Historical calibration efforts used One- Factor-at-a-Time (OFAT) test matrices, with a concentration on conditions of interest to test customers. With the introduction of DOE, the number of test points collected during the calibration decreased, and were not necessary located at historical calibration points. To validate the use of DOE for calibration purposes, the 4-ft Aerodynamic Wind Tunnel 4T was calibrated using both DOE and OFAT methods. The results from the OFAT calibration were compared to model developed from the DOE data points and it was determined that the DOE model sufficiently captured the tunnel behavior within the desired levels of uncertainty. DOE analysis also showed that within Tunnel 4T, systematic errors are insignificant as indicated by agreement noted between the two methods. Based on the results of this calibration, a decision was made to apply DOE methods to future tunnel calibrations, as appropriate. The development of the DOE matrix in Tunnel 4T required the consideration of operational limitations, measurement uncertainties, and differing tunnel behavior over the performance map. Traditional OFAT methods allowed tunnel operators to set conditions efficiently while minimizing time consuming plant configuration changes. DOE methods, however, require the use of randomization which had the potential to add significant operation time to the calibration. Additionally, certain tunnel parameters, such as variable porosity, are only of interest in a specific region of the performance map. In addition to operational concerns, measurement uncertainty was an important consideration for the DOE matrix. At low tunnel total pressures, the uncertainty in the Mach number measurements increase significantly. Aside from introducing non-constant variance into the calibration model, the large uncertainties at low pressures can increase overall uncertainty in the calibration in high pressure regions where the uncertainty would otherwise be lower. At high pressures and transonic Mach numbers, low Mach number uncertainties are required to meet drag count uncertainty requirements. To satisfy both the operational and calibration requirements, the DOE matrix was divided into multiple independent models over the tunnel performance map. Following the Tunnel 4T calibration, AEDC calibrated the Propulsion Wind Tunnel 16T, Hypersonic Wind Tunnels B and C, and the National Full-Scale Aerodynamics Complex (NFAC). DOE techniques were successfully applied to the calibration of Tunnel B and NFAC, while a combination of DOE and OFAT test methods were used in Tunnel 16T because of operational and uncertainty requirements over a portion of the performance map. Tunnel C was calibrated using OFAT because of operational constraints. The cost of calibrating these tunnels has not been significantly reduced through the use of DOE, but the characterization of test condition uncertainties is firmly based in statistical methods.

For the past several years, researchers at NASA Langley have been engaged in a series of projects to study the degree to which existing facilities and capabilities, originally created for work on full-scale aircraft, are extensible to smaller scales – those of the small unmanned aerial systems (sUAS, also UAVs and, colloquially, `drones’) that have been showing up in the nation’s airspace. This paper follows an effort that has led to an initial human-subject psychoacoustic test regarding the annoyance generated by sUAS noise. This effort spans three phases: 1. the collection of the sounds through field recordings, 2. the formulation and execution of a psychoacoustic test using those recordings, 3. the analysis of the data from that test. The data suggests a lack of parity between the noise of the recorded sUAS and that of a set of road vehicles that were also recorded and included in the test, as measured by a set of contemporary noise metrics.

As systems and missions become increasingly complex, the roles of humans throughout the mission life cycle is evolving. In areas, such as maintenance and repair, hands-on tasks still dominate, however, new technologies have changed many tasks. For example, some critical human tasks have moved from manual control to supervisory control, often of systems at great distances (e.g., remotely piloting a vehicle, or science data collection on Mars). While achieving mission success remains the key human goal, almost all human performance metrics focus on failures rather than successes. This talk will examine the role of humans in creating mission success as well as new approaches for system validation testing needed to keep up with evolving systems and human roles.

Humans are not produced in quality-controlled assembly lines, and we typically are much more variable than the mechanical systems we employ. This mismatch means that when characterizing the effectiveness of a system, the system must be considered in the context of its users. Accurate measurement is critical to this endeavor, yet while human variability is large, effort to reduce measurement error of those humans is relatively small. The following talk discusses the importance of using multiple measurement methods—triangulation—to reduce error and increase confidence when characterizing the quality of HSI. A case study from an operational test of an attack helicopter demonstrates how triangulation enables more actionable recommendations.

The importance of the incorporation of Uncertainty Quantification (UQ) techniques into the design and analysis of Army systems is discussed. Relevant examples are presented where UQ would have been extremely useful. The intent of the presentation is to show the broad relevance of UQ and how, in the future, it will greatly improve the time to fielding and quality of developmental items.

To identify the robust settings of the control factors, it is very important to understand how they interact with the noise factors. In this article, we propose space-filling designs for computer experiments that are more capable of accurately estimating the control-by-noise interactions. Moreover, the existing space-filling designs focus on uniformly distributing the points in the design space, which are not suitable for noise factors because they usually follow non-uniform distributions such as normal distribution. This would suggest placing more points in the regions with high probability mass. However, noise factors also tend to have a smooth relationship with the response and therefore, placing more points towards the tails of the distribution is also useful for accurately estimating the relationship. These two opposing effects make the experimental design methodology a challenging problem. We propose optimal and computationally efficient solutions to this problem and demonstrate their advantages using simulated examples and a real industry example involving a manufacturing packing line.

Uncertainty is an inescapable reality that can be found in nearly all types of engineering analyses. It arises from sources like measurement inaccuracies, material properties, boundary and initial conditions, and modeling approximations. For example, the increasing use of numerical simulation models throughout industry promises improved design and insight at significantly lower costs and shorter timeframes than purely physical testing. However, the addition of numerical modeling has also introduced complexity and uncertainty to the process of generating actionable results. It has become not only possible, but vital to include Uncertainty Quantification (UQ) in engineering analysis. The competitive benefits of UQ include reduced development time and cost, improved designs, better understanding of risk, and quantifiable confidence in analysis results and engineering decisions. Unfortunately, there are significant cultural and technical challenges which prevent organizations from utilizing UQ methods and techniques in their engineering practice. This presentation will introduce UQ methodology and discuss the past and present strategies for addressing these challenges, making it possible to use UQ to enhance engineering processes with fewer resources and in more situations. Looking to the future, anticipated challenges will be discussed along with an outline of the path towards making UQ a common practice in engineering.

Mixed models are ideally suited to analyzing nested data from within-persons designs, designs that are advantageous in applied research. Mixed models have the advantage of enabling modeling of random effects, facilitating an accounting of the intra-person variation captured by multiple observations of the same participants and suggesting further lines of control to the researcher. However, the sampling requirements for mixed models are prohibitive to other areas which could greatly benefit from them. This simulation study examines the impact of small sample sizes in both levels of the model on the fixed effect bias, type I error, and power of a simple mixed model analysis. Despite the need for adjustments to control for type I error inflation, findings indicate that smaller samples than previously recognized can be used for mixed models under certain conditions prevalent in applied research. Examination of the marginal benefit of increases in sample subject and observation size provides applied researchers with guidance for developing mixed-model repeated measures designs that maximize power.

A standard paradigm for assessing the quality of model simulations is to compare what these models produce to experimental or observational samples of what the models seek to predict. Often these comparisons are based on simple summary statistics, even when the objects of interest are time series. Here, we propose a method of evaluation through probabilities derived from tests of hypotheses that model-simulated and observed time sequences share common signals. The probabilities are based on the behavior of summary statistics of model output and observational data, over ensembles of pseudo-realizations. These are obtained by partitioning the original time sequences into signal and noise components, and using a parametric bootstrap to create pseudo-realizations of the noise. We demonstrate with an example from climate model evaluation for which this methodology was developed.

Binomial (pass-fail) response metrics are more far more commonly used in test, requirements, quality and engineering than they need to be. In fact, there is even an engineering school of thought that they’re superior to continuous-variable metrics. This is a serious, even dangerous problem in aerospace and other industries: think the Space Shuttle Challenger accident. There are better ways. This talk will cover some examples of methods available to engineers and statisticians in common statistical software. It will not dig far into the mathematics of the methods, but will walk through where each method might be most useful and some of the pitfalls inherent in their use – including potential sources of misinterpretation and suspicion by your teammates and customers. The talk is geared toward engineers, managers and professionals in the –ilities who run into frustrations dealing with pass-fail data and thinking.

We have developed a new sensor fusion approach called Doppler Assisted Sensor Fusion (DASF), which pairs a range rate profile from one moving sensor with location accuracy with another range rate profile from another sensor with high location accuracy. This paring provides accurate identification, location, and tracking of moving emitters, with low association latency. The approach we use for data fusion is distinct from previous approaches. In the conventional approach, post detection data from the each sensor is overlaid with data from another sensor in an attempt to associate the data outputs. For the DASF approach the fusion is at the sensor level, the first sensor collects data and provides the standard identification in addition a unique emitter range rate profile. This profile is used to associate the emitter signature to a range-rate signature obtained by the geolocation sensor. The geolocation sensor then provides the desired location accuracy. We will provide results using real tracking data scenarios.

Principal Component Analysis is a standard tool in an analyst’s toolbox. The standard practice of rescaling each column can be reframed as a copula-based decomposition in which the marginal distributions are fit with a univariate Gaussian distribution and the joint distribution is modeled with a Gaussian copula. In this light, we present an alternative to traditional PCA we call XPCA by relaxing the marginal Gaussian assumption and instead fit each marginal distribution with the empirical distribution function. Interval-censoring methods are used to account for the discrete nature of the empirical distribution function when fitting the Gaussian copula model. In this talk, we derive the XPCA estimator and inspect the differences in fits on both simulated and real data applications.

The terms “Data Science”, “Machine Learning”, and “Artificial Intelligence” have become increasingly common in popular media, professional publications, and even in the language used by DoD leadership. But these terms are often not well understood, and may be used incorrectly and interchangeably. Even a textbook definition of these fields is unlikely to help with the distinction, as many definitions tend to lump everything under the umbrella of computer science or introduce unnecessary buzzwords. Leveraging a framework first proposed by David Robinson, Chief Data Scientist at DataCamp, we forgo the textbook definitions and instead focus on practical distinctions between the work of practitioners in each field, using examples relevant to the test and evaluation community where applicable.

The growing use of simulations in the engineering design process promises to reduce the need for extensive physical testing, decreasing both development time and cost. However, as mathematician and statistician George E. P. Box said, “Essentially, all models are wrong, but some are useful.” There are many factors that determine simulation or, more broadly, model accuracy. These factors can be condensed into noise, bias, parameter uncertainty, and model form uncertainty. To counter these effects and ensure that models faithfully match reality to the extent required, simulation models must be calibrated to physical measurements. Further, the models must be validated, and their accuracy must be quantified before they can be relied on in lieu of physical testing. Bayesian calibration provides a solution for both requirements: it optimizes tuning of model parameters to improve simulation accuracy, and estimates any remaining discrepancy which is useful for model diagnosis and validation. Also, because model discrepancy is assumed to exist in this framework, it enables robust calibration even for inaccurate models. In this paper, we present a case study to investigate the potential benefits of using Bayesian calibration, sensitivity analyses, and Monte Carlo analyses for model improvement and validation. We will calibrate a 7-parameter k-𝜎 CFD turbulence model simulated in COMSOL Multiphysics®. The model predicts coefficient of lift and drag for an airfoil defined using a 6049-series airfoil parameterization from the National Advisory Committee for Aeronautics (NACA). We will calibrate model predictions using publicly available wind tunnel data from the University of Illinois Urbana-Champaign’s (UIUC) database. Bayesian model calibration requires intensive sampling of the simulation model to determine the most likely distribution of calibration parameters, which can be a large computational burden. We greatly reduce this burden by following a surrogate modeling approach, using Gaussian process emulators to mimic the CFD simulation. We train the emulator by sampling the simulation space using a Latin Hypercube (LHD) Design of Experiment (DOE), and assess the accuracy of the emulator using leave-oneout Cross Validation (CV) error. The Bayesian calibration framework involves calculating the discrepancy between simulation results and physical test results. We also use Gaussian process emulators to model this discrepancy. The discrepancy emulator will be used as a tool for model validation; characteristic trends in residual errors after calibration can indicate underlying model form errors which were not addressed via tuning the model calibration parameters. In this way, we will separate and quantify model form uncertainty and parameter uncertainty. The results of a Bayesian calibration include a posterior distribution of calibration parameter values. These distributions will be sampled using Monte Carlo methods to generate model predictions, whereby new predictions have a distribution of values which reflects the uncertainty in the tuned calibrated parameter. The resulting output distributions will be compared against physical data and the uncalibrated model to assess the effects of the calibration and discrepancy model. We will also perform global, variance based sensitivity analysis on the uncalibrated model and the calibrated models, and investigate any changes in the sensitivity indices from uncalibrated to calibrated.

In today’s manufacturing, inspection, and testing world, understanding the capability of the measurement system being used via the use of Measurement Systems Analyses (MSA) is a crucial activity that provides the foundation for the use of Design of Experiments (DOE) and Statistical Process Control (SPC). Although undesirable, there are times when human observation is the only measurement system available. In these types of situations, traditional MSA tools are often ineffectual due to the nature of the data collected. When there are no other alternatives, we need some method for assessing the adequacy and effectiveness of the human observations. When multiple observers are involved, Attribute Agreement Analyses are a powerful tool for quantifying the Agreement and Effectiveness of a visual inspection system. This talk will outline best practices and rules of thumb for Attribute Agreement Analyses, and will highlight a recent Army case study to further demonstrate the tool’s use and potential.

Detecting and diagnosing aircraft system health poses a unique challenge as system complexity increases and software is further integrated. Anomaly detection algorithms systematically highlight unusual patterns in large datasets and are a promising methodology for detecting aircraft system health. The F-35A fighter aircraft is driven by complex, integrated subsystems with both software and hardware components. The F-35A operational flight program is the software that manages each subsystem within the aircraft and the flow of required information and support between subsystems. This information and support are critical to the successful operation of many subsystems. For example, the radar system supplies information to the fusion engine, without which the fusion engine would fail. ACC operational testing can be thought of as equivalent to beta testing for operational flight programs. As in other software, many faults result in minimal loss of functionality and are often unnoticed by the user. However, there are times when a software fault might result in catastrophic functionality loss (i.e., subsystem shutdown). It is critical to catch software problems that will result in catastrophic functionality loss before the flight software is fielded to the combat air forces. Subsystem failures and degradations can be categorized and quantified using simple system health data codes (e.g., degrade, fail, healthy). However, because the integrated nature of the F-35A, a subsystem degradation may be caused by another subsystem. The 59th Test and Evaluation Squadron collects autonomous system data, pilot questionnaires, and health report codes for F-35A subsystems. Originally, this information was analyzed using spreadsheet tools (i.e., Microsoft Excel). Using this method, analysts were unable to examine all subsystems or attribute cause for subsystem faults. The 59 TES is developing a new process that leverages anomaly detection algorithms to isolate flights with unusual patterns of subsystem failures and within those flights, highlight what subsystem faults are correlated with increased subsystem failures. This presentation will compare the performance of several anomaly detection algorithms (e.g., K-means, K-nearest neighbors, support vector machines) using simulated F-35A data.

Over the years the aerospace industry has continued to implement design of experiments and metamodeling (e.g., response surface methodology) in order to shift the knowledge curve forward in the systems design process. While the adoption of these methods is still incomplete across aerospace sub-disciplines, they comprise the state-of-the-art during systems design and for design evaluation using modeling and simulation or ground testing. In the context of modeling and simulation, while national infrastructure in high performance computing becomes higher performance, so do the demands placed on those resources in terms of simulation fidelity and number of researchers. Furthermore, with recent emphasis placed on the uncertainty quantification of aerospace system design performance, the number of simulation cases needed to properly characterize a system’s uncertainty across the entire design space increases by orders of magnitude, further stressing available resources. This leads to advanced development groups either sticking to ad hoc estimates of uncertainty (e.g., subject matter expert estimates based on experience) or neglecting uncertainty quantification all together. Advancing the state-of-the-art of aerospace systems design and evaluation requires a practical adaptive sampling scheme that responds to the characteristics of the underlying design or uncertainty space. For example, when refining a system metamodel gradually, points should be chosen for design variable combinations that are located in high curvature regions or where metamodel uncertainty is the greatest. The latter method can be implemented by defining a functional form of the metamodel variance and using it to define the next best point to sample. For schemes that require n points to be sampled simultaneously, considerations can be made to ensure proper sample dispersion. The implementation of adaptive sampling schemes to the design and evaluation process will enable similar fidelity with fewer samples of the design space compared to fixed or ad hoc sampling methods (i.e., shorter time or human resources required). Alternatively, the uncertainty of the design space can be reduced to a greater extent for the same number of samples or with fewer samples using higher fidelity simulations. The purpose of this presentation will be to examine the benefits of adaptive sampling as applied to challenging design problems. Emphasis will be placed on methods that are accessible to engineering

Over the years the aerospace industry has continued to implement design of experiments and metamodeling (e.g., response surface methodology) in order to shift the knowledge curve forward in the systems design process. While the adoption of these methods is still incomplete across aerospace sub-disciplines, they comprise the state-of-the-art during systems design and for design evaluation using modeling and simulation or ground testing. In the context of modeling and simulation, while national infrastructure in high performance computing becomes higher performance, so do the demands placed on those resources in terms of simulation fidelity and number of researchers. Furthermore, with recent emphasis placed on the uncertainty quantification of aerospace system design performance, the number of simulation cases needed to properly characterize a system’s uncertainty across the entire design space increases by orders of magnitude, further stressing available resources. This leads to advanced development groups either sticking to ad hoc estimates of uncertainty (e.g., subject matter expert estimates based on experience) or neglecting uncertainty quantification all together. Advancing the state-of-the-art of aerospace systems design and evaluation requires a practical adaptive sampling scheme that responds to the characteristics of the underlying design or uncertainty space. For example, when refining a system metamodel gradually, points should be chosen for design variable combinations that are located in high curvature regions or where metamodel uncertainty is the greatest. The latter method can be implemented by defining a functional form of the metamodel variance and using it to define the next best point to sample. For schemes that require n points to be sampled simultaneously, considerations can be made to ensure proper sample dispersion. The implementation of adaptive sampling schemes to the design and evaluation process will enable similar fidelity with fewer samples of the design space compared to fixed or ad hoc sampling methods (i.e., shorter time or human resources required). Alternatively, the uncertainty of the design space can be reduced to a greater extent for the same number of samples or with fewer samples using higher fidelity simulations. The purpose of this presentation will be to examine the benefits of adaptive sampling as applied to challenging design problems. Emphasis will be placed on methods that are accessible to engineering

The Army relies heavily on the voice of the customer to develop and refine technical requirements for developmental systems, but too often the approach is reactive. The ARDEC (Armament Research, Development & Engineering Center) Statistics Group at Picatinny Arsenal, NJ, working closely with subject matter experts, has been implementing market research and web development techniques and Design of Experiments (DOE) best practices to design and analyze surveys that provide insight into the customer’s perception of utility for various developmental commodities. Quality organizations tend to focus on ensuring products meet technical requirements, with far less of an emphasis placed on whether or not the specification actually captures customer needs. The employment of techniques and best practices spanning the fields of Market Research, Design of Experiments, and Web Development (choice design, conjoint analysis, contingency analysis, psychometric response scales, stratified random sampling) converge towards a more proactive and risk-mitigating approach to the development of technical and training requirements, and encourages strategic decision-making when faced with the inarticulate nature of human preference. Establishing a hierarchy of customer preference for objective and threshold values of key performance parameters enriches the development process of emerging systems by making the process simultaneously more effective and more efficient.

Infrastructure refers to the structures, utilities, and interconnected roadways that support the work carried out at a given facility. In the case of the Lawrence Livermore National Laboratory infrastructure is considered exclusive of scientific apparatus, safety and security systems. LLNL inherited it’s infrastructure management policy from the University of California which managed the site during LLNL’s first 5 decades. This policy is quite different from that used in commercial property management. Commercial practice weighs reliability over cost by replacing infrastructure at industry standard lifetimes. LLNL practice weighs overall lifecycle cost seeking to mitigate reliability issues through inspection. To formalize this risk management policy a careful statistical study was undertaken using 20 years of infrastructure replacement data. In this study care was taken to adjust for left truncation as-well-as right censoring. 57 distinct infrastructure class data sets were fitted using MLE to the Generalized Gamma distribution. This distribution is useful because it produces a weighted blending of discrete failure (Weibull model) and complex system failure (Lognormal model). These parametric fittings then yielded median lifetimes and conditional probabilities of failure. From conditional probabilities bounds on budget costs could be computed as expected values. This has provided a scientific basis for rational budget management as-well-as aided operations by prioritizing inspection, repair and replacement activities.

Infrastructure refers to the structures, utilities, and interconnected roadways that support the work carried out at a given facility. In the case of the Lawrence Livermore National Laboratory infrastructure is considered exclusive of scientific apparatus, safety and security systems. LLNL inherited it’s infrastructure management policy from the University of California which managed the site during LLNL’s first 5 decades. This policy is quite different from that used in commercial property management. Commercial practice weighs reliability over cost by replacing infrastructure at industry standard lifetimes. LLNL practice weighs overall lifecycle cost seeking to mitigate reliability issues through inspection. To formalize this risk management policy a careful statistical study was undertaken using 20 years of infrastructure replacement data. In this study care was taken to adjust for left truncation as-well-as right censoring. 57 distinct infrastructure class data sets were fitted using MLE to the Generalized Gamma distribution. This distribution is useful because it produces a weighted blending of discrete failure (Weibull model) and complex system failure (Lognormal model). These parametric fittings then yielded median lifetimes and conditional probabilities of failure. From conditional probabilities bounds on budget costs could be computed as expected values. This has provided a scientific basis for rational budget management as-well-as aided operations by prioritizing inspection, repair and replacement activities.

Several statistical methods have been used effectively to model workforce behavior, specifically attrition due to retirement and voluntary separation[1]. Additionally various authors have introduced career development[2] as a meaningful aspect of workforce planning. While both general and more specific attrition modeling techniques yield useful results only limited success has followed attempts to quantify career stage transition probabilities. A complete workforce model would include quantifiable flows both vertically and horizontally in the network described pictorially here at a single time point in Figure 1. The horizontal labels in Figure 1 convey one possible meaning assignable to career stage transition – in this case, competency. More formal examples might include rank within a hierarchy such as in a military organization or grade in a civil service workforce. In the case of the Nuclear Weapons labs knowing that the specialized, classified knowledge needed to deal with Stockpile Stewardship is being preserved as evidenced by the production of Masters, individuals capable of independent technical work, is also of interest to governmental oversight. In this paper we examine the allocation of labor involved in a specific Life Extension program at LLNL. This growing workforce is described by discipline and career stage to determine how well the Norden-Rayleigh development cost model[3] fits the data. Since this model underlies much budget estimation within both DOD and NNSA the results should be of general interest. Data is also examined as a possible basis for quantifying horizontal flows in Figure 1.

Optimization under uncertainty increases the complexity of a problem as well as the computing resources required to solve it. As the amount of uncertainty is increased, these difficulties are exacerbated. However, when optimizing for mission-level objectives, rather than component- or system-level objectives, an increase in uncertainty is inevitable. Previous research has found methods to perform optimization under uncertainty, such as robust design optimization or reliability-based design optimization. These are generally executed at a product component quality level, to minimize variability and stay within design tolerances but are not tailored to capture the high amount of variability in a mission-level problem. . In this presentation, an approach for formulating and solving highly stochastic mission-level optimization problems is described. A case study is shown using an unmanned aerial system (UAS) on a search mission while an “enemy” UAS attempts to interfere. This simulation, modeled in the Unity Game Engine, has highly stochastic outputs, where the time to mission success varies by multiple orders of magnitude, but the ultimate goal is a binary output representing mission success or failure. The results demonstrate the capabilities and challenges of optimization in these types of mission scenarios.

Operational Test (OT) is occasionally conducted after a system is already fielded. Unlike a traditional test based on Design of Experiments (DOE) principles, it is often not possible to vary the levels of the factors of interest. Instead the test is of an observational nature. Test planning for observational studies involves choosing where, when, and how long to evaluate a system in order to observe the possible combinations of factor levels that define the battlespace. This presentation discusses a test-planning method that uses Poisson process modeling as a way to estimate the length of time required to observe factor level combinations in the operational environment.

Due to financial and production constraints, it has become increasingly common for analysts and test planners in defense applications to find themselves working with smaller amounts of data than seen in industry. These same analysts are also being asked to make strong statistical statements based on this limited data. For example, a common goal is ‘demonstrating’ a high reliability requirement with sparse data. In such situations, strong modeling assumptions are often used to achieve the desired precision. Such model-driven actions contain levels of risk that customers may not be aware of and may be too high to be considered acceptable. There is a need to articulate and mitigate risk associated with model form error in statistical reliability analysis. In this work, we review different views on model credibility from the statistical literature and discuss how these notions of credibility apply in data-limited settings. Specifically, we consider two different perspectives on model credibility: (1) data-driven credibility metrics for model fit, (2) credibility assessments based on consistency of analysis results with prior beliefs. We explain how these notions of credibility can be used to drive test planning and recommend an approach to presenting analysis results in data-limited settings. We apply this approach to two case studies from reliability analysis: Weibull analysis and Neyer-D optimal test plans.

The current ban on commercial overland supersonic flight may be replaced by a noise limit on sonic boom sound level. NASA is establishing a quiet sonic boom database to guide the new regulation. The database will consist of multiple community surveys used to model the dose-response relationship between sonic boom sound levels and human annoyance. There are multiple candidate dose-response modeling techniques, such as classical logistic regression and multilevel modeling. To plan for these community surveys, recommendations for data collection will be developed from pilot community test data. Two important aspects are selecting sample size and sound level range. Selection of sample size must be strategic as large sample sizes are costly whereas small sample sizes may result in more uncertainty in the estimates. Likewise, there are trade-offs associated with selection of the sound level range. If the sound level range includes excessively high sound levels, the public may misunderstand the potential impact of quiet sonic booms, resulting in a negative backlash against a promising technological advancement. Conversely, a narrow range that includes only low sound levels might exclude the eventual noise limit. This presentation will focus on recommendations for sound level range given the expected shape of the dose-response curve.

The Army is currently developing a cluster munition that is required to meet functional reliability requirements of 99%. This effort focuses on the design process for a setback lock within the safe and arm (S&A) device in the submunition fuze. This lock holds the arming rotor in place, thus preventing the fuze from beginning its arming sequence until the setback lock detracts during a launch event. Therefore, the setback lock is required to not arm (remain in place) during a drop event (safety) and to arm during a launch event (reliability). In order to meet these requirements, uncertainty quantification techniques were used to evaluate setback lock designs. We designed a simulation experiment, simulated the setback lock behavior in a drop event and in a launch event, fit a model to the results, and optimized the design for safety and reliability. Currently, 8 candidate designs that meet the requirements are being manufactured, and adaptive sensitivity testing is planned to inform the surrogate models and improve their predictive capability. A final optimized design will be chosen based on the improved models, and realistic drop safety and arm reliability predictions will be obtained using Monte-Carlo simulations of the surrogate models.

A vulnerability assessment, which evaluates the ability of an armored combat vehicle and its crew to withstand the damaging effects of an anti-armor weapon, presents a unique challenge because vehicles are expensive and testing is destructive. This limits the number of full-up-system-level tests to quantities that generally do not support meaningful statistical inference. The prevailing solution to this problem is to obtain test data that is more affordable from sources which include component- and subsystem-level testing. This creates a new challenge that forms the premise of this paper: how can lower-level data sources be connected to provide a credible system-level prediction of vehicle vulnerability? This paper presents a case study that demonstrates an approach to this problem that emphasizes the use of fundamental statistical techniques --- design of experiments, statistical modeling, and propagation of uncertainty --- in the context of a combat scenario that depicts a ground vehicle being engaged by indirect artillery.

Sterility Assurance Level (SAL) is the probability that a product, after being exposed to a given sterilization process, contains one or more viable organisms. The SAL is a standard way of defining cleanliness requirements on a product or acceptability of a sterilization procedure in industry and regulatory agencies. Since the SAL acknowledges the inherent probabilistic nature of detecting sterility – that we cannot be absolutely sure that sterility is achieved – a probabilistic approach to its assessment is required that considers the actual end-to-end process involved with demonstrating sterility. Provided here is one such approach. We assume the process of demonstrating sterility is based on the scientific method, and therefore starts with a scientific hypothesis of the model that generates the life/death outcomes of the organisms that will be observed in experiment. Experiments are then designed (e.g. environmental conditions determined, reference/indicator organisms selected, number of samples/replicates, instrumentation) and performed, and an initial conclusion regarding the appropriate model is drawn from observed numbers of organisms remaining once exposed to the sterilization process. Ideally, this is then validated by future experiment by independent scientific inquiry, or the results are used to develop new hypotheses and the process repeats. Ultimately, a decision is made regarding the validity of the sterilization process by means of comparing with a SAL. Bayesian statistics naturally lends itself to develop a probability distribution from this process to compare against a given SAL, which is the approach taken in this paper. We provide an example application to provide a simple demonstration of this from actual experiments performed, and discuss its relevance to the future NASA mission, Mars Sample Return.

Biological cleanliness of launched spacecraft destined to celestial bodies is governed by International Planetary Protection Guidelines. In particular, spacecraft that are targeting bodies which are of scientific interest in understanding the origins of life or thought to harbor extant life must undergo microbial reduction and recontamination prevention regimes throughout the hardware assembly, test and launch operations that result in a direct verification of biological cleanliness of spacecraft surfaces. As a result of this verification associated biologicals are enumerated on petri dishes and then numerically treated to account for sample volumes, sample device efficiency, and laboratory processing efficiencies to arrive at a bioburden density. The current NASA approach utilizes a 1950’s Viking-based mathematical treatment which factors in the raw colony count, sample device and processing efficiency and fraction of extract analyzed. Historically per NASA direction, samples are grouped based upon flight hardware structural and functional proximity and if the value of raw colony counts is zero it is changed to one. In previous missions that launched from 1996 to 2018, a combination of Poisson and Gaussian statistics that evolved from mission to mission were utilized. In 2019, the statistical approach for performing bioburden accounting and verification reporting was re-evaluated to develop a technique that is both mathematically and biologically valid. Multiple mission datasets have been analyzed at high level and it has been determined that since there is a significant data set for each mission with low incidence rates of biological counts, a Bayesian model would be appropriate. Data from the InSight mission was then utilized to demonstrate the application of these models and approach on spacecraft sampled data using both informed and non-informed priors and subsequently compared to the current and historical mission mathematical treatments. The Bayesian models were within family to the previous and heritage approaches, and as an added benefit were able to provide a range and associated confidence intervals for the reported values. From the preliminary work, we propose that these models present a valid mathematical and biological approach for reporting spacecraft bioburden to be utilized in final requirements reporting and as an input into the initial bioburden population used for probabilistic risk assessments. Further development of the models will include a full spacecraft bioburden verification comparison as well as the utilization of ground truth experiments, as deemed necessary.

With the advancement of computational modeling, there is a push to reduce historically necessary experimental testing requirements for assessing vehicle component and system level performance. As a result, uncertainty quantification is a necessary part of predictive modeling, particularly regarding the modeling approach. In the absence of experimental data, model-form uncertainty may not be easily determined and model calibration may not be possible. Therefore, quantifying the potential variability as a result of model selection is required to accurately quantify performance, robustness, and reliability. This talk will outline a proposed approach to quantifying uncertainty in a variety vehicle applications with focus given particularly to supersonic flow applications. The aim is to first identify key sources of uncertainty in computational modeling, such as spatial and temporal discretization and turbulence modeling. Then, the classification and treatment of uncertainty sources is discussed, along with the potential impact of these uncertainties on performance predictions. Lastly, a description of five upcoming tests in the NASA Langley Unitary Plan Wind Tunnel designed to test predicative capability are briefly described.

Co-authors: Benjamin Ashwell, Edward Beall, and V. Bram Lillard Bottom-up emulations of real sustainment systems that explicitly model spares, personnel, operations, and maintenance are a powerful way to tie funding decisions to their impact on readiness, but they are not widely used. The simulations require extensive data to properly model the complex and variable processes involved in a sustainment system, and the raw data used to populate the simulation are often scattered across multiple organizations. The Navy has encountered challenges in keeping sustaining the desired number of F/A-18 Super Hornets in mission capable states. IDA was asked to build an end-to-end model of the Super Hornet sustainment system using the OPUS/SIMLOX suite of tools to investigate the strategic levers that drive readiness. IDA built an R package (“honeybee”) that aggregates and interprets Navy sustainment data using statistical techniques to create component-level metrics, and a second R package (“stinger”) that uses these metrics to create a high-fidelity representation of the Navy’s operational tempo for OPUS/SIMLOX.

During the fall of 2019, AFOTEC led a cross-OTA rapid improvement event to identify and mitigate challenges Operational Test (OT) teams from all services are facing in the evolving acquisition environment.  Surveys were sent out to all of the service OTAs, AFOTEC compiled the results, and a cross-OTA meeting was held at the Institute for Defense Analysis (IDA) to determine the most significant challenges to address.  This presentation discusses the selected challenges and the proposed mitigations that were briefed to the Director, Operational Test and Evaluation (DOT&E) and the Service OTA Commanders at the OTA Roundtable in November 2019.

Much work has been conducted over the last two decades on standardizing usability surveys for determining usability of a new system. In this presentation, we analyze not what we ask users, but rather how we report results from standard usability surveys such as the System Usability Scale (SUS). When the number of individuals surveyed is large, classical statistical techniques can be leveraged; however, as we will demonstrate, due to the skewness in the data these techniques may be suboptimal when the number of individuals surveyed is small. In such small-sample circumstances, we argue for use of the bias-corrected and accelerated bootstrap confidence interval. We further demonstrate how Bayesian inference can be leveraged to take advantage of the over 10 years worth of data that exists for SUS surveys. Finally, we demonstrate an online app that we have built to aid practitioners in quantifying the uncertainty in their SUS surveys.

Uncertainty quantification has been performed in various NASA Glenn Research Center (GRC) facilities over the past several years, primarily in the wind tunnel facilities. Uncertainty propagation analysis has received a bulk of the focus and effort put forth to date, and while it provides a vital aspect of the overall uncertainty picture, it must be supplemented by consistent data analysis and check standard programs in order to achieve and maintain a statistical basis for proven data quality. This presentation will briefly highlight the uncertainty propagation effort at NASA GRC, its usefulness, and the questions that remain unanswered. It will show how performing regular check standard testing fills in the gaps in the current UQ effort, will propose high-level test plans in two to three GRC facilities, and discuss considerations that need to be addressed in the planning process.

The Scientific Test and Analysis Techniques (STAT) Center of Excellence (COE) and Science of Test Research Consortium Advancements in Test and Evaluation of Autonomous Systems (ATEAS) Workshop, held 29-31 October 2019, at the Wright Brothers Institute, is part of a study being conducted on behalf of the Office of the Secretary of Defense (OSD).The goal of the study is to determine the current state of autonomous systems used within the Department of Defense (DoD), industry, and academia, with a focus on the test, evaluation, verification, and validation of those systems. The workshop addressed two overarching study objectives: 1) identify and develop methods and processes, and identify lessons learned needed to enable rigorous test and evaluation (T&E) of autonomous systems; and 2) refine current challenges and gaps in DoD methods, processes, and test ranges to rigorously test and evaluate autonomous systems. The workshop also introduced the STAT COE’s data call to the DoD, industry, and academia. This data call further informs the autonomy community on the specific efforts of, and challenges faced by, collaborators in T&E of autonomous systems. Finally, the workshop provided STAT COE members with an excellent opportunity to form partnerships with members of the DoD with the intent of finding an autonomous system pilot program in each service branch that could benefit from STAT COE support and provide first-hand experience in T&E of autonomous systems. Major takeaways from the presentations and panels included updated information on the challenges originally identified by the STAT COE in 2015, as well as new challenges related to modeling and simulation and data ownership, storage, and sharing within the DoD. The workshop also yielded information about current efforts for T&E of autonomous systems in DoD, industry, and academia; two completed data calls; targeted population to which to send the data call; and several potential pilot programs which the STAT COE could support. The next steps of the STAT COE will be to distribute the workshop report and data call, and engage with pilot programs to get first-hand experience in T&E of autonomous systems and attempt to find solutions to the challenges currently identified.

Pivotal to current US military operations is the quick identification of buildings on Gridded Reference Graphics (GRGs), gridded satellite images of an objective. At present, the United States Special Operations Command (SOCOM) identifies these buildings by hand through the work of individual intelligence officers. Recent advances in Convolutional Neural Networks (CNNs), however, allow the possibility for this process to be streamlined through the use of object detection algorithms. In this presentation, we describe an object detection algorithm designed to quickly identify and highlight every building present on a GRG. Our work leverages both the U-Net and the Mask R-CNN architectures as well as a four-city dataset to produce an algorithm that accurately identifies a large breadth of buildings. Our model reports an accuracy of 87% and is capable of detecting buildings on a diverse set of test images.

Reliability, in colloquial terms, is the ability of a system or piece of equipment to perform some required function when and where we need it to. We argue that new and unproven military equipment can benefit from a statistical approach for modeling reliability growth. The modern “standard” for these programs is the AMSAA Planning Model based on Projection Methodology (PM2). We describe how to augment PM2 with a statistical perspective to make reliability prediction more “data informed.” We have developed teaching “modules” to help elucidate this process from the ground up.

Graph link prediction is an important task in cyber-security: relationships between entities within a computer network, such as users interacting with computers, or clients connecting to servers, can provide key insights into adversary behaviour. Poisson matrix factorization (PMF) is a popular model for link prediction in large networks, particularly useful for its scalability. An extension to PMF to include scenarios that are commonly encountered in cyber-security applications are presented. Specifically, an extension is proposed to include known covariates associated with the graph nodes as well as a seasonal variation to handle dynamic networks.

Every service Chief in the DoD is urging that the key to winning future wars is having the ability to rapidly understand our enemies and integrate maneuver across domains of land, sea, air and space - what is commonly referred to as multi-domain operations (MDO). Each service has some notion of how they will accomplish this and what it means for the pace and resilience of US warfighting but there is no universally agreed upon method to get there or even to define what “there” is. However, as far back as 2011, the DoD recognized the strategic significance of managing the disparate missions of communication and control, strike, surveillance, navigation, and electronic warfare as a single, integrated electromagnetic spectrum (ES) “maneuver space” or new strategic domain to span the traditional warfighting domains. This ES domain will be managed across many platforms (manned and unmanned) with many unique ES enabled payloads across vast distances in land, sea, air and space. Only by extending these platforms some degree of autonomy and providing them with the ability to collaborate will we achieve dominance in the ES domain. In essence, collaborative autonomous software services will form the backbone of MDO. We are already seeing this play-out in a number of defense and commercial efforts and there is little doubt that we (and our adversaries) are already on the path. Using traditional DoD methods of test and evaluation may be the greatest obstacle to deploying these capabilities. The DevOps (compound of Development Operations) movement emerged in earnest in the early 2010's as a way for large enterprise software service companies (e.g. Amazon, Google, Netflix) to continuously innovate and deliver new products to the market. It has since proliferated across the globe. Three major ideas underlying the DevOps culture are of interest to us: (1) breaking down monolithic autonomous software services into multiple loosely coupled “microservices;” (2) containerizing services for portability and baked in security; and (3) performing “continuous testing.” In this presentation, we will discuss how adopting widely available DevOps tools and methodologies will make verification and validation of collaborative autonomous systems faster, more robust and transparent and enable the DoD’s goal of achieving truly effective and scalable multi domain operations.

Testing today’s complex systems often requires advanced statistical methods to properly characterize and optimize measures of performance as a function of input factors. One promising area to more precisely model system behavior when the response is a curve or function over several measured time units is Functional Data Analysis (FDA). Some input factors such as sensor data could also be functions rather than held constant as is often assumed over the duration of the test run. Recent enhancements in common statistical software programs used across DoD and NASA now make FDA much more accessible to the analytical test community. This presentation will address the fundamental principles, workflow, and interpretation of results from FDA using a designed experiment for an autonomous system as an example. Additionally, we will address how to use FDA to establish a level of equivalence for modeling & simulation verification & validation efforts.

The mission of the Human Behavior and Cybersecurity Capability at MITRE is to leverage human behavior to reduce cybersecurity risk using behavioral sciences to understand and strengthen the human firewall. The Capability consists of a team of experienced behavioral scientists who bring applied subject-matter-expertise in human behavior to cybersecurity challenges, with demonstrated successes and thought leadership in insider threat and usable security. This presentation will introduce the four focus areas of the Capability: Insider Threat; Usable Security and Technology Adoption; Cybersecurity Assessment and Exercise Support; and Cybersecurity Risk Perceptions and Awareness. We will then discuss the methods and metrics used to study human behavior within each of these focal areas. Insider Threat Assessment: Our insider threat behavioral risk assessments use qualitative research practices to elicit discussion and disclosure of risks from the perspective of potential insiders. MITRE’s Insider Threat Framework, developed to address the challenges experienced by the National Critical Infrastructure in classifying and assessing insider threats, is a data-driven framework that includes psycho-social and cyber-physical characteristics that could be common, observable indicators for insider attacks. The continuous approach to developing this framework includes consistently structuring, hand-coding, collating and analyzing a large dataset (5,000-10,000) of raw insider threat investigation case files shared directly from multiple organizations. The aggregated framework, but not the sensitive raw data, is shared with insider threat programs to operationalize and facilitate the identification, prevention, detection and mitigation of insider threats. Usable Security and Technology Adoption: Our goal of studying the human aspects of usable security technologies is to improve decision making and enhance technology adoption by users. We will present examples of how we have applied psychological principles and behavioral methods to design valid and reliable approaches to evaluate the feasibility of new security programs, products, and resources such as AI-enabled security technology. Cybersecurity Assessment and Exercise Support: Complex collaborations among cybersecurity teams are becoming increasingly important but can trigger new challenges that impact mission success. We assess, evaluate and train individuals and teams to improve knowledge among cyber professionals during cybersecurity exercises. Examples of our work include cognitive adaptability and exercise assessments, development of team collaboration and performance metrics, team sense-making, and cyber team resilience. Cybersecurity Risk Perceptions and Awareness: Perceptions of cybersecurity risks and threats, and resulting decisions about how to approach or mitigate them have the potential to impact the effectiveness of cybersecurity programs. We evaluate how people, processes and technology impact tactical and strategic risk-based decisions and apply behavioral concepts to inform the cybersecurity risk framework and awareness programs.

Recent developments in artificial intelligence have captured the popular imagination and the attention of governments and businesses across the world. From digital assistants, to smart homes, to self-driving cars, AI appears to be on the verge of taking over many parts of our daily lives. Meanwhile, as the world becomes more networked, industry, governments, and individuals are facing a growing array of cybersecurity threats. In this talk, we will discuss the intersection of artificial intelligence, machine learning, and cybersecurity. In particular, we will look at how some popular machine learning methods will and will not change how we do cybersecurity, we will separate what in the AI and ML space is realistic from what is science fiction, and we will attempt to identify the true potential of ML to positively impact cybersecurity.

We are given a time series of graphs, for example those defined by connections between computers in network flows. Several questions are relevant to cyber security: 1) What are the natural groupings (clustering) of the computers on the network, and how do these evolve in time? 2) Is the graph "abnormal" for this day/time and hence indicative of a large scale problem? 3) Are certain nodes acting "abnormally" or "suspiciously"? 4) Given that some computers cannot be uniquely resolved (due to various types of dynamic IP address assignments), can we pair a newly observed computer with it's previous instantiation in an earlier hour? In this talk, I will give a very brief introduction to some spectral graph methods that have shown promise for answering some of these questions, and present some preliminary results. These methods are much more widely applicable, and if time permits I will discuss some of the areas to which they are currently being applied.

Linear regression is often extended to either (1) multilevel models in which response data cannot be assumed independent, e.g., nested data, or (2) generalized linear models in which response data is not normally distributed. Applying both extensions to binary responses results in generalized linear multilevel models in which a sigmoid link function gives the relationship between the mean of the response variable and a linear combination of predictors. Such models are well-suited to analyze psychophysical experiments, in which fitted sigmoids give the relationship between physical stimulus level and the probability of a human perceptual response. In this work, a generalized linear multilevel model is applied to a three-alternative forced-choice psychoacoustic test in which human test subjects were asked to identify a sound signal presented at different levels relative to a background noise. Since the guessing rate at low stimulus levels must converge to 1/3, a custom link function is applied. In this test, the grouping variable is the subject, because within-subject responses are assumed to be more alike than between-subject responses. This leads to readily available information about population-level parameters, such as how auditory thresholds are distributed within the group, how steep the psychometric functions are and if the differences are statistically significant. The multilevel model also demonstrates the effect of shrinkage in which the partially-pooled regression parameters are closer to the population mean than parameters found by un-pooled analyses.

Challenges in adapting the system design process, including test and evaluation, to autonomous systems have been well documented. These include the inability to verify autonomous behaviors, designing systems for assurance, and dealing with adaptive systems post-fielding. This talk will briefly recap those challenges, and then propose a path toward overcoming these challenges in the case of Army ground autonomous systems. The path begins with unmanned systems designed for test. Training and experimentation within a safe infrastructure can then be used to specify system characteristics and inform decisions as to what behaviors should be autonomous. Assurance comes in the form of assurance arguments updated as the system technology and understanding increases.

Machine learning is becoming increasingly imbedded in military systems, so how do we know if it’s working, or as the name implies learning? This presentation and demonstration is a thought experiment for tester’s to consider how to measure and test systems with imbedded machine learning. This presentation involves a physical demonstration of elements of the popular casino game craps. Craps is a fast-paced dice game that offers players the opportunity to try and beat the odds, which are stacked in the casino’s favor. But what if a player could flip the odds on the casino by cheating? Can the audience detect the cheater? How about a machine learning algorithm? The audience will be able to make their own guesses and compare them to A-Dell, a machine learning algorithm designed to find craps cheaters.

A common challenge in the cybersecurity realm is the proper handling of high-volume streaming data. Typically in this setting, analysts are restricted to techniques with computationally cheap model-fitting and prediction algorithms. In many situations, however, it would be beneficial to use more sophisticated techniques. In this talk, a general framework is proposed that adapts a broad family of statistical and machine learning techniques to the streaming setting. The techniques of interest are those that can generate computationally cheap predictions, but which require iterative model-fitting procedures. This broad family of techniques includes various clustering, classification, regression, and dimension reduction algorithms. We discuss applied and theoretical issues that arise when using these techniques for streaming data whose distribution is evolving over time.

This study assesses the relationship between psychometric screening of candidates for our partner special operations unit and successful completion of the unit’s candidate selection program. Improving the candidate selection program is our primary goal for this study. Our partner unit maintains a comprehensive database of summary information on previous candidates but has not yet conducted a robust analysis of candidate attributes. We sought to achieve this goal by using statistical methods to determine predictors associated with successful completion of the selection program. Our results suggest that we may identify predictors associated with success but may struggle in constructing an effective predictive model due to the inherent differences of candidates. Our predictors are scales from standardized psychometric evaluations administered by the selection program intended originally to identify candidates with psychopathologies. Our outcome is a binary variable indicating successful completion of the program. Analyzing the demographics of the candidate selection population is our secondary goal for this study. Our analysis helped our partner unit improve its selection program by identifying characteristics associated with success. Other members of the special operations community may generalize our results to improve their respective programs as well. We do not intend for units to use these results to draw definitive conclusions regarding the ability of a candidate to pass a selection program.

The Monte Carlo method of uncertainty analysis was used to characterize the uncertainty of tunnel conditions within the calibration of a wind tunnel. To calibrate the tunnel, a long static pipe, consisting of 444 static pressure ports, was used. Data from this calibration was used to generate the data needed to be implemented into the analysis. Monte Carlo analysis specifically looks at all potential possibilities using this generated data by propagating it through the equations for tunnel conditions. Any method of uncertainty analysis would normally encompass precision, bias, and fossilized uncertainty, however, for this particular analysis, precision uncertainty is assumed to be negligible. All remaining uncertainties were calculated using a sigma value of 2 with a 95% confidence level over a normal Gaussian distribution. At the end of the analysis, a comparison was done to see the effect of fossilized uncertainty on the overall uncertainty, which showed that this often overlooked portion of uncertainty does cause a noticeable change in the overall uncertainty of the value. With the consideration of fossilized uncertainty, the Monte Carlo method of analysis is a useful method for characterizing the uncertainty of tunnel conditions for a wind tunnel calibration.

In this presentation we will discuss a methodology to automatically extract road networks from aerial images using machine learning algorithms. There are many civilian applications for this technology, such as maintaining GPS maps, real-estate monitoring and disaster relief, but this presentation is aimed a military applications for analyzing remote sensing data. The algorithm we propose identifies road networks and classifies each road as either improved or unimproved. Implementing this system in a military context will require our model to work across a wide range of environments. We will discuss the effectiveness of the model in a military context.

We discuss several shock-physics applications we are pursuing using deep neural network approaches. These range from equation-of-state (EOS) inference for simple flyer-plate impact experiments when velocity time series are observed to radiographic inversion of high-speed impact simulations with composite materials. We put forward a methodology to leverage privileged information available from high-fidelity simulations to train a deep network prior to application on experimental observation. Within this process several sources of uncertainty are active and we discuss ways to mitigate these by careful structuring of the simulated training set. Once a network is trained the credibility of its inference, beyond performance on a test set, must be assessed. Without robust feature detection tools available for deep neural networks we show that much can be gained by applying classical sensitivity analysis techniques to the trained network. We show some results of this sensitivity analysis for our physics applications and discuss the caveats and pitfalls that arise when applying sensitivity analysis to deep machine learning algorithms.

We study the problem of post-hoc calibration of machine learning classifiers. We introduce the following desiderata for uncertainty calibration: (a) accuracy-preserving, (b) data-efficient, and (c) high expressive power. We show that none of the existing methods satisfy all three requirements, and demonstrate how our proposed calibration strategies can help achieve dramatically better data efficiency and expressive power while provably preserving classification accuracy of the original classifier. When calibrating a 50-layer Wide ResNet on ImageNet classification task, the proposed strategies improve the expressivity of temperature scaling by 17% and data efficiency of isotonic regression by a factor of 258, while preserving classification accuracy.

Across application domains, analysts are tasked with an untenable situation of manually completing a big data analysis of a mix of quantitative and qualitative information sets. Human decision-making requires that evidence gathered from sources such as experiments, engineering analysis, and expert judgment be transformed into an appropriate format and presentation style. Distillation and interpretation of multi-source data can be supported through tools or decision-support systems that include automated features to reduce the mental burden on human analysts. Analysts benefit from a data-informed support tool that provides the correct information, at the right time, to arrive at the correct solution under uncertainty. My research has coupled a process/workflow-oriented view with knowledge and skill elicitation techniques to predict information analysts need and how they interact with and transform that information. Thus, this data-informed approach allows for a mapping of process steps and tools that will benefit analysts. As the state of data as it exists today is only likely to grow, not diminish over time, an approach to efficiently organizing and interpreting the data is crucial. Ultimately, improved decision making is realized across an entire workflow that is sustainable across time. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Machine learning methods have attracted a lot of research attention in scientific applications. However, model credibility remains a major challenge toward the reliable deployment of these models to the field in costly or risky use cases. In this talk, we explore uncertainty quantification techniques to asses the quality of neural network predictions in regression and classification problems. To understand model variability, we examine different sources of randomness associated with training samples, data observation order, weight initialization, dropout, and ensemble formations. Motivated by typical scientific computing applications, we assume a limited sample budget and suggest approaches for reporting and possibly reducing uncertainty.

According to Forbes, merchants in the United States lose approximately $190 billion annually to credit card fraud (Shaughnessy 2012). Nordstrom, a leading retailer in the fashion industry, alone incurs losses upwards of $100 million every year. While these losses hurt the company financially, the more pressing concern for Nordstrom is the negative impact on the customer. If fraud is incorrectly flagged and an account is unnecessarily frozen, a customer will be dissatisfied with their experience. Conversely, if legitimate fraud goes undetected, valued customers can experience monetary loss and lose faith in the company. To minimize losses while maximizing customer satisfaction, Nordstrom Card Services created a fraud detection machine that analyzes every transaction and will take one of four actions based on the transaction’s characteristics: approve, approve and send to a queue, decline and send to a queue, or decline. Once a transaction is sent to the queue, it is assessed by either a human analyst or a virtual analyst. Those analysts will determine if any further action must be taken within the customer’s account. This project focuses on how to appropriately assign human and virtual decision-makers to maximize accuracy when determining whether or not a credit card transaction is fraud.

As defense systems become more complex, multi-domain, and interdependent, the problem arises: What is the best way to determine what we need to test, and how much testing is adequate? A methodology, based on systems engineering, was developed specifically for use in Live Fire Test and Evaluation (LFT&E); however, the process can be applied to operational or developmental testing as well. The use of Systems Engineering principles involves understanding the prioritized warfighter (user) needs, and applies a definition process that identifies critical test issues. The main goals of this methodology are to clearly define the system performance priorities, based on the critical mission risks if performance is insufficient, and then to generate various solutions that will mitigate those associated risks. The systems engineering process also helps develop a common language and framework so all parties can discuss tradeoffs between cost, schedule and performance. It also produces specific products for each step of the process that ensure that each step has been adequately addressed. Most importantly, the methodology should enable better communication among and within the program, test, and oversight teams by clarifying mission and test priorities, clarifying test objectives, evaluating risks from proposed testing and demonstrated performance, and reporting decision-quality information. The result of implementing the methodology is two-fold: It produces a test strategy that prioritizes testing based on the criticality and uncertainty of the system’s performance; and it guides the development of a system evaluation that clearly links test outcomes to overall mission risks.

The field of human reliability analysis (HRA) seeks to predict sources of human error for safety-critical systems. Much of the dominant research in HRA was historically conducted for nuclear power, where regulations required risk models of hardware and humans to ensure the safe operation of nuclear power plants in the face of potential accident situations. A challenge for HRA is that the incidence of nuclear accidents has fortunately been extremely rare. Thus, actuarial data do not serve as a good source for informing predictions. Simulators make it possible to train performance for rare events, but data collection has largely focused on performance where human errors have occurred, omitting a denominator of good and bad performance that would be useful for prediction. As HRA has branched into new domains, there is the additional challenge to what extent those data that are available from primarily control room operations can be generalized to other types of human activities. To address these shortcomings, new data collection efforts are underway, focusing on (1) more comprehensive logging of operator performance in simulators, (2) development of new testbeds for collecting data, and (3) data mining of existing human performance data that were not originally intended for human error. In this talk, I’ll review underlying assumptions of HRA and map new data sources to solving data needs in new domains such defense and aerospace.

The DOD’s ability to collect data is far outstripping its ability to convert that data into actionable information. This problem is not unique to the military. With vast improvements in our ability to collect data, many organizations are drowning in that data. As a result, the need for advanced algorithms (Artificial Intelligence, Machine Learning, etc.) to support human work has never been greater. However, algorithm performance and the performance of the larger work system, composed of human and machine agents, are not synonymous. In order to build and field high performance work systems that consistently provide positive mission impact, the S&T community must measure the performance of the entire Joint Cognitive System (JCS). Algorithm performance, by itself, is necessary but not sufficient for predicting the performance of these systems in the field. Better predicting JCS performance once a system is fielded requires modelling, or at least learning about, the humans’ cognitive work and how the proposed technology will change that work. For example, providing practitioners with a new Machine Learning enabled support tool may impose unintended cognitive work as practitioners calibrate themselves to the performance envelope of the new tools. In addition, the practitioners’ cognitive work is shaped by many “soft constraints” which are often not captured by technologists. For example, real world time constraints may lead practitioners to use simple decision making heuristics, rather than deliberative hypothesis testing, when making critical decisions. The technology requirements needed to support both deliberate and heuristic decision making may be very different. This talk will discuss several approaches to gain insight on JCS performance as part of a larger cycle of iteratively discovering, building, and testing these systems. Additionally, this talk will give an overview of a small pilot study conducted by the Air Force Research Lab to measure the impact of a simulated Machine Learning agent designed to support AF intelligence analysts exploiting Full Motion Video data.

Structural Equation Modeling (SEM) is an analytical framework that offers unique opportunities for investigating human-system interactions. SEM is used heavily in the social and behavioral sciences, where emphasis is placed on (1) explanation rather than prediction, and (2) measuring variables that are not observed directly. The framework facilitates modeling of survey data through confirmatory factor analysis and latent (i.e., unobserved) variable regression models. We provide a general introduction to SEM by describing what it is, the unique features it offers to analysts and researchers, and how it is easily implemented in JMP Pro 15.1. The introduction relies on a fun example everyone can relate to. Then, we shed light on a few published studies that have used SEM to unveil insights on human performance factors and the mechanisms by which performance is affected. The key goal of this presentation is to provide general exposure to a modeling tool that is likely new to most in the fields of defense and aerospace.

Having multiple objectives is common is experimental design. However, a design that is optimal for a normal response can be very different from a design that is optimized for a nonnormal response. This application uses a weighted optimality criterion to identify an optimal design with continuous factors for three different response distributions. Both linear and nonlinear models are incorporated with normal, binomial and Poisson response variables. A JMP script employs a coordinate exchange algorithm that seeks to identify a design that is useful for all three of these responses. The impact from varying the prior distributions on the nonlinear parameters as well as changing the weights on the responses in the criterion is considered.

The Air Force maintains a space catalog of orbital information on tens-of-thousands of space objects, including active satellites and satellite debris. They also computationally project where they expect objects to be in the future. Each day, the Air Force issues warnings to satellite owner-operators about potential conjunctions (space objects passing near each other), which often results in one or both of the satellites maneuvering (if possible) for safety of flight. This problem grows worse as mega-constellations, such as SpaceX's Starlink, are launched.

Introduction: Information Operations (IO) are a key component of our adversaries' strategy to undermine U.S. military power without escalating to more traditional (and more easily identifiable) military strikes. Social media activity is one method of IO. In 2017 and 2018, Twitter suspended thousands of accounts likely belonging to the Kremlin-backed Internet Research Agency (IRA). Clemson University archived a large subset of these tweets (2.9M tweets posted by over 2800 IRA accounts), tagged each tweet with metadata (date, time, language, supposed geographical region, number of followers, etc.), and published this dataset on the polling aggregation website FiveThirtyEight.

With the rise of machine learning and artificial intelligence, there has been a huge surge in data-driven approaches to solve computational science and engineering problems. In the context of uncertainty quantification (UQ), a common use case for machine learning is in the construction of efficient surrogate models (i.e., response surfaces) to replace expensive, physics-based simulations. However, relying solely on data-driven models for UQ without any further recourse to the original high-fidelity simulation will generally produce biased estimators and can yield unreliable or non-physical results, especially when training data is sparse or predictions are required outside of the training data domain.

Co-Authors: Luis Cortes (MITRE) and Alethea Duhon (Air Force Agency for Modeling and Simulation)

This talk highlights the advantage of a multi-criteria decision analysis (MCDA) technique that utilizes the one-sided tolerance interval (OSTI) for the analysis of response data resulting from design of experiments (DOE) structured testing.  Tolerance Intervals (TI) can be more intuitive for representing data and, when combined with DOE structured tests results, are excellent for providing decision quality information.  The use of statistical techniques in planning provides a rigorous approach for the analysis of test data, an ideal input for a MCDA technique.  This article’s findings demonstrate the value of constructing the OSTI for the interpretation of DOE structured tests.  We also demonstrate the utility of using an optimized model combined with the OSTI as part of a MCDA technique for choosing between alternatives, which offers an efficient and powerful method to sift through test results rapidly and efficiently.  This technique provides an alternative for analyzing data across multiple alternatives when traditional analysis techniques, such as the descriptive statistics (including the Confidence Interval [CI]), potentially obscure information from the untrained eye.  Finally, the technique provides a level playing field--a critical feature given today’s acquisition protest culture.

The challenge of identifying test cases that maximize the chance of discovering faults at minimal cost is a challenge that software test engineers constantly face. Combinatorial testing is an effective test case selection strategy to address this challenge. The idea is to select test cases that ensure that all possible combinations of settings from two (or more) inputs are accounted for, regardless of which subset of inputs are selected. This is accomplished by using a covering array as the test case selection mechanism. However, for any given testing scenario, there are usually several alternative covering arrays that may satisfy budgetary and coverage requirements. Unfortunately, it is often unclear how to choose from these alternatives. Practitioners often want a way to explore how the input space is being covered before deciding which alternative is best suited for the testing scenario. In this presentation we will provide an overview of a new graphical method that may be used to visualize the input space of covering arrays. We use the phrase "design fractals" to refer to this graphical method.

Authors: Lawrence J. Prinzel III, Jon B. Holbrook, Kyle E. Ellis, Chad L. Stephens New innovative technologies and operational concepts will be required to meet the ever increasing global demands on air transportation. The NASA System-Wide Safety (SWS) project is focused on how future aviation advances can meet demand needs while maintaining today’s ultra-safe system safety levels. Aviation safety as it evolves shall require new ways of thinking about safety, integrating a wide-range of existing and new safety systems and practices, creating and enhancing tools and technologies, leveraging the access to system-wide data and data fusion, improving data analysis capabilities, and developing new methods for in-time risk monitoring and detection, hazard prioritization and mitigation, safety assurance decision-support, and in-time integrated system analytics [NRC, 2018]. To meet these needs, the SWS project has developed research priorities including In-time System-wide Safety Assurance (ISSA) and development of In-time Aviation Safety Management System (IASMS) [Ellis et al., 2019]. As part of this effort, the concepts of “resilience” and “productive safety” are being studied. Traditional approaches to aviation safety have focused on what can go wrong and how to prevent it. Another approach to thinking about system safety should reflect not only “avoiding things that go wrong” (protective safety) but also “ensuring that things go right” (productive safety) that together enables a system to exhibit resilience. On-going SWS research is focused on application of these concepts for ISSA and design of IASMS. NASA identified significant challenges and research needs for ISSA and IASMS for Urban Air Mobility (UAM) [NRC, 2018] [Ellis et al., 2019]. UAM is an emerging concept of operation that features small aircraft providing on-demand transportation over relatively shorter distances within urban areas. The SWS project has focused on development of UAM-domain safety monitoring and alerting tools, integrated predictive technologies for UAM-level application, and adaptive in-time safety threat management [Ellis et al., 2019]. Significant research challenges include how to identify data sources and indicators for in-time safety critical risks, how to analyze those data to detect and prioritize risks, and how to optimize safety awareness and safety action decision support. Because UAM is also being used to evaluate the safety paradigm of “work-as-imagined” - characterizing how people think their work is done in comparison to how work is actually done as they are all too often not the same. The challenges associated with ISSA and development of IASMS are significant even for existing air transportation system operations where work-as-imagined and work-as-done can actually be compared. However, because UAM currently exists only as work-as-imagined, the safety challenges are far greater for meeting ISSA needs and design of IASMS. The present proposal shall discuss resilience and productive safety considerations for in-time safety assurance and safety management systems for UAM. Topics include challenges of collecting productive safety in-time data, granularity of data types and measurement, the need for new analytical methods, issues for identifying in-time productive safety metrics and indicators, and potential approaches toward quantification of resilience indices. Recommendations and future research directions shall also be described.

Recently, a Department of Energy (DOE) report was released on the concept of scientic machine learning (SML), which is broadly dened as "a computational tech- nology that can be trained, with scientic data, to augment or automate human skills." [1] As the demand for machine learning (ML) in science and engineering rapidly in- creases, it is important to have condence that the output of the ML algorithm is representative of the phenomena, processes, or physics being modeled. This is espe- cially important in high-stakes elds such as defense and aerospace. In the DOE report, three research themes were highlighted with the aim of providing condence in ML im- plementations. In particular, ML algorithms should be domain-aware, interpretable, and robust. Deep learning has become a ubiquitous term over the past decade due to its ability to model high-dimensional complex processes, but domain awareness, interpretability, and robustness in these large neural networks (NNs) are often hard to achieve. Recent advances in physics-informed neural networks (PINNs) are promising in that they can provide both domain awareness and a degree of interpretability [2, 3, 4]. These al- gorithms take advantage of the breadth of scientic knowledge built over centuries by fusing governing partial dierential equations into the NN training process. In this way, PINNs output physically admissible solutions. However, PINNs are generally deter- ministic, meaning interpretability and robustness suffer as it is unclear how uncertainty affects the model. Another noteworthy deep learning algorithm is the generative adversarial network (GAN). GANs are capable of modeling probability distributions in both forward and inverse problems, and thus have received a ood of inerest with over 15,000 citations of the seminal paper [5] in six years. A natural next step is to combine both PINNs and GANs to address all three themes laid out in [1]. The resultant physics-informed GAN (PI-GAN) is capable of both modeling physical processes and simultaneously quantifying uncertainty. A limited number of works have already demonstrated the success of PI-GANs [6, 7]. This talk will present an introduction to PI-GANs as well as an example of current NASA research implementing these networks.

Recent advances in brain computer interfaces (BCI) has brought interest in exploring the possibility of BCI like technology for future armament systems. In an experiment, conducted at the Tactical Behavioral Lab (TBRL), electroencephalogram (EEG) data was collected during simulated engagement scenarios in order to study the relationship between a soldier's state of mind and his biophysical signals. One of the goals was to determine if it was possible to anticipate the decision to fire a gun. The nature of EEG data presents a unique set of challenges. For example, the high sensitivity of EEG electrodes coupled with their close proximity on the scalp means recorded data is noisy and highly correlated. Special attention needs to be payed to data pre-processing and feature engineering in building predictive models.

Electronic warfare systems generate diverse signals in a complex environment. This briefing covers how a particular system employed DOE to develop the overall strategy, determined which tests would utilize DOE, and generated specific designs. It will cover the technical and resource challenges encountered throughout the process leading up to testing. We will discuss the difficulty in defining responses and factors and ideas on how to resolve these issues. Lastly we will cover impacts from shortened schedules and how DOE enabled a quantitative risk assessment.

When analyzing binary responses, logistic regression is sometimes difficult when some scenarios tested result in only successes or failures; this is called separation in the data. Using typical frequentist methods, logistic regression often fails because of separation. However, Bayesian logistic regression does not suffer from this limitation. This talk will walk through a Bayesian logistic regression of data with separation using the R brms package.

Requirements specification flaws are still the biggest contributing factor to most accidents related to software. Most NASA projects have safety-critical requirements that, if implemented incorrectly, could lead to serious safety implications and/or mission-ending scenarios. There are normally thousands of system-/subsystem-/component-level requirements that need to be analyzed for safety criticality early in the project development life cycle. Manually processing such requirements is typically time-consuming and prone to error. To address this, we implemented and tested text classification models to identify requirements that are safety-critical within project documentation. We found that a naïve Bayes classifier was able to identify all safety-critical requirements with an average false positive rate of 41.35%. Future models trained on larger project requirement datasets may achieve even better performance, reducing the burden of processing requirements on safety and mission assurance personnel and improving the safety of NASA projects.

Because of the formidable challenge of observing the time-evolving full-depth global ocean circulation, numerical simulations play an essential role in quantifying the ocean’s role in climate variability and long-term change. For the same reason, predictive capabilities are confounded by the high-dimensional space of uncertain variables (initial conditions, internal parameters, external forcings, and model inadequacy). Bayesian inverse methods (loosely known in approximate form as data assimilation) that optimally extract and merge information from sparse, heterogeneous observations and models are powerful tools to enable rigorously calibrated and initialized predictive models to optimally learn from the sparse data. A key enabling computational approach is the use of derivative (adjoint and Hessian) methods for solving a deterministic nonlinear least-squares optimization problem. Such a parameter and state estimation system is practiced by the NASA-supported Estimating the Circulation and Climate of the Ocean (ECCO) consortium. An end-to-end system that propagates uncertainties from observational data to relevant oceanographic metrics or quantities of interest within an inverse modeling framework should address, within a joint approach, all sources of uncertainties, including those in (1) observations (measurement, sampling and representation errors), (2) the model (parametric and structural model error), (3) the data assimilation method (algorithmic approximations and data ingestion), (4) initial and boundary conditions (external forcings, bathymetry), and (5) the prior knowledge (error covariances). Here we lay out a vision for such an end-to-end framework. Recent developments in computational science and engineering are beginning to render theoretical concepts practical in real-world applications.

Nonregular designs are a preferable alternative to regular resolution IV designs because they avoid confounding two-factor interactions. As a result nonregular designs can estimate and identify a few active two-factor interactions. However, due to the sometimes complex alias structure of nonregular designs, standard screening strategies can fail to identify all active effects. In this talk, two-level nonregular screening designs with orthogonal main effects will be discussed. By utilizing knowledge of the alias structure, we propose a design based model selection process for analyzing nonregular designs. Our Aliased Informed Model Selection (AIMS) strategy is a design specific approach that is compared to three generic model selection methods; stepwise regression, Lasso, and the Dantzig selector. The AIMS approach substantially increases the power to detect active main effects and two-factor interactions versus the aforementioned generic methodologies.

The Europa Clipper mission must comply with the NASA Planetary Protection requirement in NASA Procedural Requirement (NPR) 8020.12D, which states: “The probability of inadvertent contamination of an ocean or other liquid water body must be less than 1×10-4 per mission”. Mathematical approaches designed to assess compliance with this requirement have been offered in the past, but no accepted methodology was in-place to trace the end-to-end probability of contamination: from a terrestrial microorganism surviving in a non-terrestrial ocean, to the impact scenario that put them there, to potential flight system failures that led to impact, back to the initial bioburden launched with the spacecraft. As a result, hardware could presumably be either over- or under-cleaned. Over-specified microbial reduction protocols can greatly add to the cost (and schedule) of a project. On the other hand, if microbes on hardware are not sufficiently eliminated, there is increased risk of potentially contaminating another body with terrestrial organisms – adversely affecting scientific exploration and possibly conflicting with international treaty. The anticipated Mars Sample Return Campaign would be subject to a similar challenge regarding returning Martian material to Earth. A proposed requirement is “The MSR campaign shall have a probability of releasing unsterilized Martian particles, with diameters ≥ 50 nanometers (TBC), into Earth’s biosphere ≤ 1x10-6”. A similar question arises: what is required in terms of ensuring sterilization or containing Martian particles in order to meet the requirement? The mathematical framework and other interesting sensitivities that the analysis has revealed for each mission are discussed.

Equipment Failure Reports (EFRs) describe equipment failures and the steps taken as a result of these failures. EFRs contain both structured and unstructured data. Currently, analysts manually read through EFRs to understand failure modes and make recommendations to reduce future failures. This is a tedious process where important trends and information can get lost. This motivated the creation of an interactive dashboard that extracts relevant information from the unstructured (i.e. free-form text) data and combines it with structured data like failure date, corrective action and part number. The dashboard is an RShiny application that utilizes numerous text mining and visualization packages, including tm, plotly, edgebundler, and topicmodels. It allows the end-user to filter to the EFRs that they care about and visualize meta-data, such as geographic region where the failure occurred, over time allowing previously unknown trends to be seen. The dashboard also applies topic modeling to the unstructured data to identify key themes. Analysts are now able to quickly identify frequent failure modes and look at time and region-based trends in these common equipment failures. DISTRIBUTION STATEMENT A - APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED.

Supervised learning competitions such as those hosted by Kaggle provide a quantitative, objective way to evaluate and compare algorithms. However, the typical format of evaluating competitors against a fixed set of data can encourage overfitting to that particular set. If the competition host is interested in motivating development of approaches that will perform well against a more general “in the wild” problem outside of the competition, the winning algorithms of these competitions might fall short because they’re tuned too closely to the competition data. Furthermore, the idea of having a single final ranking of the competitors based on the competition data set ignores the possibility that that ranking might change substantially if a different data set were used, even one that has the same statistical characteristics as the original.

We present an approach for designing training and test sets that reward more robust algorithms and discourage overfitting with the intent of improved performance for scenarios beyond the competition. These carefully designed sets also enable a rich and nuanced analysis and comparison of the performance of competing algorithms, including a more flexible final ranking. We illustrate these methods with two competitions recently designed and hosted by Los Alamos National Laboratory to improve detection, identification, and location of radiological threats in urban environments.

The ability of spaceborne remote sensing data to address important Earth and climate science problems rests crucially on how well the underlying geophysical quantities can be inferred from these observations. Remote sensing instruments measure parts of the electromagnetic spectrum and use computational algorithms to infer the unobserved true physical states. However, the accompanying uncertainties, if they are provided at all, are usually incomplete. There are many reasons why including but not limited to unknown physics, computational artifacts and compromises, unknown uncertainties in the inputs, and more.

In this talk I will describe a practical methodology for uncertainty quantification of physical state estimates derived from remote sensing observing systems. The method we propose combines Monte Carlo simulation experiments with statistical modeling to approximate conditional distributions of unknown true states given point estimates produced by imperfect operational algorithms. Our procedure is carried out post-hoc; that is, after the operational processing step because it is not feasible to redesign and rerun operational code. I demonstrate the procedure using four months of data from NASA's Orbiting Carbon Observatory-2 mission, and compare our results to those obtained by validation against data from the Total Carbon Column Observing Network where it exists.

The problem of augmenting an existing sensor network can be solved by considering how best to answer the question of interest. In nuclear explosion monitoring, this could include where best to place a new seismic sensor to most accurately and precisely predict the unknown location of a seismic event. In this talk, we will solve this problem using a Bayesian Design of Experiments (DoE) approach where each design consists of the existing sensor network coupled with a new, possible location. We will incorporate complex computer simulation and experimental data to rank the possible sensor locations with respect to their ability to provide accurate and precise event location estimates. This will result in a map of desirability for new locations, so that if the most highly ranked location is not available or possible, regions of similar predictability can be identified. The Bayesian DoE approach also allows for incorporating denied access locations due to either geographical or political constraints.

The calibration of complex models to data is both a challenge and an opportunity. It can be posed as an Inverse Problem. This work focuses on the interface of Ensemble Kalman algorithms used for inversion or posterior sampling (EKI/EKS) , Gaussian process emulation (GPE) and Markov Chain Monte Carlo (MCMC) for the calibration of, and quantification of uncertainty in, parameters learned from data. The goal is to perform uncertainty quantification in predictions made from complex models, reflecting uncertainty in these parameters, with relatively few computationally expensive forward model evaluations. This is achieved by propagating approximate posterior samples obtained by judicious combination of ideas from EKI/EKS, GPE and MCMC. The strategy will be illustrated with idealized models related to climate modeling.

Recurrence analysis models the frequency of recurrent events, such as break downs or repairs, to obtain the total repairs per unit as a function of time. Text analytics is used to extract useful summary information from maintenance records. The coupling of techniques results in dynamic, actionable information used to ensure optimal management of test instruments. This session provides a practical example of recurrence analyses and text analytics on data collected from multiple units of chromatography instrumentation. Analyses include the liberal use of data visualization and practical interpretation of results giving attendees enlightenment intended to extract the best performance from test equipment. Testing and evaluation typically involves the use of sophisticated instrumentation. The information contributed from instruments must be precise, accurate, and reliable due to critical decisions that are made from the data. Regular repairs and maintenance are needed to ensure robust instruments that operate properly. Recurrence analysis is used to obtain a mean cumulative function (MCF) to better understand instrument performance over time. The MCF can be used to estimate maintenance costs, explain repair tendencies, and compare units. Test instrumentation typically includes a large amount of documentation within use logs and maintenance records. Text analytics is used to quickly summarizes documented information into word clouds, document term matrices, or clusters for enhanced understanding. Common themes that arise from text analytics help to focus preventive maintenance and to ensure that the most needed parts and resources are available to mitigate testing delays. Latent semantic analyses enhance the information to better understand the topic vectors present within the documents. The dynamic linking of techniques provides for optimal understanding of the performance of test instrumentation over the life of the equipment. The results of the combined analyses allow for data-driven maintenance planning, sound justification for instrument replacement, and the assurance of robust test results.

The growing complexity of interactive systems requires increasing amounts of effort to ensure reliability and usability. Testing is an effective approach for finding and correcting problems with implemented systems. However, testing is often regarded as the most intellectual-demanding, time-consuming, and expensive part of system development. Furthermore, it can be difficult (if not impossible) for testers to anticipate all of the conditions that need to be evaluated. This is especially true of human-machine systems. This is because the human operator (who is attempting to achieve his or her task goals) is an additional concurrent component of the system and one whose behavior is not strictly governed by the implementation of designed system elements. To address these issues, researchers have developed approaches for automatically generating test cases. Among these are formal methods: rigorous, mathematical languages, tools, and techniques for modeling, specifying, and verifying (proving properties about) systems. These support model-based approaches (almost exclusively used in computer engineering) for creating tests that are efficient and provide guarantees about their completeness (at least with respect to the model). In particular, model checking can be used for automated test case generation. In this, efficient and exhaustive algorithms search a system model to find traces (test cases) through that model that satisfy specified coverage criteria: descriptions of the conditions the tests should encounter during execution. This talk focuses on a formal automated test generation method developed in my lab for creating cases for human-system interaction. This approach makes use of task models. Task models are a standard human factors method for describing how humans normatively achieve goals when interacting with a system. When these models are given formal semantics, they can be paired with models of system behavior to account for human-system interaction. Formal, automated test case generation can then be performed for coverage criteria asserted over the system (for example, to cover the entire human interface) or human task (to ensure all human activities or actions are performed). Generated tasks, when manually executed with the system, can serve two purposes. First, testers can observe whether the human behavior in test always produces the system behavior from the test. This can help analysts validate the models and, if no problems are found, be sure that any desirable properties exhibited by the model hold in the actual system. Second, testers will be able to use their insights about system usability and performance to subjectively evaluate the system under all of conditions contained in the tests. Given the coverage guarantees provided by the process, this means that testers can be confident they have seen every system condition relevant to the coverage criteria. In this talk, I will describe this approach to automated test case generation and illustrate its utility with a simple example. I will then describe how this approach could be extended to account for different dimensions of human cognitive performance and emerging challenges in human-autonomy interaction.

Dr. Bolton is an Associate Professor of Industrial and Systems Engineering at the University at Buffalo (UB). He obtained his Ph.D. in Systems Engineering from the University of Virginia, Charlottesville, in 2010. Before joining UB, he worked as a Senior Research Associate at NASA’s Ames Research Center and as an Assistant Professor of Industrial Engineering at the University of Illinois at Chicago. Dr. Bolton is an expert on the use of formal methods in human factors engineering and has published widely in this area. He has successfully applied his research to safety-critical applications in aerospace, medicine, defense, and cybersecurity.  He has received funding on projects sponsored by the European Space Agency, NSF, NASA, AHRQ, and DoD.  This includes a Young Investigator Program Award from the Army Research Office. He is an associate editor for the IEEE Transactions on Human Machine Systems and the former Chair of the Human Performance Modeling Technical Group for the Human Factors and Ergonomics Society. He was appointed as a Senior Member of IEEE in 2015 and received the Human Factors and Ergonomics Society’s William C. Howell Young Investigator award in 2018.

In reliability, methods used to estimate failure probability are often limited by the costs associated with model evaluations. Many of these methods, such as multi-fidelity importance sampling (MFIS), rely upon a cheap, surrogate model like a Gaussian process (GP) to quickly generate predictions. The quality of the GP fit, at least in the vicinity of the failure region(s), is instrumental in propping up such estimation strategies. We introduce an entropy-based GP adaptive design that, when paired with MFIS, provides more accurate failure probability estimates and with higher confidence. We show that our greedy data acquisition scheme better identifies multiple failure regions compared to existing contour-finding schemes. We then extend the method to batch selection. Illustrative examples are provided on benchmark data as well as an application to the impact damage simulator of a NASA spacesuit design.

Austin Cole is a statistics PhD candidate at Virginia Tech. He previously taught high school math and statistics courses, and holds a Bachelor’s in Mathematics and Master’s in Secondary Education from the College of William and Mary. Austin has worked with dozens of researchers as a lead collaborator in Virginia Tech’s Statistical Applications and Innovations Group (SAIG). Under the supervision of Dr. Robert Gramacy, Austin has conducted research in the area of computer experiments with focuses on Bayesian optimization, sparse covariance matrices, and importance sampling. He is currently collaborating with researchers at NASA Langley, to evaluate the safety of the next generation of spacesuits.

Human-system interaction is a critical yet often neglected aspect of the system development process. It is mostly commonly incorporated into system performance assessments late in the design process leaving little opportunity for any substantive changes to be made to ensure satisfactory system performance achieved. As a result, workarounds and compromises become a patchwork of “corrections” that end up in the final fielded system. But what if mission outcomes, the work context, and performance expectations can be articulated earlier in the process, thereby influencing the development process throughout? This presentation will discuss how a formative method from the field of cognitive systems engineering, cognitive work analysis, can be leveraged to derive design requirements compatible with traditional systems engineering processes. This method establishes not only requirements from which system designs can be constructed, but also how system performance expectations can be more acutely defined a priori to guide the validation and verification process. Cognitive work analysis methods will be described to highlight how ‘cognitive work’ and ‘information relationship’ requirements can be derived and will be showcased in a case-study application of building a decision support system for future human spaceflight operations. Specifically, a description of the testing campaign employed to verify and validate the fielded system will be provided. In summary, this presentation will cover how system requirements can be established early in the design phase, guide the development of design solutions, and subsequently be used to assess the operational performance of the solutions within the context of the work domain it is intended to support.

Matthew J. Miller is an Exploration Research Engineer within the Astromaterials Research and Exploration Sciences (ARES) division at NASA Johnson Space Center. His work focuses on advancing present-day tools, technologies and techniques to improve future EVA operations by applying cognitive systems engineering principles. He has over seven years of EVA flight operations and NASA analog experience where he has developed and deployed various EVA support systems and concept of operations. He received a B.S. (2012), M.S. (2014) and Ph.D. (2017) in aerospace engineering from the Georgia Institute of Technology.

The spread of COVID-19 across the United States provides an interesting case study in the modeling of spatio-temporal data. In this breakout session we will provide an overview of commonly used spatio-temporal models and demonstrate how Bayesian Inference can be performed using both exact and approximate inferential techniques. Using COVID data, we will demonstrate visualization techniques in R and introduce "off-the-shelf" spatio-temporal models. We will introduce participants to the Integrated Nested LaPlace approximation (INLA) methodology and show how results from this technique compare to using Markov Chain Monte Carlo (MCMC) techniques. Finally, we will demonstrate the short-falls in using "off-the-shelf" models and show how epidemiological motivated partial differential equations can be used to generate spatio-temporal models and discuss inferential issues when we move away from common models.

LTC Nicholas Clark is an Academy Professor at the United States Military Academy where he heads the Center for Data Analysis and Statistics. Nick received his PhD in Statistics from Iowa State University and his research interests include spatio-temporal statistics and Bayesian methodology.

The zoonotic emergence of the coronavirus SARS-CoV-2 at the beginning of 2020 and the subsequent global pandemic of COVID-19 has caused massive disruptions to economies and health care systems, particularly in the United States. Using the results of serology testing, we have developed true prevalence estimates for COVID-19 case counts in the U.S. over time, which allows for more clear estimates of infection and case fatality rates throughout the course of the pandemic. In order to elucidate policy, demographic, weather, and behavioral factors that contribute to or inhibit the spread of COVID-19, IDA compiled panel data sets of empirically derived, publicly available COVID-19 data and analyzed which factors were most highly correlated with increased and decreased spread within U.S. states and counties. These analyses lead to several recommendations for future pandemic response preparedness.

Dr. Emily Heuring received her PhD in Biochemistry, Cellular, and Molecular Biology from the Johns Hopkins University School of Medicine in 2004 on the topic of human immunodeficiency virus and its impact on the central nervous system. Since that time, she has been a Research Staff Member at the Institute for Defense Analyses, supporting operational testing of chemical and biological defense programs. More recently, Dr. Heuring has supported OSD-CAPE on Army and Marine Corps programs and the impact of COVID-19 on the general population and DOD.

Certification by analysis (CBA) involves the supplementation of expensive physical testing with modeling and simulation. In high-risk fields such as defense and aerospace, it is critical that these models accurately represent the real world, and thus they must be verified, validated and provide measures of uncertainty. While machine learning (ML) algorithms such as deep neural networks have seen significant success in low-risk sectors, they are typically opaque, difficult to interpret and often fail to meet these stringent requirements. Recently, a Department of Energy (DOE) report was released on the concept of scientific machine learning (SML) [1] with the aim of generally improving confidence in ML and enabling broader use in the scientific and engineering communities. The report identified three critical attributes that ML algorithms should possess: domain-awareness, interpretability, and robustness. Recent advances in physics-informed neural networks (PINNs) are promising in that they can provide both domain awareness and a degree of interpretability [2, 3, 4] by using governing partial differential equations as constraints during training. In this way, PINNs output physically admissible, albeit deterministic, solutions. Another noteworthy deep learning algorithm is the generative adversarial network (GAN), which can learn probability distributions [5] and provide robustness through uncertainty quantification. A limited number of works have recently demonstrated success by combining these two methods into what is referred to as a physics-informed GAN, or PIGAN [6, 7]. The PIGAN has the ability to produce physically admissible, non-deterministic predictions as well as solve non-deterministic inverse problems, potentially meeting the goals of domain awareness, interpretability, and robustness. This talk will present an introduction to PIGANs as well as an example of current NASA research implementing these networks. REFERENCES [1] Nathan Baker, Frank Alexander, Timo Bremer, Aric Hagberg, Yannis Kevrekidis, Habib Najm, Manish Parashar, Abani Patra, James Sethian, Stefan Wild, Karen Willcox, and Steven Lee. Workshop report on basic research needs for scientific machine learning: Core technologies for artificial intelligence. Technical report, USDOE Office of Science (SC) Washington, DC (United States), 2019. [2] Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. [3] Alexandre Tartakovsky, Carlos Ortiz Marrero, Paris Perdikaris, Guzel Tartakovsky, and David Barajas-Solano. Learning parameters and constitutive relationships with physics informed deep neural networks. arXiv preprint arXiv:1808.03398, 2018. [4] Julia Ling, Andrew Kurzawski, and Jeremy Templeton. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155-166, 2016. [5] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems, pages 2672-2680, 2014. [6] Liu Yang, Dongkun Zhang, and George Karniadakis. Physics-informed generative adversarial networks for stochastic differential equations. arXiv preprint arXiv:1811.02033, 2018. [7] Yibo Yang and Paris Perdikaris. Adversarial uncertainty quantification in physics-informed neural networks. Journal of Computational Physics, 394:136-152, 2019.

Dr. Patrick Leser is a researcher in the Durability, Damage Tolerance, and Reliability Branch (DDTRB) at NASA Langley Research Center (LaRC) in Hampton, VA. After receiving a B.S. in Aerospace Engineering from North Carolina State University (NCSU), he became a NASA civil servant under the Pathways Intern Employment Program in 2014. In 2017, he received his PhD in Aerospace Engineering, also from NCSU. Dr. Leser’s research focuses primarily on uncertainty quantification (UQ), model calibration, and fatigue crack growth in metallic materials. Dr. Leser has applied these interests to various topics including structural health management, digital twin, additive manufacturing, battery health management and various NASA Engineering Safety Center (NESC) assessments, primarily focusing on the fatigue life of composite overwrapped pressure vessels (COPVs).  A primary focus of his work has been the development of computationally-efficient UQ methods that utilize fields such as high performance computing and machine learning.

Informed enterprise and program decision making is central to DoD’s Digital Engineering’s purpose statement. Decision analysis serves as a key mechanism to link the voice of the sponsor/end user with the voice of the engineer and the voice of the budgetary analyst in order to enable a closed loop requirements writing approach that is informed by rigorous assessments of a broad range of system-level alternatives across a thorough set of stakeholder value criteria to include life-cycle costs, schedule, performance, and long term viability. . The decision analytics framework employed by the U.S. Army’s Combat Capabilities Development Command (CCDC) Armaments Center (AC) is underpinned by a state-of-the-art modeling and simulation framework called PRISM (Performance Related and Integrated Suite of Models) developed at CCDC-AC. PRISM was designed in a way to allow performance estimates of a weapon system to evolve as more information and higher fidelity representations of those systems become available. PRISM provides the most up to date performance estimates into the decision analysis framework so that decision makers have the best information available when making complex strategic decisions. This briefing will unpack PRISM and highlight the model design elements that make it the foundation of CCDC-AC’s weapon system architecture and design decision making process.

Mr. Greco is the lead architect and developer of the Performance Related and Integrated Suite of Models (PRISM) modeling and simulation framework. The PRISM tool has been used to support the analysis of many projects locally for CCDC-AC and externally for the larger Army analytical community. Mr. Greco has over 10 years of experience working in force effectiveness analysis with the use of operational and system performance simulations.

The recent emergence of affordable, high-quality augmented-, mixed-, and virtual-reality (AR, MR, VR), technologies presents an opportunity to dramatically change the way users consume and interact with information. It has been shown that these immersive systems can be leveraged to enhance comprehension and accelerate decision-making in situations where data can be linked to spatial information, such as maps or terrain models. Furthermore, when immersive technologies are networked together, they allow for decentralized collaboration and provide perspective-taking not possible with traditional displays. However, enabling this shared space requires novel techniques in intelligent information management and data exchange. In this experiment, we explored a framework for leveraging distributed AI/ML processing to enable clusters of low-power, limited-functionality devices to deliver complex capabilities in aggregate to users distributed across the country collaborating simultaneously in a shared virtual environment. We deployed a motion detecting camera and triggered detection events to send information using a distributed request/reply worker framework to a remotely located YOLO image classification cluster. This work demonstrates the capability for various IoT and IoBT systems to invoke functionality without a priori knowledge of the specific endpoint to use to execute that functionality but by submitting a request based on a desired capability concept (e.g. image classification) with requiring only: 1) the knowledge of the broker location, 2) valid public/private key pair required to authenticate with the broker, and 3) the capability concept UUID and knowledge of request/reply formats used by that concept.

Mark Dennison is a research psychologist with DEVCOM U.S. Army Research Laboratory in the Computational and Information Sciences Directorate, Battlefield Information Systems Branch. He leads a team of government researchers and contractors focused on enabling cross-reality technologies to enhance lethality across domains through information management across echelons. Dr. Dennison graduated with a bachelor’s degree from the University of California at Irvine, and earned his Master’s and Ph.D. degrees from the University of California at Irvine, all in the field of psychology with a specialization in cognitive neuroscience. He is stationed at ARL-West in Playa Vista, CA.

Nearly all land, air, and sea maneuver systems (e.g. vehicles, ships, aircraft, and missiles) are becoming more software-reliant and blending internal communication across both Internet Protocol (IP) and non-IP buses. IP communication is widely understood among the cybersecurity community, whereas expertise and available test tools for non-IP protocols such as Controller Area Network (CAN) and MIL-STD-1553 are not as commonplace. However, a core tenet of operational cybersecurity testing is to asses all potential pathways of information exchange present on the system, to include IP and non-IP. In this presentation, we will introduce a few non-IP protocols (e.g. CAN, MIL-STD-1553) and provide a live demonstration of how to attack a CAN network using malicious message injection. We will also discuss how potential cyber effects on non-IP busses can lead to catastrophic mission effects to the target system.

Peter Mancini works at the Institute for Defense Analyses, supporting the Director, Operational Test and Evaluation (DOT&E) as a Cybersecurity OT&E analyst.

This session will focus on the novel application of a design of experiments approach to optimize a propulsion charge configuration for a U.S. Army artillery round. The interdisciplinary design effort included contributions from subject matter experts in statistics, propulsion charge design, computational physics and experimentation. The process, which we will present in this session, consisted of an initial, low fidelity modeling and simulation study to reduce the parametric space by eliminating inactive variables and reducing the ranges of active variables for the final design. The final design used a multi-tiered approach that consolidated data from multiple sources including low fidelity modeling and simulation, high fidelity modeling and simulation and live test data from firings in a ballistic simulator. Specific challenges of the effort that will be addressed include: integrating data from multiple sources, a highly constrained design space, functional response data, multiple competing design objectives and real-world test constraints. The result of the effort is a final, optimized propulsion charge design that will be fabricated for live gun firing.

Sarah Longo is a data scientist in the US Army CCDC Armaments Center's Systems Analysis Division. She has a background in Chemical and Mechanical Engineering and ten years experience in gun propulsion and armament engineering. Ms. Longo's gun-propulsion expertise has played a part in enabling the successful implementation of Design of Experiments, Empirical Modeling, Data Visualization and Data Mining for mission-critical artillery armament and weapon system design efforts.

This session will focus on the novel application of a design of experiments approach to optimize a propulsion charge configuration for a U.S. Army artillery round. The interdisciplinary design effort included contributions from subject matter experts in statistics, propulsion charge design, computational physics, and experimentation. The process, which we will present in this session, consisted of an initial, low fidelity modeling and simulation study to reduce the parametric space by eliminating inactive variables and reducing the ranges of active variables for the final design. The final design used a multi-tiered approach that consolidated data from multiple sources including low fidelity modeling and simulation, high fidelity modeling and simulation and live test data from firings in a ballistic simulator. Specific challenges of the effort that will be addressed include: integrating data from multiple sources, a highly constrained design space, functional response data, multiple competing design objectives, and real-world test constraints. The result of the effort is a final, optimized propulsion charge design that will be fabricated for live gun firing.

Melissa Jablonski is a statistician at the US Army Combat Capabilities Development Command Armaments Center.  She graduated from Stevens Institute of Technology with a Bachelor’s and Master’s degree in Mechanical Engineering and started her career in the area of finite element analysis.  She now works as a statistical consultant focusing in the areas of Design and Analysis of Computer Experiments (DACE) and Uncertainty Quantification (UQ).  She also acts as a technical expert and consultant in Design of Experiments (DOE), Probabilistic System Optimization, Data Mining/Machine Learning, and other statistical analysis areas for munition and weapon systems.  She is currently pursuing a Master’s degree in Applied Statistics from Pennsylvania State University.

Verification and validation represent important steps for appropriate use of CFD codes and it is presently considered the user’s responsibility to ensure that these steps are completed. Inconsistent definitions and use of these terms in aerospace complicate the effort. For industrial-use CFD codes, there are a number of challenges that can further confound these efforts including varying grid topology, non-linearities in the solution, challenges in isolating individual components, and difficulties in finding validation experiments. In this presentation, a number of these challenges will be reviewed with some specific examples that demonstrate why verification is much more involved and challenging than typically implied in numerical method courses, but remains an important exercise. Some of the challenges associated with validation will also be highlighted using a range of different cases, from canonical flow elements to complete aircraft models. Benchmarking is often used to develop confidence in CFD solutions for engineering purposes, but falls short of validation in the absence of being able to predict bounds on the simulation error. The key considerations in performing benchmarking and validation will be highlighted and some current shortcomings in practice will be presented, leading to recommendations for conducting validation exercises. CFD workshops have considerably improved in their application of these practices, but there continues to be need for additional steps.

Andrew Cary is a technical fellow of the Boeing Company in CFD and is the focal for the BCFD solver. In this capacity, he has a strong focus on supporting users of the code across the Boeing enterprise as well as leading the development team. These responsibilities align with his interests in verification, validation, and uncertainty quantification as an approach to ensure reliable results as well as in algorithm development, CFD-based shape optimization, and unsteady fluid dynamics. Since hiring into the CFD team in 1996, he has led CFD application efforts across a full range of Boeing products as well as working in grid generation methods, flow solver algorithms, post-processing approaches, and process automation. These assignments have given him the opportunity to work with teams around the world, both inside and outside Boeing. Andrew has been an active member of the American Institute of Aeronautics and Astronautics, serving in multiple technical committees, including his present role on the CFD Vision 2030 Integration Committee. Andrew has also been an adjunct professor at Washington University since 1999, teaching graduate classes in CFD and fluid dynamics. Andrew received a Ph.D. (97) in Aerospace Engineering from the University of Michigan and a B.S. (92) and M.S. (97) in Aeronautical and Astronautical Engineering from the University of Illinois Urbana-Champaign.

As advanced technologies and capabilities are enabling machines to engage in tasks that only humans have done previously, new challenges have emerged for the rigorous testing and evaluation (T&E) of human-machine teaming (HMT) concepts. We differentiate the distinction between a HMT and a human using a tool, and new challenges are enumerated: Agents’ mental models are opaque, machine-to-human communications need to be evaluated, and self-tasking and autonomy need to be evaluated. We argue that a focus on mission outcomes cannot fully characterize team performance due to the increased problem space evaluated and that the T&E community needs to develop and refine new metrics for agents of teams and teammate interactions. Our IDA HMT framework outlines major categories for HMT evaluation, emphasizing team metrics and parallelizing agent metrics across humans and machines. Major categories are tied to the literature and proposed as a starting point for additional T&E metric specification for robust evaluation.

Brian is a Research Staff Member at the Institute for Defense Analyses where he applies rigorous statistics and study design to evaluate, test, and report on various programs. Dr. Vickers holds a Ph.D. from the University of Michigan, Ann Arbor where he researched various factors that influence decision making, with a focus on how people allocate their money, time, and other resources.

Physics-based simulation of nondestructive evaluation (NDE) inspection can help to advance the inspectability and reliability of mechanical systems. However, NDE simulations applicable to non-idealized mechanical components often require large compute domains and long run times. This has prompted development of custom NDE simulation software tailored to high performance computing (HPC) hardware. Verification and validation (V&V) is an integral part of developing this software to ensure implementations are robust and applicable to inspection problems, producing tools and simulations suitable for computational NDE research. This presentation addresses factors common to V&V of several elastodynamic simulation codes applicable to ultrasonic NDE. Examples are drawn from in-house simulation software at NASA Langley Research Center, ranging from ensuring reliability in a 1D heterogeneous media wave equation solver to the V&V needs of 3D cluster-parallel elastodynamic software. Factors specific to a research environment are addressed, where individual simulation results can be as relevant as the software product itself. Distinct facets of V&V are discussed including testing to establish software reliability, employing systematic approaches for consistency with fundamental conservation laws, establishing the numerical stability of algorithms, and demonstrating concurrence with empirical data. This talk also addresses V&V practices for small groups of researchers. This includes establishing resources (e.g. time and personnel) for V&V during project planning to mitigate and control the risk of setbacks. Similarly, we identify ways for individual researchers to use V&V during simulation software development itself to both speed up the development process and reduce incurred technical debt.

Erik Frankforter is a research engineer in the Nondestructive Evaluation Sciences branch at NASA Langley Research Center. His research areas include the development of high performance computing nondestructive evaluation simulation software, advancement of inspection mechanics in advanced material systems, and application of physics-based simulation for inspection guidance. In 2017, he obtained a Ph. D. in mechanical engineering from the University of South Carolina, and served as a postdoctoral research scholar at the National Institute of Aerospace.

With the rise of machine learning and artificial intelligence, there has been a huge surge in data-driven approaches to solve computational science and engineering problems. In the context of uncertainty propagation, machine learning is often employed for the construction of efficient surrogate models (i.e., response surfaces) to replace expensive, physics-based simulations. However, relying solely on surrogate models without any recourse to the original high-fidelity simulation will produce biased estimators and can yield unreliable or non-physical results. This talk discusses multi-model Monte Carlo methods that combine predictions from both fast, low-fidelity models with reliable, high-fidelity simulations to enable efficient and accurate uncertainty propagation. For instance, the low-fidelity models could arise from coarsened discretizations in space/time (e.g., Multilevel Monte Carlo - MLMC) or from general data-driven or reduced order models (e.g., Multifidelity Monte Carlo - MFMC; Approximate Control Variates - ACV). Given a fixed computational budget and a collection of models of varying cost/accuracy, the goal of these methods is to optimally allocate and combine samples across the models. The talk will also present a NASA-developed open-source Python library that acts as a general multi-model uncertainty propagation capability. The effectiveness of the discussed methods and Python library is demonstrated on a trajectory simulation application. Here, orders of magnitude computational speedup and accuracy are obtained for predicting the landing location of an umbrella heat shield under significant uncertainties in initial state, atmospheric conditions, etc.

Dr. Geoffrey Bomarito is a Materials Research Engineer at NASA Langley Research Center.  Before joining NASA in 2014, he earned a PhD in Computational Solid Mechanics from Cornell University.  He also holds an MEng from the Massachusetts Institute of Technology and a BS from Cornell University, both in Civil and Environmental Engineering.  Dr. Bomarito's work centers around machine learning and uncertainty quantification as applied to aerospace materials and structures.  His current topics of interest are physics informed machine learning, symbolic regression, additive manufacturing, and trajectory simulation.

Under the Artemis program, the Exploration Extravehicular Mobility Unit (xEMU) spacesuit will ensure the safety of NASA astronauts during the targeted 2024 return to the moon. Efforts are currently underway to finalize and certify the xEMU design. There is a delicate balance between producing a spacesuit that is robust enough to safely withstand potential fall events while still satisfying stringent mass and mobility requirements. The traditional approach of considering worst case-type loading and applying conservative factors of safety (FoS) to account for uncertainties in the analysis was unlikely to meet the narrow design margins. Thus, the xEMU design requirement was modified to include a probability of no impact failure (PnIF) threshold that must be verified through probabilistic analysis. As part of a broader one year effort to help integrate modern uncertainty quantification (UQ) methodology into engineering practice at NASA, the certification of the xEMU spacesuit was selected as the primary case study. The project, led by NASA Langley Research Center (LaRC) under the Engineering Research & Analysis (R&A) Program in 2020, aimed to develop an end-to-end UQ workflow for engineering problems and to help facilitate reliability-based design at NASA. The main components of the UQ workflow included 1) sensitivity analysis to identify the most influential model parameters, 2) model calibration to quantified model parameter uncertainties using experimental data, and 3) uncertainty propagation for producing probabilistic model predictions and estimating reliability. Particular emphasis was placed on overcoming the common practical barrier of prohibitive computational expense associated with probabilistic analysis by leveraging state-of-the-art UQ methods and high performance computing (HPC). In lieu of mature computational models and test data for the xEMU at the time of the R&A Program, the UQ workflow for estimating PnIF was demonstrated using existing models and data from the previous generation of spacesuits (the Z-2). However, the lessons learned and capabilities developed in the process of the R&A are directly transferable to the ongoing xEMU certification effort and are currently being integrated in 2021. This talk provides an overview of the goals of and findings under NASA's UQ R&A project, focusing on the spacesuit certification case study. The steps of the UQ workflow applied to the Z-2 spacesuit using the available finite element method (FEM) models and impact test data will be detailed. The ability to quantify uncertainty in the most influential subset of FEM model input parameters and then propagate that uncertainty to estimates of PnIF is demonstrated. Since the FEM model of the full Z-2 assembly took nearly 1 day to execute just once, the advanced UQ methods and HPC utilization required to make the probabilistic analysis tractable are discussed. Finally, the lessons learned from conducting the case study are provided along with planned ongoing/future work for the xEMU certification in 2021.

Dr. James Warner joined NASA Langley Research Center (LaRC) in 2014 as a Research Computer Engineer after receiving his PhD in Computational Solid Mechanics from Cornell University. Previously, he received his B.S. in Mechanical Engineering from SUNY Binghamton University and held temporary research positions at the National Institute of Standards and Technology and Duke University. Dr. Warner is a member of the Durability, Damage Tolerance, and Reliability Branch (DDTRB) at LaRC, where he focuses on developing computationally-efficient approaches for uncertainty quantification for a range of applications including structural health management and space radiation shielding design. His other research interests include high performance computing, inverse methods, and topology optimization.

As statisticians, we are always working on new ways to explain statistical methodologies to non-statisticians. It is in this realm that we never underestimate the value of graphics and patience! In this presentation, we present a case study that involves stress rupture data where a Weibull regression is needed to estimate the parameters. The context of the case study results from a multi-stage project supported by NASA’s Engineering Safety Center (NESC) where the objective was to assess the safety of composite overwrapped pressure vessels (COPVs). The analytical team was tasked with devising a test plan to model stress rupture failure risk in carbon fiber strands that encase the COPVs with the goal of understanding the reliability of the strands at use conditions for the expected mission life. While analyzing the data, we found that the proper analysis contradicts accepted theories about the stress rupture phenomena. In this talk, we will introduce ways to graph the stress rupture data to better explain the proper analysis and also explore assumptions.

Anne Ryan Driscoll is an Associate Collegiate Professor in the Department of Statistics at Virginia Tech. She received her PhD in Statistics from Virginia Tech. Her research interests include statistical process control, design of experiments, and statistics education. She is a member of ASQ and ASA.

Fragmentation analysis is a critical piece of the live fire test and evaluation (LFT&E) of lethality and vulnerability aspects of warheads. But the traditional methods for data collection are expensive and laborious. New optical tracking technology is promising to increase the fidelity of fragmentation data, and decrease the time and costs associated with data collection. However, the new data will be complex, three dimensional 'fragmentation clouds', possibly with a time component as well. This raises questions about how testers can effectively summarize spatial data to draw conclusions for sponsors. In this briefing, we will discuss the Bayesian spatial models that are fast and effective for characterizing the patterns in fragmentation data, along with several exploratory data analysis techniques that help us make sense of the data. Our analytic goals are to - Produce simple statistics and visuals that help the live fire analyst compare and contrast warhead fragmentations; - Characterize important performance attributes or confirm design/spec compliance; and - Provide data methods that ensure higher fidelity data collection translates to higher fidelity modeling and simulation down the line. This talk is a version of the first-step feasibility study IDA is taking – hopefully much more to come as we continue to work on this important topic.

Dr. John Haman is a statistician at the Institute for Defense Analyses, where he develops methods and tools for analyzing test data. He has worked with a variety of Army, Navy, and Air Force systems, including counter-UAS and electronic warfare systems. Currently, John is providing technical support on operational testing to the Joint Artificial Intelligence Center.

Biomechanical experiments investigating the failure modes of biological tissue require a significant investment of time and money due to the complexity of procuring, preparing, and testing tissue. Furthermore, the potentially destructive nature of these tests makes repeated testing infeasible. This leads to experiments with notably small sample sizes in light of the high variance common to biological material. When the goal is to estimate parameters for an analytic artifact such as an injury risk curve (IRC), which relates an input quantity to a probability of injury, small sample sizes result in undesirable uncertainty. One way to ameliorate this effect is through a Bayesian approach, incorporating expert opinion and previous experimental data into a prior distribution. This has the advantage of leveraging the information contained in expert opinion and related experimental data to obtain faster convergence to an appropriate parameter estimation with a desired certainty threshold. We explore several ways of implementing Bayesian methods in a biomechanical setting, including permutations on the use of expert knowledge and prior experimental data. Specifically, we begin with a set of experimental data from which we generate a reference IRC. We then elicit expert predictions of the 10th and 90th quantiles of injury, and use them to formulate both uniform and normal prior distributions. We also generate priors from qualitatively similar experimental data, both directly on the IRC parameters and on the injury quantiles, and explore the use of weighting schemes to assign more influence to better datasets. By adjusting the standard deviation and shifting the mean, we can create priors of variable quality. Using a subset of the experimental data in conjunction with our derived priors, we then re-fit the IRC and compare it to the reference curve. For all methods we will measure the certainty, speed of convergence, and accuracy relative to the reference IRC, with the aim of recommending a best practices approach for the application of Bayesian methods in this setting. Ultimately an optimized approach for handling small samples sizes with Bayesian methods has the potential to increase the information content of individual biomechanical experiments by integrating them into the context of expert knowledge and prior experimentation.

Amanda French is a data scientist at Johns Hopkins University Applied Physics Laboratory. She obtained her PhD in mathematics from UNC Chapel Hill and went on to perform data science for a variety of government agencies, including the Department of State, Military Health System, and Department of Defense. Her expertise includes statistics, machine learning, and experimental design.

The Scientific Test and Analysis Techniques (STAT) process is designed to provide structure for a test team to progress from a requirement to decision quality information. The four phases of the STAT process are Plan, Design, Execute, and Analyze. Within the Test and Evaluation (T&E) community we tend to focus on the quantifiable metrics and the hard science of testing, which are the Design and the Analyze phases. At the STAT Center of Excellence (COE) we have emphasized an increased focus on the planning phase and in this presentation we focus on the elements necessary for a comprehensive planning session. In order to efficiently and effectively test a system it is vital that the test team understand the requirements, the System Under Test (SUT) to include any subsystems that will be tested, and the test facility. To accomplish this the right team members with the necessary knowledge must be in the room and prepared to present their information and have an educated discussion to arrive at a comprehensive agreement about the desired end stated of the test. Our recommendations for the initial planning meeting are based on a thorough study of the STAT process and on lessons learned from actual planning meetings.

Mr. Ramert is a graduate of the US Naval Academy and the Naval Postgraduate School and a 20 year veteran of the Marine Corps. During his career in the Marines he served tours in operational air and ground units as well as academic assignments. He joined the Scientific Test and Analysis Techniques (STAT) Center of Excellence (COE) in 2016 where he works with major acquisition programs with the Department of Defense to apply rigor and efficiency to their test and evaluation methodology through the application of the STAT process.

Receiver Operating Characteristic curves (ROC curves) are often used to assess the performance of detection and classification systems. ROC curves can have unexpected subtleties that make them difficult to interpret. For example, the Strategic Environmental Research and Development Program and the Environmental Security Technology Certification Program (SERDP/ESTCP) is sponsoring the development of novel systems for the detection and classification of Unexploded Ordnance (UXO) in underwater environments. SERDP is also sponsoring underwater testbeds to demonstrate the performance of these novel systems. The Institute for Defense Analyses (IDA) is currently designing and implementing the scoring process for these underwater demonstrations that addresses the subtleties of ROC curve interpretation. This presentation will provide an overview of the main considerations for ROC curve parameter selection when scoring underwater demonstrations for UXO detection and classification.

Jacob Bartel is a Research Associate at the Institute for Defense Analyses (IDA). His research focuses on computational modeling and verification and validation (V&V), primarily in the field of nuclear engineering. Recently, he has worked with SERDP/ESTCP to develop and implement scoring processes for testing underwater UXO detection and classification systems. Prior to joining IDA, his graduate research focused on the development of novel algorithms to model fuel burnup in nuclear reactors. Jacob earned his master’s degree in Nuclear Engineering and his bachelor’s degree in Physics from Virginia Tech.

The Collaborative Human AI Red Teaming (CHART) project is an effort to develop an AI Collaborator which can help human test engineers quickly develop test plans for AI systems. CHART was built around processes developed for cybersecurity red-teaming. Using a goal-focused approach based upon iteratively testing and attacking a system then updating the testers model to discover novel failure modes not discovered by traditional T&E processes. Red teaming is traditionally a time intensive process which requires subject matter expert to study the system they are testing for months in order to develop attack strategies. CHART will accelerate this process by guiding the user through the process of diagraming the AI system under test and drawing upon a pre-established body of knowledge to identify the most probably vulnerabilities. CHART was provided internal seedling funds during FY20 to perform a feasibility study of the technology. During this period the team developed a taxonomy of AI vulnerabilities and an ontology of AI irruptions. Irruptions being events (either caused by a malicious actor or unintended consequences) which trigger the vulnerability and lead to an undesirable result. Using this taxonomy we built a threat modeling tool that allows users to diagram their AI system and identifies all the possible irruptions which could occur. This initial demonstration was based around two scenarios. An smartphone-based ECG system for telemedicine and a UAV trained reinforcement learning to avoid mid-air collisions. In this talk we will first discuss how Red Teaming differs from adversarial machine learning and traditional testing and evaluation. Next, we will provide an overview of how industry is approaching the problem of AI Red Teaming and how our approach differs. Finally, we will discuss how we developed our taxonomy of AI vulnerabilities, how to apply goal-focused testing to AI systems, and our strategy for automatically generating test plans.

Dr. Galen Mullins is a senior staff scientist in the Robotics Group of the Intelligent Systems branch at the Johns Hopkins Applied Physics Laboratory. His research is focused on developing intelligent testing techniques and adversarial tools for finding the vulnerabilities of AI systems. His recent project work has included the development of new imitation learning frameworks for modeling the behavior of autonomous vehicles, creating algorithms for generating adversarial environments, and developing red teaming procedures for AI systems.  He is the secretary for the IEEE/RAS working group on Guidelines for Verification of Autonomous Systems and teaches the Introduction to Robotics course at the Johns Hopkins Engineering for Professionals program. Dr. Galen Mullins received his B.S degrees in Mechanical Engineering and Mathematics respectively from Carnegie Mellon University in 2007 and joined APL the same year. Since then he earned his M.S. in Applied Physics from Johns Hopkins University in 2010, and his Ph.D in Mechanical Engineering from the University of Maryland in 2018. His doctoral research was focused on developing active learning algorithms for generating adversarial scenarios for autonomous vehicles.

Introduction: Information Operations (IO) are a key component of our adversaries’ strategy to undermine U.S. military power without escalating to more traditional (and more easily identifiable) military strikes. Social media activity is one method of IO. In 2017 and 2018, Twitter suspended thousands of accounts likely belonging to the Kremlin-backed Internet Research Agency (IRA). Clemson University archived a large subset of these tweets (2.9M tweets posted by over 2800 IRA accounts), tagged each tweet with metadata (date, time, language, supposed geographical region, number of followers, etc.), and published this dataset on the polling aggregation website FiveThirtyEight. Methods: Machine Learning researchers at the Institute for Defense Analyses (IDA) downloaded Clemson’s dataset from FiveThirtyEight and analyzed both the content of the IRA tweets and their accompanying metadata. Using unsupervised learning techniques (Latent Dirichlet Allocation), IDA researchers mapped out how the patterns in the IRA’s tweet topics evolved over time. Results: Results showed that the IRA started tweeting in/before February 2012, but ramped up significantly in May/June 2015. Most tweets were in English, and most likely targeted the U.S. The IRA created new accounts after the first Twitter suspension in November 2017, with each new account quickly establishing an audience. Between at least January 2015 and October 2017, the IRA’s English tweet topics evolved over time, becoming tighter, more specific, more negative, and more polarizing, with the final pattern emerging in late 2015. Discussion: The United States government must expect that our adversaries' social media activity will continue to evolve over time. Efficient processing pipelines are needed for semi-automated analyses of time-evolving social media activity.

Emily Parrish is a Research Associate in the Science and Technology Division at the Institute for Defense Analyses (IDA).  Her research focuses at IDA range from model verification and validation, countermine system developmental testing, and using natural language processing (NLP) to interpret collections of data of interest to DoD sponsors.  Emily graduated from the College of William and Mary in 2015 with a B.S. in chemistry and is currently earning her M.S. in data analytics at The George Washington University, with a focus on machine learning and NLP.

Dr. Peter Parker is Team Lead for Advanced Measurement Systems at the National Aeronautics and Space Administration’s Langley Research Center in Hampton, Virginia. He serves an Agency-wide statistical expert across all of NASA’s mission directorates of Exploration, Aeronautics, and Science to infuse statistical thinking, engineering, and methods. His expertise is in collaboratively integrating research objectives, measurement sciences, modeling and simulation, and test design to produce actionable knowledge that supports rigorous decision-making for aerospace research and development. After eight years in private industry, Dr. Parker joined Langley Research Center in 1997. He holds a B.S. in Mechanical Engineering from Old Dominion University, a M.S. in Applied Physics and Computer Science from Christopher Newport University, and a M.S. and Ph.D. in Statistics from Virginia Tech. He is a licensed Professional Engineer in the Commonwealth of Virginia. Dr. Parker is a senior member of the American Institute of Aeronautics and Astronautics, American Society for Quality, and American Statistical Association. He currently serves as Chair-Elect of the International Statistical Engineering Association. Dr. Parker is the past-Chair of the American Society for Quality’s Publication Management Board and Editor Emeritus of the journal Quality Engineering.

Dr. Alex Varbanov is a Principal Scientist at Procter & Gamble. He was born in Bulgaria. He graduated with Ph.D. from the School of Statistics, University of Minnesota in 1999. Dr. Varbanov has been with P&G R&D for 21 years and provides statistical support for variety of company business units (e.g., Fabric & Home Care) and brands (e.g., Tide, Swiffer). He performs experimental design, statistical analysis, and advanced modeling for many different areas including genomics, consumer research, and product claims. He is currently developing scent character end-to-end models for perfume optimization in P&G consumer products.

Problems faced in defense and aerospace often go well beyond textbook problems presented in academic settings. Textbook problems are typically very well defined, and can be solved through application of one "correct" tool. Conversely, many defense and aerospace problems are ill-defined, at least initially, and require an overall strategy to attack, one involving multiple tools and often multiple disciplines. Statistical engineering is an approach recently developed for addressing large, complex, unstructured problems, particularly those for which data can be effectively utilized. This session will present a brief overview of statistical engineering, and how it can be applied to engineer solutions to complex problems. Following this introduction, two case studies of statistical engineering will be presented, to illustrate the concepts.

Dr. Roger W. Hoerl is the Brate-Peschel Associate Professor of Statistics at Union College in Schenectady, NY. Previously, he led the Applied Statistics Lab at GE Global Research. While at GE, Dr. Hoerl led a team of statisticians, applied mathematicians, and computational financial analysts who worked on some of GE’s most challenging research problems, such as developing personalized medicine protocols, enhancing the reliability of aircraft engines, and management of risk for a half-trillion dollar portfolio. Dr. Hoerl has been named a Fellow of the American Statistical Association and the American Society for Quality, and has been elected to the International Statistical Institute and the International Academy for Quality. He has received the Brumbaugh and Hunter Awards, as well as the Shewhart Medal, from the American Society for Quality, and the Founders Award and Deming Lectureship Award from the American Statistical Association. While at GE Global Research, he received the Coolidge Fellowship, honoring one scientist a year from among the four global GE Research and Development sites for lifetime technical achievement. His book with Ron Snee, Statistical Thinking: Improving Business Performance, now in its 3rd edition, was called “the most practical introductory statistics textbook ever published in a business context” by the journal Technometrics.

Problems faced in defense and aerospace often go well beyond textbook problems presented in academic settings. Textbook problems are typically very well defined, and can be solved through application of one "correct" tool. Conversely, many defense and aerospace problems are ill-defined, at least initially, and require an overall strategy to attack, one involving multiple tools and often multiple disciplines. Statistical engineering is an approach recently developed for addressing large, complex, unstructured problems, particularly those for which data can be effectively utilized. This session will present a brief overview of statistical engineering, and how it can be applied to engineer solutions to complex problems. Following this introduction, two case studies of statistical engineering will be presented, to illustrate the concepts.

Analysis-based means of compliance for airplane and engine certification, commonly known as “Certification by Analysis” (CbA), provides a strong motivation for the development and maturation of current and future flight and engine modeling technology. The most obvious benefit of CbA is streamlined product certification testing programs at lower cost while maintaining equivalent levels of safety. The current state of technologies and processes for analysis is not sufficient to adequately address most aspects of CbA today, and concerted efforts to drastically improve analysis capability are required to fully bring the benefits of CbA to fruition. While the short-term cost and schedule benefits of reduced flight and engine testing are clearly visible, the fidelity of analysis capability required to realize CbA across a much larger percentage of product certification is not yet sufficient.  Higher-fidelity analysis can help reduce the product development cycle and avoid costly and unpredictable performance and operability surprises that sometimes happen late in the development cycle. Perhaps the greatest long-term value afforded by CbA is the potential to accelerate the introduction of more aerodynamically and environmentally efficient products to market, benefitting not just manufacturers, but also airlines, passengers, and the environment. A far-reaching vision for CbA has been constructed to offer guidance in developing lofty yet realizable expectations regarding technology development and maturity through stakeholder involvement. This vision is composed of the following four elements: The ability to numerically simulate the integrated system performance and response of full-scale airplane and engine configurations in an accurate, robust, and computationally efficient manner. The development of quantified flight and engine modeling uncertainties to establish appropriate confidence in the use of numerical analysis for certification. The rigorous validation of flight and engine modeling capabilities against full-scale data from critical airplane and engine testing. The use of flight and engine modeling to enable Certification by Simulation. Key technical challenges include the ability to accurately predict airplane and engine performance for a single discipline, the robust and efficient integration of multiple disciplines, and the appropriate modeling of system-level assessment. Current modeling methods lack the capability to adequately model conditions that exist at the edges of the operating envelope where the majority of certification testing generally takes place.  Additionally, large-scale engine or airplane multidisciplinary integration has not matured to the level where it can be reliably used to efficiently model the intricate interactions that exist in current or future aerospace products. Logistical concerns center primarily on the future High Performance Computing capability needed to perform the large number of computationally intensive simulations needed for CbA. Complex, time-dependent, multidisciplinary analyses will require a computing capacity increase several orders of magnitude greater than is currently available. Developing methods to ensure credible simulation results is critically important for regulatory acceptance of CbA. Confidence in analysis methodology and solutions is examined so that application validation cases can be properly identified. Other means of measuring confidence such as uncertainty quantification and “validation-domain” approaches may increase the credibility and trust in the predictions. Certification by Analysis is a challenging long-term endeavor that will motivate many areas of simulation technology development, while driving the potential to decrease cost, improve safety, and improve airplane and engine efficiency. Requirements to satisfy certification regulations provide a measurable definition for the types of analytical capabilities required for success. There is general optimism that CbA is a goal that can be achieved, and that a significant amount of flight testing can be reduced in the next few decades.

For the past 20 years, Timothy Mauery has been involved in the development of low-speed CFD design processes. In this capacity, he has had the opportunity to interact with users and provide CFD support and training throughout the product development cycle. Prior to moving to the Commercial Airplanes division of The Boeing Company, he worked at the Lockheed Martin Aircraft Center, providing aerodynamic liaison support on a variety of military modification and upgrade programs. At Boeing, he has had the opportunity to support both future products as well as existing programs with CFD analysis and wind tunnel testing. Over the past ten years, he has been closely involved in the development and evaluation of analysis-based certification processes for commercial transport vehicles, for both derivative programs as well as new airplanes. Most recently he was the principal investigator on a NASA research announcement for developing requirements for airplane certification by analysis. Timothy received his bachelor's degree from Brigham Young University, and his master's degree from The George Washington University, where he was also a research assistant at NASA-Langley.

Existing software reliability growth models are limited to parametric models that characterize the number of defects detected as a function of testing time or the number of vulnerabilities discovered with security testing. However, the amount and types of testing effort applied are rarely considered. This lack of detail regarding specific testing activities limits the application of software reliability growth models to general inferences such as the additional amount of testing required to achieve a desired failure intensity, mean time to failure, or reliability (period of failure free operation). This presentation provides an overview of an open source software reliability tool implementing covariate software reliability models [1] to aid DoD organizations and their contractors who desire to quantitatively measure and predict the reliability and security improvement of software. Unlike traditional software reliability growth models, the models implemented in the tool can accept multiple discrete time series corresponding to the amount of each type of test activity performed as well as dynamic metrics computed in each interval. When applied in the context of software failure or vulnerability discovery data, the parameters of each activity can be interpreted as the effectiveness of that activity to expose reliability defects or security vulnerabilities. Thus, these enhanced models provide the structure to assess existing and emerging techniques in an objective framework that promotes thorough testing and process improvement, motivating the collection of relevant metrics and precise measurements of the time spent performing various testing activities. References [1] Vidhyashree Nagaraju, Chathuri Jayasinghe, Lance Fiondella, Optimal test activity allocation for covariate software reliability and security models, Journal of Systems and Software, Volume 168, 2020, 110643.

Jacob Aubertine is pursuing a MS degree in the Department of Electrical and Computer Engineering at the University of Massachusetts Dartmouth, where he also received his BS (2020) in Computer Engineering. His research interests include software reliability, performance engineering, and statistical modeling.

Equipment Failure Reports (EFRs) describe equipment failures and the steps taken as a result of these failures. EFRs contain both structured and unstructured data. Currently, analysts manually read through EFRs to understand failure modes and make recommendations to reduce future failures. This is a tedious process where important trends and information can get lost. This motivated the creation of an interactive dashboard that extracts relevant information from the unstructured (i.e. free-form text) data and combines it with structured data like failure date, corrective action and part number. The dashboard is an RShiny application that utilizes numerous text mining and visualization packages, including tm, plotly, edgebundler, and topicmodels. It allows the end-user to filter to the EFRs that they care about and visualize meta-data, such as geographic region where the failure occurred, over time allowing previously unknown trends to be seen. The dashboard also applies topic modeling to the unstructured data to identify key themes. Analysts are now able to quickly identify frequent failure modes and look at time and region-based trends in these common equipment failures.

Robert Molloy is a data scientist for the Johns Hopkins University Applied Physic Laboratory's Systems Analysis Group, where he supports a variety of projects including text mining on unstructured text data, applying machine learning techniques to text and signal data, and implementing and modifying existing natural language models. He graduated from the University of Maryland, College Park in May 2020 with a dual degree in computer science and mathematics with a concentration in statistics.

When developing a system, it is important to consider system performance from a user perspective. This can be done through operational testing---assessing the ability of representative users to satisfactorily accomplish tasks or missions with the system in operationally-representative environments. This process can be expensive and time-consuming, but is critical for evaluating a system. We show how an existing design of experiments (DOE) process for operational testing can be leveraged to construct a Bayesian adaptive design. This method, nested within the larger design created by the DOE process, allows interim analyses using predictive probabilities to stop testing early for success or futility. Furthermore, operational environments with varying probabilities of encountering are directly used in product evaluation. Representative simulations demonstrate how these interim analyses can be used in an operational test setting, and reductions in necessary test events are shown. The method allows for using either weakly informative priors when data from previous testing is not available, or for priors built using developmental testing data when it is available. The proposed method for creating priors using developmental testing data allows for more flexibility in which data can be incorporated into analysis than the current process does, and demonstrates that it is possible to get more precise parameter estimates. This method will allow future testing to be conducted in less time and at less expense, on average, without compromising the ability of the existing process to verify the system meets the user's needs.

Victoria R.C. Sieck is a PhD Candidate in Statistics at the University of New Mexico. She is also an Operations Research Analyst in the US Air Force (USAF), with experiences in the USAF testing community as a weapons and tactics analyst and an operational test analyst. Her research interests include design of experiments and improving operational testing through the use of Bayesian methods.

Cybersecurity Metrics and Quantification is a fundamental but notoriously hard problem. It is one of the pillars underlying the emerging Science of Cybersecurity. In this talk, I will describe a number of cybersecurity metrics quantification research problems that are encountered in evaluating the effectiveness of a range of cyber defense tools. I will review the research results we have obtained over the past years. I will also discuss future research directions, including the ones that are undertaken in my research group.

Shouhuai Xu is the Gallogly Chair Professor in the Department of Computer Science, University of Colorado Colorado Springs (UCCS). Prior to joining UCCS, he was with the Department of Computer Science, University of Texas at San Antonio. He pioneered a systematic approach, dubbed Cybersecurity Dynamics, to modeling and quantifying cybersecurity from a holistic perspective. This approach has three orthogonal research thrusts: metrics (for quantifying security, resilience and trustworthiness/uncertainty, to which this talk belongs), cybersecurity data analytics, and cybersecurity first-principle modeling (for seeking cybersecurity laws). His research has won a number of awards, including the 2019 worldwide adversarial malware classification challenge organized by the MIT Lincoln Lab. His research has been funded by AFOSR, AFRL, ARL, ARO, DOE, NSF and ONR. He co-initiated the International Conference on Science of Cyber Security (SciSec) and is serving as its Steering Committee Chair. He has served as Program Committee co-chair for a number of international conferences and as Program Committee member for numerous international conferences.  He is/was an Associate Editor of IEEE Transactions on Dependable and Secure Computing (IEEE TDSC), IEEE Transactions on Information Forensics and Security (IEEE T-IFS), and IEEE Transactions on Network Science and Engineering (IEEE TNSE). More information about his research can be found at https://xu-lab.org.

Development of new technology always incorporates model testing. This is certainly true for hypersonics, where flight tests are expensive and testing of component- and system-level models has significantly advanced the field. Unfortunately, model tests are often limited in scope, being only approximations of reality and typically only partially covering the range of potential realistic conditions. In this talk, we focus on the problem of real-time detection of the shock train leading edge in high-speed air-breathing engines, such as dual-mode scramjets. Detecting and controlling the shock train leading edge is important to the performance and stability of such engines, and a problem that has seen significant model testing on the ground and some flight testing. Often, methods developed for shock train detection are specific to the model used. Thus, they may not generalize well when tested in another facility or in flight as they typically require a significant amount of prior characterization of the model and flow regime. A successful method for shock train detection needs to be robust to changes in features like isolator geometry, inlet and combustor states, flow regimes, and available sensors. Such data can be difficult or impossible to obtain if the isolator operating regime is large. To this end, we propose the an approach for real-time detection of the isolator shock train. Our approach uses real-time pressure measurements to adaptively estimate the shock train position in a data-driven manner. We show that the method works well across different isolator models, placement of pressure transducers, and flow regimes. We believe that a data-driven approach is the way forward for bridging the gap between testing and reality, saving development time and money.

Greg is an interdisciplinary researcher that builds scientific tools. He is trained as a statistician, mathematician and computer scientist. Currently he work on a diverse set of problems in biology, physics, and engineering.

In design of experiments, setting exact replicates of factor settings enables estimation of pure-error; a model-independent estimate of experimental error useful in communicating inherent system noise and testing of model lack-of-fit. Often in practice, the factor levels for replicates are precisely measured rather than precisely set, resulting in near-replicates. This can result in inflated estimates of pure-error due to uncompensated set-point variation. In this article, we review previous strategies for estimating pure-error from near-replicates and propose a simple alternative. We derive key analytical properties and investigate them via simulation. Finally, we illustrate the new approach with an application.

The Advanced Joint Effectiveness Model (AJEM) is a joint forces model developed by the U.S. Army that is used in vulnerability and lethality (V/L) predictions for threat/target interactions. This complex model primarily generates a probability response for various components, scenarios, loss of capabilities, or summary conditions. Sensitivity analysis (SA) and uncertainty quantification (UQ), referred to jointly as SA/UQ, are disciplines that provide the working space for how model estimates changes with respect to changes in input variables. A comparative measure that will be used to characterize the effect of an input change on the predicted outcome was developed and is reviewed and illustrated in this presentation. This measure provides a practical context that stakeholders can better understand and utilize. We show graphical and tabular results using this measure.

Craig Andres is a Mathematical Statistician at the recently formed DEVCOM Data & Analysis Center in the Materiel M&S Branch working primarily on the uncertainty quantification, as well as the verification and validation, of the AJEM vulnerability model.  He is currently on developmental assignment with the Capabilities Projection Team.  He has a master's degrees in Applied Statistics from Oakland University and a master's degree in Mathematics from Western Michigan University.

Generating aerodynamic coefficients can be computationally expensive, especially for the viscous CFD solvers in which multiple complex models are iteratively solved. When filling large design spaces, utilizing only a high accuracy viscous CFD solver can be infeasible. We apply state-of-the-art methods for design and analysis of computer experiments to efficiently develop an emulator for high-fidelity simulations. First, we apply a cokriging model to leverage information from fast low-fidelity simulations to improve predictions with more expensive high-fidelity simulations. Combining space-filling designs with a Gaussian process model-based sequential sampling criterion allows us to efficiently generate sample points and limit the number of costly simulations needed to achieve the desired model accuracy. We demonstrate the effectiveness of these methods with an aerodynamic simulation study using a conic shape geometry. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. Release Number: LLNL-ABS-818163

Structural dynamic programming models are a powerful tool to help guide policy under uncertainty. By creating a mathematical representation of the intertemporal optimization problem of interest, these models can answer questions that static models cannot address. Applications can be found from military personnel policy (how does future compensation affect retention now?) to inventory management (how many aircraft are needed to meet readiness objectives?). Recent advances in statistical methods and computational algorithms allow us to develop dynamic programming models of complex real-world problems that were previously too difficult to solve.

Mikhail earned his PhD in Economics from Johns Hopkins University in 2017 and recently joined the Strategy, Forces, and Resources Division at the Institute for Defense Analyses after spending several years at CNA.  He specializes in structural and nonparametric econometrics, computational statistics, and machine learning, and his research has focused on questions related to retention and other personnel related decisions for the DOD.

Digital anthropometry obtained from 3D body scanners has already revolutionized the clothing and fitness industries. Within seconds, these scanners collect hundreds of anthropometric measurements which are used by tailors to customize an article of clothing or by fitness trainers to track their client’s progress towards a goal. Three-dimensional body scanners have also been used in military applications, such as predicting injuries at Army basic training and checking a solder’s compliance with body composition standards. In response this increased demand, several 3D body scanners have become commercially available, each with a proprietary algorithm for measuring specific body parts. Individual scanners may suffice to collect measurements from a small population; however, they are not practical for use in creating large data sets necessary to train artificial intelligence (AI) or machine learning algorithms. This study fills the gap between these two applications by correlating body circumferences taken from a small population (n = 109) on three different body scanners and creating a standard scale for pooling data from the different scanners into one large AI ready data set. This data set is then leveraged in a separate application to understand the relationship between body shape and performance on the Army Combat Fitness Test (ACFT).

Nicholas (Nick) Ashby is a fourth-year cadet at the United States Military Academy. He was born in California’s Central Coast and grew up in Charlottesville, VA. At West Point he is pursuing a B.S. in Applied Statistics and Data Science and will commission as an Army Aviation officer in May. In his free time Nick works as a student manager and data analyst for Army’s NCAA Division 1 baseball team and he enjoys playing golf as well. His research, under advisor Dr. Diana M. Thomas, has focused on body shape, body composition, and performance on the U.S. Army Combat Fitness Test (ACFT).

A Human-Machine Team (HMT) is a group of agents consisting of at least one human and at least one machine, all functioning collaboratively towards one or more common objectives. As industry and defense find more helpful, creative, and difficult applications of AI-driven technology, the need to effectively and accurately model, simulate, test, and evaluate HMTs will continue to grow and become even more essential. Going along with that growing need, new methods are required to evaluate whether a human-machine team is performing effectively as a team in testing and evaluation scenarios. You cannot predict team performance from knowledge of the individual team agents, alone; interaction between the humans and machines – and interaction between team agents, in general – increases the problem space and adds a measure of unpredictability. Collective team or group performance, in turn, depends heavily on how a team is structured and organized, as well as the mechanisms, paths, and substructures through which the agents in the team interact with one another – i.e. the team’s topology. With the tools and metrics for measuring team structure and interaction becoming more highly developed in recent years, we will propose and discuss a practical, topological HMT modeling framework that not only takes into account but is actually built around the team’s topological characteristics, while still utilizing the individual human and machine performance measures.

Dr. Wilkins received his Ph.D. in Mathematics from the University of Tennessee, and then spent several years as a postdoctoral fellow and professor in academia before moving into the realm of defense research.  After three years at the US Army Research Office (part of the US Army Research Lab) managing the Mathematical Analysis and Complex Systems Program, he came to IDA as a research staff member in January 2020.  His mathematical background is in applied topological and geometric analysis, particularly the application of topology and geometry to the solution of problems in networking and optimal transport.  His current project work at IDA includes team modeling and simulation, sequential testing of chemical agent detectors, and helicopter pilot guidance and firing interfaces.

Heather Wojton is Director of Data Strategy and Chief Data Officer at IDA, a role she assumed in 2021. In this position, Heather provides strategic leadership, project management, and direction for the corporation’s data strategy. She is responsible for enhancing IDA’s ability to efficiently and effectively accomplish research and business operations by assessing and evolving data systems, data management infrastructure, and data-related practices.

Heather joined IDA in 2015 as a researcher in the Operational Evaluation Division of IDA’s Systems and Analyses Center. She is an expert in quantitative research methods, including test design and program evaluation. She held numerous research and leadership roles before being named an assistant director in the Operational Evaluation Division.

As a researcher at IDA, Heather led IDA’s test science research program that facilitates data-driven decision-making within the Department of Defense (DOD) by advancing statistical, behavioral, and data science methodologies and applying them to the evaluation of defense acquisition programs. Heather’s other accomplishments include advancing methods for test design, modeling and simulation validation, data management and curation, and artificial intelligence testing. In this role, she worked closely with academic and DOD partners to adapt existing test design and evaluation methods for DoD use and develop novel methods where gaps persist.

Heather has a doctorate in experimental psychology from the University of Toledo and a bachelor’s degree in research psychology from Marietta College, where she was a member of the McDonough International Leadership Program. She is a graduate of the George Washington University National Security Studies Senior Management Program and the Maxwell School National Security Management Course at Syracuse University.

Boolean responses are common for both tangible and simulation experiments. Well known approaches to fit models to Boolean responses include ordinary regression with normal approximations or variance stabilizing transforms, and logistic regression. Less well known is kernel regression. This session will present properties of kernel regression, its application to Bernoulli trial experiments, and other lessons learned from using kernel regression in the wild. Kernel regression is a non-parametric method. This requires modifications to many analyses, such as the required sample size. Unlike ordinary regression, the experiment design and model solution interact with each other. Consequently, the number of experiment samples for a desired modeling accuracy depends on the true state of nature. There has been trend in increasingly large simulation sample sizes as computing horsepower has grown. With kernel regression there is a point of diminishing return on sample sizes. That is, an experiment is better off with more data sites once a sufficient sample size is reached. Confidence interval accuracy is also dependent on the true state of nature. Parsimonious model tuning is required for accurate confidence intervals. Kernel tuning to build a parsimonious model using cross validation methods will be illustrated.

Dr. John Lipp received his Electrical Engineering PhD from Michigan Technological University in the area of stochastic signal processing.  He currently is employed at Lockheed Martin where he holds the position of Fellow.  He teaches statistics and probability for engineers, Kalman filtering, design of experiments, and statistical verification and validation.

The Department of Defense (DoD) spends the majority of its annual budget on making sure that systems are ready to perform when called to action. Even with large investments, though, maintaining adequate system readiness poses a major challenge for the DoD. Here, we discuss why readiness is so difficult to maintain and introduce the tools IDA has developed to aid readiness and supply chain analysis and decision-making. Particular emphasis is placed on “honeybee,” the tool developed to clean, assemble, and mine data across a variety of sources in a well-documented and reproducible way. Using a notional example, we demonstrate the utility of this tool and others like it in our suite; these tools lower the barrier to performing meaningful analysis, constructing and estimating input data for readiness models, and aiding the DoD’s ability to tie resources to readiness outcomes.

Megan Gelsinger is a Research Staff Member in the Operational Evaluation Division of IDA.  Her work focuses on weapons system sustainment and readiness modeling.  Prior to joining IDA in August 2021, she completed her PhD in statistics at Cornell University. Her graduate work focused on developing computationally efficient methods for implementing big data spatio-temporal statistical models.

Dr. Laura Freeman is a Research Associate Professor of Statistics and dual hatted as the Director of the Intelligent Systems Lab, Virginia Tech National Security Institute and the Director of the Information Sciences and Analytics Division, Virginia Tech Applied Research Corporation (VT-ARC).   Her research leverages experimental methods for conducting research that brings together cyber-physical systems, data science, artificial intelligence (AI), and machine learning to address critical challenges in national security.  She develops new methods for test and evaluation focusing on emerging system technology.  In her role with VT-ARC she focuses on transitioning emerging research in these areas to solve challenges in Defense and Homeland Security. She is also a hub faculty member in the Commonwealth Cyber Initiative and leads research in AI Assurance. She is the Assistant Dean for Research for the College of Science, in that capacity she works to shape research directions and collaborations in across the College of Science in the Greater Washington D.C. area. Previously, Dr. Freeman was the Assistant Director of the Operational Evaluation Division at the Institute for Defense Analyses.  Dr. Freeman has a B.S. in Aerospace Engineering, a M.S. in Statistics and a Ph.D. in Statistics, all from Virginia Tech.  Her Ph.D. research was on design and analysis of experiments for reliability data.

Machine learning is being deployed into the real-world, yet the body of knowledge on testing, evaluating, and maintaining machine learning models is overwhelmingly centered on component-level analysis. But, machine learning and engineered systems are tightly coupled. This is evidenced by extreme sensitivity to of ML to changes in system structure and behavior. Thus, reliability, prognostics, and other efforts related to test and evaluation for ML cannot be divorced from the system. That is, machine learning and its system go hand-in-hand. Any other way makes an unjustified assumption about the existence of an independent variable. This talk explores foundational reasons for this phenomena, and the foundational challenges it poses to existing practice. Cases in machine health monitoring and in cyber defense are used to motivate the position that machine learning is not independent of physical changes to the system with which it interacts, and ML is not independent of the adversaries it defends against. By acknowledging these couplings, systems and mission engineers can better align test and evaluation practices with the fundamental character of ML.

We describe a potential framework for applying DOE to cyber testing and provide an example of its application to testing of a hypothetical command and control system.

Dr. James M. Gilmore "Mike" Research Staff Member | IDA SED   Fields of Expertise Test and evaluation; cost analysis; cost-effectiveness analysis.   Education 1976 - 1980 Doctor of Philosophy in Nuclear Engineering at University of Wisconsin 1972 - 1976 Bachelor of Science in Physics at M.I.T.   Employment 2017 - 2018 Principal Physical Scientist, RAND Corporation Performed various analyses for federal government clients 2009 - 2017 Director, Operational Test and Evaluation, Department of Defense Senate-confirmed Presidential appointee serving as the principal advisor to the Secretary of Defense regarding the operational effectiveness of all defense systems 2001 - 2009 Assistant Director for National Security , Congressional Budget Office Responsible for the CBO division performing analyses of a broad array of issues in national security for committees of the U.S. Congress 1994 - 2001 Deputy Director for General Purpose Programs, OSD Program Analysis and Evaluation Responsible for four divisions performing analyses and evaluations of all aspects of DoD's conventional forces and associated programs 1993 - 1994 Division Director, OSD Progam Analysis and Evaluation Responsible for divisions in the Cost Analysis Improvement Group performing independent cost analyses of major defense acquisition programs 1990 - 1993 Analyst, OSD Program Analysis and Evaluation Performed analysis of strategic defense systems and command, control, and communications systems 1989 - 1990 Analyst, Falcon Associates Performed various analyses for DoD clients 1985 - 1989 Analyst, McDonnell Douglas Performed analysis involving issues in command, control, communications, and intelligence 1981 - 1985 Scientist, Lawrence Livermore National Laboratory Modelled nuclear fusion experiments

Summary overview of Discrete Event Simulations (DES) for optimizing scheduling operations in a high mix, low volume, job shop environment. The DES model employs Monte Carlo simulation to minimize schedule conflicts and prioritize work, while taking into account competition for limited resources. Iterative simulation balancing to dampen model results and arrive at a globally optimized schedule plan will be contrasted with traditional deterministic scheduling methodologies.

Master's of Science in Systems Engineering from University of Denver with emphasis in discrete event simulation and optimization Bachelor's of Science in Manufacturing Engineering from Brigham Young University with emphasis in statistics and applied research +13 years of experience with Lockheed Martin Corporation employed in statistical analysis, material science, and manufacturing engineering

Monitoring noisy profiles for changes in the behavior can be used to validate whether the process is operating under normal conditions over time. Change-point detection and estimation in sequences of multivariate functional observations is a common method utilized in monitoring such profiles. A nonparametric method utilizing Classification and Regression Trees (CART) to build a sequence of regression trees is proposed which makes use of the Kolmogorov-Smirnov statistic to monitor profile behavior. Our novel method compares favorably to existing methods in the literature.

Effective training of the broad set of users/operators of systems has downstream impacts on usability, workload, and ultimate system performance that are related to mission success. In order to measure training effectiveness, we designed a survey called the Operational Assessment of Training Scale (OATS) in partnership with the Army Test and Evaluation Center (ATEC). Two subscales were designed to assess the degrees to which training covered relevant content for real operations (Relevance subscale) and enabled self-rated ability to interact with systems effectively after training (Efficacy subscale). The full list of 15 items were given to over 700 users/operators across a range of military systems and test events (comprising both developmental and operational testing phases). Systems included vehicles, aircraft, C3 systems, and dismounted squad equipment, among other types. We evaluated reliability of the factor structure across these military samples using confirmatory factor analysis. We confirmed that OATS exhibited a two-factor structure for training relevance and training efficacy. Additionally, a shortened, six-item measure of the OATS with three items per subscale continues to fit observed data well, allowing for quicker assessments of training. We discuss various ways that the OATS can be applied to one-off, multi-day, multi-event, and other types of training events. Additional OATS details and information about other scales for test and evaluation are available at the Institute for Defense Analyses’ web site, https://testscience.mystagingwebsite.com/validated-scales-repository/.

Dr. Brian Vickers received his PhD in Cognition and Cognitive Neuroscience from the University of Michigan in 2015 on the topic of how decision architectures and human factors influence people’s decisions about time, money, and material options. Since that time, he was worked in data and decision science, and is currently a Research Staff Member at the Institute for Defense Analyses supporting topics including operational testing, test science, and artificial intelligence.

Wireless and radio frequency (RF) technology are ubiquitous in our daily lives, including laptops, key fobs, remote sensors, and antennas. These devices, while oftentimes portable and convenient, can potentially be susceptible to adversarial attack over the air. This breakout session will provide a short introduction into wireless hacking concepts such as passive scanning, active injection, and the use of software defined radios to flexibly sample the RF spectrum. We will also ground these concepts in live demonstrations of attacks against both wireless and wired systems.

Dr. Jason Schlup received his Ph.D. in Aeronautics from the California Institute of Technology in 2018.  He is now a Research Staff Member at the Institute for Defense Analyses and provides analytical support to the Director, Operational Test and Evaluation’s Cyber Assessment Program.  Jason also contributes to IDA’s Cyber Lab capability.

For AI-enabled and autonomous systems, issues of safety, security, and mission effectiveness are not separable—the same underlying data and software give rise to interrelated risks in all of these dimensions. If treated separately, there is considerable unnecessary duplication (and sometimes mutual interference) among efforts needed to satisfy commanders, operators, and certification authorities of the systems’ dependability. Assurances cases, pioneered within the safety and cybersecurity communities, provide a structured approach to simultaneously verifying all dimensions of system dependability with minimal redundancy of effort. In doing so, they also provide a more concrete and useful framework for system development and explanation of behavior than is generally seen in discussions of “transparency” and “trust” in AI and autonomy. Importantly, trust generally cannot be “built in” to systems, because the nature of the assurance arguments needed for various stakeholders requires iterative identification of evidence structures that cannot be anticipated by developers.

David A. Sparrow received a Ph.D. in physics in 1974, and spent 12 years as an academic physicist. He joined IDA in 1986 and has been a Research Staff Member ever since, with brief forays into management and government service. He was the first Director of the IDA Simulation Center from 1989 to 1990, and Assistant Director of the Science and Technology Division from 1993 to 1997. He then joined the government for a two-year stint as Science Advisor on Modeling and Simulation to the Director, Operational Test and Evaluation. Since returning to IDA, he has focused on technical issues in system development, especially ground combat systems—expansively defined to include unexploded ordinance (UXO), counter mine, and, occasionally, missile defense. Recent emphasis has been on AI-enabled autonomous systems for defense applications.  He has authored ~100 refereed papers and invited talks on various academic and national security topics.

To properly test and evaluate today’s advanced military systems, the T&E community must utilize big data analytics (BDA) and techniques to quickly process, visualize, understand, and report on massive amounts of data. This tutorial/presentation/TBD will inform the audience how to transform the current T&E data infrastructure and analysis techniques to one employing enterprise BDA and Knowledge Management (BDKM) that supports the current warfighter T&E needs and the developmental and operational testing of future weapon platforms. The TRMC enterprise BDKM will improve acquisition efficiency, keep up with the rapid pace of acquisition technological advancement, and ensure that effective weapon systems are delivered to warfighters at the speed of relevance – all while enabling T&E analysts across the acquisition lifecycle to make better and faster decisions using data previously inaccessible or unusable. This capability encompasses a big data architecture framework – its supporting resources, methodologies, and guidance – to properly address the current and future data needs of systems testing and analysis, as well as an implementation framework, the Cloud Hybrid Edge-to-Enterprise Evaluation and Test Analysis Suite (CHEETAS). In combination with the TRMC’s Joint Mission Environment Test Capability (JMETC) which provides readily-available connectivity to the Services’ distributed test capabilities and simulations, the TRMC has demonstrated that applying enterprise-distributed BDA tools and techniques to distributed T&E leads to faster and more informed decision-making – resulting in reduced overall program cost and risk.

Dr. Edward T. Powell is a lead architect and systems engineer for the Test Resource Management Center.  He has worked in the military simulation, intelligence, and Test and Evaluation fields during his thirty-year career, specializing in systems and simulation architecture and engineering.  His current focus is on integrating various OSD and Service big data analysis initiatives into a single seamless cloud-based system-of-systems.  He holds a PhD in Astrophysics from Princeton University is the principal of his own consulting company based in Northern Virginia.

Planning Full-Up System-Level (FUSL) Live Fire tests is a complex process that has historically relied solely on subject matter expertise. In particular, there is no established method to determine the appropriate number of FUSL tests necessary for a given program. We developed a novel method that is analogous to the Design of Experiments process that is used to determine the scope of Operational Test events. Our proposed methodology first requires subject matter experts (SMEs) to define all potential FUSL shots. For each potential shot, SMEs estimate the severity of that shot, the uncertainty of that severity estimate, and the similarity of that shot to all other potential shots. We developed a numerical optimization algorithm that uses the SME inputs to generate a prioritized list of FUSL events and a corresponding plot of the total information gained with each successive shot. Together, these outputs can help analysts determine the adequate number of FUSL tests for a given program. We illustrate this process with an example on a notional ground vehicle. Future work is necessary prior to implementation on a program of record.

Dr. Lindsey Butler holds a B.S. in Chemical Engineering from Virginia Tech and a Ph.D. in Biomedical Engineering from the University of South Carolina. She has worked at the Institute for Defense Analyses for 5 years where she supports the Director of Operational Test and Evaluation. Dr. Butler is the Deputy Live Fire lead at the Institute for Defense Analyses. Her primary projects focus on assessing the survivability of body armor and armored vehicles against operationally realistic threats. She also has an expertise evaluating casualty assessments to personnel after live fire tests.

Sustainment for weapon systems involves multiple components that influence readiness outcomes through a complex array of interactions. While military leadership can use simple analytical approaches to yield insights into current metrics (e.g., dashboard for top downtime drivers) or historical trends of a given sustainment structure (e.g., correlative studies between stock sizes and backorders), they are inadequate tools for guiding decision-making due to their inability to quantify the impact on readiness. In this talk, we discuss the power of IDA’s end-to-end modeling and simulation (M&S) approach that estimates time-varying readiness outcomes based on real-world data on operations, supply, and maintenance. These models are designed to faithfully emulate fleet operations at the level of individual components and operational units, as well as to incorporate the multi-echelon inventory system used in military sustainment. We showcase a notional example in which our M&S approach produces a set of recommended component-level investments and divestments in wholesale supply that would improve the readiness of a weapon system. We argue for the urgency of increased end-to-end M&S efforts across the Department of Defense to guide the senior leadership in its data-driven decision-making for readiness initiatives.

Andrew Flack is a Research Staff Member in the Operational Evaluation division at IDA. His work focuses on weapons system sustainment and readiness modeling. Prior to joining IDA in 2016, Andrew was an analyst at the Defense Threat Reduction Agency (DTRA) studying M&S tools for chemical and biological defense.

Autocorrelated sequences arise in many modern-day industrial applications. In this paper, our focus is on estimating the time of sudden shift in the location or scale of a continuous ergodic-stationary sequence following a genuine signal from a statistical process control chart. Our general approach involves “clipping” the continuous sequence at the median or interquartile range (IQR) to produce a binary sequence, and then modeling the joint mass function for the binary sequence using a Bahadur approximation. We then derive a maximum likelihood estimator for the time of sudden shift in the mean of the binary sequence. Performance comparisons are made between our proposed change point estimator and two other viable alternatives. Although the literature contains existing methods for estimating the time of sudden shift in the mean and/or variance of a continuous process, most are derived under strict independence and distributional assumptions. Such assumptions are often too restrictive, particularly when applications involve Industry 4.0 processes where autocorrelation is prevalent and the distribution of the data is likely unknown. The change point estimation strategy proposed in this work easily incorporates autocorrelation and is distribution-free. Consequently, it is widely applicable to modern-day industrial processes.

Zhi Wang is currently a data scientist contractor at Bayer Crop Science focused on advancing field operation using modern data analytic tools and methods. Dr. Wang's research interests include changepoint detection and estimation, statistical process monitoring, business analytics, and geospatial environmental modeling.

Machine Learning models have been incredibly impactful over the past decade; however, testing those models and comparing their performance has remained challenging and complex. In this presentation, I will demonstrate novel methods for measuring the performance of computer vision object detection models, including running those models against still imagery and video. The presentation will start with an introduction to the pros and cons of various metrics, including traditional metrics like precision, recall, average precision, mean average precision, F1, and F-beta. The talk will then discuss more complex topics such as tracking metrics, handling multiple object classes, visualizing multi-dimensional metrics, and linking metrics to operational impact. Anecdotes will be shared discussing different types of metrics that are appropriate for different types of stakeholders, how system testing fits in, best practices for model integration, best practices for data splitting, and cloud vs on-prem compute lessons learned. The presentation will conclude by discussing what software libraries are available to calculate these metrics, including the MORSE-developed library Charybdis.

Dr. Anna Rubinstein serves as the Director of Test and Evaluation for a Department of Defense (DoD) Artificial Intelligence (AI) program. She directs testing for AI models spanning the fields of computer vision, natural language processing, and other forms of machine perception. She leads teams developing metrics and assessing capabilities at the algorithm, system, and operational level, with a particular interest in human-machine teaming. Dr. Rubinstein has spent the last five years supporting national defense as a contractor, largely focusing on model and system evaluation. In her previous role as a Science Advisor in the Defense Advanced Research Projects Agency’s (DARPA) Information Innovation Office (I2O), she provided technical insight to research programs modeling cyber operations in the information domain and building secure software-reliant systems. Before that, Dr. Rubinstein served as a Research Staff Member at the Institute for Defense Analyses (IDA), leading efforts to provide verification and validation of nuclear weapons effects modeling codes in support of the Defense Threat Reduction Agency (DTRA). Dr. Rubinstein also has several years of experience developing algorithms for atmospheric forecasting, autonomous data fusion, social network mapping, anomaly detection, and pattern optimization. Dr. Rubinstein holds an M.A. in Chemical Engineering and a Ph.D. in Chemical Engineering and Materials Science from Princeton University, where she was a National Science Foundation Graduate Research Fellow. She also received a B.S. in Chemical Engineering, a B.A. in Chemistry, and a B.A. in Chinese, all from the University of Mississippi, where she was a Barry M. Goldwater Scholar.

Current disease outbreak surveillance practices reflect underlying delays in the detection and reporting of disease cases, relying on individuals who present symptoms to seek medical care and enter the health care system. To accelerate the detection of outbreaks resulting from possible bioterror attacks, we introduce a novel two-tier, human sentinel network (HSN) concept composed of wearable physiological sensors capable of pre-symptomatic illness detection, which prompt individuals to enter a confirmatory stage where diagnostic testing occurs at a certified laboratory. Both the wearable alerts and test results are reported automatically and immediately to a secure online platform via a dedicated application. The platform aggregates the information and makes it accessible to public health authorities. We evaluated the HSN against traditional public health surveillance practices for outbreak detection of 80 Bacillus anthracis (Ba) release scenarios in mid-town Manhattan, NYC. We completed an end-to-end modeling and analysis effort, including the calculation of anthrax exposures and doses based on computational atmospheric modeling of release dynamics, and development of a custom-built probabilistic model to simulate resulting wearable alerts, diagnostic test results, symptom onsets, and medical diagnoses for each exposed individual in the population. We developed a novel measure of network coverage, formulated new metrics to compare the performance of the HSN to public health surveillance practices, completed a Design of Experiments to optimize the test matrix, characterized the performant trade-space, and performed sensitivity analyses to identify the most important engineering parameters. Our results indicate that a network covering greater than ~10% of the population would yield approximately a 24-hour time advantage over public health surveillance practices in identifying outbreak onset, and provide a non-target-specific indication (in the form of a statistically aberrant number of wearable alerts) of approximately 36-hours; these earlier detections would enable faster and more effective public health and law enforcement responses to support incident characterization and decrease morbidity and mortality via post-exposure prophylaxis.

Jane Valentine received her B.S. in Mathematics and French, and Ph.D. in Biomedical Engineering, both from Carnegie Mellon University. She then completed a post-doc in Mechanical Engineering at the University of Illinois, and a data science fellowship in the United Kingdom, working with a pharmaceutical company.  She has been working at the Johns Hopkins University Applied Physics Laboratory since 2020, where she works on mathematical modeling and simulation, optimization, and data science, particularly in the areas of biosensors, knowledge graphs, and epidemiological modeling.

The DoD’s challenge to provide test results at the “Speed of Relevance” has generated many new strategies to accelerate data collection, adjudication, and analysis. As a result, the Air Force Operational Test and Evaluation Center (AFOTEC), in conjunction with the Air Force Chief Data Office’s Visible, Accessible, Understandable, Linked and Trusted Data Platform (VAULT), is developing a Survey Application. This new cloud-based application will be deployable on any AFNET-connected computer or tablet and merges a variety of tools for collection, storage, analytics, and decision-making into one easy-to-use platform. By placing cloud-computing power in the hands of operators and testers, authorized users can view report-quality visuals and statistical analyses the moment a survey is submitted. Because the data is stored in the cloud, demanding computations such as machine learning are run at the data source to provide even more insight into both quantitative and qualitative metrics. The T-7A Red Hawk will be the first operational test (OT) program to utilize the Survey Application. Over 1000 flying and simulator test points have been loaded into the application, with many more coming from developmental test partners. The Survey app development will continue as USAF testing commences. Future efforts will focus on making the Survey Application configurable to other research and test programs to enhance their analytic and reporting capabilities.

irst Lieutenant Timothy Dawson is an operational test analyst assigned with the Air Force Operational Test and Evaluation Center, Detachment 5, at Edwards AFB, Ca. The lieutenant serves as the lead AFOTEC Mobility Training Operations analyst, splitting his work between the T-7A Red Hawk high performance trainer, KC-46A Pegasus tanker, and VC-25B presidential transport. Lieutenant Dawson also serves alongside the 416^th Flight Test Squadron as a flight test engineer on the T-38C Talon.   Lieutenant Dawson, originally from Olympia, Wa., received his commission as a second lieutenant upon completing ROTC at the University of California, Berkeley in 2019. He served as a student pilot at Vance AFB, Ok., leading data analysis and software development projects before arriving to his current duty location at Edwards.

It is often said that many research findings -- from social sciences, medicine, economics, and other disciplines -- are false. This fact is trumpeted in the media and by many statisticians. There are several reasons that false research is published, but to what extent should we be worried about them in defense testing and in particular modeling and simulation validation studies? In this talk I will present several recommendations for actions that statisticians and data scientists can take to improve the quality of our validations and evaluations.

Dr. John Haman is a research staff member at the Institute for Defense Analyses, where he leads a team of analysts that develop methods and tools for analyzing test data. He has worked with a variety of Army, Navy, and Air Force systems, including counter-UAS and electronic warfare systems.

The DoD sustainment system is responsible for managing the supply of millions of different spare parts, most of which are infrequently and inconsistently requisitioned, and many of which have procurement lead times measured in years. The DoD must generally buy items in anticipation of need, yet it simply cannot afford to buy even one copy of every unique part it might be called upon to deliver. Deciding which items to purchase necessarily involves taking risks, both military and financial. However, the huge scale of the supply system makes these risks difficult to quantify. We have developed methods that use raw supply data in new ways to support this decision making process. First, we have created a method to identify areas of potential overinvestment that could safely be reallocated to areas at risk of underinvestment. Second, we have used raw requisition data to create an item priority list for individual weapon systems in terms of importance to mission success. Together, these methods allow DoD decision makers to make better-informed decisions about where to take risks and where to invest scarce resources.

Joseph Fabritius earned his Bachelor's degree in Physics from Rochester Institute of Technology in 2012. He earned his Master's degree in Physics from Drexel University 2017, and he earned his PhD in Physics from Drexel University in 2021. He currently is a Research Staff Member at the Institute for Defense Analyses where he works on sustainment analyses.

Advances in open-source software have brought powerful machine learning and data analysis tools requiring little more than a few coding basics. Unfortunately, the very nature of rapidly changing software can contribute to legitimate concerns surrounding the reproducibility of research and analysis. Borrowing from current practices in data science and software engineering fields, a more robust process using the R ecosystem to produce a version-controlled data analysis pipeline is proposed. By integrating the data cleaning, model generation, manuscript writing, and presentation scripts, a researcher or data analyst can ensure small changes at any step will automatically be reflected throughout using the Rmarkdown, targets, renv, and xaringan R packages.

Andrew G. Farina is an Assistant Professor at the United States Military Academy, Department of Behavioral Sciences and Leadership. He has ten combat deployments, serving with both conventional and special operations units. His research interests include leadership, character development, and risk-taking propensity.

Uncontrolled model extrapolation leads to two serious kinds of errors: (1) the model may be completely invalid far from the data, and (2) the combinations of variable values may not be physically realizable. Optimizing models that are fit to observational data can lead to extrapolated solutions that are of no practical use without any warning. In this presentation we introduce a general approach to identifying extrapolation based on a regularized Hotelling T-squared metric. The metric is robust to certain kinds of messy data and can handle models with both continuous and categorical inputs. The extrapolation model is intended to be used in parallel with a machine learning model to identify when the machine learning model is being applied to data that are not close to that model training set or as a non-extrapolation constraint when optimizing the model. The methodology described was introduced into the JMP Pro 16 Profiler.

The DOD has to maintain readiness across a staggeringly diverse array of modern weapon systems, yet no single person or organization in the DOD has an end-to-end picture of the sustainment system that supports them. This shortcoming can lead to bad decisions when it comes to allocating resources in a funding-constrained environment. The underlying problem is driven by stovepiped databases, a reluctance to share data even internally, and a reliance on tribal knowledge of often cryptic data sources. Notwithstanding these difficulties, we need to create a comprehensive picture of the sustainment system to be able to answer pressing questions from DOD leaders. To that end, we have created a documented and reproducible workflow that shepherds raw data from DOD databases through cleaning and curation steps, and then applies logical rules, filters, and assumptions to transform the raw data into concrete values and useful metrics. This process gives us accurate, up-to-date data that we use to support quick-turn studies, and to rapidly build (and efficiently maintain) a suite of readiness models for a wide range of complex weapon systems.

Dr. Benjamin Ashwell has been a Research Staff Member at the Institute for Defense Analyses (IDA) since 2015. A founding member of IDA’s Sustainment Analysis Group, he leads the NAVAIR Sustainment analysis task. This work combines deep data analysis with end-to-end stochastic simulations to tie resource investments to flight line readiness outcomes. Before moving to the Sustainment Group in 2019, Dr. Ashwell spent three years supporting the Director of Operational Test and Evaluation’s (DOT&E’s) analysis of the Navy’s Littoral Combat Ship, specializing in surface warfare and system reliability.  Dr. Ashwell received his PhD in Chemistry from Northwestern University in 2015.

Machine learning algorithms can help to distill vast quantities of information to support decision making. However, machine learning also presents unique legal, moral, and ethical concerns – ranging from potential discrimination in personnel applications to misclassifying targets on the battlefield. Building on foundational principles in ethical philosophy, this presentation summarizes key legal, moral, and ethical criteria applicable to machine learning and provides pragmatic considerations and recommendations.

Alan earned his PhD in Economics from the University of Iowa in 2014 and currently leads the Human Capital Group in the Strategy, Forces, and Resources Division at the Institute for Defense Analyses.  He specializes in microeconomics, game theory, experimental and behavioral economics, and machine learning, and his research focuses on personnel attrition and related questions for the DOD.

Enabling Enhanced Validation of NDE Computational Models and Simulations William C. Schneck, III, Ph.D. Elizabeth D. Gregory, Ph.D. NASA Langley Research Center Computer simulations of physical processes are increasingly used in the development, design, deployment, and life-cycle maintenance of many engineering systems [1] [2]. Non-Destructive Evaluation (NDE) and Structural Health Monitoring (SHM) must employ effective methods to inspect increasingly complex structural and material systems developed for new aerospace systems. Reliably and comprehensively interrogating this multidimensional [3] problem domain from a purely experimental perspective can become cost and time prohibitive. The emerging way to confront these new complexities in a timely and cost-effective manner is to utilize computer simulations. These simulations must be Verified and Validated [4] [5] to assure reliable use for these NDE/SHM applications. Beyond the classical use of models for engineering applications for equipment or system design efforts, NDE/SHM are necessarily applied to as-built and as-used equipment. While most structural or CFD models are applied to ascertain performance of as-designed systems, the performance of an NDE/SHM system is necessarily tied to the indications of damage/defects/deviations (collectively, flaws) within as-built and as-used structures and components. Therefore, the models must have sufficient fidelity to determine the influence of these aberrations on the measurements collected during interrogation. To assess the accuracy of these models, the Validation data sets must adequately encompass these flaw states. Due to the extensive parametric spaces that this coverage would entail, this talk proposes an NDE Benchmark Validation Data Repository, which should contain inspection data covering representative structures and flaws. This data can be reused from project to project, amortizing the cost of performing high quality Validation testing. Works Cited [1] Director, Modeling and Simulation Coordination Office, "Department of Defense Standard Practice: Documentation of Verification, Validation, and Accredation (VV&A) for Models and Simulations," Department of Defense, 2008. [2] Under Secretary of Defense (Acquisition, Technology and Logistics), "DoD Modeling and Simulation (M&S) Verification, Validation, and Accredation (VV&A)," Department of Defense, 2003. [3] R. C. Martin, Clean Architecture: A Craftsman's Guide to Software Structure and Design, Boston: Prentice Hall, 2018. [4] C. J. Roy and W. L. Oberkampf, "A Complete Framework for Verification, Validation, and Uncertainty Quantification in Scientific Computing (Invited)," in 48th AIAA Aerospace Sciences Meeting, Orlando, 2010. [5] ASME Performance Test Code Committee 60, "Guide for Verification and Validation in Computational Solid Mechanics," ASME International, New York, 2016.

A case study will be presented where patients wearing continuous glycemic monitoring systems provide sensor stream data of their glucose levels before and after consuming 1 of 5 different types of snacks. The goal is to be able to better predict a new patient’s glycemic-response-over-time trace after being given a particular type of snack. Functional Data Analysis (FDA) is used to extract eigenfunctions that capture the longitudinal shape information of the traces and principal component scores that capture the patient-to-patient variation. FDA is used twice. First it is used on the “before” baseline glycemic-response-over-time traces. Then a separate analysis is done on the snack-induced “after” response traces. The before FPC scores and the type of snack are then used to model the after FPC scores. This final FDA model will then be used to predict the glycemic response of new patients given a particular snack and their existing baseline response history. Although the case study is for medical sensor data, the methodology employed would work for any sensor stream where an event perturbs the system thus affecting the shape of the sensor stream post event.

Tom Donnelly works as a Systems Engineer for JMP Statistical Discovery LLC supporting users of JMP in the Defense and Aerospace sector.  He has been actively using and teaching Design of Experiments (DOE) methods for the past 38 years to develop and optimize products, processes, and technologies.  Donnelly joined JMP in 2008 after working as an analyst for the Modeling, Simulation & Analysis Branch of the US Army’s Edgewood Chemical Biological Center (now CCDC CBC).  There, he used DOE to develop, test, and evaluate technologies for detection, protection, and decontamination of chemical and biological agents.  Prior to working for the Army, Tom was a partner in the first DOE software company for 20 years where he taught over 300 industrial short courses to engineers and scientists.  Tom received his PhD in Physics from the University of Delaware.

Designed experiments can be a powerful tool for gaining fundamental understanding of systems and processes or maintaining or optimizing systems and processes. There are usually multiple performance and quality metrics that are of interest in an experiment, and these multiple responses may include data from nonnormal distributions, such as binary or count data. A design that is optimal for a normal response can be very different from a design that is optimal for a nonnormal response. This work includes a two-phase method that helps experimenters identify a hybrid design for a multiple response problem. Mixture and optimal design methods are used with a weighted optimality criterion for a three-response problem that includes a normal, a binary, and a Poisson model, but could be generalized to an arbitrary number and combination of responses belonging to the exponential family. A mixture design is utilized to identify the optimal weights in the criterion presented.

Brittany Fischer is a PhD candidate in industrial engineering at Arizona State University. Prior to ASU, she received her bachelor’s and master’s degrees in statistics from Pennsylvania State University and worked as a statistical engineer for 5 years at Corning Incorporated.

Traditional hypothesis testing is used extensively in test and evaluation (T&E) to determine if there is a difference between two or more populations. For example, we can analyze a designed experiment using t-tests to determine if a factor affects the response or not. Rejecting the null hypothesis would provide evidence that the factor changes the response value. However, there are many situations in T&E where the goal is to actually show that things didn’t change; the response is actually the same (or nearly the same) after some change in the process or system. If we use traditional hypothesis testing to assess this scenario, we would want to “fail to reject” the null hypothesis; however, this doesn’t actually provide evidence that the null hypothesis is true. Instead, we can orient the analysis to the decision that will be made and use equivalence testing. Equivalence testing initially assumes the populations are different; the alternative hypothesis is that they are the same. Rejecting the null hypothesis provides evidence that the populations are the same, matching the objective of the test. This talk provides an overview of equivalence testing with examples demonstrating its applicability in T&E. We also discuss additional considerations for planning a test where equivalence testing will be used including: sample size and what does “equivalent” really mean.

Sarah Burke is an analyst for The Perduco Group, A Linquest Company. She earned her doctorate in industrial engineering at Arizona State University. Her research interests include design of experiments, response surface methodology, multi‐criteria decision making, and statistical engineering.

Getting Responsible AI (RAI) right is difficult and demands expertise. All AI-relevant skill sets, including ethics, are in high demand and short supply, especially regarding AI’s intersection with test and evaluation (T&E). Frameworks, guidance, and tools are needed to empower working-level personnel across DOD to generate RAI assurance cases with support from RAI SMEs. At a high level, framework should address the following points: 1. T&E is a necessary piece of the RAI puzzle--testing provides a feedback mechanism for system improvement and builds public and warfighter confidence in our systems, and RAI should be treated just like performance, reliability, and safety requirements. 2. We must intertwine T&E and RAI across the cradle-to-grave product life cycle. Programs must embrace T&E and RAI from inception; as development proceeds, these two streams must be integrated in tight feedback loops to ensure effective RAI implementation. Furthermore, many AI systems, along with their operating environments and use cases, will continue to update and evolve and thus will require continued evaluation after fielding. 3. The five DOD RAI principles are a necessary north star, but alone they are not enough to implement or ensure RAI. Programs will have to integrate multiple methodologies and sources of evidence to construct holistic arguments for how much the programs have reduced RAI risks. 4. RAI must be developed, tested, and evaluated in context--T&E without operationally relevant context will fail to ensure that fielded tools achieve RAI. Mission success depends on technology that must interact with warfighters and other systems in complex environments, while constrained by processes and regulation. AI systems will be especially sensitive to operational context and will force T&E to expand what it considers.

In the current moment, autonomous and artificial intelligence (AI) systems are emerging at a dizzying pace. Such systems promise to expand the capacity and capability of individuals by delegating increasing levels of decision making down to the agent-level. In this way, operators can set high-level objectives for multiple vehicles or agents and need only intervene when alerted to anomalous conditions. Test and evaluation efforts at the Join AI Center are focused on exercising a prescribed test strategy for AI-enabled systems. This new AI T&E Framework recognizes the inherent complexity that follows from incorporating dynamic decision makers into a system (or into a system-of-systems). The AI T&E Framework is composed of four high-level types of testing that examine at an AI-enabled system from different angles to provide as complete a picture as possible of the system's capabilities and limitations, including algorithmic, system integration, human-system integration, and operational tests. These testing categories provides stakeholders with appropriate qualitative and quantitative assessments that bound the system’s use cases in a meaningful way. The algorithmic tests characterize the AI models themselves against metrics for effectiveness, security, robustness and responsible AI principles. The system integration tests the system itself to ensure it operates reliably, functions correctly, and is compatible with other components. The human-machine testing asks what do human operators think of the system, if they understand what the system is telling them, and if they trust the system under appropriate conditions. All of which culminates in an operational test that evaluates how the system performs in a realistic environment with realistic scenarios and adversaries. Interestingly, counter to traditional approaches, this framework is best applied during and throughout the development of an AI-enabled system. Our experience is that programs that conduct independent T&E alongside development do not suffer delays, but instead benefit from the feedback and insights gained from incremental and iterative testing, which leads to the delivery of a better overall capability.

Lt Col Brian Woolley is a U.S. Air Force officer currently serving as the Test and Evaluation Enclave Manager at the DoD’s Joint Artificial Intelligence Center, Arlington, Virginia. He earned his Doctoral degree in Computer Engineering from the University of Central Florida and a Masters of Science in Software Engineering from the Air Force Institute of Technology. During his 19-year military career, Brian has served as a Cyber Operations Officer supporting the Air Force Weather Agency, the Joint Headquarters for the DoD Information Networks, U.S. Cyber Command, and as the Deputy Directory for the Autonomy and Navigation Center at the Air Force Institute of Technology.

Combat Capabilities Development Command Armaments Center (DEVCOM AC) is developing the next generation breaching munition, a replacement for the M58 Mine Clearing Line Charge. A series of M&S experiments were conducted to aid with the design of mine-neutralizing submunitions, utilizing space-filling designs, support vector machines, and hyper-parameter optimization. A probabilistic meta-model of the FEA-simulated performance data was generated with Platt Scaling in order to facilitate optimization, which was implemented to generate several candidate designs for follow-up live testing. This paper will detail the procedure used to iteratively explore and extract information from a deterministic process with a binary response.

Eli Golden, GStat is a statistician in the Systems Analysis Division of US Army Combat Capabilities Development Command Armaments Center (DEVCOM AC). He is an experienced practitioner of Design of Experiments, empirical model-building, and data visualization focusing in the domains of conventional munition development, market research, advanced manufacturing, and modelling and simulation, and is an instructor/content curator for the Probability and Statistics courses at the Armament Center's Armament Graduate School (AGS). Mr. Golden has an M.S. in Applied Statistics from New Jersey Institute of Technology, M.S. in Mechanical Engineering from Stevens Institute of Technology, and a B.S. in Mechanical Engineering with a minor in Mathematics from Lafayette College.

Adopting uncertainty quantification (UQ) has become a prerequisite for providing credibility in modeling and simulation (M&S) applications. It is well known, however, that UQ can be computationally prohibitive for problems involving expensive high-fidelity models, since a large number of model evaluations is typically required. A common approach for improving efficiency is to replace the original model with an approximate surrogate model (i.e., metamodel, response surface, etc.) using machine learning that makes predictions in a fraction of the time. While surrogate modeling has been commonplace in the UQ field for over a decade, many practitioners still remain hesitant to rely on “black box” machine learning models over trusted physics-based models (e.g., FEA) for their analyses. This talk discusses the role of machine learning in enabling computational speedup for UQ, including traditional limitations and modern efforts to overcome them. An overview of surrogate modeling and its best practices for effective use is first provided. Then, some emerging methods that aim to unify physics-based and data-based approaches for UQ are introduced, including multi-model Monte Carlo simulation and physics-informed machine learning. The use of both traditional surrogate modeling and these more advanced machine learning methods for UQ are highlighted in the context of applications at NASA, including trajectory simulation and spacesuit certification.

Dr. James (Jim) Warner joined NASA Langley Research Center (LaRC) in 2014 as a Research Computer Engineer after receiving his PhD in Computational Solid Mechanics from Cornell University. Previously, he received his B.S. in Mechanical Engineering from SUNY Binghamton University and held temporary research positions at the National Institute of Standards and Technology and Duke University. Dr. Warner is a member of the Durability, Damage Tolerance, and Reliability Branch (DDTRB) at LaRC, where he focuses on developing computationally-efficient approaches for uncertainty quantification for a range of applications including structural health management, additive manufacturing, and trajectory simulation. Additionally, he works to bridge the gap between UQ research and NASA mission impact, helping to transition state-of-the-art methods to solve practical engineering problems. To that end, he is currently involved in efforts to certify the xEMU spacesuit and develop guidance systems for entry, descent, and landing for Mars landing. His other research interests include machine learning, high performance computing, and topology optimization.

For analysis of military Developmental Test (DT) data, frequentist statistical models are increasingly challenged to meet the needs of analysts and decision-makers. Bayesian models have the potential to address this challenge. Although there is a substantial body of research on Bayesian reliability estimation, there appears to be a paucity of Bayesian applications to issues of direct interest to DT decision makers. To address this deficiency, this research accomplishes two tasks. First, this work provides a motivating example that analyzes reliability for a notional but representative system. Second, to enable the motivated analyst to apply Bayesian methods, it provides a foundation and best practices for Bayesian reliability analysis in DT. The first task is accomplished by applying Bayesian reliability assessment methods to notional DT lifetime data generated using a Bayesian reliability growth planning methodology (Wayne 2018). The tested system is assumed to be a generic complex system with a large number of failure modes. Starting from the Bayesian assessment methodology of (Wayne and Modarres, A Bayesian Model for Complex System Reliability 2015), this work explores the sensitivity of the Bayesian results to the choice of the prior distribution and compares the Bayesian results for the reliability point estimate and uncertainty interval with analogous results from traditional reliability assessment methods. The second task is accomplished by establishing a generic structure for systematically evaluating relevant statistical Bayesian models. It identifies what have been implicit reliability issues for DT programs using a structured poll of stakeholders combined with interviews of a selected set of Subject Matter Experts. Secondly, candidate solutions are identified in the literature. Thirdly, solutions matched to issues using criteria designed to evaluate the capability of a solution to improve support for decision-makers at critical points in DT programs. The matching process uses a model taxonomy structured according to decisions at each DT phase, plus criteria for model applicability and data availability. The end result is a generic structure that allows an analyst to identify and evaluate a specific model for use with a program and issue of interest. Wayne, Martin. 2018. "Modeling Uncertainty in Reliability Growth Plans." 2018 Annual Reliability and Maintainability Symposium (RAMS). 1-6. Wayne, Martin, and Mohammad Modarres. 2015. "A Bayesian Model for Complex System Reliability." IEEE Transactions on Reliability 64: 206-220.

Dr. Paul Fanto is a Research Staff Member at the Institute for Defense Analyses.  He received a Ph.D. in Physics from Yale University, where he worked on the application of Monte Carlo methods and high-performance computing to the modeling of atomic nuclei.  His current work involves the study of space systems and the application of Bayesian statistical methods to defense system testing.

Based on observations gathered from the IDA Forum on Orbital Debris (OD) Risks and Challenges (October 8-9, 2020), DOT&E needed first-order predictive tools to evaluate the effects of orbital debris on mission risk, catastrophic collision, and collateral damage to DOD spacecraft and other orbital assets – either from unintentional or intentional [Anti-Satellite (ASAT)] collisions. This lack of modeling capability hindered DOT&E’s ability to evaluate the risk to operational effectiveness and survivability of individual satellites and large constellations, as well as risks to the overall use of space assets in the future. Part 1 of this presentation describes an IDA-derived Excel-based tool (SatPen) for determining the probability and mission effects of >1mm orbital debris impacts and penetration on individual satellites in low Earth orbit (LEO). IDA estimated the likelihood of satellite mission loss using a Starlink-like satellite as a case study and NASA’s ORDEM 3.1 orbital debris environment as an input, supplemented with typical damage prediction equations to support mission loss predictions. Part 2 of this presentation describes an IDA-derived technique (DebProp) to evaluate the debris propagating effects of large, trackable debris (>5 cm) or antisatellite weapons colliding with satellites within constellations. IDA researchers again used a Starlink-like satellite as a case study and worked with Stellingwerf Associates to modify the Smooth Particle Hydrodynamic Code (SPHC) in order to predict the number and direction of fragments following a collision by a tracked satellite fragment. The result is a file format that is readable as an input file for predicting orbital stability or debris re-entry for thousands of created particles, and predict additional, short-term OD-induced losses to other satellites in the constellation. By pairing these techniques, IDA can predict additional, short-term and long-term OD-induced losses to other satellites in the constellation, and conduct long-term debris growth studies.

FIELDS OF EXPERTISE Air and space vehicle survivability, missile lethality, LFT&E, ballistic response, active protection systems, hypervelocity impact, space debris, crew and passenger casualty assessment EDUCATION HISTORY 1993 Doctor of Philosophy in Systems Engineering at University of Alabama, Huntsville 1989 Master of Science in Engineering Management at University of Alabama, Huntsville 1983 Bachelor of Science in Mechanical Engineering at University of Nebraska EMPLOYMENT HISTORY 2003 - Present Research Staff Member, IDA, OED 1998 - 2003 Director, Center for Space Systems Survivability, University of Denver 1987 - 1998 Spacecraft Survivability Design , NASA-Marshall Space Flight Center, NASA 1983 - 1987 U.S. Army Missile Command, Research Development and Engineering Center, Warhead Design, U.S. Army PROFESSIONAL ACTIVITIES American Institute of Aeronautics and Astronautics (Chair, Survivability Technical Committee, 2001-2003) Tau Beta Pi Engineering Honorary Society Pi Tau Sigma Mechanical Engineering Honorary Society HONORS IDA Welch Award, 2020. National AIAA Survivability Award, 2012. Citation reads, "For outstanding achievement in enhancing spacecraft, aircraft, and crew survivability through advanced meteoroid/orbital debris shield designs, on-orbit repair techniques, risk assessment tools, and live fire evaluation." NASA Astronauts' Personal Achievement Award (Silver Snoopy), 2001. NASA Exceptional Achievement Medal, Spacecraft Survivability Analysis, 1995. Army Research and Development Achievement Award, 1985. Patents and Statutory Invention Registrations: Enhanced Hypervelocity Impact Shield, 1997. Joint. Patents and Statutory Invention Registrations: Pressure Wall Patch, 1994. Joint. Patents and Statutory Invention Registrations: Advanced Anti-Tank Airframe Configuration Tandem Warhead Missile, 1991. Joint. Patents and Statutory Invention Registrations: Extendible Shoulder Fired Anti-tank Missile, 1990. Joint. Patents and Statutory Invention Registrations: Particulated Density Shaped Charge Liner, 1987. Patents and Statutory Invention Registrations: High Velocity Rotating Shaped Charge Warhead, 1986. Patents and Statutory Invention Registrations: Missile Canting Shaped Charge Warhead, 1985. Joint. NASA Group Achievement Awards (Space Station), 1992-1994. NASA Group Achievement Awards (Hubble System Review Team) 1989, 1990. Outstanding Performance Awards, 1984-1988, 1990, 1992-1997. First NASA-MSFC representative to International Space University, 1989.  

David Harrison and Kelsey Cannon from Lockheed Martin Space will present on STAT and UQ implementation lessons learned within Lockheed Martin. Faced with training 60,000 engineers in statistics, David and Kelsey formed a plan to make STAT and UQ processes the standard at Lockheed Martin. The presentation includes a range of information from initial communications plan, to obtaining leader adoption, to training engineers across the corporation. Not all programs initially accepted this process, but implementation lessons have been learned over time as many compounding successes and savings have been recorded. ©2022 Lockheed Martin, all rights reserved

Kelsey Cannon is a Senior Research Scientist at Lockheed Martin Space, previously completing a Specialty Engineering rotation program where she worked in a variety of environments and roles. Kelsey currently works with David Harrison, the statistical engineering SME at LM, to implement technical principles and a communications plan throughout the corporation. Kelsey holds a BS in Metallurgical and Materials Engineering from the Colorado School of Mines and is nearing completion of a MS in Computer Science and Data Science.

NASA will soon begin a series of tests that will collect nationally representative data on how people perceive low noise supersonic overflights. For half a century, civilian aircraft have been required to fly slower than the speed of sound over land to prevent “creating an unacceptable situation” on the ground due to sonic booms. However, new aircraft shaping techniques have led to dramatic changes in how shockwaves from supersonic flight merge together as they travel to the ground. What used to sound like a boom on the ground will be transformed into a thump. NASA is now building a full-scale, piloted demonstration aircraft called the X-59 to demonstrate low noise supersonic flight. In 2024, the X-59 aircraft will commence a national series of community overflight tests to collect data on how people perceive “sonic thumps.” The community response data will be provided to national and international noise regulators as they consider creating new standards that allow supersonic flight over land at acceptably low noise levels.

Jonathan Rathsam is a Senior Research Engineer at NASA’s Langley Research Center in Hampton, Virginia.  He conducts laboratory and field research on human perceptions of low noise supersonic overflights.  He currently serves as Technical Lead of Survey Design and Analysis for Community Test Planning and Execution within NASA’s Commercial Supersonic Technology Project.  Recently he served as co-chair for the annual Defense and Aerospace Test and Analysis Workshop (DATAWorks) and as chair for a NASA Source Evaluation Board.  He holds a Ph.D. in Engineering from the University of Nebraska, a B.A. in Physics from Grinnell College in Iowa, and completed postdoctoral research in acoustics at Ben-Gurion University in Israel.