DATAWorks Sessions Archive

NASA has instituted requirements for establishing Agency-level safety thresholds and goals that define “long-term targeted and maximum tolerable levels of risk to the crew as guidance to developers in evaluating ‘how safe is safe enough’ for a given type of mission.” With the adoption of this policy for human space flight and with ongoing Agency efforts to increase formality in the development and review of the basis for risk acceptance, the decision-support demands placed on risk models are becoming more stringent at NASA. While these models play vital roles in informing risk acceptance decisions, they are vulnerable to incompleteness of risk identification, as well as to incomplete understanding of probabilities of occurrence, potentially leaving a substantial portion of the actual risk unaccounted for, especially for new systems. This presentation argues that management of uncertainty about the “actual” safety performance of a system must take into account the contribution of unknown and/or underappreciated (UU) risk. Correspondingly, responsible risk-acceptance decision-making requires the decision-maker to think beyond the model and address factors (e.g., organizational and management factors) that live outside traditional engineering risk models. This presentation advocates the use of a safety-case approach to risk acceptance decision-making.

One use for Modeling and Simulation (M&S) in Test and Evaluation (T&E) is to produce weapon miss distances to evaluate the effectiveness of a weapons. This is true for the Air Intercept Missile-9X (AIM-9X) T&E community. Since flight testing is expensive, the test program uses relatively few flight tests at critical conditions, and supplements those data with large numbers of miss distances from simulated tests across the weapons operational space. However, before the model and simulation is used to predict performance it must first be validated. Validation is an especially daunting task when working with a limited number of live test data. In this presentation we shown that even with a limited number of live test points (e.g. 16 missile fires), we can still perform a statistical analysis for the validation. Specifically, we introduce a validation technique known as Fisher’s Combined Probability Test and we show how to apply Fisher’s test to validate the AIM-9X model and simulation.

Randomization is one of the basic principles of experimental design and the associated statistical methods. In Navy operational testing, complete randomization is often not possible due to scheduling or execution constraints. Given these constraints, operational test designers often utilize split-plot designs to accommodate the hard-to-change nature of various factors of interest. Several case studies will be presented to provide insight into the challenges associated with Navy operational test design and execution.

With nearly continuous recording of sensor values now common, a new type of data called “functional data” has emerged. Rather than the individual readings being modeled, the shape of the stream of data over time is being modeled. As an example, one might model many historical vibration-over-time streams of a machine at start-up to identify functional data shapes associated with the onset of system failure. Functional Principal Components (FPC) analysis is a new and increasingly popular method for reducing the dimensionality of functional data so that only a few FPCs are needed to closely approximate any of a set of unique data streams. When combined with Design of Experiments (DoE) methods the response to be modeled in as fewest tests as possible is now the shape of a stream of data over time. Example analyses will be shown where the form of the curve is modeled as the function of several input variables allowing one to determine the input settings associated with shapes indicative of good or poor system performance. This allows the analyst to predict the shape of the curve as a function of the input variables.

This tutorial will show how to compare and choose experimental designs based on multiple criteria. Answers to questions like “Which Design of Experiments (DOE) is better/best?” will be answered by looking at both data and graphics that show the relative performance of the designs based on multiple criteria, including; power of the designs for different model terms, how well the designs minimize predictive variance across the design space, to what level are model terms confounded or correlated, what are the relative efficiencies that measure how well coefficients are estimated or how well predictive variance is minimized. Many different case studies of screening, response surface, and screening augmented to response surface designs will be compared. Designs with both continuous and categorical factors, and with constraints on the experimental region will also be compared.

This presentation will use a notional armament system case-study to illustrate the use of M&S DOE, surrogate modeling, sensitivity analysis, multi-objective optimization and model calibration during early lifecycle development and design activities in the context of a new armament system. In addition to focusing on the statistician’s, data scientist’s, or analyst’s role and the key statistical techniques in engineering DoD systems, this presentation will also emphasize the non-statistical / engineering domain-specific aspects in a multidisciplinary design and development process which make uses of these statistical approaches at the subcomponent and subsystem-level as well as the end-to-end system modeling. A statistical engineering methodology which emphasizes the use of ‘virtual’ DOE-based model emulators developed at the subsystem-level and integrated using a systems-engineering architecture framework can yield a more tractable engineering problem compared to traditional ‘design-build-test-fix’ cycles or direct simulation of computationally expensive models. This supports a more informed prototype design for physical experimentation while providing a greater variety of materiel solutions thereby reducing development and testing cycles and time to field complex systems.

One of the major pillars of experimental design is sequential learning. The experimental design should not be viewed as a “one-shot” effort, but rather as a series of experiments where each stage builds upon information learned from the previous study. It is within this realm of sequential learning that experimentation soundly supports the application of the scientific method. This presentation illustrates the value of sequential experimentation and also the connection between the scientific method and experimentation through a discussion of a multi-stage project supported by NASA’s Engineering Safety Center (NESC) where the objective was to assess the safety of composite overwrapped pressure vessels (COPVs). The analytical team was tasked with devising a test plan to model stress rupture failure risk in carbon fiber strands that encase the COPVs with the goal of understanding the reliability of the strands at use conditions for the expected mission life. This presentation highlights the recommended experimental design for the strand tests and then discusses the benefits that resulted from the suggested sequential testing protocol.

An important property of any experimental design is its ability to detect active factors. For supersaturated designs, in which model parameters outnumber experimental runs, power is even more critical. In this talk, we review several popular supersaturated design construction criteria and analysis methods. We then demonstrate how simulation studies can be useful for practitioners in selecting a supersaturated design with regards to power to detect active factors. One of our findings based on an extensive simulation study is that although differences clearly exist among analysis methods, most supersaturated design construction methods are indistinguishable in terms of power. This conclusion can be reassuring for practitioners as supersaturated designs can then be sensibly chosen based upon convenience. For instance, the Bayesian D-optimal supersaturated designs can be easily constructed in JMP and SAS for any run size and number of factors. On the other hand, software for constructing E(s2)-optimal supersaturated designs is not as accessible.

Within NASA’s Air Traffic Management Technology Demonstration # 1 (ATD-1), Interval Management (IM) is a flight deck tool that enables pilots to achieve or maintain a precise in-trail spacing behind a target aircraft. Previous research has shown that violations of aircraft spacing requirements can occur between an IM aircraft and its surrounding non-IM aircraft when it is following a target on a separate route. This talk focuses on the experimental design and analysis of a computer experiment which models the airspace configuration of interest in order to determine airspace/aircraft conditions leading to spacing violations during IM operation. We refer to multi-layered nested continuous factors as those that are continuous and ordered in their selection; they can only be selected sequentially with a level selected for one factor affecting the range of possible values for each subsequently nested factor. While each factor is nested within another factor, the exact nesting relationships have no closed form solution. In this talk, we describe our process of engineering an appropriate space-filling design for this situation. Using this space-filling design and Gaussian process modeling, we found that aircraft delay assignments and wind profiles significantly impact the likelihood of spacing violations and the interruption of IM operations.

Small screening designs are frequently used in the initial stages of experimentation with the goal of identifying important main effects as well as to gain insight on potentially important two-factor interactions. Commonly utilized experimental designs for screening (e.g., resolution III or IV two-level fractional factorials, Plackett-Burman designs, etc.) are unreplicated and as such, provide no unbiased estimate of experimental error. However, if statistical inference is considered an integral part of the experimental analysis, one view is that inferential procedures should be performed using the unbiased pure error estimate. As full replication of an experiment may be quite costly, partial replication offers an alternative for obtaining a model independent error estimate. Gilmour and Trinca (2012, Applied Statistics) introduce criteria for the design of optimal experiments for statistical inference (providing for the optimal selection of replicated design points). We begin with an extension of their work by proposing a Bayesian criterion for the construction of partially replicated screening designs with less dependence on an assumed model. We then consider the use of the proposed criterion within the context of multi-criteria design selection where estimation and protection against model misspecification are considered. Insights for analysis and model selection in light of partial replication will be provided.

Physics-based simulation of nondestructive evaluation (NDE) inspection can help to advance the inspectability and reliability of mechanical systems. However, NDE simulations applicable to non-idealized mechanical components often require large compute domains and long run times. This has prompted development of custom NDE simulation software tailored to high performance computing (HPC) hardware. Verification and validation (V&V) is an integral part of developing this software to ensure implementations are robust and applicable to inspection problems, producing tools and simulations suitable for computational NDE research. This presentation addresses factors common to V&V of several elastodynamic simulation codes applicable to ultrasonic NDE. Examples are drawn from in-house simulation software at NASA Langley Research Center, ranging from ensuring reliability in a 1D heterogeneous media wave equation solver to the V&V needs of 3D cluster-parallel elastodynamic software. Factors specific to a research environment are addressed, where individual simulation results can be as relevant as the software product itself. Distinct facets of V&V are discussed including testing to establish software reliability, employing systematic approaches for consistency with fundamental conservation laws, establishing the numerical stability of algorithms, and demonstrating concurrence with empirical data. This talk also addresses V&V practices for small groups of researchers. This includes establishing resources (e.g. time and personnel) for V&V during project planning to mitigate and control the risk of setbacks. Similarly, we identify ways for individual researchers to use V&V during simulation software development itself to both speed up the development process and reduce incurred technical debt.

For chemical/biological testing, test chambers are sometimes designed with a vapor or aerosol homogeneity requirement. For example, a test community may require that the difference in concentration between any two test locations in a chamber be no greater than 20 percent. To validate the chamber, testers must demonstrate that such a requirement is met with a specified amount of certainty, such as 80 percent. With a validated chamber, multiple systems can be simultaneously tested at different test locations with the assurance that each system is exposed to nearly the same concentration. In some cases, however, homogeneity requirements are difficult to achieve. This presentation demonstrates a valid Bayesian method for testing probability of detection as a function of concentration in a chamber that fails to meet a homogeneity requirement. The demonstrated method of analysis is based on recent experience with an actual test chamber. Multiple systems are tested simultaneously at different locations in the chamber. Because systems tested in the chamber are exposed to different concentrations depending on these locations, the differences must be quantified to the greatest extent possible. To this end, data from the failed validation efforts are used to specify informative prior distributions for probability-of-detection modeling. Because these priors quantify and incorporate uncertainty in model parameters, they ensure that the final probability-of-detection model constitutes a valid comparison of the performance of the different systems.

Many software reliability models characterize the number of faults detected during the testing process as a function of testing time, which is performed over multiple stages. Typically, the later stages are progressively more expensive because of the increased number of personnel and equipment required to support testing as the system nears completion. Such transitions from one stage of testing to the next change in the operational environment. One statistical approach to combine software reliability growth models in a manner capable of characterizing multi-stage testing is the concept of a change-point process, where the intensity of a process experiences a distinct change at one or more discrete times during testing. Thus, change-point processes can be used to model change in the failure rate of software due to changes in the testing strategy and environment, integration testing, and resource allocation as it proceeds through multiple stages of testing. This presentation generalizes change-point models to the heterogeneous case, where fault detection before and after a change-point can be characterized by distinct nonhomogeneous Poisson processes (NHPP). Experimental results suggest that heterogeneous change-point models better characterize some failure data sets, which can improve the applicability of software reliability models to large-scale software systems that are tested over multiple stages.