DATAWorks Sessions Archive

Two responses to an expensive, time consuming test on a final product will be referred to as “adhesion” and “strength”. A screening test was performed on compounds that comprise the final product. These screening tests are multivariate profile measurements. Previous models to predict the expensive, time consuming test lacked accuracy and precision. Data visualization was used to guide a statistical engineering model that makes use of multiple statistical techniques. The modeling approach raised some interesting statistical questions for partial least square models regarding over-fitting and cross validation. Ultimately, the model interpretation and the visualization both make engineering sense and led to interesting insights regarding the product development program and screening compounds.

To identify the robust settings of the control factors, it is very important to understand how they interact with the noise factors. In this article, we propose space-filling designs for computer experiments that are more capable of accurately estimating the control-by-noise interactions. Moreover, the existing space-filling designs focus on uniformly distributing the points in the design space, which are not suitable for noise factors because they usually follow non-uniform distributions such as normal distribution. This would suggest placing more points in the regions with high probability mass. However, noise factors also tend to have a smooth relationship with the response and therefore, placing more points towards the tails of the distribution is also useful for accurately estimating the relationship. These two opposing effects make the experimental design methodology a challenging problem. We propose optimal and computationally efficient solutions to this problem and demonstrate their advantages using simulated examples and a real industry example involving a manufacturing packing line.

When developing a system, it is important to consider system performance from a user perspective. This can be done through operational testing—assessing the ability of representative users to satisfactorily accomplish tasks or missions with the system in operationally-representative environments. This process can be expensive and time-consuming, but is critical for evaluating a system. We show how an existing design of experiments (DOE) process for operational testing can be leveraged to construct a Bayesian adaptive design. This method, nested within the larger design created by the DOE process, allows interim analyses using predictive probabilities to stop testing early for success or futility. Furthermore, operational environments with varying probabilities of encountering are directly used in product evaluation. Representative simulations demonstrate how these interim analyses can be used in an operational test setting, and reductions in necessary test events are shown. The method allows for using either weakly informative priors when data from previous testing is not available, or for priors built using developmental testing data when it is available. The proposed method for creating priors using developmental testing data allows for more flexibility in which data can be incorporated into analysis than the current process does, and demonstrates that it is possible to get more precise parameter estimates. This method will allow future testing to be conducted in less time and at less expense, on average, without compromising the ability of the existing process to verify the system meets the user’s needs.

Over the past 10 years, a number of activities have incorporated statistical methods for the purpose of database development and associated uncertainty modeling. This presentation will highlight approaches taken in aerodynamic database development for space vehicle projects, specifically the Orion spacecraft and abort vehicle, and the Space Launch System (SLS) launch vehicle. Additionally, statistical methods have been incorporated into the Commercial Supersonic Transport Project for test technique development and optimization as well as a certification prediction methodology, which is planned to be verified with the Low-Boom Flight Demonstrator data. Discussion will conclude with the use of statistical methods for quality control and assurance in the NASA Langley Research Center ground testing facilities related to our Check Standard Project and characterization and calibration testing.

Scientific computing has undergone extraordinary growth in sophistication in recent years, enabling the simulation of a wide range of complex multiphysics and multiscale phenomena. Along with this increase in computational capability is the growing recognition that uncertainty quantification (UQ) must go hand-in-hand with numerical simulation in order to generate meaningful and reliable predictions for engineering applications. If not rigorously considered, uncertainties due to manufacturing defects, material variability, modeling assumptions, etc. can cause a substantial disconnect between simulation and reality. Packaging these complex computational models within an UQ framework, however, can be a significant challenge due to the need to repeatedly evaluate the model when even a single evaluation is time-consuming. This talk discusses efforts at NASA Langley Research Center (LaRC) to enable rapid UQ for problems with expensive computational models. Under the High Performance Computing Incubator (HPCI) program at LaRC, several open-source software libraries are being developed and released to provide access to general-purpose, state-of-the-art UQ algorithms. The common denominator of these methods is that they all expose parallelism among the model evaluations needed for UQ and, as such, are implemented to leverage HPC resources when available to achieve tremendous computational speedup. While the methods and software presented are broadly applicable, they will be demonstrated in the context of applications that have particular interest at NASA, including structural health management and trajectory simulation.

Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.

Over the past decade, uncertainty quantification has become an integral part of engineering design and analysis. Both NASA and the DoD are making significant investments to advance the science of uncertainty quantification, increase the knowledge base, and strategically expanding its use. This increased use of uncertainty based results improves investment strategies and decision making. However, in complex systems, many challenges still exist when dealing with uncertainty in cases that have sparse, unreliable, poorly understood, and/or conflicting data. Often times, assumptions are made regarding the statistical nature of data that may not be well grounded and the impact of those assumptions is not well understood, which can lead to ill-informed decision making. This talk will focus on the quantification of uncertainty when both well characterized, aleatory, and not well known, epistemic, uncertainty sources exist. Particular focus is given to the treatment and management of epistemic uncertainty. A summary of non-probabilistic methods will be presented along with the propagation of mixed uncertainty and optimization under uncertainty. A discussion of decision making under uncertainty is also included to illustrate the use of uncertainty quantification.

This presentation discusses a methodology to use knowledge of cyber attacks and defender responses from operational assessments to gain insights into the defensive posture and to inform a strategy for improvement. The concept is to use the attack thread as the instrument to probe and measure the detection capability of the cyber defenses. The data enable a logistic regression approach to provide a quantitative basis for the analysis and recommendations.

This paper presents the results of two hypervelocity impact failure probabilistic risk assessments for critical wire bundles exposed aboard the Joint Polar Satellite System (JPSS-1) to an increased orbital debris environment at its 824 km, 98.8 deg inclination orbit. The first “generic” approach predicted the number of wires broken by orbital debris ejecta emerging from normal impact with multi-layer insulation (MLI) covering 36-, 18-, and 6-strand wire bundles at a 5 cm standoff using the Smooth Particle Hydrodynamic (SPH) code. This approach also included a mathematical approach for computing the probability that redundant wires were impacted then severed within the bundle. Based in part on the high computed risk of a critical wire bundle failure from the generic approach, an enhanced orbital debris protection design was examined, consisting of betacloth-reinforced MLI suspended at a 5 cm standoff over a seven layer betacloth and Kevlar blanket, draped over the exposed wire bundles. A second SPH-based risk assessment was conducted that also included the beneficial effects from the high (75 degree) obliquity of orbital debris impact and shadowing by other spacecraft components, and resulted in a considerably reduced likelihood of critical wire bundle failure compared to the original baseline design.

One of the most powerful features of Bayesian analyses is the ability to combine multiple sources of information in a principled way to perform inference. For example, this feature can be particularly valuable in assessing the reliability of systems where testing is limited for some reason (e.g., expense, treaty). At their most basic, Bayesian methods for reliability develop informative prior distributions using expert judgment or similar systems. Appropriate models allow the incorporation of many other sources of information, including historical data, information from similar systems, and computer models. I will introduce the approach and then consider examples from defense acquisition and lifecycle extension, focusing on the strengths and weaknesses of the Bayesian analyses.

NASA’s Airspace Technology Demonstration (ATD) project was developed to facilitate the transition of mature air traffic management technologies from the laboratory to operational use. The first ATD focused on an integrated set of advanced NASA technologies to enable efficient arrival operations in high-density terminal airspace. This integrated arrival solution was validated and verified in laboratories and transitioned to a field prototype for an operational demonstration. Within NASA, this was a collaborative effort between Ames and Langley Research Centers involving a multi-year iterative experimentation process consisting of a series of sequential batch computer simulations and human-in-the-loop experiments, culminating in a flight test. Designing and analyzing the flight test involved a number of statistical challenges. There were several variables which are known to impact the performance of the system, but which could not be controlled in an operational environment. Changes in the schedule due to weather and the dynamic positioning of the aircraft on the arrival routes resulted in the need for a design that could be modified in real-time. This presentation describes a case study from a recent NASA flight test, highlights statistical challenges, and discusses lessons learned.

Traffic Aware Strategic Aircrew Requests (TASAR) is a NASA-developed operational concept for flight efficiency and route optimization for the near-term airline flight deck. TASAR provides the aircrew with a cockpit automation tool that leverages a growing number of information sources on the flight deck to make fuel- and time-saving route optimization recommendations while in route. In partnership with a commercial airline, a research prototype software that implements TASAR has been installed on three aircraft to enable the evaluation of this software in operational use. During the flight trials, data are being collected to quantify operational performance, which will enable NASA to improve algorithms and enhance functionality in the software based on real-world user experience. This presentation highlights statistical challenges and discusses lessons learned during the initial stages of the operational evaluation.

Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used.