DATAWorks Sessions Archive

The key to acquiring a cybersecure system is the ability to drive considerations about security from the operational level into the tactics and procedures. For T&E to support development and acquisition decisions, we must also adopt the perspective that a cyberattack is an attack on the mission using technology. A well-defined process model linking tools, tasks, and operators to mission performance supports this perspective. We will discuss an approach based on best practices learned from various DHS programs.

There are many testing situations that historically involve a large number of runs. The use of experimental design methods can reduce the number of runs required to obtain the information desired. Example applications include wind tunnel test campaigns, computational experiments and live fire tests. In this work we present three case studies conducted under the auspices of the Science of Test Research Consortium comparing the information obtained via a historical experimental approach with the information obtained via an experimental design approach. The first case study involves a large scale wind tunnel experimental campaign. The second involves a computational fluid dynamics model of a missile through various speeds and angles of attack. The third case involves ongoing live fire testing involve hot surface testing. In each case, results suggest a tremendous opportunity to reduce experimental test efforts without losing test information.

An important property of any experimental design is its ability to detect active factors. For supersaturated designs, in which model parameters outnumber experimental runs, power is even more critical. In this talk, we review several popular supersaturated design construction criteria and analysis methods. We then demonstrate how simulation studies can be useful for practitioners in selecting a supersaturated design with regards to power to detect active factors. One of our findings based on an extensive simulation study is that although differences clearly exist among analysis methods, most supersaturated design construction methods are indistinguishable in terms of power. This conclusion can be reassuring for practitioners as supersaturated designs can then be sensibly chosen based upon convenience. For instance, the Bayesian D-optimal supersaturated designs can be easily constructed in JMP and SAS for any run size and number of factors. On the other hand, software for constructing E(s2)-optimal supersaturated designs is not as accessible.

Over the past 10 years, a number of activities have incorporated statistical methods for the purpose of database development and associated uncertainty modeling. This presentation will highlight approaches taken in aerodynamic database development for space vehicle projects, specifically the Orion spacecraft and abort vehicle, and the Space Launch System (SLS) launch vehicle. Additionally, statistical methods have been incorporated into the Commercial Supersonic Transport Project for test technique development and optimization as well as a certification prediction methodology, which is planned to be verified with the Low-Boom Flight Demonstrator data. Discussion will conclude with the use of statistical methods for quality control and assurance in the NASA Langley Research Center ground testing facilities related to our Check Standard Project and characterization and calibration testing.

https://s3.amazonaws.com/workshop-archives-2016/IDA+Workshop+2016/testsciencemeeting.ida.org/pdfs/1b-ExperimentalDesignMethodsandApplications.pdf

Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used.

NASA has instituted requirements for establishing Agency-level safety thresholds and goals that define “long-term targeted and maximum tolerable levels of risk to the crew as guidance to developers in evaluating ‘how safe is safe enough’ for a given type of mission.” With the adoption of this policy for human space flight and with ongoing Agency efforts to increase formality in the development and review of the basis for risk acceptance, the decision-support demands placed on risk models are becoming more stringent at NASA. While these models play vital roles in informing risk acceptance decisions, they are vulnerable to incompleteness of risk identification, as well as to incomplete understanding of probabilities of occurrence, potentially leaving a substantial portion of the actual risk unaccounted for, especially for new systems. This presentation argues that management of uncertainty about the “actual” safety performance of a system must take into account the contribution of unknown and/or underappreciated (UU) risk. Correspondingly, responsible risk-acceptance decision-making requires the decision-maker to think beyond the model and address factors (e.g., organizational and management factors) that live outside traditional engineering risk models. This presentation advocates the use of a safety-case approach to risk acceptance decision-making.

Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used.

The American National Standards Institute (ANSI) maintains a set of test standards that provide methods to characterize and determine the acceptability of radiation detection systems for use in Homeland security. With a focus on the environmental, electromagnetic, and mechanical functionality tests, we describe the test formulation and discuss challenges faced in administering the standard to include the assurance of comparable evaluations across multiple test facilities and the handling of systems that provide a non-standard, unit-less response. We present proposed solutions to these difficulties that are currently being considered in updated versions of the ANSI standards. We briefly describe a decision analytic approach that could allow for the removal of minimum performance requirements from the standards and enable the end user to determine system suitability based on operation-specific requirements.

A question about whether how post-processing affects the flammability of a given metal an oxygen enriched environment could be answered with the sensitivity of a standard test that is typically used to measure differences in flammability between different metals. The principal investigator was familiar with design of experiments (DOE) and wanted to optimize the value of the information to be gained. This talk will focus on the interchange between the engineer and a statistician. Their negotiations will illustrate the process of clarifying the problem, planning the test including choosing factors and responses, running the experiment and analyzing and reporting on the data. It will focus on the choices made to help squeeze extra knowledge from each test run and leverage statistics to result in increased engineering insight.