DATAWorks Sessions Archive

In this case study we demonstrate an application of Pratt & Whitney’s “Design for Variation” discipline applied to the task of a roller bearing design. The ultimate goal, in this application, was to utilize test data from the “real world” to calibrate a computer model used for design and ensure that roller bearing designs obtained from this model were optimized for maximum robustness to major sources of variation in bearing manufacture and operation. The “Design for Variation” process provides engineers with many useful analysis results even before real world data is applied: high fidelity sensitivity analysis, uncertainty analysis (quantifying a baseline risk of failing to meet design intent) and model verification. The combining of real world data and Bayesian statistical methods that Design for Variation employs to calibrate models, however, goes a step further, validating the accuracy of the model’s outputs and quantifying any bias between the model and the real world. As a result of this application, the designers were able to identify the sources of the bias and correct the model’s physics-based aspects to more accurately model reality. The improved model is now integrated into all successive bearing design activities. The benefits of this method, which only required a small amount of high quality test data, are now available to all present and future roller bearing designs.

Co-authors: Adam L. Pintar, Blaza Toman, and Dennis Leber. A task of the Domestic Nuclear Detection Office (DNDO) is the evaluation of radiation and nuclear (rad/nuc) detection systems used to detect and identify illicit rad/nuc materials. To obtain estimated system performance measures, such as probability of detection, and to determine system acceptability, the DNDO sometimes conduct large scale field tests of these systems at great cost. Typically, non adaptive designs are employed where each rad/nuc test source is presented to each system under test a predetermined and fixed number of times. This approach can lead to unnecessary cost if the system is clearly acceptable or unacceptable. In this presentation, an adaptive design with Bayesian decision theoretic foundations is discussed as an alternative to, and contrasted with, the more common single stage design. Although the basis of the method is Bayesian decision theory, designs may be tuned to have desirable type I and II error rates. While the focus of the presentation is a specific DNDO example, the method is applicable widely. Further, since constructing the designs is somewhat compute intensive, software in the form of an R package will be shown and is available upon request.

Modeling and simulation (M&S) is often an important element of operational evaluations of effectiveness, suitability, survivability, and lethality. For example, the testing of new systems designed to operate against advanced foreign threats, as well as the testing of systems of systems, will involve the use of M&S to examine scenarios that cannot be created using live testing. In order to have an adequate understanding of, and confidence in, the results obtained from M&S, statistically rigorous techniques should be applied to the validation process wherever possible. Design of experiments methodologies should be employed to determine what live and simulation data are needed to support rigorous validation, and formal statistical tests should be used to compare live and simulated data. This talk will discuss the importance of M&S in operational testing through a few examples, and outline several statistically rigorous techniques for validation.