DATAWorks Sessions Archive

Target detection in hyperspectral images (HSI) has broad value in defense applications, and neural networks have recently begun to be applied for this problem. A common criticism of neural networks is they give a point estimate with no uncertainty quantification (UQ). In defense applications, UQ is imperative because the cost of a false positive or negative is high. Users desire high confidence in either “target” or “not target” predictions, and if high confidence cannot be achieved, more inspection is warranted. One possible solution is Bayesian neural networks (BNN). Compared to traditional neural networks which are constructed by choosing a loss function, BNN take a probabilistic approach and place a likelihood function on the data and prior distributions for all parameters (weights and biases), which in turn implies a loss function. Training results in posterior predictive distributions, from which prediction intervals can be computed, rather than only point estimates. Heatmaps show where and how much uncertainty there is at any location and give insight into the physical area being imaged as well as possible improvements to the model. Using pytorch and pyro software, we test BNN on a simulated HSI scene produced using the Rochester Institute of Technology (RIT) Digital Imaging and Remote Sensing Image Generation (DIRSIG) model. The scene geometry used is also developed by RIT and is a detailed representation of a suburban neighborhood near Rochester, NY, named “MegaScene.” Target panels were inserted for this effort, using paint reflectance and bi-directional reflectance distribution function (BRDF) data acquired from the Nonconventional Exploitation Factors Database System (NEFDS). The target panels range in size from large to subpixel, with some targets only partially visible. Multiple renderings of this scene are created under different times of day and with different atmospheric conditions to assess model generalization. We explore the uncertainty heatmap for different times and environments on MegaScene as well as individual target predictive distributions to gain insight into the power of BNN.

Since its publication, Statistical Methods for Reliability Data by W. Q. Meeker and L. A. Escobar has been recognized as a foundational resource in analyzing failure time to and survival data. Along with the text, the authors provided an S-Plus software package, called SPLIDA, to help readers utilize the methods presented in the text. Today, R is the most popular statistical computing language in the world, largely supplanting S-Plus. The SMRD package is the result of a multi-year effort to completely rebuild SPLIDA, to take advantage of the improved graphics and workflow capabilities available in R. This presentation introduces the SMRD package, outlines the improvements and shows how the package works seamlessly with the rmarkdown and shiny packages to dramatically speed up your workflow. The presentation concludes with a discussion on what improvements still need to be made prior to publishing the package on the CRAN.

Uncertainty appears in many aspects of systems design including stochastic design parameters, simulation inputs, and forcing functions. Uncertainty Quantification (UQ) has emerged as the science of quantitative characterization and reduction of uncertainties in both simulation and test results. UQ is a multidisciplinary field with a broad base of methods including sensitivity analysis, statistical calibration, uncertainty propagation, and inverse analysis. Because of their ability to bring greater degrees of confidence to decisions, uncertainty quantification methods are playing a greater role in test, evaluation, and modeling and simulation in defense and aerospace. The value of UQ comes with better understanding of risk from assessing the uncertainty in test and modeling and simulation results. The presentation will provide an overview of UQ and then discuss the use of some advanced statistical methods, including DOEs and emulation for multiple simulation solvers and statistical calibration, for efficiently quantifying uncertainties. These statistical methods effectively link test, evaluation and modeling and simulation by coordinating the valuation of uncertainties, simplifying verification and validation activities.

Combining physical measurements with computational models is key to many investigations involving validation and uncertainty quantification (UQ). This talk surveys some of the many approaches taken for validation and UQ, with large-scale computational models. Experience with such applications suggests classifications of different types of problems with common features (e.g. data size, amount of empiricism in the model, computational demands, availability of data, extent of extrapolation required, etc.). More recently, social and social-technical systems are being considered for similar analyses, bringing new challenges to this area. This talk will approaches for such problems and will highlight what might be new research directions for application and methodological development in UQ.

Over the past decade, uncertainty quantification has become an integral part of engineering design and analysis. Both NASA and the DoD are making significant investments to advance the science of uncertainty quantification, increase the knowledge base, and strategically expanding its use. This increased use of uncertainty based results improves investment strategies and decision making. However, in complex systems, many challenges still exist when dealing with uncertainty in cases that have sparse, unreliable, poorly understood, and/or conflicting data. Often times, assumptions are made regarding the statistical nature of data that may not be well grounded and the impact of those assumptions is not well understood, which can lead to ill-informed decision making. This talk will focus on the quantification of uncertainty when both well characterized, aleatory, and not well known, epistemic, uncertainty sources exist. Particular focus is given to the treatment and management of epistemic uncertainty. A summary of non-probabilistic methods will be presented along with the propagation of mixed uncertainty and optimization under uncertainty. A discussion of decision making under uncertainty is also included to illustrate the use of uncertainty quantification.

The Advanced Joint Effectiveness Model (AJEM) is a joint forces model developed by the U.S. Army that is used in vulnerability and lethality (V/L) predictions for threat/target interactions. This complex model primarily generates a probability response for various components, scenarios, loss of capabilities, or summary conditions. Sensitivity analysis (SA) and uncertainty quantification (UQ), referred to jointly as SA/UQ, are disciplines that provide the working space for how model estimates changes with respect to changes in input variables. A comparative measure that will be used to characterize the effect of an input change on the predicted outcome was developed and is reviewed and illustrated in this presentation. This measure provides a practical context that stakeholders can better understand and utilize. We show graphical and tabular results using this measure.

The Boeing Company is assessing uncertainty quantification methodologies across many phases of aircraft design in order to establish confidence in computational fluid dynamics-based simulations of aircraft performance. This presentation provides an overview of several of these efforts. First, the uncertainty in aerodynamic performance metrics of a commercial aircraft at transonic cruise due to turbulence model and flight condition variability is assessed using 3D CFD with non-intrusive polynomial chaos and second order probability. Second, a sample computation of uncertainty in increments is performed for an engineering trade study, leading to the development of a new method for propagating input-uncontrolled uncertainties as well as input-controlled uncertainties. This type of consideration is necessary to account for variability associated with grid convergence on different configurations, for example. Finally, progress toward applying the computed uncertainties in forces and moments into an aerodynamic database used for flight simulation will be discussed. This approach uses a combination of Gaussian processes and multiple-fidelity Kriging meta-modeling to synthesize the required data.

This brief talk will focus on the process of human-machine trust in context of automated intelligence tools. The trust process is multifaceted and this talk will define concepts such as trust, trustworthiness, trust behavior, and will examine how these constructs might be operationalized in user studies. The talk will walk through various aspects of what might make an automated intelligence tool more or less trustworthy. Further, the construct of transparency will be discussed as a mechanism to foster shared awareness and shared intent between humans and machines.