DATAWorks Sessions Archive

When developing a system, it is important to consider system performance from a user perspective. This can be done through operational testing—assessing the ability of representative users to satisfactorily accomplish tasks or missions with the system in operationally-representative environments. This process can be expensive and time-consuming, but is critical for evaluating a system. We show how an existing design of experiments (DOE) process for operational testing can be leveraged to construct a Bayesian adaptive design. This method, nested within the larger design created by the DOE process, allows interim analyses using predictive probabilities to stop testing early for success or futility. Furthermore, operational environments with varying probabilities of encountering are directly used in product evaluation. Representative simulations demonstrate how these interim analyses can be used in an operational test setting, and reductions in necessary test events are shown. The method allows for using either weakly informative priors when data from previous testing is not available, or for priors built using developmental testing data when it is available. The proposed method for creating priors using developmental testing data allows for more flexibility in which data can be incorporated into analysis than the current process does, and demonstrates that it is possible to get more precise parameter estimates. This method will allow future testing to be conducted in less time and at less expense, on average, without compromising the ability of the existing process to verify the system meets the user’s needs.

To identify the robust settings of the control factors, it is very important to understand how they interact with the noise factors. In this article, we propose space-filling designs for computer experiments that are more capable of accurately estimating the control-by-noise interactions. Moreover, the existing space-filling designs focus on uniformly distributing the points in the design space, which are not suitable for noise factors because they usually follow non-uniform distributions such as normal distribution. This would suggest placing more points in the regions with high probability mass. However, noise factors also tend to have a smooth relationship with the response and therefore, placing more points towards the tails of the distribution is also useful for accurately estimating the relationship. These two opposing effects make the experimental design methodology a challenging problem. We propose optimal and computationally efficient solutions to this problem and demonstrate their advantages using simulated examples and a real industry example involving a manufacturing packing line.

Two responses to an expensive, time consuming test on a final product will be referred to as “adhesion” and “strength”. A screening test was performed on compounds that comprise the final product. These screening tests are multivariate profile measurements. Previous models to predict the expensive, time consuming test lacked accuracy and precision. Data visualization was used to guide a statistical engineering model that makes use of multiple statistical techniques. The modeling approach raised some interesting statistical questions for partial least square models regarding over-fitting and cross validation. Ultimately, the model interpretation and the visualization both make engineering sense and led to interesting insights regarding the product development program and screening compounds.

We have developed a new sensor fusion approach called Doppler Assisted Sensor Fusion (DASF), which pairs a range rate profile from one moving sensor with location accuracy with another range rate profile from another sensor with high location accuracy. This paring provides accurate identification, location, and tracking of moving emitters, with low association latency. The approach we use for data fusion is distinct from previous approaches. In the conventional approach, post detection data from the each sensor is overlaid with data from another sensor in an attempt to associate the data outputs. For the DASF approach the fusion is at the sensor level, the first sensor collects data and provides the standard identification in addition a unique emitter range rate profile. This profile is used to associate the emitter signature to a range-rate signature obtained by the geolocation sensor. The geolocation sensor then provides the desired location accuracy. We will provide results using real tracking data scenarios.

Aerodynamic databases for space flight vehicles rely on wind-tunnel tests utilizing precision force measurement systems (FMS). Recently, NASA’s Space Launch System (SLS) program has conducted numerous wind-tunnel testing. This presentation will focus on the calibration of a unique booster FMS through the use of design of experiments (DoE) and regression modeling. Utilization of DoE resulted in a sparse, time-efficient, design with results exceeding researcher’s expectations.

In an experiment with a single factor with three levels, treatments A, B, and C, a single treatment is to be applied to each of several experimental units selected from some set of units. The response variable is continuous, and differences in its value show the relative effectiveness of the treatments. An experimental design will dictate which treatment is applied to what units. Since differences in the response variable are used to judge differences between treatments, the most important goal of the design is to prevent the treatment effect being masked by some unrelated property of the experimental units. Second important function of the design is to ensure power, that is, that if the treatments are not equally effective, the differences in the response variable are likely to be larger than background noise. Classical experimental design theory uses three principles: replication, randomization, and blocking, to produce an experimental design. Replication refers to how many units are used, blocking is a possible grouping of the units to reduce between unit heterogeneity, and randomization governs the assignment of units to treatment. Classical experimental designs are balanced as much as possible, that is, the three treatments are applied the same number of times, in each potential block of units. Bayesian experimental design aims to make use of additional related information, often called prior information, to produce a design. The information may be in the form of related experimental results, for example, treatments A and B may have been previously studied. It could be additional information about the experimental units, or about the response variable. This additional information could be used to change the usual blocking, to reduce the number of units assigned to treatments A and B compared to C, and/or reduce the total number of units needed to ensure power. This talk aims to explain Bayesian design concepts and illustrate them on realistic examples.

The Icing Research Tunnel (IRT) at NASA Glenn Research Center follows the recommended practice for icing tunnel calibration outlined in SAE’s ARP5905 document. The calibration team has followed the schedule of a full calibration every five years with a check calibration done every six months following. The liquid water content of the IRT has maintained stability within in the specifications presented to customers that the variation is within +/- 10% of the calibrated, target measurement. With recent measurements and instrumentation errors, a more thorough assessment of error source was desired. By constructing statistical process control charts, the ability to determine how the instrument varies in the short term, mid term, and long term was gained. The control charts offer a view of instrument error, facility error, or installation changes. It was discovered that there was a shift from target to mean baseline thus leading to the study of the overall capability indices of the liquid water content measuring instrument to perform within specifications defined in the IRT. This presentation describes data processing procedures for the Multi-Element Sensor in the IRT, including collision efficiency corrections, canonical correlation analysis, Chauvenet’s Criterion for rejection of data, distribution check of data, and mean, median and mode for construction of control charts. Further data is presented to describe the repeatability of the IRT with the Multi-Element Sensor and the ability to maintain a stable process for the defined calibration schedule.

The U.S. Army Research Laboratory’s (ARL) Essential Research Program (ERP) on Artificial Intelligence & Machine Learning (AI & ML) seeks to research, develop and employ a suite of AI-inspired and ML techniques and systems to assist teams of soldiers and autonomous agents in dynamic, uncertain, complex operational conditions. Systems will be robust, scalable, and capable of learning and acting with varying levels of autonomy, to become integral components of networked sensors, knowledge bases, autonomous agents, and human teams. Three specific research gaps will be examined: (i) Learning in Complex Data Environments, (ii) Resource-constrained AI Processing at the Point-of-Need and (iii) Generalizable & Predictable AI. The talk will highlight ARL’s internal research efforts over the next 3-5 years that are connected, cumulative and converging to produce tactically-sensible AI-enabled capabilities for decision making at the tactical edge, specifically addressing topics in: (1) adversarial distributed machine learning, (2) robust inference & machine learning over heterogeneous sources, (3) adversarial reasoning integrating learned information, (4) adaptive online learning and (5) resource-constrained adaptive computing. The talk will also highlight collaborative research opportunities in AI & ML via ARL’s Army AI Innovation Institute (A2I2) which will harness the distributed research enterprise via the ARL Open Campus & Regional Campus initiatives.

The Air Force (AF) Human Systems Integration (HSI) program is led by the 711th Human Performance Wing’s Human Systems Integration Directorate (711 HPW/HP). 711 HPW HP provides direct support to system program offices and AF Major Commands (MAJCOMs) across the acquisition lifecycle from requirements development to fielding and sustainment in addition to providing home office support. With an ever-increasing demand signal for support, HSI practitioners within 711 HPW/HP assess HSI domain areas for human-centered risks and strive to ensure systems are designed and developed to safely, effectively, and affordably integrate with human capabilities and limitations. In addition to system program offices and MAJCOMs, 711 HPW/HP provides HSI support to AF Centers (e.g., AF Sustainment Center, AF Test Center), the AF Medical Service, and special cases as needed. The AF Global Strike Command (AFGSC) is the largest MAJCOM with several Programs of Record (POR), such as the B-1, B-2, and B-52 bombers, Intercontinental Ballistic Missiles (ICBM), Ground-Based Strategic Deterrent (GBSD), Airborne Launch Control System (ALCS), and other support programs/vehicles like the UH-1N. Mr. Anthony Thomas (711 HPW/HP), the AFGSC HSI representative, will discuss how 711 HPW/HP supports these programs at the MAJCOM headquarters level and in the system program offices.

Model Validation for Simulations of CVN-78 Sortie Generation As part of the test planning process, IDA is examining flight operations on the Navy’s newest carrier, CVN-78. The analysis uses a model, the IDA Virtual Carrier Model (IVCM), to examine sortie generation rates and whether aircraft can complete missions on time. Before using IVCM, it must be validated. However, CVN-78 has not been delivered to the Navy, and data from actual operations are to validate the model. Consequently, we will validate IVCM by comparing it to another model. This is a reasonable approach when a model is used in general analyses such as test planning, but is not acceptable when a model is used in the assessment of system effectiveness and suitability. The presentation examines the use of various statistical tools – Wilcoxon Rank Sum Test, Kolmogorov-Smirnov Test, and lognormal regression – to examine whether the results from two models provide similar results and to quantify the magnitude of any differences. From the analysis, IDA concluded that locations and distribution shapes are consistent, and that the differences between the models are less than 15 percent, which is acceptable for test planning.

Binary response experiments are common in epidemiology, biostatistics as well as in military applications. The Up and Down method, Langlie’s Method, Neyer’s method, K in a Row method and 3 Phase Optimal Design are methods used for sequential experimental design when there is a single continuous variable and a binary response. During this talk, we will discuss a new sequential experimental design approach called the Break Separation Method (BSM). BSM provides an algorithm for determining sequential experimental trials that will be used to find a median quantile and fit a logistic regression model using Maximum Likelihood estimation. BSM results in a small sample size and is designed to efficiently compute the median quantile.

Although reliability analysis is a part of Operational Test and Evaluation, it is uncommon for analysts to have a background in reliability theory or experience applying it. This presentation highlights some lessons learned from reliability analysis conducted on several AFOTEC test programs. Topics include issues related to censored data, limitations and alternatives to using the exponential distribution, and failure rate analysis using test data.

Several optimization models are described for allocating resources to different testing activities in a system’s reliability growth program. These models assume availability of an underlying reliability growth model for the system, and capture the tradeoffs associated with focusing testing resources at various levels (e.g., system, subsystem, component) and/or how to divide resources within a given level. In order to demonstrate insights generated by solving the model, we apply the optimization models to an example series-parallel system in which reliability growth is assumed to follow the Crow/AMSAA reliability growth model. We then demonstrate how the optimization models can be extended to incorporate uncertainty in Crow/AMSAA parameters.