DATAWorks Sessions Archive

Co-Author: Melanie Luperon.

Most software reliability growth models only track defect discovery. However, a practical  concern is removal of high severity defects, yet defect removal is often assumed to occur instantaneously. More recently, several defect removal models have been formulated as differential equations in terms of the number of defects discovered but not yet resolved and the rate of resolution. The limitation of this approach is that it does not take into consideration data contained in a defect tracking database.

This talk describes our recent efforts to analyze data from a NASA program. Two methods to model defect resolution are developed, namely (i) distributional and (ii) Markovian approaches. The distributional approach employs times between defect discovery and resolution to characterize the mean resolution time and derives a software defect resolution model from the corresponding software reliability growth model to track defect discovery. The Markovian approach develops a state model from the stages of the software defect lifecycle as well as a transition probability matrix and the distributions for each transition, providing a semi-Markov model. Both the distribution and Markovian approaches employ a censored estimation technique to identify the maximum likelihood estimates, in order to handle the case where some but not all of the defects discovered have been resolved. Furthermore, we apply a hypothesis test to determine if a first or second order Markov chain best characterizes the defect lifecycle. Our results indicate that a first order Markov chain was sufficient to describe the data considered and that the Markovian approach achieves modest improvements in predictive accuracy, suggesting that the simpler distributional approach may be sufficient to characterize the software defect resolution process during test. The practical inferences of such models include an estimate of the time required to discover and remove all defects.

The DOD’s ability to collect data is far outstripping its ability to convert that data into actionable information. This problem is not unique to the military. With vast improvements in our ability to collect data, many organizations are drowning in that data. As a result, the need for advanced algorithms (Artificial Intelligence, Machine Learning, etc.) to support human work has never been greater. However, algorithm performance and the performance of the larger work system, composed of human and machine agents, are not synonymous. In order to build and field high performance work systems that consistently provide positive mission impact, the S&T community must measure the performance of the entire Joint Cognitive System (JCS). Algorithm performance, by itself, is necessary but not sufficient for predicting the performance of these systems in the field. Better predicting JCS performance once a system is fielded requires modelling, or at least learning about, the humans’ cognitive work and how the proposed technology will change that work. For example, providing practitioners with a new Machine Learning enabled support tool may impose unintended cognitive work as practitioners calibrate themselves to the performance envelope of the new tools. In addition, the practitioners’ cognitive work is shaped by many “soft constraints” which are often not captured by technologists. For example, real world time constraints may lead practitioners to use simple decision making heuristics, rather than deliberative hypothesis testing, when making critical decisions. The technology requirements needed to support both deliberate and heuristic decision making may be very different. This talk will discuss several approaches to gain insight on JCS performance as part of a larger cycle of iteratively discovering, building, and testing these systems. Additionally, this talk will give an overview of a small pilot study conducted by the Air Force Research Lab to measure the impact of a simulated Machine Learning agent designed to support AF intelligence analysts exploiting Full Motion Video data.

Ballistic limit testing of armor is testing in which a kinetic energy threat is shot at armor at varying velocities. The striking velocity and whether the threat completely penetrated or partially penetrated the armor is recorded. The probability of penetration is modeled as a function of velocity using a generalized linear model. The parameters of the model serve as inputs to MUVES which is a DoD software tool used to analyze weapon system vulnerability and munition lethality.

Generally, the probability of penetration is assumed to be monotonically increasing with velocity. However, in cases in which there is a change in penetration mechanism, such as the shatter gap phenomena, the probability of penetration can no longer be assumed to be monotonically increasing and a more complex model is necessary. One such model was developed by Chang and Bodt to model the probability of penetration as a function of velocity over a velocity range in which there are two penetration mechanisms.

This paper proposes a D-optimally based sequential shot selection method to efficiently select threat velocities during testing. Two cases are presented: the case in which the penetration mechanism for each shot is known (via high-speed or post shot x-ray) and the case in which the penetration mechanism is not known. This method may be used to support an improved evaluation of armor performance for cases in which there is a change in penetration mechanism.

Equipment Failure Reports (EFRs) describe equipment failures and the steps taken as a result of these failures. EFRs contain both structured and unstructured data. Currently, analysts manually read through EFRs to understand failure modes and make recommendations to reduce future failures. This is a tedious process where important trends and information can get lost. This motivated the creation of an interactive dashboard that extracts relevant information from the unstructured (i.e. free-form text) data and combines it with structured data like failure date, corrective action and part number. The dashboard is an RShiny application that utilizes numerous text mining and visualization packages, including tm, plotly, edgebundler, and topicmodels. It allows the end-user to filter to the EFRs that they care about and visualize meta-data, such as geographic region where the failure occurred, over time allowing previously unknown trends to be seen. The dashboard also applies topic modeling to the unstructured data to identify key themes. Analysts are now able to quickly identify frequent failure modes and look at time and region-based trends in these common equipment failures. DISTRIBUTION STATEMENT A – APPROVED FOR PUBLIC RELEASE; DISTRIBUTION IS UNLIMITED.

This study assesses the relationship between psychometric screening of candidates for our partner special operations unit and successful completion of the unit’s candidate selection program. Improving the candidate selection program is our primary goal for this study. Our partner unit maintains a comprehensive database of summary information on previous candidates but has not yet conducted a robust analysis of candidate attributes. We sought to achieve this goal by using statistical methods to determine predictors associated with successful completion of the selection program. Our results suggest that we may identify predictors associated with success but may struggle in constructing an effective predictive model due to the inherent differences of candidates. Our predictors are scales from standardized psychometric evaluations administered by the selection program intended originally to identify candidates with psychopathologies. Our outcome is a binary variable indicating successful completion of the program. Analyzing the demographics of the candidate selection population is our secondary goal for this study. Our analysis helped our partner unit improve its selection program by identifying characteristics associated with success. Other members of the special operations community may generalize our results to improve their respective programs as well. We do not intend for units to use these results to draw definitive conclusions regarding the ability of a candidate to pass a selection program.

The challenge of identifying test cases that maximize the chance of discovering faults at minimal cost is a challenge that software test engineers constantly face. Combinatorial testing is an effective test case selection strategy to address this challenge. The idea is to select test cases that ensure that all possible combinations of settings from two (or more) inputs are accounted for, regardless of which subset of inputs are selected. This is accomplished by using a covering array as the test case selection mechanism. However, for any given testing scenario, there are usually several alternative covering arrays that may satisfy budgetary and coverage requirements. Unfortunately, it is often unclear how to choose from these alternatives. Practitioners often want a way to explore how the input space is being covered before deciding which alternative is best suited for the testing scenario. In this presentation we will provide an overview of a new graphical method that may be used to visualize the input space of covering arrays. We use the phrase “design fractals” to refer to this graphical method.

Supervised learning competitions such as those hosted by Kaggle provide a quantitative, objective way to evaluate and compare algorithms. However, the typical format of evaluating competitors against a fixed set of data can encourage overfitting to that particular set. If the competition host is interested in motivating development of approaches that will perform well against a more general “in the wild” problem outside of the competition, the winning algorithms of these competitions might fall short because they’re tuned too closely to the competition data. Furthermore, the idea of having a single final ranking of the competitors based on the competition data set ignores the possibility that that ranking might change substantially if a different data set were used, even one that has the same statistical characteristics as the original.

We present an approach for designing training and test sets that reward more robust algorithms and discourage overfitting with the intent of improved performance for scenarios beyond the competition. These carefully designed sets also enable a rich and nuanced analysis and comparison of the performance of competing algorithms, including a more flexible final ranking. We illustrate these methods with two competitions recently designed and hosted by Los Alamos National Laboratory to improve detection, identification, and location of radiological threats in urban environments.

The Transportation Security Administration (TSA) uses several types of screening technologies for the purposes of threat detection at airports and federal facilities across the country.   Computed Tomography (CT) systems afford TSA personnel in the Checked Baggage setting a quick and effective method to screen property with less need to physically inspect property due to their advanced imaging capabilities.  Recent reductions in size, cost, and processing speed for CT systems spurred an interest in incorporating these advanced imaging systems at the Checkpoint to increase the speed and effectiveness of scanning personal property as well as passenger satisfaction during travel.  The increase in speed and effectiveness of scanning personal property with fewer physical property inspections stems from several qualities native to CT imaging that current 2D X-Ray based Advanced Technology 2 (AT2) systems typically found at Checkpoints lack.  Specifically, the CT offers rotatable 3D images and advanced identification algorithms that allow TSA personnel to more readily identify items requiring review on-screen without requesting that passengers remove them from their bag. The introduction of CT systems at domestic airports led to the identification of a few key Human Factors issues, however.  Several vendors used divergent strategies to produce the CT systems introduced at domestic airport Checkpoints.  Each system offered users different 3D visualizations, informational displays, and identification algorithms, offering a range of views, tools, layouts, and material colorization for users to sort through. The disparity in system similarity and potential for multiple systems to operate at a single airport resulted in unnecessarily complex training, testing, certification, and operating procedures.  In response, a group of human factors engineers (HFEs) was tasked with creating requirements for a single common Graphical User Interface (GUI) for all CT systems that would provide a standard look, feel, and interaction across systems. We will discuss the development and analytic process used to 1.) gain an understanding of the tasks that CT systems must accomplish at the Checkpoint (i.e. focus groups), 2.) identify what tools Transportation Security Officers (TSOs) tend to use and why (i.e. focus groups and rank-ordered surveys), and 3.) determine how changes during iterative testing effects performance (i.e. A/B testing while collecting response time, accuracy, and tool usage). The data collection effort described here resulted in a set of requirements that produced a highly usable CT interface as measured by several valid and reliable objective and subjective measures.  Perceptions of the CGUI’s usability (e.g., the System Usability Scale; SUS) were aligned with TSO performance (i.e., P_d, P_FA, and Throughput) during use of the CGUI prototype. Iterative testing demonstrated an increase in the SUS score and performance measures for each revision of the requirements used to produce the common CT interface.  User perspectives, feedback, and performance data also offered insight toward the determination of necessary future efforts that will increase user acceptance of the redesigned CT interface.  Increasing user acceptance offers TSA the opportunity to improve user engagement, reduces errors, and the likelihood that the system will stay in service without a mandate.

Recent advances in brain computer interfaces (BCI) has brought interest in exploring the possibility of BCI like technology for future armament systems. In an experiment, conducted at the Tactical Behavioral Lab (TBRL), electroencephalogram (EEG) data was collected during simulated engagement scenarios in order to study the relationship between a soldier’s state of mind and his biophysical signals. One of the goals was to determine if it was possible to anticipate the decision to fire a gun. The nature of EEG data presents a unique set of challenges. For example, the high sensitivity of EEG electrodes coupled with their close proximity on the scalp means recorded data is noisy and highly correlated. Special attention needs to be payed to data pre-processing and feature engineering in building predictive models.

A common challenge in the cybersecurity realm is the proper handling of high-volume streaming data. Typically in this setting, analysts are restricted to techniques with computationally cheap model-fitting and prediction algorithms. In many situations, however, it would be beneficial to use more sophisticated techniques. In this talk, a general framework is proposed that adapts a broad family of statistical and machine learning techniques to the streaming setting. The techniques of interest are those that can generate computationally cheap predictions, but which require iterative model-fitting procedures. This broad family of techniques includes various clustering, classification, regression, and dimension reduction algorithms. We discuss applied and theoretical issues that arise when using these techniques for streaming data whose distribution is evolving over time.

Electronic warfare systems generate diverse signals in a complex environment. This briefing covers how a particular system employed DOE to develop the overall strategy, determined which tests would utilize DOE, and generated specific designs. It will cover the technical and resource challenges encountered throughout the process leading up to testing. We will discuss the difficulty in defining responses and factors and ideas on how to resolve these issues. Lastly we will cover impacts from shortened schedules and how DOE enabled a quantitative risk assessment.

We study the problem of post-hoc calibration of machine learning classifiers. We introduce the following desiderata for uncertainty calibration: (a) accuracy-preserving, (b) data-efficient, and (c) high expressive power. We show that none of the existing methods satisfy all three requirements, and demonstrate how our proposed calibration strategies can help achieve dramatically better data efficiency and expressive power while provably preserving classification accuracy of the original classifier. When calibrating a 50-layer Wide ResNet on ImageNet classification task, the proposed strategies improve the expressivity of temperature scaling by 17% and data efficiency of isotonic regression by a factor of 258, while preserving classification accuracy.

Co-Authors: Luis Cortes (MITRE) and Alethea Duhon (Air Force Agency for Modeling and Simulation)

This talk highlights the advantage of a multi-criteria decision analysis (MCDA) technique that utilizes the one-sided tolerance interval (OSTI) for the analysis of response data resulting from design of experiments (DOE) structured testing.  Tolerance Intervals (TI) can be more intuitive for representing data and, when combined with DOE structured tests results, are excellent for providing decision quality information.  The use of statistical techniques in planning provides a rigorous approach for the analysis of test data, an ideal input for a MCDA technique.  This article’s findings demonstrate the value of constructing the OSTI for the interpretation of DOE structured tests.  We also demonstrate the utility of using an optimized model combined with the OSTI as part of a MCDA technique for choosing between alternatives, which offers an efficient and powerful method to sift through test results rapidly and efficiently.  This technique provides an alternative for analyzing data across multiple alternatives when traditional analysis techniques, such as the descriptive statistics (including the Confidence Interval [CI]), potentially obscure information from the untrained eye.  Finally, the technique provides a level playing field–a critical feature given today’s acquisition protest culture.

Machine learning is becoming increasingly imbedded in military systems, so how do we know if it’s working, or as the name implies learning? This presentation and demonstration is a thought experiment for tester’s to consider how to measure and test systems with imbedded machine learning. This presentation involves a physical demonstration of elements of the popular casino game craps. Craps is a fast-paced dice game that offers players the opportunity to try and beat the odds, which are stacked in the casino’s favor. But what if a player could flip the odds on the casino by cheating? Can the audience detect the cheater? How about a machine learning algorithm? The audience will be able to make their own guesses and compare them to A-Dell, a machine learning algorithm designed to find craps cheaters.

Linear regression is often extended to either (1) multilevel models in which response data cannot be assumed independent, e.g., nested data, or (2) generalized linear models in which response data is not normally distributed. Applying both extensions to binary responses results in generalized linear multilevel models in which a sigmoid link function gives the relationship between the mean of the response variable and a linear combination of predictors. Such models are well-suited to analyze psychophysical experiments, in which fitted sigmoids give the relationship between physical stimulus level and the probability of a human perceptual response. In this work, a generalized linear multilevel model is applied to a three-alternative forced-choice psychoacoustic test in which human test subjects were asked to identify a sound signal presented at different levels relative to a background noise. Since the guessing rate at low stimulus levels must converge to 1/3, a custom link function is applied. In this test, the grouping variable is the subject, because within-subject responses are assumed to be more alike than between-subject responses. This leads to readily available information about population-level parameters, such as how auditory thresholds are distributed within the group, how steep the psychometric functions are and if the differences are statistically significant. The multilevel model also demonstrates the effect of shrinkage in which the partially-pooled regression parameters are closer to the population mean than parameters found by un-pooled analyses.