DATAWorks Sessions Archive

Since its publication, Statistical Methods for Reliability Data by W. Q. Meeker and L. A. Escobar has been recognized as a foundational resource in analyzing failure time to and survival data. Along with the text, the authors provided an S-Plus software package, called SPLIDA, to help readers utilize the methods presented in the text. Today, R is the most popular statistical computing language in the world, largely supplanting S-Plus. The SMRD package is the result of a multi-year effort to completely rebuild SPLIDA, to take advantage of the improved graphics and workflow capabilities available in R. This presentation introduces the SMRD package, outlines the improvements and shows how the package works seamlessly with the rmarkdown and shiny packages to dramatically speed up your workflow. The presentation concludes with a discussion on what improvements still need to be made prior to publishing the package on the CRAN.

One of the most powerful features of Bayesian analyses is the ability to combine multiple sources of information in a principled way to perform inference. For example, this feature can be particularly valuable in assessing the reliability of systems where testing is limited for some reason (e.g., expense, treaty). At their most basic, Bayesian methods for reliability develop informative prior distributions using expert judgment or similar systems. Appropriate models allow the incorporation of many other sources of information, including historical data, information from similar systems, and computer models. I will introduce the approach and then consider examples from defense acquisition and lifecycle extension, focusing on the strengths and weaknesses of the Bayesian analyses.

The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals. The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface. When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help. This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.

The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals. The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface. When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help. This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.

Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.

Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.

NASA’s Airspace Technology Demonstration (ATD) project was developed to facilitate the transition of mature air traffic management technologies from the laboratory to operational use. The first ATD focused on an integrated set of advanced NASA technologies to enable efficient arrival operations in high-density terminal airspace. This integrated arrival solution was validated and verified in laboratories and transitioned to a field prototype for an operational demonstration. Within NASA, this was a collaborative effort between Ames and Langley Research Centers involving a multi-year iterative experimentation process consisting of a series of sequential batch computer simulations and human-in-the-loop experiments, culminating in a flight test. Designing and analyzing the flight test involved a number of statistical challenges. There were several variables which are known to impact the performance of the system, but which could not be controlled in an operational environment. Changes in the schedule due to weather and the dynamic positioning of the aircraft on the arrival routes resulted in the need for a design that could be modified in real-time. This presentation describes a case study from a recent NASA flight test, highlights statistical challenges, and discusses lessons learned.

Many modern processes generate complex data records not readily analyzed by traditional techniques. For example, a single observation from a process might be a radar signal consisting of n pairs of bivariate data described via some functional relation between reflection and direction. Methods are examined here for detecting changes in such sequences from some known or estimated nominal state. Additionally, estimates of the degree of change (scale, location, frequency, etc.) are desirable and discussed. The proposed methods are designed to take advantage of all available data in a sequence. This can become unwieldy for long sequences of large-sized observations, so dimension reduction techniques are needed. In order for these methods to be as widely applicable as possible, we make limited distributional assumptions and so we propose new nonparametric and Bayesian tools to implement these estimators.

Spectrograms (i.e., squared magnitude of short-time Fourier transform) are commonly used as features to classify audio signals in the same way that social media companies (e.g., Google, Facebook, Yahoo) use images to classify or automatically tag people in photos. However, a serious problem arises when using spectrograms to classify acoustic signals, in that the user must choose the input parameters (hyperparameters), and such choices can have a drastic effect on the accuracy of the resulting classifier. Further, considering all possible combinations of the hyperparameters is a computationally intractable problem. In this study, we simplify the problem making it computationally tractable, explore the utility of response surface methods for sampling the hyperparameter space, and find that response surface methods are a computationally efficient means of identifying the hyperparameter combinations that are likely to give the best classification results.

The process for testing military systems which are largely software intensive involves techniques and procedures often different from those for hardware-based systems. Much of the testing can be performed in laboratories at many of the acquisition stages, up to operational testing. Testing software systems is not different from testing hardware-based systems in that testing earlier and more intensively benefits the acquisition program in the long run. Automated testing of software systems enables more frequent and more extensive testing, allowing for earlier discovery of errors and faults in the code. Automated testing is beneficial for unit, integrated, functional and performance testing, but there are costs associated with automation tool license fees, specialized manpower, and the time to prepare and maintain the automation scripts. This presentation discusses some of the features unique to automated software testing and offers a framework organizations can implement to make the business case for, to organize for, and to execute and benefit from automating the right aspects of their testing needs. Automation has many benefits in saving time and money, but is most valuable in freeing test resources to perform higher value tasks.

The process for testing military systems which are largely software intensive involves techniques and procedures often different from those for hardware-based systems. Much of the testing can be performed in laboratories at many of the acquisition stages, up to operational testing. Testing software systems is not different from testing hardware-based systems in that testing earlier and more intensively benefits the acquisition program in the long run. Automated testing of software systems enables more frequent and more extensive testing, allowing for earlier discovery of errors and faults in the code. Automated testing is beneficial for unit, integrated, functional and performance testing, but there are costs associated with automation tool license fees, specialized manpower, and the time to prepare and maintain the automation scripts. This presentation discusses some of the features unique to automated software testing and offers a framework organizations can implement to make the business case for, to organize for, and to execute and benefit from automating the right aspects of their testing needs. Automation has many benefits in saving time and money, but is most valuable in freeing test resources to perform higher value tasks.

In recent years, software testing techniques based on formal methods have made their way into industrial practice as a supplement to system and unit testing. I will discuss three core techniques that have proven particularly amenable to transition: 1) Concolic execution, which enables the automatic generation of high-coverage test suites; 2) Property-based randomized testing, which automatically checks sequences of API calls to ensure that expected high-level behavior occurs; and 3) Bounded model checking, which enables systematic exploration of both concrete systems and high-level models to check temporal properties, including ordering of events and timing requirements.

Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced

Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced

Surveys are commonly used to evaluate the quality of human-system interactions during the operational testing of military systems. Testers use Likert-type response options to measure the intensity of operators’ subjective experiences (e.g., usability, workload) while operating the system. Recently, appropriate methods for analyzing Likert data have become a point of contention within the operational test community. Some argue that Likert data can be analyzed with parametric techniques whereas others argue that only non-parametric techniques should be used. However, the reasons stated for holding a particular view are rarely tied to findings in the empirical literature. This presentation sheds light on this debate by reviewing existing research on how parametric statistics affect the conclusions drawn from Likert data and debunk common myths and misunderstandings about the nature of Likert data within the operational test community and academia.

The System Usability Scale (SUS) was developed by John Brooke in 1986 “to take a quick measurement of how people perceived the usability of (office) computer systems on which they were working.” The SUS is a 10-item, generic usability scale that is assumed to be system agnostic, and it results in a numerical score that ranges from 0-100. It has been widely employed and researched with non-military systems. More recently, it has been strongly recommended for use with military systems in operational test and evaluation, in part because of its widespread commercial use, but largely because it produces a numerical score that makes it amendable to statistical operations. Recent lessons learned with SUS in operational test and evaluation strongly question its use with military systems, most of which differ radically from non-military systems. More specifically, (1) usability measurement attributes need to be tailored to the specific system under test and meet the information needs of system users, and (2) a SUS numerical cutoff score of 70—a common benchmark with non-military systems—does not accurately reflect “system usability” from an operator or test team perspective. These findings will be discussed in a psychological and human factors measurement context, and an example of system-specific usability attributes will be provided as a viable way forward. In the event that the SUS is used in operational test and evaluation, some recommendations for interpreting the outcomes will be provided.

Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.

Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.

Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.

“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning. Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”

“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning. Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”