DATAWorks Sessions Archive

The US Army ARDEC has recently established an initiative to integrate statistical and probabilistic techniques into engineering modeling and simulation (M&S) analytics typically used early in the design lifecycle to guide technology development. DOE-driven Uncertainty Quantification techniques, including statistically rigorous model verification and validation (V&V) approaches, enable engineering teams to identify, quantify, and account for sources of variation and uncertainties in design parameters, and identify opportunities to make technologies more robust, reliable, and resilient earlier in the product’s lifecycle. Several recent armament engineering case studies – each with unique considerations and challenges – will be discussed.

The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals. The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface. When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help. This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.

Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced

Binary response experiments are common in epidemiology, biostatistics as well as in military applications. The Up and Down method, Langlie’s Method, Neyer’s method, K in a Row method and 3 Phase Optimal Design are methods used for sequential experimental design when there is a single continuous variable and a binary response. During this talk, we will discuss a new sequential experimental design approach called the Break Separation Method (BSM). BSM provides an algorithm for determining sequential experimental trials that will be used to find a median quantile and fit a logistic regression model using Maximum Likelihood estimation. BSM results in a small sample size and is designed to efficiently compute the median quantile.

This brief talk will focus on the process of human-machine trust in context of automated intelligence tools. The trust process is multifaceted and this talk will define concepts such as trust, trustworthiness, trust behavior, and will examine how these constructs might be operationalized in user studies. The talk will walk through various aspects of what might make an automated intelligence tool more or less trustworthy. Further, the construct of transparency will be discussed as a mechanism to foster shared awareness and shared intent between humans and machines.

Teams of people with many different talents and skills work together at NASA to improve our understanding of our planet Earth, our Sun and solar system, and the Universe. The Earth System is made up of complex interactions and dependencies of the solar, oceanic, terrestrial, atmospheric, and living components. Solar storms have been recognized as a cause of technological problems on Earth since the invention of the telegraph in the 19th century. Solar flares, coronal holes, and coronal mass ejections (CME’s) can emit large bursts of radiation, high speed electrons and protons, and other highly energetic particles that are released from the sun, and are sometimes directed at Earth. These particles and radiation can damage satellites in space, shutdown power grids on earth, cause GPS outages, and have serious health concerns to humans flying at high altitudes on earth, as well as astronauts in space. NASA builds and operates a fleet of satellites to study the sun and a fleet of satellites and aircraft to observe the Earth system. NASA’s Computer Models combine the observations with numerical models, to understand how these systems work. Using satellite observations alongside computer models we can combine many pieces of information to form a coherent view of Earth and the Sun. NASA research helps us understand how processes combine to affect life on Earth: this includes severe weather, health, changes in climate, and space weather. The Scientific Visualization Studio wants you to learn about NASA programs through visualization. The SVS works closely with scientists in the creation of data visualizations, animations, and images in order to promote a greater understanding of Earth and Space Science research activities at NASA and within the academic research community supported by NASA.

Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.

Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced

Model Validation for Simulations of CVN-78 Sortie Generation As part of the test planning process, IDA is examining flight operations on the Navy’s newest carrier, CVN-78. The analysis uses a model, the IDA Virtual Carrier Model (IVCM), to examine sortie generation rates and whether aircraft can complete missions on time. Before using IVCM, it must be validated. However, CVN-78 has not been delivered to the Navy, and data from actual operations are to validate the model. Consequently, we will validate IVCM by comparing it to another model. This is a reasonable approach when a model is used in general analyses such as test planning, but is not acceptable when a model is used in the assessment of system effectiveness and suitability. The presentation examines the use of various statistical tools – Wilcoxon Rank Sum Test, Kolmogorov-Smirnov Test, and lognormal regression – to examine whether the results from two models provide similar results and to quantify the magnitude of any differences. From the analysis, IDA concluded that locations and distribution shapes are consistent, and that the differences between the models are less than 15 percent, which is acceptable for test planning.

To enable quiet supersonic passenger flight overland, NASA is providing national and international noise regulators with a low-noise sonic boom database. The database will consist of dose-response curves, which quantify the relationship between low-noise sonic boom exposure and community annoyance. The recently-updated international standard for environmental noise assessment, ISO 1996-1:2016, references multiple fitting methods for dose-response analysis. One of these fitting methods, Fidell’s community tolerance level method, is based on theoretical assumptions that fix the slope of the curve, allowing only the intercept to vary. This fitting method is applied to an existing pilot sonic boom community annoyance data set from 2011 with a small sample size. The purpose of this exercise is to develop data collection and analysis recommendations for future sonic boom community annoyance surveys.

In May 2016, the NASA Langley Research Center’s Engineering Director stood up a group consisting of employees within the directorate to assess the current state of engineering being done by the organization. The group was chartered to develop ideas, through investigation and benchmarking of other organizations within and outside of NASA, for how engineering should look in the future. This effort would include brainstorming, development of recommendations, and some detailed implementation plans which could be acted upon by the directorate leadership as part of an enduring activity. The group made slow and sporadic progress in several specific, self-selected areas including: training and development; incorporation of non-traditional engineering disciplines; capturing and leveraging historical data and knowledge; revolutionizing project documentation; and more effective use of design reviews. The design review investigations have made significant progress by leveraging lessons learned and techniques gained by collaboration with operations research analysts within the local Lockheed Martin Center for Innovation (the “Lighthouse”) and pairing those techniques with advanced data analysis tools available through the IBM Watson Content Analytics environment. Trials with these new techniques are underway but show promising results for the future of providing objective, quantifiable data from the design review environment – an environment which to this point has remained essentially unchanged for the past 50 years.

The NASA Big Data, Big Think team jump-starts coordination, strategy, and progress for NASA applications of Big Data Analytics techniques, fosters collaboration and teamwork among centers and improves agency-wide understanding of Big Data research techniques & technologies and their application to NASA mission domains. The effort brings the Agency’s Big Data community together and helps define near term projects and leverages expertise throughout the agency. This presentation will share examples of Big Data activities from the Agency and discuss knowledge areas and experiences, including data management, data analytics and visualization.

Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.

More often than not, the data we analyze for the military is plagued with statistical issues. Multicollinearity, small sample sizes, quasi-experimental designs, and convenience samples are some examples of what we commonly see in military data. Many of these complications can be resolved either in the design or analysis stage with appropriate statistical procedures. But, to keep our work useful, usable, and transparent to the military leadership who sponsors it, we must strike the elusive balance between explaining and justifying our design and analysis techniques and not inundating our audience with unnecessary details. It can be even more difficult to get military leadership to understand the statistical problems and solutions so well that they are enthused and supportive of our approaches. Using literature written on the subject as well as a variety of experiences, we will showcase several examples, as well as present ideas for keeping our clients actively engaged in statistical methodology discussions.

This panel will share status, experiences and expectations within DoD and NASA for transitioning Systems Engineering to a more integrated digital engineering domain. A wide range of perspectives will be provided, covering the implementation waterfront of practitioner, management, research and strategy. Panelist will also be prepared to discuss more focused areas of digital systems engineering, such as test and evaluation, and engineering statistics.

When Modeling and Simulation (M&S) is used as part of operational evaluations of effectiveness, suitability, survivability, or lethality, the M&S capability should first be rigorously validated to ensure it is representing the real world accurately enough for the intended use. Specifically, we need to understand and characterize the usefulness and limitations of the M&S, especially in terms of uncertainty! Many statistical techniques are available to compare M&S output with live test data. This presentation will describe and present results from a simulation study conducted to determine which techniques provide the highest statistical power to detect differences in mean and variance between live and sim for a variety of data types and sizes.

Due to small Tactical Data Link testing windows, only commonly used messages are tested resulting in the evaluation of only a small subset of all possible Link 16 messages. To increase the confidence that software design and implementation issues are discovered in the earliest phases of government acceptance testing, Marine Corps Tactical Systems Support Activity (MCTSSA) Instrumentation and Data Management Section (IDMS) successfully implemented an extension of the traditional form of Design of Experiments (DOE), called Combinatorial Testing (CT). CT was utilized to reduce the human bias and inconsistencies involved in Link 16 testing and replace them with a thorough test that can validate a system’s ability to properly consume all of the possible valid combinations of Link 16 message field values. MCTSSA’s unique team of subject matter experts was able to bring together the tenants of virtualization, automation, C4I Air systems testing, tactical data link testing, and Design of Experiments methodology, to invent a testing paradigm that will exhaustively evaluate tactical Air systems. This presentation will give an overview of how CT was implemented for the test.

Data within a project, investigation or test series are often seen as a bunch of numbers that were produced. While this is part of the story, it forgets the most important part: the data’s users. This more powerful process begins with early focus on planning, executing and managing data flow within a test or project as a system, treating each handoff between internal and external stakeholders a system interface. This presentation will persuade you why data production should be replaced by the idea of a data supply chain focused on goals and customers. The presenter will outline how this could be achieved in your team. The talk is aimed at not only project and data managers, but also team members who produce or use data. Retooling team thinking and processes along these lines will help communication, facilitate availability, display and understanding of data by any stakeholder, make data verification, validation and analysis easier, and help keep team members focused on what is necessary and important: solving the problem at hand.