DATAWorks Themes Archive

1: Progress and Challenges in Program Evaluation

The Department of Defense (DoD) and NASA develop and acquire some of the world’s most sophisticated technological systems. In cooperation with these organizations, we face challenges in ensuring that these systems undergo testing and evaluation (T&E) that is both adequate and efficient prior to their use in the field.
Sessions with this theme will feature case study examples highlighting advancements in T&E methodologies that we have made as a community.
Topics covered may include modeling and simulation validation, uncertainty quantification, experimental design usage, Bayesian analysis, among others. Sessions will also discuss the current challenges and areas in the field that require more research.

2: Machine Learning and Artificial Intelligence

With the future of automation trending towards using machine learning (ML) and artificial intelligence (AI), we aim to provide resources, set precedent, and inspire innovation in these fields.
Sessions with this theme will discuss how we can quantify confidence that ML and/or AI algorithms function as intended. To do this we explore whether the functions are free of vulnerabilities that are either intentionally or unintentionally integrated as part of the data or algorithm.
Topics covered may include testing and evaluation (T&E) metrics and methods for AI algorithms, T&E metrics and methods for cyber-physical systems with AI algorithms, threat portrayal for ML/AI algorithms, and robust ML/AI algorithm design.

3: Cybersecurity and Software Test & Evaluation

The number of known cyber-attacks has been steadily increasing over the past ten years, exposing millions of data records. With the help of leading minds in the cyber field, we work to not only to discover new ways of identifying these attacks, but also ways to counter future attacks.
Sessions with this theme will discuss metrics and test methodologies for testing cyber-physical systems with an emphasis on characterizing cyber resiliency.
Topics covered may include identifying cybersecurity threats (IP and non-IP) across critical operational missions, development and testing secure architectures, identifying critical system components that enable attack vectors, developing and testing countermeasures to cyber-attacks, and testing methods for cyber-physical systems that include machine learning algorithms.

4: Modeling the Human-System Interaction

Mission outcomes are heavily determined by how effectively operators interact with systems under operational conditions as decided by their objective performance and their perceived performance. In order to effectively interpret these systems, we analytically observe team dynamics to predict how changing operating conditions will affect the dynamic and therefore productivity.
Sessions with this theme will discuss methods used to collect and model the quality of human-system interaction and its impact on operational performance.
Topics covered will include survey development/test design, statistical methods for surveys, scale validation, and network analysis.

5: Science Applications of Test and Evaluation

Scientists use a variety of tools to perform analyses within their areas of expertise. Our goal is to demonstrate knowledge of how to improve analyses that they already perform and to introduce new tools to increase analysis efficiency.
Sessions with this theme will cover simulation, prediction, uncertainty quantification, and inference for physical and physical-statistical modeling of geophysical processes.
Topics covered may include Earth Science problems (e.g. ice sheet evolution models, atmospheric and ocean processes, the carbon cycle, land surface processes, natural hazards), astronomy and cosmology (e.g. exoplanet detection, galactic formation, cosmic microwave background), and planetary science (e.g. planetary atmospheres, formation).

1: Verification, Validation, and Uncertainty Quantification

These sessions will focus on model verification and validation including case study examples as well as the use of physics and computational based modeling and simulation to benefit system evaluation.

2: Data Integration and Sequential Testing

These sessions will focus on integrating, to the extent possible all available information, including previous test outcomes, to sequentially update and refine future testing.

3: Autonomy, Artificial Intelligence, Machine Learning

These sessions will focus on introducing new tools for evaluating how operators interact with autonomous systems and demonstrate how members of the defense and aerospace communities have applied statistical approaches to evaluate autonomous system performance.

4: Data Curation and Engineering

These sessions will focus on best practices for storing, curating, cleaning and making data ready for analysis.

5: Improving System Cybersecurity

These sessions will focus on methods for evaluating cyber survivability, specifically cyber-attack resiliency for weapon systems.

6: Human-System Integration

These sessions demonstrate the importance of human-centered design in defense and aerospace systems and introduce the range of test design and statistical methods available for evaluating human-system integration.

1: Statistical Engineering for Defense and Aerospace

2: Statistical Engineering for Defense and Aerospace

3: Humans are Part of the System: Implications for Test Planning and Analysis

4: Rigorous Use of Modeling and Simulation

5: Statistical Engineering for Autonomous Systems

6: Data Science

1: Improving Computational Modeling: CFD Vision 2030

This session will focus on the statistics in improving existing and developing new computational modeling capabilities.

2: Experimental Design Methods for System Design and Characterization

This session will focus on the value of a structured experimental design process in both the design and evaluation of systems.

3: Statistical Engineering for Defense and Aerospace

This session will include cases studies that illustrate how straightforward statistical approaches linked together in the right way and targeted towards answering the right question can be useful in addressing the complex data and design challenges in defense and aerospace.

4: Rigorous Use of Modeling and Simulation

This session will focus on the use of physics and computational based modeling and simulation to benefit system evaluation and group/flight/live testing.

5: Improving System Reliability and Reliability Test Planning

This session will focus on advanced techniques in testing long term system reliability.

6: Humans are Part of the System: Implications for test planning and analysis.

This session will focus on the design and test methods for assessing the coupled interaction of the human and the system.