Sessions Archive

Developing AI solutions requires the alignment of business needs, data availability, digital tools, and subject matter expertise. The Army Test and Evaluation Command (ATEC) is an organization with extensive data and operational requirements but is still developing its AI capabilities. To address this gap, ATEC launched an annual AI Challenge—an enterprise-wide, education-focused initiative aimed at solving real-world business problems using the new ATEC Data Mesh digital ecosystem. The 2024 challenge focused on automating defect detection in X-ray scans of body armor. Over three months, 153 participants across 29 teams—including internal and external partners—developed computer vision solutions to autonomously identify manufacturing defects such as cracks, foreign debris, and voids. The winning team achieved a remarkable 92% accuracy in defect detection. This effort not only resulted in a valuable tool that enhances operational capacity and efficiency but also significantly advanced AI expertise across the organization. Participants gained hands-on experience with cloud infrastructure, while ATEC refined its methodologies for testing and evaluating AI-enabled systems. The AI Challenge exemplifies how combining educational resources and competition can foster innovation towards real capabilities.

The Department of Defense (DoD) Test and Evaluation (T&E) community has fully embraced digital engineering, as defined in the 2018 Digital Engineering Strategy, motivating the ongoing development and adoption of model-based testing methodologies. This article expands upon existing grey box model-driven test design (MDTD) approaches by leveraging model-based systems engineering (MBSE) artifacts to generate flight test planning models and documents. A baseline model of a system under test (SUT) and two additional system case studies are used to assess a Model-Driven Test Design (MDTD) process. The paper illustrates the method's applicability to these case studies, assesses the benefits of MDTD by applying novel metrics of model element reuse, and discusses the relevance to operational flight testing. This approach is novel within the flight-testing community, as it is the first implementation of MDTD in USAF operational testing applications. Whereas previous studies have explored SysML model reuse in small-scale problems or product families, MBSE model management for operational tests at flight-system scale and assessment of reuse in the T&E phase of the SE lifecycle are unresearched to date. This methodology and the case studies will be of particular interest to those involved in developing, executing, and reporting on flight test plans in the context of the DoD Digital Engineering transformation.

A strategic goal of the DoD test and evaluation community is to combine information from across the acquisition lifecycle, enabling better understanding of systems earlier and design of tests to maximize information later. This talk will provide a systematic comparison of methods for integrating such information, ranging from hierarchical methods to informative priors to normalized power priors, leveraging as a motivating example a notional model of a defense system for which behavior and test factors evolve as the system develops. Though large-scale simulation experiments testing a variety of situations and assumptions, we will illustrate how the techniques work and their promise for improving understanding of systems, while highlighting best practices as well as potential implementation pitfalls. The comparison illuminates best practices for integrated system evaluation and illustrates how modeling assumptions affect estimates of system parameters.

The evolution from a traditional waterfall software release model to a continuous and iterative release model allows operational testers to conduct operational testing earlier in the development lifecycle. To enable operationally realistic software testing before deploying to users, programs create preproduction environments designed to replicate the hardware and software infrastructure of production environments. However, it can be challenging for testers to assess the similarity between the preproduction and production software environments and determine what data can be used from the preproduction environment to support operational evaluations. We present a general framework for the Verification, Validation, and Accreditation (VV&A) of preproduction environments that aims to be rigorous yet flexible enough to meet the needs of acquisition programs conducting operational testing in preproduction environments. This framework includes a three-stage VV&A process, composed of an initial VV&A, followed by a set of automated and continuous verification and validation (V&V) checks, and a VV&A renewal if major differences between the environments appear. We describe the data needed to verify and validate the environment and how to customize the VV&A process to fit the needs of each program.

The advent of advanced machine learning (ML) capabilities has dramatically increased the need for data to train, test, and validate models. At the same time, systems and models are being asked to operate in increasingly diverse environments. With the high cost of data collection and labeling, or even the complete lack of available data, synthetic data has become an attractive alternative. Synthetic data promises many potential benefits over traditional datasets including reduced cost, improved coverage of edge-cases, more balanced data sets, reduced data collection time, and in many cases, it may be the only data that is available for a target environment. At the same time, it introduces potential risks including lack of realism, bias amplification, overfitting to potentially synthetic features, missing real-world variability, and lack of confidence in the results. The degree to which these benefits and risks manifest depends greatly on the type of problem and the way in which synthetic data is generated and utilized. In this paper we propose a principled systematic approach to testing the effectiveness of synthetic data for specific classes of problems. We illustrate our approach on an image classifier using a flower type database. We first establish a model baseline by training and testing the classifier model with real data, then measure its performance. We then establish a synthetic dataset baseline by attempting to train binary classifiers to distinguish each synthetic dataset from the real dataset. Poor performance of the binary classifier indicates that the corresponding synthetic dataset is a better representation of the real data. We then conduct two core sets of experiments evaluating the effectiveness of the synthetic data in training (replacement and augmentation), and another set evaluating the effectiveness of synthetic data for testing. In the replacement experiments we gradually replace real data with synthetic data and measure the degradation in performance for each synthetic dataset. In the augmentation experiments we augment the available real data with additional synthetic data and measure the improvement in performance. Finally, we conduct a set of experiments to evaluate the usefulness of synthetic data for testing. We do this by comparing performance metrics calculated with different subsets of real data against different subsets of synthetic data. In addition, we perturb the model by deliberately degrading the training data (e.g., by deliberately mislabeling subsets) and verifying that the resulting degradation in performance as calculate with synthetic data tracks the degradation as calculated with real data. For each of the synthetic data sets, we compare the results with the original synthetic data quality evaluation we calculated in our baseline.

According to AV-Test, roughly 450,000 new malicious programs are detected each day adding to a total number of malware signatures that stands at almost 1.5 billion. These known signatures are analyzed and given labels by antivirus companies to classify the malware. These classifications allow security operation centers or antivirus programs to more easily take action to prevent or stop costly damage. In recent years, polymorphic malware, malware that intentionally obfuscates its behavior and signature, has seen a rise in prevalence. We aim to show that current antivirus classifications inefficiently group malware, especially in the case of polymorphic malware, that share enough intrinsic similarities with other malware to justify consolidation into broader groupings we are calling tribes. We hypothesize that the consolidation of these labels will reduce the time it takes for analysts to classify malware thus lowering incident response time. This generalized labeling will be implemented through the use of a transformer-based sequence-to-sequence variational autoencoder that takes in a malware binary and produces a clustering based on its distinct characteristics. We are naming this method Tribal Relational Inferential Encoder (TRIBE). The use of autoencoders in malware classification has shown promise in accurately labeling malware. TRIBE will perform unsupervised learning to independently create these tribes, generalized labels, and compare the results to existing labeling schemes. We estimate three outcomes from research: a data set of existing malware antivirus families with associated malware, a trainable autoencoder tool that will produce robust malware tribes, and a classifier that will make use of tribes to label malware.

Multi-label classification is a crucial tool in various applications such as image classification and document categorization. Unlike single-label classification, the evaluation of multi-label classification performance is complex because a model's prediction can be partially correct—capturing some labels while missing others. This lack of a straightforward binary correct/incorrect outcome introduces challenges in model testing and evaluation. Model developers rely on a collection of metrics to fine-tune these models for optimal performance; however, existing metrics are mostly adapted from single-label contexts. This has led to large array of metrics which are difficult to interpret and may not adequately reflect model performance in a practical manner.

To address this issue, we designed an experiment which replaced multi-label classification algorithms with a random process to evaluate how metrics perform relative to prescribed classifier and dataset characteristics. The relationships between metrics and how those relationships change relative to dataset attributes were investigated with the goal to down-select to a spanning set. Additionally, we explored the potential for developing more interpretable metrics by incorporating dataset characteristics into model evaluation.

Over the past three decades, an evolution has occurred in popular terminology, beginning with “simulation-based acquisition” (SBA), transitioning to “digital engineering,” and then to “model-based systems engineering.” Although these three terms emphasize different aspects, the consistent, unspoken, element that underlies all of them is credible modeling, simulation, and analysis (CMSA). It is CMSA that enables predictable delivery. We use the word “delivery” to encompass three aspects that are relevant to SBA: (a) satisfying design intent, (b) meeting the cost/schedule goal, and (c) reaching the production throughput target. The word “credible” implies using the language of probability to quantify uncertainty to support decision-making within estimated risk/reward bounds to achieve predictable delivery.

We introduce a Predictability Bayes Net (PBN) that represents, from a contractor’s perspective, top-level dependencies between modeling and simulation (M&S) activities and standardized workflows across a population of programs. The PBN describes how to meet SBA objectives, or to diagnose and learn why objectives were not met. For example, given failure to deliver at-cost and on-time, the PBN computes marginal probabilities that suggest the most likely sequence of events leading to this failure. The PBN does this by linking verified and validated M&S to standardized workflows, thereby transitioning from CMSA to assured physical delivery.

Our previous publications focused on CMSA for design engineering, using Bayes Nets to assess compliance with system performance requirements. We have now expanded CMSA to include production cost estimation and factory throughput modeling. The PBN includes top-level elements such as mission stability, technology stretch, workforce experience level, standardized workflow adherence, and supplier responsiveness.

A Bayes Net includes a set of event nodes and node state definitions. These definitions become metrics to hold ourselves accountable for product delivery. The PBN is a joint probability distribution; it includes conditional probability estimates, which are based on a combination of opinions from subject matter experts and data from developmental and operational test events. Sufficient, relevant data from these test events is crucial. It supports M&S verification and validation and, when standardized workflows are followed, embodies CMSA. Without this data, M&S is non-credible, and predictable delivery becomes impossible.

The PBN is of mutual interest to both the DoD and contractors. Early phases in a program cast the largest shadow over a system’s ultimate cost, performance, and production throughput. Given imperfect, limited, or partially shared information available in early program phases, probabilistic inference becomes critical for optimal decision-making using this information. The PBN facilitates SBA through CMSA by (a) first bringing clarity to fine-grained communication using the language of probability, and (b) then quantifying the uncertainty that exists at a specific moment of decision. The PBN also serves as a starting point for building lower-level Bayes Nets to answer targeted queries regarding a program’s execution, including aspects of design, supply chain, and production. The PBN is the mechanism for achieving predictable delivery.

Well characterized aerodynamic databases are necessary for accurate simulation of flight dynamics. High-fidelity CFD is computationally expensive and thus we use a surrogate model to represent the database. By utilizing active learning, we can efficiently generate samples for the database and target areas of the design space that are most useful for flight simulation. Here we focus on trimmed aero databases where our goal is to find regions where multiple moments are simultaneously zero and use a novel contour active learning to achieve this goal. Entropy-based methods are well explored for computer experiment research about reliability, however sequential design for estimation for multiple contours is less studied. We compare multiple entropy-based approaches for estimating contours for multiple responses simultaneously. This task requires the development of new metrics to evaluate the performance of the active learning strategies. We apply these active learning methods and metrics for both Gaussian Process and Deep Gaussian Process surrogate models. The performance of this approach is evaluated with multiple examples and applied to a reference vehicle as a simulation study.

Ambiguity in how to associate system detections with ground truth measurements poses difficulty in interpreting system evaluation tests. As an example, we discuss the tests performed for the Strategic Environmental Research and Development Program and the Environmental Security Technology Certification Program (SERDP/ESTCP) meant to assess novel systems which detect & classify underwater Unexploded Ordnances (UXO). Due to the larger uncertainties associated with underwater environments, these tests frequently have ambiguities. The Institute of Defense Analyses (IDA), tasked with scoring these SERDP/ESTCP tests, developed and implemented a scoring methodology which interpret tests with ambiguities.

This talk will introduce the basics of non-ambiguous detection & classification scoring, discuss methods IDA has used to address ambiguity, and discuss how this is approach to applied to produce graphs and tables which can be interpreted by relevant stakeholders.

Regardless of test design method (combinatoric, Design of Experiments, a narrow robustness study, etc.), a scientifically rigorous experiment must understand, manage, and control the variables that impact test outcomes. For most scientific fields, this is settled science with decades – even centuries – of formalism and honed methodology. For the emerging field of Large Language Models (LLM) in military weapon systems, it is the wild west. This presentation will survey the factors and conditions that impact LLM test outcomes, along with supporting literature and practical methods, models, and measures for your use in tests. The presentation will also highlight: 1) The statistical assumptions that underly the common LLM performance metrics and how to test those assumptions; 2) How to evaluate a benchmark for its utility in addressing measures of performance, as well as checking the benchmark’s statistical validity; 3) Practical models, and supporting literature, for binning factors into levels of severity (conditions); 4) Resources for ensuring a User-centered test design; and 5) Incorporating selected adversarial techniques. These resources and techniques are immediately actionable (you can even try them out on your device and favorite LLM during the session) and will equip you to navigate the complexity of scientific test design for LLM-enabled systems.

The integration of Artificial Intelligence (AI), Machine Learning (ML), and Uncertainty Quantification (UQ) is transforming aerospace systems engineering by improving how systems are designed, tested, and operated. This presentation explores the transformative role of large language models (LLMs) and UQ in tackling the high stakes and inherent uncertainty of space systems. LLMs streamline requirements analysis, enable intelligent creation and querying of design documents, and accelerate development timelines, while UQ provides robust risk assessments, predictive modeling, and cost-saving opportunities throughout the mission lifecycle.

Effectively managing uncertainty is critical at every stage of the project lifecycle, from early design formulation to on-orbit operations. This presentation highlights practical applications of UQ in space mission formulation and science data pipelines, as well as its role in assessing risk and enhancing system reliability. It also examines how LLMs improve the development and analysis of system documentation, enabling more agile and informed decision-making in complex projects.

By integrating LLMs and UQ into systems engineering, aerospace teams can better manage complexity, enhance system resilience, and achieve cost-effective solutions. This presentation offers key insights, lessons learned, and future opportunities for advancing systems engineering with AI/ML and UQ.

Material supply chains are complex, global systems that drive the production of goods and services, from raw material extraction to manufacturing processes. As the U.S. becomes increasingly dependent on foreign production of strategic and critical materials (S&CMs), our nation’s security will demand effective analyses of material supply chains to identify potential shortages and suggest ways to alleviate them. The Institute for Defense Analyses (IDA) developed the Risk Assessment and Mitigation Framework for Strategic Materials (RAMF-SM) to help the Department of Defense identify and resolve potential shortfalls of S&CMs in the National Defense Stockpile Program. The Stockpile Sizing Module (SSM) is the main computational vehicle in the RAMF-SM suite of models and is used to estimate shortfalls of S&CMs during national emergencies. This talk presents a multicommodity flow model that extends the SSM’s network linear programming formulation by explicitly representing two stages of supply—commonly categorized as mining and refining—and tracking material throughout these stages while incorporating decrements to supply. This more accurate representation offers a robust framework for analyzing material production dynamics, enabling precise shortfall calculations and the identification of bottlenecks. While focused on two production stages, this work lays critical groundwork for future extensions to a comprehensive multi-stage production model.

The Department of Defense has realigned its approach to data management. Prior to 2020, data was viewed as a strategic risk for the Department. Now it is seen as a strategic asset that will position the Department for joint all-domain operations and artificial intelligence applications. Commensurate with Department-level data policy, Director, Operation Test and Evaluation has published new policy that test programs shall create data management plans to make test data VAULTIS (visible, accessible, understandable, linked, trustworthy, interoperable, and secure). In this briefing, I will motivate the necessity for testers to take an intentional approach to data management, tour the new policies for test data, and provide an overview of the Data Management Plan Guidebook – an approach to planning for test data management that is in line with DOD’s VAULTIS framework.

Bayesian neural networks (BNNs) combine the remarkable flexibility of deep learning models with principled uncertainty quantification. However, poor scalability of traditional Bayesian inference methods such as MCMC has limited the utility of BNNs for uncertainty quantification (UQ). In this talk, we focus on recent advances in approximate Bayesian inference for BNNs and seek to evaluate, in the context of a real application, how useful these approximate inference methods are in providing UQ for scientific applications. As an example application, we consider prediction of chemical composition from laser-induced breakdown spectroscopy measured by the ChemCam instrument on the Mars rover Curiosity, which was designed to characterize Martian geology. We develop specialized BNNs for this task and apply multiple existing approximate inference methods. We evaluate the quality of the posterior predictive distribution under different inference algorithms and touch on the utility of approximate inference schemes for other tasks, including model selection.

Joseph B. Lyons, a member of the scientific and professional cadre of senior executives, is the Senior Scientist for Human-Machine Teaming, 711^th Human Performance Wing, Human Effectiveness Directorate, Air Force Research Laboratory, Wright-Patterson AFB, Ohio.  He serves as the principal scientific authority and independent researcher in the research, development, adaptation, and application of Human-Machine Teaming.

 Dr. Lyons began his career with the Air Force in 2005 in the Human Effectiveness Directorate, Wright-Patterson AFB, Ohio. Dr. Lyons has served as a thought leader for the DoD in the areas of trust in autonomy and Human-Machine Teaming. Dr. Lyons has published over 100 technical publications including 64 journal articles in outlets focused on human factors, human-machine interaction, applied psychology, robotics, and organizational behavior. Dr. Lyons also served as Co-Editor for the 2020 book, Trust in Human-Robot Interaction. Dr. Lyons is an AFRL Fellow, a Fellow of the American Psychological Association, and a Fellow of the Society for Military Psychologists. Prior to assuming his current position, Dr. Lyons served as a Program Officer for the Air Force Office of Scientific Research and was a Principal Research Psychologist within the Human Effectiveness Directorate.

David Salvagnini serves as the Chief Data Officer at NASA. Since joining NASA in June 2023, David’s role recently expanded to include his appointment in May 2024 as the Chief Artificial Intelligence Officer. In these roles, David will yield synergies between these critical roles especially in assuring data readiness in response to responsible and transparent artificial intelligence (AI).

David formerly served as the Director of the Intelligence Community Chief Information Officer (IC CIO) Architecture and Integration Group (AIG) and Chief Architect. In these roles, he worked with Intelligence Community elements and 5-Eye Enterprise (5EEE) international partners on the development and implementation of reference architectures for interoperability, data sharing and technical advancement of Information Technology (IT) infrastructure, data services, foundational AI services, and other mission capabilities.

Before joining the IC CIO, Mr. Salvagnini held a variety of positions at the Defense Intelligence Agency (DIA), to include Chief Information Office (CIO) Technical Director, Chief Data Officer, and Deputy Chief of the Enterprise Cyber and Infrastructure Services Division. In these roles, David supported the deployment of AI capabilities and the development of analytic tradecraft related to AI use as part of intelligence production. He was appointed to the Senior Executive Service as the Senior Technical Officer for Enterprise IT and Cyber Operations at DIA in June 2016.

Mr. Salvagnini joined DIA as a civil servant in May 2005. Prior to his selection as a senior executive, he served on a joint duty assignment as the Chief Architect for the IC Desktop Environment (IC DTE). In that position, he was responsible for DIA and National Geospatial-Intelligence Agency (NGA) adoption of DTE services, and the transitioning of over 57,000 personnel to the IC Information Technology Enterprise (IC ITE). Previously, Mr. Salvagnini held numerous key leadership positions, to include Deputy Chief, Infrastructure Integration; Deputy Chief, Applications Operations; and Acting Chief, Infrastructure Innovation Division. Mr. Salvagnini's experience includes all aspects of enterprise IT service delivery, including research, engineering, testing, security, and operations.

Mr. Salvagnini retired from the Air Force as a Communications and Computer Systems Officer in May 2005 after having served in a variety of leadership assignments during his 21-year career.

He is a native of Setauket, New York and resides with his family in Falls Church, Virginia.

Army Test and Evaluation Command (ATEC) has been an advocate of Design of Experiments (DOE) since the inception of DOTE DOE policies in the early 2000s. During that time, test designs were primarily D-optimal designs for Operational Tests. As ATEC has endorsed a shift-left mindset, meaning collecting/utilizing Developmental Testing to prove out performance metrics, more statistical analysis models, and therefore more test design options became applicable. With the emphasis on these new avenues for test and evaluation, ATEC has expanded their use of DOE outside of what the test community generally considers the standard Operational and even Developmental Test events. This talk will include test designs for deterministic data, sampling from models, modeling and simulation vs live data, and initial artificial intelligence test cases within ATEC. The designs discussed are not novel in their statistical approach but do shed light on the advances ATEC has made in implementing DOE across multiple stages of test and evaluation.

Frequent user engagements are critical to the success of modern software development, but documentation lacks precise and structured guidance on when and how programs should obtain user feedback. The Software Acquisition Pathway (SWP) was introduced for software-intensive systems as part of the new Adaptive Acquisition Framework (AAF). DOD Instruction 5000.87, “Operation of the Software Acquisition Pathway,” recognizes the unique characteristics of software development and acquisition, including the rapid pace of change in software technologies and the iterative and incremental nature of modern software development methodologies. Success in developing software-intensive systems will require agile and iterative development processes that incorporate user feedback throughout the development process.
This work merges established survey principles with the agile, iterative methods necessary to facilitate rapid delivery of a software capability to the user. Four critical milestones in the software development process were identified to gather user feedback throughout the SWP. Examples are given for building effective surveys to gain insight from the user. This work is presented as a framework for collecting actionable user feedback and generating analysis plans and automated reports at the identified key points. Incorporating user feedback early and throughout software development reduces the risk of developing the wrong product.

Model validation is critical to using models to evaluate systems but requires resource-limited live test to validate. If a simulated model shows the system meets requirements, then the model needs to be validated with live data. Bayes Factors can be applied to determine whether live data is more consistent with the model or with an alternative that the system does not meet requirements. If the Bayes Factor is sufficiently large, then the evidence shows that the model aligns better with the data and therefore adequately represents the system.

This presentation shows how Bayes Factors can be used to validate a model and how to size live tests when using this method. This approach is demonstrated for a model that predicts success and failures. Examples illustrating the methods will be shown and the effect of the factors influencing the required number of tests will be discussed. The simulation results can be represented with a beta distribution that captures the probability of success and compared against an alternative distribution.

A common challenge for the Department of Defense is determining how long a system should be tested for to adequately assess system reliability. Unlike traditional reliability demonstration tests, which rely solely on current test data and often demand an impractical amount of testing, Bayesian methods aim to improve test plan efficiency by incorporating previous knowledge of a system's reliability. Although prior information is often available and has the potential to improve test plan efficiency, evaluators face challenges in applying Bayesian methods to test planning and analysis due to the lack of readily available tools. To address this gap, we developed an easy-to-use R Shiny application which automatically generates recommended test plans to assess system reliability using all available information about a system. Researchers can also use the application to estimate a system’s reliability using Bayesian methods. Through a case study, we show leveraging prior information into the analysis of test data can yield a narrower range of uncertainty for reliability estimates compared to traditional methods.

High-precision mass spectrometry (MS) is a key technology in advancing nuclear nonproliferation analytical capabilities and enabling mission organizations at Savannah River National Laboratory (SRNL). Despite the centrality of precision MS to SRNL projects and organizations, no software currently exists with either basic or advanced data analysis tools and data interactivity functions necessitated by modern high-precision MS, including thermal ionization mass spectrometry (TIMS) and multicollector–inductively coupled plasma–mass spectrometry (MC-ICP-MS). The absence of transparent, optimizable, multifaceted data management and analytics tools in commercial MS software is further compounded by the non-user-friendly and cumbersome experience of using manufacturer-supplied software, whose interface is often buggy and prone to copy/paste errors, mislabeling, and misassignment of samples in a serialized autosampler queue (MC-ICP-MS) or filament turret (TIMS). If left unchecked, such seemingly small user “quality of life” considerations can lead to loss of precious instrument time and/or spuriously reported customer sample results. Insofar as open-source alternatives are concerned, the relevance of the few open-source data tools that do exist is inherently limited. Thus, we aim to develop a comprehensive data analytics software package and R Shiny graphical user interface (GUI) that focuses on flexibility, transparency, and reproducibility.
Within the GUI, the implementation of a Bayesian framework supports the national security focus of DATAWorks 2025 by allowing for cross validation, more conservative uncertainty classification, and improved model performance. We suggest the implementation of Markov chain Monte Carlo (MCMC) and other sampling algorithms to better standardize and quantify the distribution of traceable isotope ratios in support of high precision mass spectrometry data processing. Additionally, we demonstrate the implementation of various priors including hierarchical, uninformative, and informative to allow for further flexibility in model construction. In doing so, we aim to emphasize how mass spectrometrists at US National Laboratories, academia, and beyond can seamlessly implement Bayesian, data-driven analysis into their own research.

Aerodynamic design optimization and database generation have seen a growing need for surrogate aerodynamic models based on computational data. Traditionally, the accuracy and quality of surrogate models is analyzed by examining performance compared to test data. However, in some situations this is not possible such as when cost makes the acquisition of additional data impractical. In that case alternative methods are needed to evaluate the quality of a surrogate model.

Resampling of computational data (e.g., cross validation and bootstrapping) is a technique that can be used to inform the quality of surrogate models and downstream analysis (e.g., structural loading, control robustness). Recent work has shown that modal decomposition-based methods (Principal Component Analysis/Proper Orthogonal Decomposition) enable this type of analysis. By taking subsamples of the snapshot matrix used to generate the model, a distribution of predictions can be generated that reflects the sensitivity of the model to the quality and quantity of input data. A wider spread gives evidence of the need for more data collection.

One of the underlying problems is that resampling the snapshot matrix is discrete by nature, which complicates the identification of a representative, continuous output distribution. The presented work employs quadrature weighting to make discrete matrix resampling continuous, with an arbitrarily small amount of input data neglected instead of an arbitrary number of discrete snapshots. In this way, size in the variations can be controlled by the user to give a better understanding of how the system reacts to both small and large changes in the system. This results in more coherent output distributions, which then yield more meaningful uncertainties on the surrogate models of interest.

To explore the validity and usefulness of the approach, data from a simplified three body launch vehicle were used. Three parameters are explored: aerodynamic angle of attack varying from 0 deg. to 90 deg., vehicle roll from 0 deg. to 90 deg., and freestream Reynolds number from 0.6 to 10 million at low Mach numbers. The flow characteristics in this domain are both variable, making for a challenging test problem, and relevant to contemporary applications. The separated wakes in a majority of the incidence range mandate costly, high-fidelity simulations; in such computationally expensive regimes, the use of surrogate models is essential to be able to characterize the entire parameter space. The presented method thus enables surrogate model uncertainty quantification of both input data scope and significance, where no other data sources are available.

Follow-on operational test and evaluation (FOT&E) and agile approaches to system development need rigorous ways to determine the amount of testing required. These systems change in small increments over time, and traditional frequentist methods for sizing tests that do not account for prior data result in unrealistically large sample sizes for testing the latest update. Bayesian methods mathematically account for previous system data, enabling reduced sample sizes while maintaining scientific rigor. This presentation will demonstrate a Bayesian power prior framework for evaluating binary (pass/fail) reliability of systems undergoing agile development. Using this framework, the presentation will discuss how to plan the number of trials required for a future update. Additionally, the results and lessons learned from application to a DOD program are discussed.

Combinatorial testing (CT) is a black-box approach for software integration testing utilizing test suites like covering arrays that guarantee to identify the presence of faults caused by up to a fixed number of interacting components while minimizing the number of tests required. CT makes different assumptions from other DOE approaches, such as requiring components to have discrete levels. CT has applications in testing Artificial Intelligence-enabled systems (AIES) such as constructing test sets with measurable coverage, identifying factors that define a model’s operating envelope, augmenting training datasets for fine-tuning in transfer learning, testing for fairness and bias, and explainable AI. This short course is intended to introduce how to apply CT to testing AIES to the test practitioner through examples that are recognizable within the DOD and NASA as a collaboration between Virginia Tech (VT), National Institute of Standards and Technology (NIST), and Institute for Defense Analyses (IDA). Topics to be covered at a conceptual level will include: CT theoretical background including measures as well as empirical results from the software testing community; differences between CT and other design of experiments approaches; CT applications for AIES across the AI development lifecycle; and how to know when to use CT within a broader test program and at what level of test. Participants will be guided through hands-on exercises with CT tools including NIST’s Automated Combinatorial Testing for Software Tool (ACTS) for test generation and VT’s Coverage of Data Explorer (CODEX) tool for characterizing the AI model input space. The group will also work through how CT may be applied to practical problems within the DOD. To support the hands-on and practical components of this short course, participants should expect to bring a personal laptop on which they have permission to download and run software tools and, if possible, provide real-world (or surrogate) example problems to the organizers prior to the course. Information regarding the software and request for inputs will be sent to registered participants in advance.

Combinatorial testing (CT) is a black-box approach for software integration testing utilizing test suites like covering arrays that guarantee to identify the presence of faults caused by up to a fixed number of interacting components while minimizing the number of tests required. CT makes different assumptions from other DOE approaches, such as requiring components to have discrete levels. CT has applications in testing Artificial Intelligence-enabled systems (AIES) such as constructing test sets with measurable coverage, identifying factors that define a model’s operating envelope, augmenting training datasets for fine-tuning in transfer learning, testing for fairness and bias, and explainable AI. This short course is intended to introduce how to apply CT to testing AIES to the test practitioner through examples that are recognizable within the DOD and NASA as a collaboration between Virginia Tech (VT), National Institute of Standards and Technology (NIST), and Institute for Defense Analyses (IDA). Topics to be covered at a conceptual level will include: CT theoretical background including measures as well as empirical results from the software testing community; differences between CT and other design of experiments approaches; CT applications for AIES across the AI development lifecycle; and how to know when to use CT within a broader test program and at what level of test. Participants will be guided through hands-on exercises with CT tools including NIST’s Automated Combinatorial Testing for Software Tool (ACTS) for test generation and VT’s Coverage of Data Explorer (CODEX) tool for characterizing the AI model input space. The group will also work through how CT may be applied to practical problems within the DOD. To support the hands-on and practical components of this short course, participants should expect to bring a personal laptop on which they have permission to download and run software tools and, if possible, provide real-world (or surrogate) example problems to the organizers prior to the course. Information regarding the software and request for inputs will be sent to registered participants in advance.

Combinatorial testing (CT) is a black-box approach for software integration testing utilizing test suites like covering arrays that guarantee to identify the presence of faults caused by up to a fixed number of interacting components while minimizing the number of tests required. CT makes different assumptions from other DOE approaches, such as requiring components to have discrete levels. CT has applications in testing Artificial Intelligence-enabled systems (AIES) such as constructing test sets with measurable coverage, identifying factors that define a model’s operating envelope, augmenting training datasets for fine-tuning in transfer learning, testing for fairness and bias, and explainable AI. This short course is intended to introduce how to apply CT to testing AIES to the test practitioner through examples that are recognizable within the DOD and NASA as a collaboration between Virginia Tech (VT), National Institute of Standards and Technology (NIST), and Institute for Defense Analyses (IDA). Topics to be covered at a conceptual level will include: CT theoretical background including measures as well as empirical results from the software testing community; differences between CT and other design of experiments approaches; CT applications for AIES across the AI development lifecycle; and how to know when to use CT within a broader test program and at what level of test. Participants will be guided through hands-on exercises with CT tools including NIST’s Automated Combinatorial Testing for Software Tool (ACTS) for test generation and VT’s Coverage of Data Explorer (CODEX) tool for characterizing the AI model input space. The group will also work through how CT may be applied to practical problems within the DOD. To support the hands-on and practical components of this short course, participants should expect to bring a personal laptop on which they have permission to download and run software tools and, if possible, provide real-world (or surrogate) example problems to the organizers prior to the course. Information regarding the software and request for inputs will be sent to registered participants in advance.

Combinatorial testing (CT) is a black-box approach for software integration testing utilizing test suites like covering arrays that guarantee to identify the presence of faults caused by up to a fixed number of interacting components while minimizing the number of tests required. CT makes different assumptions from other DOE approaches, such as requiring components to have discrete levels. CT has applications in testing Artificial Intelligence-enabled systems (AIES) such as constructing test sets with measurable coverage, identifying factors that define a model’s operating envelope, augmenting training datasets for fine-tuning in transfer learning, testing for fairness and bias, and explainable AI. This short course is intended to introduce how to apply CT to testing AIES to the test practitioner through examples that are recognizable within the DOD and NASA as a collaboration between Virginia Tech (VT), National Institute of Standards and Technology (NIST), and Institute for Defense Analyses (IDA). Topics to be covered at a conceptual level will include: CT theoretical background including measures as well as empirical results from the software testing community; differences between CT and other design of experiments approaches; CT applications for AIES across the AI development lifecycle; and how to know when to use CT within a broader test program and at what level of test. Participants will be guided through hands-on exercises with CT tools including NIST’s Automated Combinatorial Testing for Software Tool (ACTS) for test generation and VT’s Coverage of Data Explorer (CODEX) tool for characterizing the AI model input space. The group will also work through how CT may be applied to practical problems within the DOD. To support the hands-on and practical components of this short course, participants should expect to bring a personal laptop on which they have permission to download and run software tools and, if possible, provide real-world (or surrogate) example problems to the organizers prior to the course. Information regarding the software and request for inputs will be sent to registered participants in advance.

Combinatorial testing (CT) is a black-box approach for software integration testing utilizing test suites like covering arrays that guarantee to identify the presence of faults caused by up to a fixed number of interacting components while minimizing the number of tests required. CT makes different assumptions from other DOE approaches, such as requiring components to have discrete levels. CT has applications in testing Artificial Intelligence-enabled systems (AIES) such as constructing test sets with measurable coverage, identifying factors that define a model’s operating envelope, augmenting training datasets for fine-tuning in transfer learning, testing for fairness and bias, and explainable AI. This short course is intended to introduce how to apply CT to testing AIES to the test practitioner through examples that are recognizable within the DOD and NASA as a collaboration between Virginia Tech (VT), National Institute of Standards and Technology (NIST), and Institute for Defense Analyses (IDA). Topics to be covered at a conceptual level will include: CT theoretical background including measures as well as empirical results from the software testing community; differences between CT and other design of experiments approaches; CT applications for AIES across the AI development lifecycle; and how to know when to use CT within a broader test program and at what level of test. Participants will be guided through hands-on exercises with CT tools including NIST’s Automated Combinatorial Testing for Software Tool (ACTS) for test generation and VT’s Coverage of Data Explorer (CODEX) tool for characterizing the AI model input space. The group will also work through how CT may be applied to practical problems within the DOD. To support the hands-on and practical components of this short course, participants should expect to bring a personal laptop on which they have permission to download and run software tools and, if possible, provide real-world (or surrogate) example problems to the organizers prior to the course. Information regarding the software and request for inputs will be sent to registered participants in advance.

Combinatorial testing (CT) is a black-box approach for software integration testing utilizing test suites like covering arrays that guarantee to identify the presence of faults caused by up to a fixed number of interacting components while minimizing the number of tests required. CT makes different assumptions from other DOE approaches, such as requiring components to have discrete levels. CT has applications in testing Artificial Intelligence-enabled systems (AIES) such as constructing test sets with measurable coverage, identifying factors that define a model’s operating envelope, augmenting training datasets for fine-tuning in transfer learning, testing for fairness and bias, and explainable AI. This short course is intended to introduce how to apply CT to testing AIES to the test practitioner through examples that are recognizable within the DOD and NASA as a collaboration between Virginia Tech (VT), National Institute of Standards and Technology (NIST), and Institute for Defense Analyses (IDA). Topics to be covered at a conceptual level will include: CT theoretical background including measures as well as empirical results from the software testing community; differences between CT and other design of experiments approaches; CT applications for AIES across the AI development lifecycle; and how to know when to use CT within a broader test program and at what level of test. Participants will be guided through hands-on exercises with CT tools including NIST’s Automated Combinatorial Testing for Software Tool (ACTS) for test generation and VT’s Coverage of Data Explorer (CODEX) tool for characterizing the AI model input space. The group will also work through how CT may be applied to practical problems within the DOD. To support the hands-on and practical components of this short course, participants should expect to bring a personal laptop on which they have permission to download and run software tools and, if possible, provide real-world (or surrogate) example problems to the organizers prior to the course. Information regarding the software and request for inputs will be sent to registered participants in advance.

The joint risk analysis of cost and schedule are typically conducted either via parametric methods or using a detailed bottom-up approach by resource-loading schedule networks. Earned value data are not typically used as part of this process. However, there is significant value to utilizing earned value data in order to improve the accuracy of joint cost and schedule risk analyses. This paper will explain how to use a joint cost and schedule risk analysis as an input and then as earned value data are collected, the joint cost and schedule risk probability distributions will be updated using this information. The specific technique used is Bayesian Parameter Learning, which provides a rigorous mathematical framework for updating probability distributions using new information. This presentation is an extension of a prior work that applies the same principle of Bayesian Parameter Learning to improve the predictive accuracy of estimate at completion for cost.

Ensemble methods are comprised of multiple individual models each producing a prediction. Voting schemes are used to weigh the decisions of the individual models, namely classifiers, to predict class. A well-formed ensemble should be formed from classifiers with diverse assumptions, e.g., differing underlying training data, feature space selection, and therefore decision boundaries. Voting scheme methods are often based on consideration of underlying feature space and individual reported classifier confidence in predictions. Diversity across the classifiers is to an advantage, but is not being fully exploited with existing voting schemes. The purpose of the described concept is to enhance current voting scheme approaches by weighing the individual model competence measures ensuring that input data are appropriate to the prediction space of the individual classifiers. Consideration of the individual classifiers in the voting for the specified input will be based on achieving a threshold model competence measure. This approach appends confidence-based schemes with ensuring that inputs are consistent with the training data of the individual models. An application employing random forest classifiers will be demonstrated.

Operationally realistic data to inform machine learning models can be costly to gather. An example is collecting aerial images of rare objects to train an image classifier. Before collecting new data, it is helpful to understand where your model is deficient. For example, it may not be good at identifying rare objects in seasons not well represented in the training data. We offer a way of informing subsequent data acquisition to maximize model performance by leveraging both the toolkit of computer experiments and the metadata describing the circumstances under which the training data was collected (e.g. time of day, location, source). We do this by treating the composition of metadata and the performance of the learner, respectively, as the inputs and outputs of a Gaussian process (GP). The resulting GP fit shows which metadata features yield the best learner performance. We take this a step further by using the GP to inform new data acquisitions, recommending the best circumstances to collect future data. Our method for active learning offers improvements to learner performance as compared to data with randomly selected metadata, which we illustrate on image classification and detection benchmarks.

The Lake City Army Ammunition Plant (LCAAP) in Independence, Missouri produces and tests millions of rounds of small arms ammunition daily, in many cases using decades-old procedures. While these testing methods are effective, the long history of high manufacturing quality for most products suggests they are significantly over-testing the lots in production. To address this issue, we have developed a skip-lot testing procedure that uses a Bayesian approach to estimate the true quality of each lot in production, even as some lots are skipped. By using this updated approach, we can reduce the total number of tests required while controlling the risk of accepting low-quality lots. Simulation results demonstrate that this process both reduces the number of tests required and meets the production facility’s standards for risk exposure.

The current talent pool is struggling to keep pace with the demand for cyber professionals. The rapid pace of technological advancement coupled with the evolving nature of cyber threats has created a constant need for relevant skills. To make cyber careers more accessible, the Office of the National Cyber Director (ONCD) has outlined several initiatives in the National Cyber Workforce and Education Strategy. ONCD policy initiatives include removing unnecessary degree requirements, transitioning to a skills-based hiring approach, expanding work-based learning opportunities, and supporting efforts to bring together employers, academia, local governments, and non-profit organizations. While the federal government has implemented measures such as the Workforce Innovation and Opportunity Act to tackle skills gaps, the aim of our study is to measure the gap between the workforce and unfilled cyber positions and decipher what parameters are necessary to close the skills gap without overfitting for unfit candidates? By implementing data cleaning and isolation of a LinkedIn job postings dataset, we found more than 124,000 cyber job descriptions. With these records, we used key words to bin cyber job descriptions into three distinct levels for each job description. We then used natural language processing and text analytics to identify the knowledge, skills, and abilities (KSAs) undergraduate students obtain from their education. Finally, we extend existing cosine similarity methods to enable us to determine how similar employer job descriptions are to the KSAs of the job applicant population. Using these methods, cyber policy makers and employers have additional tools to ensure that cyber jobs are filled by the correct candidates with applicable experience and credentials.

For more than a decade, IDA has supported DOT&E through detailed analysis of cyber attack pathways. This work has resulted in multiple IDA publications and inputs to DOT&E reports that have informed Congressional, U.S. Cyber Command, and OSD Chief Information Officer decision-making. The IDA attack pathway analysis has used the MITRE on-network ATT&CK framework as a common taxonomy and classification for cyber attack actions, and to enable the development of data standards, analysis methodologies, and reporting. However, the MITRE ATT&CK framework only applies to on-network activity, and does not include a framework for assessing physical security.
Physical penetrations parallel on-network cyber attacks in many ways; both are multi-step activity chains where steps in the attack chain enable one or more later steps. Therefore, it should be possible to analyze the data from Close Access Team (CAT) physical security assessments similarly to how IDA has historically treated on-network cyber attack data.
This presentation provides an overview of the Central Research Project (CRP) work to develop a framework for CAT social engineering and physical intrusion TTPs as an analog to the MITRE ATT&CK framework. After an overview of how IDA currently uses the MITRE ATT&CK framework and program data standards to analyze CAT mission data and produce data trend products in support of the DOT&E Cyber Assessment Program (CAP), we provide an overview of the academic literature used as a foundation for the development of IDA’s own social engineering framework the Social Spoofing Security Analysis Reference (S3AR). S3AR summarizes CAT operator TTPs into four nodes: Planning, ingress, lateral, and objective. These nods span the life cycle of a CAT operator planning their attack (planning node), entering the target location of interest (ingress node), operating within that location of interest (lateral node), and deploying a device on a network of interest (objective node). IDA leveraged academic literature as well as a tailored survey of CAT operations deployed to CAT operators spanning 5 CATs. The results of this survey and their direct tie to the continued development of lateral node of the analysis reference TTPs are presented.
In addition to the original charge to develop an analogous analysis framework to MITRE ATT&CK for physical ingress and social engineering techniques, IDA also generated data collection standards for CAT operations. The three data collection sheets presented are designed to be completed during CAT missions and each align to at least one of the analysis reference nodes: planning and prep sheet (planning node), daily actions sheet (ingress and lateral node), and systems accessed sheet (objective node).

Developmental T&E of Autonomous Systems – Consolidated Challenges and Guidance

This presentation will give an overview of challenges, methodologies, and best practices for Developmental Test and Evaluation of Autonomous Systems. This addresses the novel challenges of removing human operators from DoD systems, and empowering future autonomous systems to independently act in contested environments. These challenges, such as safety, black-box components, data, and human-machine teaming, demand iterative approaches to evaluating the growing capabilities of autonomous systems, to assure trusted mission capability across complex operational environments.

The guidance provided includes lessons learned and best practices for the full continuum of autonomy T&E, such as runtime assurance, LVC testing, continuous testing, and cognitive instrumentation. This guidance leverages emerging best practices in agile and iterative testing to extend success throughout the T&E continuum. By applying these best practices to achieve efficient, effective, and robust DT&E, autonomous DoD systems will be primed for successful operational T&E and operational employment.
The information presented is being published as a new “Developmental T&E of Autonomous Systems Guidebook” which is intended to be a living document contributed by a broad community and will adapt to ensure the best information reaches a wide audience.

The views expressed are those of the author /presenter and do not necessarily reflect the official policy or position of the Department of the Air Force, the Department of Defense, or the U.S. government

Since the Office of the Undersecretary of Defense, Research and Engineering released its Digital Engineering (DE) strategy in 2018, the services along with supporting agencies and industry have been working to transform their practices, improve tooling, and upskill their workforce to realize the vision of a digitally harmonized engineering environment that supports Department of Defense (DoD) weapon system acquisition as well as operations and sustainment of existing weapon systems. In kind, the Test and Evaluation (T&E) community within the DoD and supporting agencies is advancing and institutionalizing the application of DE methods to T&E.

The research team at the Acquisition Innovation Research Center (AIRC) have developed a short course to introduce program offices and other T&E professionals to the basics of implementing DE in support of verification, validation, and accreditation activities. This two-hour course will introduce DE concepts and lifecycles, establish the value proposition of DE methods in DoD acquisition, introduce various DE tools, and share best practices as well as potential challenges in applying DE methods to T&E efforts. The course will address basics in Model Based Mission Engineering, Model Based Systems Engineering, Digital Design, Modeling & Simulation, Model Based T&E Planning, as well as briefly explore application of Generative Artificial Intelligence to T&E. Participants will leave with a working knowledge of how DE can be applied to their specific tasks and have an awareness of some of the tooling employed across the DoD for realizing DE objectives.

Upon the release of the Digital Engineering (DE) Strategy in 2018 by the Office of the Undersecretary of Defense for Research and Engineering, the Department of Defense (DoD) and supporting agencies and industry have been working towards transforming practices, improving tooling, and enhancing a workforce that will contribute to an enhanced engineering environment. This balanced and harmonized environment supports weapon acquisition as well as the upkeep and sustainment of existing weapon systems, contributing to the Test and Evaluation (T&E) community. The advancement and commitment to applying DE methods to T&E are highlighted within the DoD and its supporting agencies.

The research team at the Acquisition Innovation Research Center (AIRC), alongside the University Affiliated Research Center (UARC) in partnership with the Director, Operational Test and Evaluation (DOT&E) have been developing and maturing a proxy as a solution for a DoD system acquisition to enhance the digital transformation of T&E. A testbed and framework have been developed with the mission defined as an Unmanned Ground Vehicle navigating through a minefield and disarming them to allow cargo and troops to be transported safely. The mission title is regarded as Operation Safe Passage (OSP). The proxy for the mission utilizes the Lego Mindstorm™ as a physical model, while digital models include Computer Aided Design (CAD) models, SysML-based models, physics-based models for analysis, and decision dashboards. This integrated approach enables rapid decision-making by connecting architecture models, test planning, and physical testing. The testbed and framework for OSP enhance the vision of prospective transformations in T&E

The digital twin (DT) is a rapidly evolving technology that draws upon a multidisciplinary foundation, integrating principles from computer science, physics, mathematics, statistics, and engineering. Its applications are diverse, spanning industries such as engineering, healthcare, biomedicine, climate changes, renewable energy, and national security. This work aims to discuss the characterization, development, and application of DT as well as to identify both the challenges and opportunities that lie ahead. The study identifies research gaps and a path forward to advance the statistical and computational foundations and applications of DT in the field of reliability engineering and preventive maintenance for the statistical quality control and assurance. Fostering innovation in the total quality management, DT is poised to transform industries. Leveraging advanced data analytics, data science, machine learning (ML), and artificial intelligence (AI), DT enables monitoring, simulation, and optimization of complex systems, ensuring higher quality, greater reliability, and improved decision-making. Addressing the challenges and opportunities, continued investment in the DT technologies will drive the next wave of engineering excellence and operational efficiency.

Data governance determines how an organization makes decisions, while data stewardship determines how these decisions are carried out. Broadly, a data steward is a person with data-related responsibilities. One might execute data-related plans made by data governors, engage with metadata, resolve data issues, or track individual data elements. Benefits of data stewardship include ensuring the quality of data. Taken on an individual or team level, this can lead to savings in time and money and increased accuracy of research and testing. However, the return on investment associated with data stewardship can be best seen if clear goals and data management bottlenecks are identified first.

This presentation will introduce data stewardship and discuss a case where the author applied data stewardship principles to their research practice. Data management strategies will be discussed. Goals, progress, challenges, and lessons learned will be presented.

Recently, analysts from OED’s live fire group reviewed multiple sources to amass a comprehensive list of aircraft combat damage events that are of interest to vulnerability assessments. Many of these events could be found in more than one source. Analysts believe there were events not known to any of the sources and are absent from the dataset. We want to estimate this number of unobserved events, which when combined with observed events would produce a better estimate of the total number of events that actually occurred.
Ecologists developed statistical techniques to estimate the sizes of hard-to-count populations, and these techniques can be applied to other domains where observations are recorded in multiple lists.
In this presentation, I will demonstrate various techniques of multiple system estimation, building on simple intuition. I will then apply it to the combat loss data using several computational tools readily available in R. The analysis will show that the total number of events is likely to be much larger than the observed count.

Eucalyptus – An Analysis Suite for Fault Trees with Uncertainty Quantification

Eucalyptus is a novel code developed by Lawrence Livermore National Laboratory to incorporate uncertainty quantification into Fault Tree Analysis. This tool addresses the challenge of imperfect knowledge in “grey-box” systems by allowing analysts to incorporate and propagate uncertainty from component-level assessments to system-level effects. Eucalyptus facilitates a consistent evaluation of the impact of subject matter expert judgment and knowledge gaps on overall system response by Monte Carlo generation of possible system fault trees, sampling probabilities of the existence of subsystems and components. The code supports the specification of fault trees through text and allows export to various formats, including auto-generated images, easing analysis and reducing errors. It has undergone extensive verification testing, demonstrating its reliability and readiness for deployment, and leverages on-node parallelism for rapid analysis. Example analyses are shown that include the identification of system failure paths and quantification of the value of further information about system components.

In support of Chief Digital and Artificial Intelligence Office (CDAO) ongoing efforts to provide best practices and methods for test and evaluation of artificial intelligence-enabled systems, I was tasked to examine metrics for computer vision models. In particular, I studied and implemented multiclass metrics for computer vision models. A table was produced and lists the strengths and weaknesses of different popular metrics. I then simulated these strengths and weaknesses by using the CDAO's Joint Artificial Intelligence Infrastructure Capability (JATIC), a python-based test and evaluation package, to evaluate models trained on the overhead MNIST2 satellite imaging dataset for the automated target recognition (ATR) use case.

Extending Analysis of Variance (ANOVA) to functional data enables researchers and analysts to better understand how categorical variables influence data that vary continuously over a common domain, such as time or frequency. Functional ANOVA (F-ANOVA) builds upon the strengths of traditional scalar ANOVA, allowing for one-way and two-way analyses to test the equality of mean and covariance functions across groups at a given statistical significance threshold. This capability is particularly valuable for uncovering meaningful insights into functional data at both the group level and the interaction level. In the absence of group and interaction effects, datasets can be confidently pooled, resulting in larger sample sizes and enhanced statistical power.

To address the lack of existing tools for performing F-ANOVA, a custom library was developed and validated, offering unique analytical capabilities while maintaining robust performance. These capabilities include two-way analyses, equality of covariance tests, and greater support when heteroscedasticity is present between groups. This library not only simplifies the application of F-ANOVA but also provides tools tailored for handling diverse functional data scenarios. Best practices for F-ANOVA will be demonstrated in the context of mechanical shock analysis, a domain where functional data is particularly beneficial. However, the library's design makes it broadly applicable to other fields, offering a versatile solution for modern functional data challenges.

Satellite-based remote sensing of greenhouse gases and carbon cycle science are examples of operational science data production use cases where thousands to millions inverse problems have to be solved daily as part of processing pipelines. These inversions are typically computationally costly, and further require rigorous Uncertainty Quantification (UQ) to ensure the reliability of data products for downstream users. Even current state-of-the-art methods face considerable challenges with downlinked data volumes, and these problems are only getting more pressing with upcoming next generation of Earth observing satellites and orders of magnitudes increase in data volume. In this talk, we present recent advances in computationally efficient statistical methods and machine learning to tackle these pressing issues. We present approaches on emulating the costly atmospheric radiative transfer physics models to lower the computational burden of inversions, as well as techniques to emulating a direct solution to the inverse problem together with well-calibrated UQ. Specifically, we focus on efficient Gaussian process regression for forward model emulation, and Gaussian mixture modeling and diffusion-based approaches for the inversion and UQ.

At the Institute for Defense Analyses (IDA), we are advancing test science through the use of machine vision and generative AI to process a database of 50,000 Test and Evaluation Master Plans (TEMPs) in PDF format. This effort is encapsulated in an air-gapped, retrieval-augmented LLM framework known as TEMP Copilot. We demonstrate our methodology using a repository of DOT&E annual reports. Starting with scanned, inaccessible PDFs, these reports are transformed by applying machine learning techniques in combination with LLM-enabled qualitative research tools. Our approach reconstructs, parses, and restructures the dataset to enable efficient querying. In this demonstration, we compare various querying techniques on our dataset and show that a combination of data-centric LLM routing and layered search algorithms outperforms traditional frameworks, such as basic retrieval-augmented generation (RAG) and GRAPH-RAG.

Are you currently NOT USING YOUR ENTIRE DATA STREAM to inform decisions?
Sensors that stream data (e.g., temperature, pressure, vibration, flow, force, proximity, humidity, intensity, concentration, etc.), as well as radar, sonar, chromatography, NMR, Raman, NIR, or mass spectroscopy, all measure a signal versus a longitudinal component like wavelength, frequency, energy, distance, or in many cases - time. Are you just using select points, peaks, or thresholds in your curved or spectral data to evaluate performance? This course will show you how use the complete data stream to improve your process knowledge and make better predictions.

Curves and spectra are fundamental to understanding many scientific and engineering processes. They are created by many types of test and manufacturing processes, as well as measurement and detection technologies. Any response varying over a continuum is functional data.

Functional Data Analysis (FDA) uses functional principal components analysis (FPCA) to break curve or spectral data into two parts - FPC Scores and Shape Components. The FPC Scores are scalar quantities (or weights) that explain function-to-function variation. The Shape Components explain the longitudinal variation. FPC Scores can then be used with a wide range of traditional modeling and machine learning methods to extract more information from curves or spectra.

When these functional data are used as part of a designed experiment, the curves and spectra can be well predicted as functions of the experimental factors. Curves and spectra can also be used to optimize or “reverse engineer” factor settings. In a machine learning application functional data analysis uses the whole curve or spectra to better predict outcomes than employing “landmark” or summary statistical analyses of individual peaks, slopes, or thresholds.

References and links will be provided for open-source tools to do FDA, but in this course JMP Pro 18 software will be used to demonstrate analyses and to illustrate multiple case studies. See how a functional model is created by fitting a B-spline, P-spline, Fourier, or Wavelets basis model to the data. One can also perform functional principal components analysis directly on the data, without fitting a basis function model first. Direct Models include several Singular Value Decomposition (SVD) approaches as well as Multivariate Curve Resolution (MCR).

Curve or spectral data can often be messy. Several data preprocessing techniques will be presented. Methods to cleanup (remove, filter, reduce), transform (center, standardize, rescale), and align data (line up peaks, dynamic time warping) will be demonstrated. Correction methods specific to spectral data including Standard Normal Variate (SNV), Multiplicative Scatter Correction (MSC), Savitzky-Golay filtering, and Baseline Correction will be shown.

Case studies will be used to demonstrate the methods discussed above.

Are you currently NOT USING YOUR ENTIRE DATA STREAM to inform decisions?
Sensors that stream data (e.g., temperature, pressure, vibration, flow, force, proximity, humidity, intensity, concentration, etc.), as well as radar, sonar, chromatography, NMR, Raman, NIR, or mass spectroscopy, all measure a signal versus a longitudinal component like wavelength, frequency, energy, distance, or in many cases - time. Are you just using select points, peaks, or thresholds in your curved or spectral data to evaluate performance? This course will show you how use the complete data stream to improve your process knowledge and make better predictions.

Curves and spectra are fundamental to understanding many scientific and engineering processes. They are created by many types of test and manufacturing processes, as well as measurement and detection technologies. Any response varying over a continuum is functional data.

Functional Data Analysis (FDA) uses functional principal components analysis (FPCA) to break curve or spectral data into two parts - FPC Scores and Shape Components. The FPC Scores are scalar quantities (or weights) that explain function-to-function variation. The Shape Components explain the longitudinal variation. FPC Scores can then be used with a wide range of traditional modeling and machine learning methods to extract more information from curves or spectra.

When these functional data are used as part of a designed experiment, the curves and spectra can be well predicted as functions of the experimental factors. Curves and spectra can also be used to optimize or “reverse engineer” factor settings. In a machine learning application functional data analysis uses the whole curve or spectra to better predict outcomes than employing “landmark” or summary statistical analyses of individual peaks, slopes, or thresholds.

References and links will be provided for open-source tools to do FDA, but in this course JMP Pro 18 software will be used to demonstrate analyses and to illustrate multiple case studies. See how a functional model is created by fitting a B-spline, P-spline, Fourier, or Wavelets basis model to the data. One can also perform functional principal components analysis directly on the data, without fitting a basis function model first. Direct Models include several Singular Value Decomposition (SVD) approaches as well as Multivariate Curve Resolution (MCR).

Curve or spectral data can often be messy. Several data preprocessing techniques will be presented. Methods to cleanup (remove, filter, reduce), transform (center, standardize, rescale), and align data (line up peaks, dynamic time warping) will be demonstrated. Correction methods specific to spectral data including Standard Normal Variate (SNV), Multiplicative Scatter Correction (MSC), Savitzky-Golay filtering, and Baseline Correction will be shown.

Case studies will be used to demonstrate the methods discussed above.

Are you currently NOT USING YOUR ENTIRE DATA STREAM to inform decisions?
Sensors that stream data (e.g., temperature, pressure, vibration, flow, force, proximity, humidity, intensity, concentration, etc.), as well as radar, sonar, chromatography, NMR, Raman, NIR, or mass spectroscopy, all measure a signal versus a longitudinal component like wavelength, frequency, energy, distance, or in many cases - time. Are you just using select points, peaks, or thresholds in your curved or spectral data to evaluate performance? This course will show you how use the complete data stream to improve your process knowledge and make better predictions.

Curves and spectra are fundamental to understanding many scientific and engineering processes. They are created by many types of test and manufacturing processes, as well as measurement and detection technologies. Any response varying over a continuum is functional data.

Functional Data Analysis (FDA) uses functional principal components analysis (FPCA) to break curve or spectral data into two parts - FPC Scores and Shape Components. The FPC Scores are scalar quantities (or weights) that explain function-to-function variation. The Shape Components explain the longitudinal variation. FPC Scores can then be used with a wide range of traditional modeling and machine learning methods to extract more information from curves or spectra.

When these functional data are used as part of a designed experiment, the curves and spectra can be well predicted as functions of the experimental factors. Curves and spectra can also be used to optimize or “reverse engineer” factor settings. In a machine learning application functional data analysis uses the whole curve or spectra to better predict outcomes than employing “landmark” or summary statistical analyses of individual peaks, slopes, or thresholds.

References and links will be provided for open-source tools to do FDA, but in this course JMP Pro 18 software will be used to demonstrate analyses and to illustrate multiple case studies. See how a functional model is created by fitting a B-spline, P-spline, Fourier, or Wavelets basis model to the data. One can also perform functional principal components analysis directly on the data, without fitting a basis function model first. Direct Models include several Singular Value Decomposition (SVD) approaches as well as Multivariate Curve Resolution (MCR).

Curve or spectral data can often be messy. Several data preprocessing techniques will be presented. Methods to cleanup (remove, filter, reduce), transform (center, standardize, rescale), and align data (line up peaks, dynamic time warping) will be demonstrated. Correction methods specific to spectral data including Standard Normal Variate (SNV), Multiplicative Scatter Correction (MSC), Savitzky-Golay filtering, and Baseline Correction will be shown.

Case studies will be used to demonstrate the methods discussed above.

Reliable AI systems require secure AI models. The proliferation of AI capabilities in civilian and government workflows creates novel attack vectors for adversaries to exploit. The Adversarial Robustness Toolbox (ART), created in 2018 by IBM Research, is designed to simulate and evaluate threats targeting modern AI systems, identify which AI models are at greatest risk, and provide methods for risk mitigation. ART is designed to support red-blue/attack-defense test and evaluation operations and contains a broad catalog of state-of-the-art methods encompassing evasion, poisoning, extraction, and inference attacks. ART is accessible as an open-source software repository via APIs that execute the evaluation, defense, certification, and verification of AI models. ART supports a wide range of AI frameworks (e.g. TensorFlow, PyTorch), tasks (e.g. classification, object detection, speech recognition) and data types (e.g. images, video, audio). This enables end users to bring their own custom datasets and AI models to assess model adversarial robustness using a variety of attack types, explore available avenues to mitigate attacks’ impacts, and harden pre-trained models via fine-tuning. Using ART, end users can better understand and ultimately mitigate vulnerabilities. ART is continuing to support mission critical enhancements such as supporting scalability across GPUs and the addition new state of the art methods, including automated detection of adversarial inputs.

In collaboration with the Department of Defense (DoD)’s Chief Digital and AI Office (CDAO), IBM has created an extension of ART to support a suite of adversarial robustness test & evaluation procedures as part of the Joint AI Test Infrastructure Capability (JATIC) program. This extension, the Hardened Extension of Adversarial Robustness Toolbox (HEART), focuses primarily on computer vision use cases including object detection and image classification to bolster model performance against evasion attacks. HEART is available via PyPi and Conda Forge and enables ART’s capabilities to be leveraged through a set of standardized protocols designed to increase ease of use and interoperability across AI model test and evaluation tooling. Specifically, HEART is developed to address key DoD use cases and integrate with DoD workflows. Recently, HEART was deployed on the Navy’s Project Harbinger as an added method of protecting the AI models.

Rising occurrences of extreme heat and humidity, combined with high-altitude conditions, present significant challenges for aviation by increasing density altitude—a critical factor in aircraft performance. Elevated temperatures, humidity, and high-altitude environments reduce air density, impacting engine efficiency, lift, and takeoff capability while extending required runway lengths. These challenges are particularly pronounced at high-altitude airports, where all three factors converge to affect operational safety and efficiency. This study utilizes high-resolution atmospheric projections and aircraft performance modeling to assess risks for global airports and proposes scalable adaptive strategies. Addressing the growing prevalence of extreme heat and humidity requires resilient infrastructure, comprehensive risk assessments, and forward-thinking policies to ensure aviation operations remain safe and reliable in evolving conditions.

Resource decisions (e.g. purchasing and positioning spares) across the DoD supply system are optimally made from a multi-echelon viewpoint, allocating resources between retail sites and a centralized wholesale in tandem to maximize readiness outcomes. Due to the size and complexity of the supply system, it is difficult to draw direct connections between resource decisions and mission outcomes. Thus, the common metrics used to grade performance do not strongly correlate to readiness and result in siloed thinking and inefficient resource allocation.

Using discrete-event simulations of the end-to-end Naval aviation sustainment system, we quantified the readiness and cost impact of sub-optimal resourcing decisions due to siloed decision-making. These inefficiencies are best avoided by taking a multi-echelon approach. However, recognizing the DoD-wide paradigm shift this would require, we identified new wholesale metrics that more strongly tie to flight line readiness to mitigate the inefficiency. Furthermore, quantifying the cost-to-readiness relationship across the supply system can serve as a powerful basis for DoD-wide resource optimization in lieu of multi-echelon approaches.

The field of nondestructive evaluation (NDE) is currently undergoing a digital transformation. NDE 4.0, mirroring the Industry 4.0 concept, seeks to improve inspections by leveraging advancements in computing hardware, machine learning/artificial intelligence, and digital thread/digital twin with large volumes of digital NDE data. These new capabilities cannot be realized without well-curated data sets. However, NDE data poses several major challenges. Data are often complex and unstructured, ranging from large multi-dimensional arrays of raw data to perhaps simplified image or one-dimensional representations. Often, meaningful data representing signals of interest are limited and are often in the presence of significant noise. Critical metadata describing the circumstances of the measurement may often be missing. Data may also be stored in a variety of proprietary formats, potentially restricting access to critical raw data needed to develop new measurement techniques or train AI/ML models. These factors have hindered research and development on these NDE 4.0 concepts. This talk will discuss efforts toward overcoming some of these hurdles and improving long term reusability of NDE data through improved data management practices. This includes development of software tools to improve metadata capture, with an emphasis on user experience and minimizing impact on workflow. Also discussed will be efforts toward development of a new data format standard for NDE data and some of the lessons learned to date.
Portions of this work are funded by the U.S. Air Force through contract FA8650-19-D-5230.

DISTRIBUTION STATEMENT A. Approved for public release: distribution is unlimited.

This mini tutorial focuses on introducing the methods and concepts used in interpretable machine learning, particularly for applications that incorporate tabular operational data.
Much of the current AI focus is on generative AI, which is deeply rooted in uninterpretable neural networks; however, there is still substantial research in interpretable machine learning that regularly outperforms neural networks for classical machine learning tasks such as regression and classification.
This course will summarize supervised vs. unsupervised machine learning tasks as well as online vs offline learning settings. Classical machine learning models are introduced and used to motivate concepts such as bias-variance trade-off, hyperparameter tuning, and optimization algorithms/convergence. Last, a swift overview of recent work in interpretable machine learning will point the audience towards state-of-the-art advances.
The only pre-requisites for this course are familiarity with mathematical notation and elementary linear algebra: mainly being familiar with matrix/vector operations, matrix inverses, and ill-conditioned matrices.

Gaussian Process (GP) models are popular tools in uncertainty quantification (UQ) because they purport to furnish functional uncertainty estimates that can be used to represent model uncertainty. It is often difficult to state with precision what probabilistic interpretation attaches to such an uncertainty, and in what way is it calibrated. Without such a calibration statement, the value of such uncertainty estimates is quite limited and qualitative. I will discuss the interpretation of GP-generated uncertainty intervals in UQ, and how one may learn to trust them, through a formal procedure for covariance kernel validation that exploits the multivariate normal nature of GP predictions, and show examples.

In this study, we use smooth particle hydrodynamics modeling to examine the creation of mission-lethal non-trackable orbital debris from impacts of 10 g, 100 g, and 10 kg spherical and cylindrical objects on small satellite bus structures at 0, 15, 45, and 75 degrees obliquity. Our simulations include impacts at velocities below, approaching, and above the energy-density threshold that typically disables satellite functionality and creates additional lethal debris. We compare the mass distributions resulting from smooth particle hydrodynamics simulations to the distributions derived from NASA’s satellite breakup model; NASA’s approximation of debris generation aligns well with our simulation results for large, trackable masses but deviates from our simulation results for small, non-trackable masses. Results also show only minor differences in satellite damage and debris generation between spherical and cylindrical 10 kg impactors.

Operational Testing and Evaluation with cyber threats informs decision making by providing examples of adversarial cyber threats and the effects those exploits cause to mission performance. This briefing examines the goals for, current problems with, and opportunities for improvement in operational testing in a cyber-contested environment. Namely, operational testing in the DoD too often does not consider the cyber effects against a unit during active missions. This briefing argues the DoD needs to move from evaluating segregated systems, to evaluating integrated systems-of-systems. This also requires operational realism with regards to personnel and system configuration, cyber threat integration with mission performance testing, dedicated and methodical data collection from operational testers, and including advanced cyber-attacks such as supply chain compromises.

This project explores transfer-learning approaches for operationalizing multimodal video captioning, focusing on effectively summarizing longer videos. Our methodology employs a Convolutional Neural Network (CNN) encoder with image, motion, and audio modes and a Long Short-Term Memory (LSTM) decoder architecture, incorporating key-frame extraction to reduce computational overhead while integrating audio features surrounding the key frames to improve caption quality. We begin by training a model on the Microsoft Research Video-to-Text (MSR-VTT) benchmark dataset, primarily containing short video clips. We then operationalize the model by evaluating performance on a context-specific dataset, the Dattalion dataset, which features war footage from the conflict in Ukraine of substantially higher length than the videos in MSR-VTT. To address challenges associated with labeling the Dattalion dataset ourselves and to enable the model to understand context-specific themes in the dataset, we apply transfer-learning by combining baseline training on MSR-VTT with fine-tuning on the Dattalion dataset to better handle context-specific videos of potentially long durations. We investigate the comparative performance before and after transfer-learning by evaluating key metrics — BLEU4, METEOR, ROUGE, and CIDEr. This research aims to provide insights into the effectiveness of transfer-learning for video captioning on longer videos in context-specific environments , with implications for improving video summarization in domains such as disaster response and defense.

Quality control for munitions manufacturing is an arduous process in which trained technicians examine and enhance X-ray scans of each round to determine whether defects are present. In this talk, we will introduce a machine learning model that would allow manufacturers to automate this process - which is particularly suited for supervised learning given its repetitive nature and clearly defined defect characteristics. Four scans are taken for each round at 30-degree deflections from one another. The three distinct zones of the round have different standards for what constitutes a defect. Our preprocessing pipeline involves applying the necessary image enhancement techniques to highlight defects in the scan and then applying an unsupervised masking algorithm to isolate and segment each zone of the round. Isolated zones from all four views are then fed into a zone-specific multiview neural network trained to detect defects. Due to different defect rates in each zone, two zones use a variational autoencoder for anomaly-based detection while one zone uses a convolutional neural network for heuristic-based detection. The implementation of this system stands to save manufacturers significant resources and man-hours devoted to quality control of their rounds.

The Atmospheric Science Data Center (ASDC) is a vital resource for the global scientific community, providing access to a wealth of atmospheric and climate-related data collected through NASA's Earth observing missions. This paper introduces the ASDC, outlining its key functions, data offerings, and user services. By facilitating the discovery, access, and utilization of extensive atmospheric datasets—including those related to radiation budget, aerosols, clouds, and air quality—the ASDC plays a crucial role in advancing Earth system science research. We highlight how researchers, educators, and decision-makers can leverage the center’s resources for applications ranging from climate modeling to air quality monitoring and public health studies. Additionally, we explore the ASDC's commitment to open science, emphasizing its data management practices, user support, and tools for ensuring data accessibility and usability across diverse scientific disciplines. This introduction aims to guide users in navigating the ASDC’s data portal and effectively integrating these datasets into their research workflows for enhanced environmental understanding and decision-making.

Non-destructive evaluation is an integral component of engineering applications that is subject to variability and uncertainty. One such example is the use of ultrasonic waves to infer bond strength of a specimen of two adhered composite materials by first inferring the specimen’s interfacial stiffness using noisy phase angle measurements and then propagating the result through an established linear regression relating interfacial stiffness to bond strength. We apply optimization-based confidence intervals to obtain the interfacial stiffness confidence interval and then propagate the result through the existing linear regression such that the final bond strength interval achieves finite-sample coverage under reasonable assumptions. Since we have access to a parameterized forward model of the ultrasonic wave propagation through the specimen along with known physical constraints on the model parameters, this technique is a particularly effective approach to leverage known information without relying on a subjective prior distribution. Applying this technique requires two methodological innovations; incorporation of unknown noise variance and the propagation of the resulting interval through an existing linear regression output. Additionally, the forward model characterizing the propagation of ultrasonic waves is highly nonlinear in the parameters, necessitating interval calibration innovation. We compute a variety of intervals and demonstrate their statistical validity via a simulation study.

Batch weighing is a common practice in the manufacture, research, development, and handling of product. Counting individual parts can be a time consuming and inefficient process, and the ability to batch weigh can save time and money. The main downside of batch weighing is the potential risk of error in the estimated quantity due to tolerance and noise stacking. The methodology highlighted in this presentation aims to directly address and alleviate this risk by quantifying it using Monte Carlo simulation and Discriminant Analysis, a supervised machine learning modeling approach. The final model can be used to inform the user of the specific risk associated with each batch based on weight and allow for less potential for misclassification. The presentation also discusses guidelines for applying the methodology, and remedial methods for certain issues that may arise during its application, using a case study to help illustrate the method’s benefits.

Space-filling designs play a critical role in efficiently exploring high-dimensional input spaces, especially in applications involving simulation, autonomous systems, and complex physical experiments. While a variety of methods exist for generating such designs, most rely on rectangular constraints, fixed weighting schemes, or limited support for nominal factors. In this presentation, we introduce a novel approach based on sliced optimal transport that addresses these limitations by enabling the creation of designs with significantly improved space-filling properties—offering enhanced minimum inter-point distances and better uniformity compared to existing methods. Our approach accommodates arbitrary non-rectangular domains and weighted target distributions, ensuring that practitioners can capture realistic constraints and focus sampling where it matters most. The sliced formulation naturally extends to multi-class domains, enabling a unified design across any number of categorical factors in which each sub-design maintains favorable space-filling properties, and the classes collectively collapse into a well-distributed design when nominal factors are ignored. Furthermore, our method is computationally efficient, readily scaling to hundreds of thousands of design points without sacrificing performance—an essential feature for testing AI and autonomous systems in high-dimensional simulation environments. We will demonstrate the theory behind sliced optimal transport, outline our algorithms, and present empirical comparisons that highlight the benefits of this method over existing space-filling approaches.

As some DoD programs extend beyond their original decommission dates, ongoing subcomponent testing is crucial to ensure system reliability and study aging effects over the extended service life. However, surveillance efforts often face challenges due to limited asset availability, as predetermined quantities are typically allocated to match the original decommission timeline. Options such as recycling assets or procuring additional units are frequently constrained by destructive testing, high costs, long lead times, or a lack of capable manufacturers. Consequently, programs face depleting supplies, leading to a halt in surveillance efforts to sustain life-extension, risking undetected performance degradation.

This work presents a simulation-based approach to optimize test quantities and testing intervals while maintaining reliable detection of performance degradation given the surveillance study design, current data characteristics and model specification. By simulating datasets that mirror the original dataset, using the parameter coefficients and residual standard deviation derived from a fitted model, the approach estimates statistical power. Starting with a fixed test size and interval (e.g., n = 1 and t = 1), an equivalent model is sequentially fitted to each dataset, adding simulations cumulatively. This process is repeated across various test sizes and intervals to determine the optimal set with sufficient power to detect degradation consistent with the original dataset’s effect size and variability. While a simple linear regression is specified for demonstration, the approach is flexible and can potentially accommodate other models that involve hypothesis testing such as nonlinear models or generalized additive models (GAMs).

A key challenge is the increased risk of inflated false-positive rates as accumulating data are analyzed over time given the sequential nature of surveillance data collection. To address this, sequential methods, commonly used in clinical trials (e.g., Haybittle-Peto and O’Brien-Fleming boundaries) can be employed and integrated into the simulation framework. These methods help balance the need for interim analyses with the risk of false positives in long-term surveillance studies, ensuring statistical rigor.

Results demonstrate that variability, effect size, test quantity and testing intervals collectively affect the ability to detect true aging trends. Visualizations highlight the idea that larger effect sizes with low variability are more likely to reveal true aging trends with less data compared to smaller effect sizes in high-variability settings. This approach can enable programs to tailor test schedules based on achieved power for individual parameters, adjusting the overall test quantity and testing frequency for a subcomponent as necessary while maintaining confidence in detecting performance degradation over time. Alternatively, the test frequency and quantity can be increased to accelerate the identification of aging trends and emerging issues. Extensions and limitations of this approach are also planned for as a discussion.

In the event of a conflict, the Department of Defense (DOD) anticipates significant disruptions to their ability to resupply deployed units with the spare components required to repair their equipment. Simply giving units enough additional spares to last the entirety of the mission without resupply is the most straightforward and risk-averse approach to ensure success. However, this approach is also the most expensive, as a complete duplicate set of spares must be purchased for each unit, reducing the number of systems that can be so augmented on a limited budget. An alternative approach would be to support multiple combatant units with a common set of forward-positioned spares, reducing the duplicative purchasing of critical items with relatively low failure rates and freeing up funding to support additional systems. This approach, while cost-effective, introduces a single point of failure, and presupposes timely local resupply.
We have used Readiness Based Sparing (RBS) tools and discrete event simulations to explore and quantify effectiveness of different strategies for achieving high availability in a contested logistics environment. Assuming that local, periodic resupply of spares is possible, we found that creating a centralized pool of forward-positioned spares dramatically decreases the overall cost for a given readiness target compared to augmenting each individual unit with additional spares. Our work ties dollars spent to readiness outcomes, giving DOD leadership the tools to make quantitative tradeoffs.

Understanding how weapon systems and platforms will perform in cyber-contested environments is crucial for making rational programmatic and engineering decisions. Understanding the cyber survivability of systems as part of full-spectrum survivability is particularly difficult.

Probabilistic attack trees, that combine attack surface analysis, vulnerability analysis, and mission loss into an overall risk picture provide a productive approach to this challenge. Attack trees can be used to show all of the individual pathways an attacker could follow to lead to a particular mission loss, and if the probabilities of the lowest level on the attack tree is determined in some way, the rest of the probabilities across the tree can be easily calculated providing the likelihood that an adversary will achieve a particular effect, and the likelihood of the various pathways an attacker might utilize to get to that effect.

Assigning the initial probabilities is the most challenging and contentious part of this approach, but it can be accomplished using a three tiered process. First, any available historical data should be used to generate probabilities of leaf nodes, including historical, test, and architectural data. Second, simple linear models can be built using a combination of data, human experts, and AI. Finally, direct assessment can be utilized for leaves that do not yet have applicable data or models built using a combination of SMEs and AI models.

The initially completed attack tree can then be analyzed to find opportunities where additional data or models can be applied, with a focus on those leaves that appear to have the greatest impact on the overall mission risk. Mitigations and design changes can be considered and the attack tree recalculated with those changes, providing an easy and compelling way to understand and present return on investment for different options.

Probabilistic attack trees have the potential to become a cornerstone of modern cyber risk assessment with quantitative results that are transparent, repeatable, and easily understood enabling defense programs and operators to make better decisions. This approach offers a reliable and scalable way to safeguard mission-critical platforms and weapon systems enabling them to continue to function as intended despite whatever an adversary may throw at them.

In 2023, the Scientific Test and Analysis Techniques Center of Excellence (STAT COE) assisted the planning of a test for the effect of storage conditions on the protective coating of launched devices. It was observed that the protective coating had developed potentially indented striations on support surfaces, which could affect the flight properties of the device in question. However, the extent of these striations under various potential storage conditions were unknown. A physics-based model to simulate the extent of striations over time was updated. In addition to validation of the physics-based model, the model was slow and expensive to run and the team wished to approximate or extrapolate results in a larger factor space than feasible with the simulation. The test team had initially planned a replicated full factorial design to characterize the behavior of the protective coating. With the assistance of the STAT COE, they were able to identify the meaningful depth of striation that would be necessary to detect and estimate the inherent variability of the system. With this information the team was able to substantially reduce testing, saving the program more than $200,000 and schedule impact while sufficiently characterizing the behavior of the system under test.

Responsible AI (RAI) is a critical framework that ensures artificial intelligence (AI) systems are designed, developed, and deployed responsibly, with trust and safety as a primary consideration, alongside ethical and legal use. As AI becomes increasingly pervasive across various domains, including business, healthcare, transportation, the military and education, it is essential that we prioritize responsible principles, policies, and practices. This RAI one-day short course ensures practitioners have the critical knowledge, skills, and analytical abilities needed to identify and address opportunities and challenges in the design, development, and deployment of systems that incorporate AI.

Responsible AI (RAI) is a critical framework that ensures artificial intelligence (AI) systems are designed, developed, and deployed responsibly, with trust and safety as a primary consideration, alongside ethical and legal use. As AI becomes increasingly pervasive across various domains, including business, healthcare, transportation, the military and education, it is essential that we prioritize responsible principles, policies, and practices. This RAI one-day short course ensures practitioners have the critical knowledge, skills, and analytical abilities needed to identify and address opportunities and challenges in the design, development, and deployment of systems that incorporate AI.

All projects are inherently uncertain. To make wise decisions, projects should incorporate an analysis of uncertainty in considering potential alternatives. One process that provides an approach to incorporating uncertainty in the decision-making process is known as risk-informed decision making (RIDM). RIDM can be applied whenever there is uncertainty involved and competing alternatives. This tutorial provides a comprehensive of everything needed to conduct RIDM analysis. Examples of technical failures, cost overruns, and schedule delays for historical project projects are provided, giving clear evidence of project uncertainty and provide strong motivation for conducting risk analysis. The terminology of risk is presented. Risk and uncertainty for a variety of sources – cost, schedule, technical, and safety – are discussed. The mathematical basis for risk analysis is discussed, along with a simple worked example. The tools and techniques necessary for the conduct of risk analysis are provided as well. Best practices and potential pitfalls are provided. The RIDM process is outlined and contrasted with a similar approach called risk-based decision making. An overview of risk management is provided and its relationship with RIDM is described. The tutorial concludes notional case studies of RIDM.

Modeling the equation of state (EOS) of chemically dissociating materials at extreme temperature and density conditions is necessary to predict their thermodynamic behavior in simulations and experiments. However, this task is challenging due to sparse experimental and theoretical data needed to calibrate the parameters of the equation of state model, such as the latent molar mass surface. In this work, we adopt semi-parametric models for the latent molar mass of the material and its corresponding free energy surface. Our method employs basis representations of the latent surfaces with regularization to address challenges in basis selection and prevent overfitting. We show with an example involving carbon dioxide that our method improves model fit over simpler representations of the molar mass surface while preserving low computational overhead. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344. LLNL-ABS-872125

There are few recommended methods to help testers plan efficient modeling and simulation studies. Space-filling designs are a rigorous choice, but one of their drawbacks is that they require the final sample size to be selected prior to testing. More efficient testing can be completed by using sequential designs, which choose test points without knowledge of the final sample size. Using sequential designs can prevent oversampling and help to augment poorly designed tests. We provide an overview of sequential space-filling designs, with the focus on designs that are most suitable for the test and evaluation community.

Space is becoming increasingly crowded with both satellites and orbital debris as more proliferated constellations come online. While research has shown that self-sustained run-away growth of orbital debris (a.k.a. Kessler Syndrome) is unlikely in the future, crowded space lanes are degrading satellites’ ability to perform their mission today. To quantify these operational impacts, we built the Space Environment Risk Prediction by Evaluating Numerical Simulations (SERPENS) model. SERPENS uses a high-fidelity, physics-based propagator to precisely calculate orbits and evaluate conjunction scenarios. Here, we use SERPENS to simulate the space environments after various debris-creating events and predict the operational consequences on particular satellites and constellations. SERPENS provides IDA with a test-bed to evaluate the operational impacts of future DoD capabilities including upgrades to space domain awareness (SDA) infrastructure and satellite collision avoidance methodologies.

Interplanetary spacecraft are exposed to meteoroid fluxes with characteristics far exceeding the physical simulation capabilities of test facilities for predicting the likelihood that a meteoroid will penetrate a spacecraft’s critical systems. Accurate risk predictions are crucial to ensuring that important interplanetary missions, such as sample returns, can survive years of exposure to the meteoroid environment and safely reenter the Earth’s atmosphere with their scientific cargo.

In this paper, we summarize a series of meteoroid impact damage computational simulations into two types of spacecraft composite protective structures using the Smooth Particle Hydrodynamics Code. We consider the effects of both meteoric materials and non-meteoric materials on a shielded forebody thermal protection system (TPS) for an extreme entry environment and on an unshielded aftbody TPS that is similar to the material covering the space shuttle’s external tank. Key to ensuring spacecraft reentry survival is understanding the potential damage from a meteoroid to a spacecraft’s TPSs, which may be housed beneath protective “garage” shielding enclosures.

This analysis was presented at the Hypervelocity Impact Symposium 2024 [2] and is a follow-on effort to work published in 2022 by Williamsen et al. [1], who evaluated select meteoroid impact simulations into two types of spacecraft composite protective structures using the Smooth Particle Hydrodynamics Code (SPHC). Both past [1] and present [2] analyses support ongoing National Aeronautics and Space Administration Engineering Safety Center tasks. Here, we present the SPHC predictions for a different type of multi-shock shield than in [1]. We also report patterns discovered in our analysis that reveal how this multi-shock shield and underlying TPS respond to physical impact.

The meteoritic impactor materials we investigated in this study are iron, ice, and dunite (chondrite); the non-meteoritic impactor material is aluminum. We discuss the insights SPHC offers regarding debris cloud characteristics and forebody TPS damage, and then we use these insights to identify recognizable trends in forebody TPS penetration depth based on impact energy and overmatch energy. We leverage a new general multi-shock shield ballistic limit equation [3] to provide ballistic limit data that are missing from our limited set of SPHC predictions. Finally, we evaluate the SPHC predictions for aluminum particles impacting the aftbody TPS at three obliquities.

[1] Williamsen, Joel, et al. “Prediction and Enhancement of Thermal Protection Systems from Meteoroid Damage Using a Smooth Particle Hydrodynamic Code.” Proceedings of 2022 Hypervelocity Impact Symposium. Paper #: HVIS2022-09XWV8VB6X. Alexandria, VA, September 18-22, 2022.

[2] Corbett, Brooke et al. "Smooth Particle Hydrodynamic Code Predictions for Meteoroid Damage to Thermal Protection Systems Shielded by Composite Structures." Proceedings of 2024 Hypervelocity Impact Symposium. Paper #: HVIS2024-013. Tsukuba, Japan, September 9-13. 2024.

[3] Schonberg, William P., et al. “Toward a More Generalized Ballistic Limit Equation for Multi-Shock Shields.” Acta Astronautica Vol. 213 (2023): pp. 307-319.

Source identification is an inferential problem that evaluates the likelihood of opposing propositions regarding the origin of items. The specific source problem refers to a situation where the researcher aims to assess if a particular source originated the items or if they originated from an alternative, unknown source. Score-based likelihood ratios offer an alternative method to assess the relative likelihood of both propositions when formulating a probabilistic model is challenging or infeasible, as in the case of pattern evidence in forensic science. However, the lack of available data and the dependence structure created by the current procedure for generating learning instances can lead to reduced performance of score likelihood ratio systems. To address these issues, we propose a resampling plan that creates synthetic items to generate learning instances under the specific source problem.
Simulation results show that our approach achieves a high level of agreement with an ideal scenario where data is not a limitation and learning instances are independent. We also present two applications in forensic sciences -handwriting and glass analysis- illustrating our approach with both a score-based and a machine learning-based score likelihood ratio system. These applications show that our method may outperform current alternatives in the literature.

Each test event, ATEC test centers collect tens, hundreds, or even thousands of Test Incident Reports (TIRs) capturing issues with systems. Currently AEC evaluators score these TIRs one by one based on their severity, mission impact, and cause, which can be extremely time-consuming and prone to inconsistency. We are developing a TIR Analytics tool (TIRANICS) to accelerate the TIR scoring process. For any one TIR to be evaluated, TIRANICS provides three main capabilities: (1) lists of similar TIRs from current and past test events, (2) a recommended score, and (3) references to the Failure Definition and Scoring Criteria (FDSC) Guide for the system under test. To provide these capabilities, TIRANICS applies natural language processing techniques, namely, term frequency and transformer-based embeddings, similarity metrics, hierarchical text splitting, and neural network classifiers, to both the TIR narratives and/or the corresponding FDSC for the system under test. In the way we have processed and represented the corpus to capture semantic meaning, this also paves the foundation for utilizing a Large Language Model with the text as a knowledge base to further provide an evaluator with a recommended score and reasoning for that score.

Orbital debris collision hazard and population evolution modeling are foundational for space safety, space sustainability, and space security. The entangling dependencies of these three domains require persistent orbital intelligence to maintain sufficient situational awareness to satisfy space operators. LeoLabs provides this service by leveraging a global network of state of the art radars, a cloud-based computational/distribution engine, physics-based utilities, advanced visual analytics and machine learning tools. These capabilities combine to provide precise alerts in near real-time to space operators and contextual insights to space planners/policymakers. The suite of tools available via LeoLabs is examined and how they combine to serve a variety of demands for situational awareness. An emphasis is placed on balancing results between reacting to tactical events all the way to identifying strategic trends – both requiring immediate attention.

Communicating technical concepts—clearly, concisely, and with purpose—is a skill that gives you the edge. Whether influencing business decisions, shaping research pathways, or simply growing as a technical professional, an ability to write distinguishes you from your peers. Learn the simple tools to make your technical content stand out.

At The Technical Pen, we’ve found that strong technical writers showcase their skills in three primary categories: document layout, written voice, and audience engagement. This short course is built around core principles in each of these categories, offering actionable techniques to improve technical writing skills. Throughout the day, we will work on building a meaningful structure, energizing your written voice, and keeping your audience on track.

The classic DoD T&E paradigm (Operational Effectiveness-Suitability-Survivability-Safety) benefits from 40 years of formalism and refinement and has produced numerous specialized testing disciplines (e.g., the -ilities), governing regulations, and rigorous analysis procedures. T&E of AI-enabled systems is still new, and the laboratory of ideas is a constant source of new testing options, the majority of which focus on performance. The classic gap between measures of performance and measures of effectiveness is very much present in AI-enabled systems and the over emphasis on performance testing means we might miss the (effectiveness) forest for the (performance) trees.

Borrowing from the classic “integrated survivability onion” conceptual model, we propose a set of integrated and nested evaluation questions for AI-enabled systems that covers the full range of classic T&E considerations, plus a few that are unique to AI technologies within the military operational context and implied by the DoD Responsible AI Guidance. All requirements for rigorous analytical and statistical techniques are preserved and new opportunities to apply test science are identified. We hope to prompt an exchange of ideas that moves the community toward filling significant T&E capability gaps – especially the gaps between performance and effectiveness – and advancing a whole-program evaluation approach.

The development of planetary entry technologies relies heavily on computational modeling due to limited ground and flight test data, which simulate the extreme environments encountered by spacecraft during atmospheric entry. Besides the inherent uncertainties associated with the hypersonic entry environment, computational models include significant uncertainties that may affect the prediction accuracy of the simulations. This makes uncertainty quantification a necessary tool for predicting the flight uncertainty with computational models and improving the robustness and reliability of entry systems. In this talk, after outlining the approach for flight uncertainty prediction of hypersonic entry systems, I will focus on presenting an overview of our research on efficient uncertainty quantification (UQ) and sensitivity analysis (SA) based on non-intrusive polynomial chaos theory applied to aerothermal prediction of entry systems. The application examples will include the demonstration of the UQ and SA methods on the stagnation point heat flux prediction of a hypersonic vehicle with a reduced order correlation and thermal protection system response of a hypersonic inflatable aerodynamic decelerator.

Our presentation explores the application of Consensus-based Distributed Ledger Technology (C-DLT) in the testing and evaluation of collaborative adaptive AI-enabled systems (CA2IS). It highlights the potential of C-DLT to enhance real-time data collection, data validation, synchronization, and security while providing a trusted framework for model & parameter sharing, access control, multi-agent data fusion, and (as emphasized for this topic area) novel methods for continuous monitoring and Test & Evaluation of CA2IS systems.
As autonomous multi-agent systems (AMAS) evolve, the underlying probabilistic foundation of increasingly AI/ML-enabled systems necessitates the adoption of new approaches to system Test and Evaluation (T&E) that extend traditional range-based & fixed scenario-based repetitive testing methods that have primarily evolved for the testing of historically deterministic systems - into capabilities to assess the performance expectations of probabilistic systems for applications in highly complex and dynamic environments where both operating conditions and system performance may be constantly changing and continuously adapting. Ideally, T&E methods would seamlessly extend from developmental testing, through acceptance and validation testing, and into missions operations (of critical importance as autonomous weapons-capable systems emerge). Additionally, these provisions may provide a mechanism for enhancing system resilience in contested environments through the inherently distributed, time ordered, and redundant records of on-board status, diagnostic & behavioral indicators, as well as, inter-agent communications and data transactions, collectively providing for continuous assessment of system performance to enable real-time characterization of system confidence as an input to well-informed decision making – whether such decisions are made via an agenic AI, or human-in-the-loop, context.
The proposed C-DLT framework addresses these considerations by enabling both onboard and in-situ monitoring combined with system-wide record synchronizations to capture the real-world context and dynamics of inter-agent behaviors within a global frame of reference model (and common operating picture). Additionally, the proposed method provides for recurring periodic testing of operational AI/ML-based systems, emphasizing and addressing the dynamic nature of collaborative adaptive Continuous Learning Systems (CLS) as they incorporate new training data, and adapt to new operational environments and changing environmental conditions. The performance trade-offs and T&E challenges that arise within this vision of CA2IS underscore the necessity for the proposed in-situ testing method.

We explore an uncertainty quantification (UQ) and design optimization pipeline for thermal analysis of spacecraft by incorporating an industry standard commercial tool, Thermal Desktop (TD), with the Sandia Dakota framework. Optimizing design elements manually can be tedious and engineers often rely on their own strategy, leading to non-standardized and often unscalable practices. Similarly, timely and efficient uncertainty quantification is limited by the lack of standard interfaces to solvers and a lack of access to robust and reliable uncertainty routines. Sandia’s Dakota framework solves many of the challenges, free of charge, and has detailed documentation. We leverage Dakota’s UQ and optimizations routines by connecting it to TD’s API called OpenTD. We explore the challenges and viability of this pipeline for performing thermal design optimization and thermal uncertainty quantification on a simplified satellite model.

As technology becomes increasingly capable in modern defense and aerospace systems, the human operator is often forced into a state of information overload. Advancements in technology, such as AI-enabled automation, has offset the cognitive overload to an automated agent; however, previous research has shown that as automation increases, situational awareness may decrease as the human moves out-of-the-loop. The interplay between fully autonomous and fully manual systems introduces a dichotomy of placing the operator either in cognitive overload (fully manual) or cognitive underload (fully autonomous), and it is important to design systems such that human agents and automated agents can collaborate optimally. Cognitive workload is difficult to measure and is often captured subjectively and intrusively (e.g., NASA-TLX); therefore, our work focuses on a non-intrusive means of capturing operator state through physiological measurement. Physiological measures, such as heart rate (HR) and heart rate variability (HRV), have shown to be associated with cognitive workload; therefore, the objective of our work is to develop a Bayesian statistical model to quantify physiological state to determine a quantifiable trigger for turning on and off automation in the flight deck.

An experimental study with 45 participants was performed using the OpenMATB environment which is an open-source version of NASA’s Multi-Attribute Task Battery (MATB). The sub-tasks in the OpenMATB environment were either in automated and manual mode, and participants interacted with the interface under 5 levels of automation ranging from no automation (level 0), partially automated (levels 1-3), and fully automated (level 4). Automation reliability was treated as a between-subjects factor with three levels: 50%, 70%, and 99.9% reliable.

A hierarchical Bayesian model was built to quantify HR data collected from an ECG strap worn by participants. The Normal-Normal conjugacy was utilized to create posterior estimates for each participant’s HR under different experimental conditions. The hierarchical model produces posterior estimates for group level (e.g. reliability condition) and participant level (e.g. automation level) factors. As the model is fed increasingly specific physiological datasets, it quantifies more of the uncertainty associated with continuous HR data. Due to its hierarchical nature, the final output of the model shows the individual differences in HR as it depends on automation level and reliability condition. Therefore, our model is able to quantify current physiological state as well as predict how the operator may change physiologically in the future as the automation and reliability change.

The research team is currently exploring how to utilize this model to trigger adaptive automation for defense and aerospace applications using pilot and soldier data. The current work shows that we can predict future physiological state; therefore, future work will investigate how to then adjust the automation level if the operator’s physiological state trends towards cognitive overload or cognitive underload. The long-term goal of this work is to design and develop an adaptive automation system that utilizes a non-invasive physiological and neurological data collection approach to quantify cognitive workload and adjust automation to fit the individual operator.

• Jerome Perkins
• Using the Logistics Composite Model (LCOM) and Bayesian Statistics to Evaluate the B 21 / B-52J Availability in Operational Test
• Test and Evaluation (Suitability)
• Abstract
This presentation addresses the integration of the Logistics Composite Model (LCOM) and Bayesian statistics within the B-21/B-52J Operational Test and Evaluation (OT&E). With the increasing complexity of modern defense systems, ensuring reliability and readiness through comprehensive testing and evaluation is crucial. The Logistics Composite Model provides a framework for assessing availability, logistics support and system sustainment, while Bayesian statistics offer a method for incorporating prior knowledge and managing uncertainties in the evaluation process. By combining these methodologies, the presentation will showcase how to enhance analysis, leading to more accurate predictions of system performance and potential shortfalls. Attendees will be introduced to techniques and examples demonstrating the effective application of LCOM and Bayesian approaches in operational test and evaluation. The goal is to equip defense analysts and testers with tools and techniques to improve evaluation of systems with small sample sizes and limited test time to ensure operational suitability.

During the 2016 Conference on Applied Statistics in Defense (CASD), we presented a paper describing “The DASE Axioms.” The paper included several “divide-and-conquer” strategies for addressing the Curse of Dimensionality that is typical in simulated systems.

Since then, a new, integrate-and-conquer approach has emerged, which applies decision-theoretic concepts from Bayesian Analysis (BA). This paper and presentation re-visit the DASE axioms from the perspective of BA.

Over the past fifteen years, we have tailored and expanded conventional design-of-experiments (DOE) principles to take advantage of the flexibility that is offered by modeling, simulation, and analysis (MSA). The result is embodied within three, high-level checklists: (a) the Model Description and Report (MDR) protocol enables iteratively developing credible models and simulation (M&S) for an evolving intended use; (b) the 7-step Design & Analysis of Simulation Experiments (DASE) protocol guides credible M&S usage; and (c) the Bayesian Analysis (BA ) protocol enables fully quantifying the uncertainty that accumulates, both when building and using M&S.

When followed iteratively by all MSA stakeholders throughout the product lifecycle, the MSA protocols result in effective and efficient risk-informed decision making.

The paper and presentation include several quantitative examples to show how the three MSA protocols interact. For example, we show how to use BA to combine Sim and field data for calibrating M&S. Thereafter, given a well-specified query, adaptive sampling is illustrated for optimizing usage of high-performance computing (HPC), either to minimize resources required to answer a specific query, or to maximize HPC utilization within a fixed time period.

The Bayesian approach to M&S development and usage reflects a shift in perspective, from viewing MSA as mainly a design tool, to being a digital test and evaluation venue. This change renders fully relevant all of the attendant operational constraints and associated risks regarding M&S scheduling, availability, cost, accuracy, and delay in analyzing inappropriately large HPC data sets. The MSA protocols employ statistical models and other aspects of Scientific Test and Analysis Techniques (STAT) that are being taught and practiced within the operational test and evaluation community.

This study employs cosine similarity topic modeling to analyze the curriculum content of AI (Artificial Intelligence) bachelor’s and master’s degrees, comparing them with Data Science bachelor’s and master’s degrees, as well as Computer Science (CS) bachelor’s degrees with concentrations in AI. 97 programs total were compared. 52 topics of interest were identified at the course level. The analysis creates a representation for each of the 52 identified topics by compiling course descriptions whose course title matches into a bag-of-words. Cosine similarity is employed to compare the topic coverage of each program against all course descriptions of required courses from within that program.

Subsequently, Kmeans and Hierarchical clustering methods are applied to the results to investigate potential patterns and similarities among the programs. The primary objective was to discern whether there are distinguishable differences in the topic coverage of AI degrees in comparison to CS bachelor’s degrees with AI concentrations and Data Science degrees.
The findings reveal a notable similarity between AI bachelor’s degrees and CS bachelor’s degrees with AI concentrations, suggesting a shared thematic focus. In contrast, both AI and CS bachelor’s programs exhibit distinct dissimilarities in topic coverage when compared to Data Science bachelor’s and master's degrees. A notable difference being that the Data Science degrees exhibit much higher coverage of math and statistics than the AI and CS bachelor’s degrees. This research contributes to our understanding of the academic landscape, and helps scope the field as public and private interest into AI is at an all-time high.

Operational test and evaluation (OT&E) of a system provides the opportunity to examine how representative individuals and units use the system to accomplish their missions, and complements functionality-focused automated testing conducted throughout development. Operational evaluations of software acquisitions need to consider more than just the software itself; they must account for the complex interactions between the software, the end users, and supporting personnel (such as maintainers, help desk staff, and cyber defenders) to support the decision-maker who uses information processed through the software system. We present a framework for meeting OT&E objectives while enabling the delivery schedule for software acquisitions by identifying potential areas for OT&E efficiencies. The framework includes continuous involvement beginning in the early stages of the acquisition program to prepare a test strategy and infrastructure for the envisioned pace of activity during the develop and deploy cycles of the acquisition program. Key early OT&E activities are to acquire, develop, and accredit test infrastructure and tools for OT&E, and embed the OT&E workforce in software acquisition program activities. Early OT&E community involvement in requirements development and program planning supports procedural efficiencies. It further allows the OT&E community to determine whether the requirements address the collective use of the system and include all potential user roles. OT&E during capability development and deployment concentrates on operational testing efficiencies via appropriately scoped, dedicated tests while integrating information from all sources to provide usable data that meets stakeholder needs and informs decisions. The testing aligns with deliveries starting with the initial capability release and continuing with risk-informed approaches for subsequent software deployments.

The DOD’s materiel commands generally rely on working capital funds (WCFs) to fund their purchases of spares. A WCF insulates the materiel commands against the disruptions of the yearly appropriations cycle, and allows for long-term planning and contracting. A WCF is expected to cover its own costs by allocating its funds judiciously and adjusting the prices it charges to the end customer, but the multi-year lead times associated with most items means that items must be ordered years in advance of anticipated need. Being financially conservative (ordering less) leads to backorders, while minimizing backorders (ordering more) often introduces financial risk by buying items that may not be sold in a timely manner. In this work, we develop an optimization framework that produces a "Buy List" of repairs and procurements for each fiscal year. The optimizer seeks to maximize a financial and readiness-minded objective function subject to constraints such as budget limitations, contract priorities, and historical variability of demand signals. Buy Lists for each fiscal year provide a concrete baseline for examining the repair/procurement decisions of real wholesale planners and comparing performance via simulation of different histories.

Recent advances in computation and statistics led to an increasing use of Federated Models for system evaluation. A federated model is a collection of sub models interconnected where the outputs of a sub-model act as inputs to subsequent models. However, the process of verifying and validating federated models is poorly understood and testers often struggle with determining how to best allocate limited test resources for model validation. We propose a graph-based representation of federated models, where the graph encodes the connections between sub-models. Vertices of the graph are given by sub-models. A directed edge between vertex a and b is drawn if a inputs into b. We characterize sub-models through vertex attributes and quantify their uncertainties through edge weights. The graph-based framework allows us to quantify the uncertainty propagated through the model and optimize resource allocation based on the uncertainties.

Modeling and simulation (M&S) validation for operational testing often involves comparing live data with simulation outputs. Statistical methods known as functional data analysis (FDA) provides techniques for analyzing large data sets ("large" meaning that a single trial has a lot of information associated with it), such as radar tracks. We preview how FDA methods could assist M&S validation by providing statistical tools handling these large data sets. This may facilitate analyses that make use of more of the data available and thus allows for better detection of differences between M&S predictions and live test results. We demonstrate some fundamental FDA approaches with a notional example of live and simulated radar tracks of a bomber’s flight.

Modern artificial intelligence relies upon foundation models (FMs), which are prodigious, multi-purpose machine learning models, typically deep neural networks, trained on a massive data corpus. Many benchmarks assess FMs by evaluating their performances on a battery of tasks for which the FMs are adapted to solve, but uncertainty is usually not accounted for in such benchmarking practices. This talk will present statistical approaches for performing uncertainty quantification with benchmarks meant to compare FMs. We demonstrate bootstrapping of task evaluation data, Bayesian hierarchical models for task evaluation data, rank aggregation techniques, and visualization of model performance under uncertainty with different task weightings. The utility of these statistical approaches is illustrated with real machine learning benchmark data, and a crucial finding is that the incorporation of uncertainty leads to less clear-cut distinctions in FM performance than would otherwise be apparent.

To enable the rapid design and evaluation of survivable reentry systems, the Johns Hopkins University Applied Physics Laboratory (JHU/APL) developed a simulation environment to quickly explore the reentry system tradespace. As part of that effort, a repeatable process for designing and assessing the tradespace was implemented utilizing experimental design and statistical modeling techniques. This talk will discuss the utilization of the fast flexible filling experimental design and maximum value-weighted squared error (MaxVSE) adaptive sequential experimental design methods and Gaussian Process modeling techniques for assessing features that impact reentry system trajectories and enabling continuous model refinements. The repeatable scripts used to implement these methods allow for integration into other software tools for a complete end-to-end simulation of reentry systems.

AVATAR Dynamic Interactive Charting Tool (ADICT) is an advanced Power BI tool that has been developed by the National Aeronautics and Space Administration (NASA) to transform budget forecasting and assist in critical budget decision-making. This innovative tool leverages the power of the M language within Power BI to provide organizations with a comprehensive 15-year budget projection system that ensures real-time accuracy and efficiency. It is housed in Power BI for the capabilities to simultaneously update to the model from our excel file named AVATAR.

One of the standout features of ADICT is its capability to allow users to define and apply rate changes. This feature empowers organizations, including NASA, to customize their budget projections by specifying rate variations, resulting in precise and adaptable financial forecasting. NASA integrates ADICT to SharePoint to house the model to avoid local drives and seamlessly update the model to adjust for any scenario. This tool is used a scenario-based planner as well, as it can provide support for workforce planning and budget decisions.

ADICT seamlessly integrates with source Excel sheets, offering dynamic updates as data evolves. This integration eliminates the need for manual data manipulation, enhancing the overall decision-making process. It ensures that financial projections remain current and reliable, enabling organizations like NASA to respond swiftly to changing economic conditions and emerging challenges.
At its core, ADICT enhances budgeting by transforming complex financial data into interactive visualizations, enabling NASA to gain deeper insights into their financial data and make agile decisions. ADICT also seamlessly integrates with source Excel sheets, offering dynamic updates as data evolves. This integration eliminates the need for manual data manipulation, enhancing the overall decision-making process. It ensures that financial projections remain current and reliable for organizations to respond swiftly to changing economic conditions and emerging challenges.

The ability to run AI at the edge can be transformative for applications that need to process data to make decisions at the location where sensing and data acquisition takes place. Deep neural networks (DNNs) have a huge number of parameters and consist of many layers, including nodes and edges that contain mathematical relationships that need to be computed when the DNN is run during deployment. This is why it is important to benchmark DNNs on edge computers which are constrained in hardware resources and usually run on a limited supply of battery power. The objective of our NASA funded project which is aligned to the mission of robotic space exploration is to enable AI through fine-tuning of convolutional neural networks (CNNs) for extraterrestrial terrain analysis. This research currently focuses on the optimization of the ResNet50 model, which consists of 4.09 GFLOPs and 25.557 million parameters, to set performance baselines on various edge devices using the Mars Science Laboratory (MSL) v2.1 dataset. Although our initial focus is on Martian terrain classification, the research is potentially impactful for other sectors where efficient edge computing is critical.

We addressed a critical imbalance in the dataset by augmenting the underrepresented class of with an additional 167 images, improving the model's classification accuracy substantially. Pre-augmentation, these images were frequently misclassified as another class, as indicated by our confusion matrix analysis. Post-augmentation, the fine-tuned ResNet50 model achieved an exceptional test accuracy of 99.31% with a test loss of 0.0227, setting a new benchmark for similar tasks.

The core objective of this project extends beyond classification accuracy; it aims to establish a robust development environment for testing efficient edge AI models suitable for deployment in resource-constrained scenarios. The fine-tuned ResNet50-MSL-v2.1 model serves as a baseline for this development. The model was converted into a TorchScript format to facilitate cross-platform deployment and inference consistency.

Our comprehensive cross-platform evaluation included four distinct hardware configurations, chosen to mirror a variety of deployment scenarios. The NVIDIA Jetson Nano achieved an average inference time of 62.04 milliseconds with 85.83% CPU usage, highlighting its utility in mobile contexts. An Intel NUC with a Celeron processor, adapted for drone-based deployment, registered an inference time of 579.87 milliseconds at near-maximal CPU usage of 99.77%. A standard PC equipped with an RTX 3060 GPU completed inference in just 6.52 milliseconds, showcasing its capability for high-performance, stationary tasks. Lastly, an AMD FX 8350 CPU-only system demonstrated a reasonable inference time of 215.17 milliseconds, suggesting its appropriateness for less demanding edge computing applications.

These results not only showcase the adaptability of ResNet50 across diverse computational environments but also emphasize the importance of considering both model complexity and hardware capabilities when deploying AI at the edge. Our findings indicate that with careful optimization and platform-specific tuning, it is possible to deploy advanced AI models like ResNet50 effectively on a range of hardware, from low-power edge devices to high-performance ground stations. Our ongoing research will use these established baselines to further explore efficient AI model deployment in resource-constrained setting.

Reproducible research principles ensure that analyses can be verified and defended by meeting the criterion that conducting the same analysis on the same data should yield identical results. Not only are reproducible analyses more defensible and less susceptible to errors, but they also enable faster iteration and yield cleaner results. In this seminar, we will delve into how to conceptualize reproducible research and explore how reproducible research practices align with government policies. Additionally, we will provide hands-on examples, using Python and MS Excel, illustrating various approaches for conducting reproducible research.

Sex trafficking remains a global problem, requiring new innovations to detect and disrupt such criminal enterprises. Our research project is an application of artificial intelligence (AI) methods, and the knowledge of social science and homeland security to the detection and understanding of the operational models of sex trafficking networks (STNs). Our purpose is to enhance the AI capabilities of software-based detection technologies and support the homeland defense community in detecting and countering human sex trafficking, including the trafficking of underage victims.
To accomplish this, we propose a novel architecture capable of jointly representing and learning from multiple modalities, including images and text. The interdisciplinary nature of this work involves the fusion of computer vision, natural language processing, and deep neural networks (DNNs) to address the complexities of sex trafficking detection from online advertisements. This research proposes the creation of a software prototype as an extension of the Image Surveillance Assistant (ISA) built by our research team, to focus on cross-modal information retrieval and context understanding critical for identifying potential sex trafficking cases. Our initiative aligns with the objectives outlined in the DHS Strategic Plan, aiming to counter both terrorism and security threats, specifically focusing on the victim-centered approach to align with security threat segments.
We leverage current AI and machine learning techniques integrated by the project to create a working software prototype. DeepFace, a DNN for biometric analysis of facial image features such as age, race, and gender from images is utilized. Few-shot text classification, utilizing the SciKit Learn Python library and Large Language Models (LLMs), is enabling the detection of written trafficking advertisements. The prime funding agency, the Department of Homeland Security (DHS) mandates the use of synthetic data for this unclassified project, so we have developed code to leverage Application Programming Interfaces (APIs) that connect to LLMs and generative AI for images to create synthetic training and test data for the DNN models. Test and evaluation with synthetic data are the core capabilities of our approach to build prototype software that can potentially be used for real applications with real data.
Ongoing work includes creating a program to fuse the outputs from AI models on a single advertisement input. The fusion program will provide the numeric value of the likelihood of the class of advertisement, ranging from classes such as legal advertisement to different categories of trafficking. This research project is a potential contribution to the development of deployment-ready software for intelligence agencies, law enforcement, and border security. We currently show high accuracy for detecting advertisements related to victims of specific demographic categories. We have identified areas where increased accuracy is needed, and we are collecting more training data to address those gaps. The AI-based capabilities emerging from our research hold promise for enhancing the understanding of STN operational models, addresses technical challenges of sex trafficking detection, and also emphasizes the broader societal impact and alignment with national security goals.

Background: Current practice on how starting pitchers are pulled vary, are inconsistent, and not transparent. With the new major league baseball (MLB) data available through Statcast, such decisions can be more consistently supported by combining measured data.
Methods: To address this gap, we scraped pitch-level data from Statcast, a new technology system that collects real time MLB game measurements using laser technology. Here, we used Statcast data within a Cox regression for survival analysis to identify measurable factors that are associated to pitcher longevity. Measurements from 696,743 pitches were extracted for analysis from the 2021 MLB season. The pitcher was considered “surviving” the pitch if they remained in the game. Mortality was defined as the pitcher’s last pitch. Analysis began at the second inning to account for high variation during the first inning.
Results: Statistically significant factors include HSR(Hits to Strike Ratio), Runs per batter faced, and total bases per batter faced yielded the highest hazard coefficients (ranging from 10-23), which means higher risk of being relieved.
Conclusions: Our findings indicate that HSR, runs per batter faced and total bases per batter faced provide decision making information for relieving the starting pitcher.

The state of the art of numerical simulation of nondestructive evaluations (NDE) has begun to transition from research to application. The simulation software, both commercial and custom, has reached a level of maturity where it can be readily deployed to solve real world problems. The next area of research that is beginning to emerge is determining when and how to NDE simulation should be applied. At NASA Langley Research Center, NDE simulations have already been utilized for several aerospace projects to facilitate or enhance understanding of the inspection optimization and interpretation of results. Researchers at NASA have identified several different scenarios where it is appropriate to utilized NDE simulations. In this presentation, we will describe these scenarios, give examples of each instance, and demonstrate how NDE simulations were applied to solve problems with an emphasis on the mechanics of integrating with other workgroups. These examples will include inspection planning for multi-layer pressure vessels as well as on-orbit inspections.

In the last decade, deep learning models have proven capable of learning complex spatiotemporal relations and producing highly accurate short-term forecasts, known as nowcasts. Various models have been proposed to forecast precipitation associated with storm events hours before they happen. More recently, neural networks have been developed to produce accurate lightning nowcasts, using various types of satellite imagery, past lightning data, and other weather parameters as inputs to their model. Furthermore, the inclusion of attention mechanisms into these spatiotemporal weather prediction models has shown increases in the model’s predictive capabilities.

However, the calibration of these models and other spatiotemporal neural networks is rarely discussed. In general, model calibration addresses how reliable model predictions are, and models are typically calibrated after the model training process using scaling and regression techniques. Recent research suggests that neural networks are poorly calibrated despite being highly accurate, which brings into question how accurate the models are.

This research develops attention-based and non-attention-based deep-learning neural networks that uniquely incorporate reliability measures into the model tuning and training process to investigate the performance and calibration of spatiotemporal deep-learning models. All of the models developed in this research prove capable of producing lightning occurrence nowcasts using common remotely sensed weather modalities, such as radar and satellite imagery. Initial results suggest that the inclusion of attention mechanisms into the model architecture improves the model’s accuracy and predictive capabilities while improving the model’s calibration and reliability.

Statistical analysis is integral to the evaluation of defense systems throughout the acquisition process. Unlike traditional frequentist statistical methods, newer Bayesian statistical analysis incorporates prior information, such as historical data and expert knowledge, into the analysis for a more integrated approach to test data analysis. With Bayesian techniques, practitioners can more easily decide what data to leverage in their analysis, and how much that data should impact their analysis results. This provides a more flexible, informed framework for decision making in the testing and evaluation of DoD systems.

However, the application of Bayesian statistical analyses is often challenging due to the advanced statistical knowledge and technical coding experience necessary for the utilization of current Bayesian programming tools. The development of automated analysis tools can help address these barriers and make modern Bayesian analysis techniques available to a wide range of stakeholders, regardless of technical background. By making new methods more readily-available, collaboration and decision-making are made easier and more effective within the T&E community.

To facilitate this, we have developed a web application with the R Shiny package in R. This application uses an intuitive user interface to enable the implementation of our Bayesian reliability analysis approach by non-technical users without the need for any coding knowledge or advanced statistical background. Users can upload reliability data from the developmental testing and operational testing stages of a system of interest and tweak parameters of their choosing to automatically generate plots and estimates of system reliability performance based on their uploaded data and prior knowledge of system behavior.

With recent advances in computing power, many Bayesian methods that were once impracticably expensive are becoming increasingly viable. Parameter recovery problems present an exciting opportunity to explore some of these Bayesian techniques. In this talk we briefly introduce Bayesian design of experiments and look at a simple case study comparing its performance to classical approaches. We then discuss a PDE inverse problem and present ongoing efforts to optimize parameter recovery in this more complicated setting. This is joint work with Justin Krometis, Nathan Glatt-Holtz, Victoria Sieck, and Laura Freeman.

In projection pursuit regression (PPR), a univariate response variable is approximated by the sum of M "ridge functions," which are flexible functions of one-dimensional projections of a multivariate input variable. Traditionally, optimization routines are used to choose the projection directions and ridge functions via a sequential algorithm, and M is typically chosen via cross-validation. We introduce a novel Bayesian version of PPR, which has the benefit of accurate uncertainty quantification. To infer appropriate projection directions and ridge functions, we apply novel adaptations of methods used for the single ridge function case (M=1), called the Bayesian Single Index Model; and use a Reversible Jump Markov chain Monte Carlo algorithm to infer the number of ridge functions $M$. We evaluate the predictive ability of our model in 20 simulated scenarios and for 23 real datasets, in a bake-off against an array of state-of-the-art regression methods. Finally, we generalize this methodology and demonstrate the ability to accurately model multivariate response variables. Its effective performance indicates that Bayesian Projection Pursuit Regression is a valuable addition to the existing regression toolbox.

Developmental programs for complex systems with limited resources often face the daunting task of predicting the time needed to achieve system reliability goals.

Traditional reliability growth plans rely heavily on operational testing. They use confidence estimates to determine the required sample size, and then work backward to calculate the amount of testing required during the developmental test program to meet the operational test goal and satisfy a variety of risk metrics. However, these strategies are resource-intensive and do not take advantage of the information present in the developmental test period.

This presentation introduces a new method for projecting the reliability growth of a discrete, one-shot system. This model allows for various corrective actions to be considered, while accounting for both the uncertainty in the corrective action effectiveness and the management strategy used to parameterize those actions. Solutions for the posterior distribution on the system reliability are found numerically, while allowing for a variety of prior distributions on the corrective action effectiveness and the management strategy. Additionally, the model can be extended to account for system degradation across testing environments. A case study demonstrates how this model can use historical data with limited failure observations to inform its parameters, making it even more valuable for real-world applications.

This work builds upon previous research in Reliability Growth planning from Drs. Brian Hall and Martin Wayne.

Murray Cantor, Ph.D., Cantor Consulting
Christian Smart, Ph.D., Jet Propulsion Laboratory, California Institute of Technology

Cost risk analysis and earned value data are typically used separately and independently to estimate Estimates at Completion (EAC). However, there is significant value to combining the two in order to improve the accuracy of EAC forecasting. In this paper, we provide a rigorous method for doing this using Bayesian methods.
In earned value management (EVM), the Estimate at Completion (EAC) is perhaps the critical metric. It is used to forecast the effort’s total work cost as it progresses. In particular, it is used to see if the work is running over or under its planned budget, specified as the budget at completion (BAC).
Separate probability distribution functions (PDF) of the EAC at the onset of the effort and after some activities have been completed show the probability that EAC will fall within the BAC, and, conversely, the probability it won’t. At the onset of an effort, the budget is fixed, and the EAC is uncertain. As the work progresses, some of the actual costs (AC) are reported. The EAC uncertainty should then decrease and the likelihood of meeting the budget should increase. If the area under the curve to the left of the BAC decreases as the work progresses, the budget is in jeopardy, and some management action is warranted.
This paper will explain how to specify the initial PDF and learn the later PDFs from the data tracked in EVM. We describe the technique called Bayesian parameter learning (BPL). We chose this technique because it is the most robust for exploiting small sets of progress data and is most easily used by practitioners. This point will be elaborated further in the paper.

Murray Cantor, Ph.D., Cantor Consulting
Christian Smart, Ph.D., Jet Propulsion Laboratory, California Institute of Technology

Cost risk analysis and earned value data are typically used separately and independently to estimate Estimates at Completion (EAC). However, there is significant value to combining the two in order to improve the accuracy of EAC forecasting. In this paper, we provide a rigorous method for doing this using Bayesian methods.
In earned value management (EVM), the Estimate at Completion (EAC) is perhaps the critical metric. It is used to forecast the effort’s total work cost as it progresses. In particular, it is used to see if the work is running over or under its planned budget, specified as the budget at completion (BAC).
Separate probability distribution functions (PDF) of the EAC at the onset of the effort and after some activities have been completed show the probability that EAC will fall within the BAC, and, conversely, the probability it won’t. At the onset of an effort, the budget is fixed, and the EAC is uncertain. As the work progresses, some of the actual costs (AC) are reported. The EAC uncertainty should then decrease and the likelihood of meeting the budget should increase. If the area under the curve to the left of the BAC decreases as the work progresses, the budget is in jeopardy, and some management action is warranted.
This paper will explain how to specify the initial PDF and learn the later PDFs from the data tracked in EVM. We describe the technique called Bayesian parameter learning (BPL). We chose this technique because it is the most robust for exploiting small sets of progress data and is most easily used by practitioners. This point will be elaborated further in the paper.

In the modern world, we underappreciate the role of uncertainty and tend to be blind to risk. As the Nobel Prize–winning economist Kenneth Arrow once wrote, “Most individuals underestimate the uncertainty of the world . . . our knowledge of the way things work, in society or in nature, comes trailing clouds of vagueness.” The resistance to recognition of risk and uncertainty includes project management. As a colleague of mine aptly put it, “Project management types, especially, have a tendency to treat plans as reality.” However, projects of all types – weapon systems, robotic and human space efforts, dams, tunnels, bridges, the Olympics, etc. – experience regular and often extreme cost growth and schedule delays. Cost overruns occur in 80% or more of project development efforts and schedule delays happen in 70% or more. For many types of projects, average cost growth is 50% or more and these costs double in more than one in every six. These widespread and enduring increases reflect the high degree of risk inherent in projects. If there were no recurring history of cost and schedule growth, there would be no need for resource risk analysis and management. The planning of such projects would be as easy as planning a trip to a local dry cleaner. Instead, the tremendous risk inherent in such projects necessitates the consideration of risk and uncertainty throughout a program’s life cycle.
In practice the underestimation of project risk manifests itself in one of four ways. First, projects often completely ignore variation and rely exclusively on point estimates. As we will demonstrate, overlooking risk in the planning stages guarantees cost growth and schedule delays. Second, even when variation is considered, there is often an exclusive reliance on averages. We will show that there is much more to risk than simple averages. Third, even when the potential consequences are considered, risk matrices are often used in place of a rigorous quantitative analysis overreliance on risk matrices. We will provide proof that qualitative risk matrices underestimate the true degree of uncertainty. Fourth, an often-overlooked weakness in quantitative applications is the human element. There is an innate bias early in a project’s lifecycle to perceive less risk than is present which leads to a significant underestimation in uncertainty until late in a project’s development, at which point many of the risks a project faces have either been avoided, mitigated, or confronted. Even the application of best practices in risk analysis often lead to uncertainty ranges that are significantly tighter than indicated by history. Reasons for this phenomenon are discussed and the calibration of risk analysis outputs to historical cost growth and schedule delays is presented as a remedy.

In the rapidly evolving landscape of technology, Artificial Intelligence (AI) and Machine Learning (ML) have emerged as powerful tools with transformative potential. However, the adoption of these advanced technologies has often been limited to individuals with coding expertise, leaving a significant portion of the population, particularly those without programming skills, on the sidelines. This shift towards user-friendly AI/ML interfaces not only enhances inclusivity but also opens new avenues for innovation. A broader spectrum of individuals can combine the benefits of these cutting-edge technologies with their own domain knowledge to solve complex problems rapidly and effectively. Bringing no-code AI/ML to subject matter experts is necessary to ensure that the massive amount of data being produced by the DoD is properly analyzed and valuable insights are captured. This presentation delves into the importance of making AI and ML accessible to individuals with no coding experience. By doing so, it opens a world of possibilities for diverse participants to engage with and reap the benefits of the AI revolution.
While the prospect of making AI and ML accessible to individuals without coding experience is promising, it comes with its own set of challenges, particularly in addressing the barriers for individuals lacking a background in data analysis. One significant hurdle lies in the complexity of AI and ML algorithms, which often require a nuanced understanding of statistical concepts, data preprocessing, and model evaluation. Individuals without a foundation in analysis may find it challenging to interpret results accurately, hindering their ability to derive meaningful insights from AI-driven applications.
Another challenge is the availability of data, especially in the defense domain. Many models require large amounts of data to be effective. Ensuring the quality and consistency of the chosen dataset is a challenge, as individuals may encounter missing values, outliers, or inaccuracies that can adversely impact the performance of their ML models. Data preprocessing steps such as categorical variable encoding, interpolation, and normalization can be performed automatically, but it is important to understand when to use these techniques and why. Applying transformations such as logarithmic or polynomial transformations can enhance model performance. However, individuals with limited experience may struggle to determine when and how to apply these techniques effectively.
The lack of familiarity with key concepts such as feature engineering, model selection, and hyperparameter tuning can impede users from effectively utilizing AI tools. The black-box nature of some advanced models further complicates matters, as users may struggle to comprehend the inner workings of these algorithms, raising concerns about transparency and trust in AI-generated outcomes. Ethical considerations and biases inherent in AI models also pose substantial challenges. Users without an analysis background may inadvertently perpetuate biases or misinterpret results, underscoring the need for education and awareness to navigate these ethical complexities.
In this talk, we delve into the multifaceted challenges of bringing AI and ML to individuals without a background in analysis, emphasizing the importance of developing solutions that empower individuals to harness the potential of these technologies while mitigating potential pitfalls.

Coming soon

CIVA NDT simulation software is a powerful tool for non-destructive testing (NDT) applications. It allows users to design, optimize, and validate inspection procedures for various NDT methods, such as ultrasonic, eddy current, radiographic, and guided wave testing. Come learn about the benefits of using CIVA NDT simulation software to improve the reliability, efficiency, and cost-effectiveness of NDT inspections.

Domestic crime, conflict, and instability pose a significant threat to many contemporary governments.
These challenges have proven to be particularly acute within modern-day Mexico. While there have been significant developments in predicting intrastate armed and electoral conflict in various contemporary settings, such efforts have thus far been limited in their use of spatial as well as temporal correlations, as well as in the features they have considered. Machine learning, especially deep learning, has been proven to be highly effective in predicting future conflicts
using word embeddings in Convolutional Neural Networks (CNN) but lacks the spatial structure and, due to the black box nature, cannot explain the importance of predictors. We develop a novel methodology using machine learning that can accurately classify future anti-government violence in Mexico. We further demonstrate that our approach can identify important leading predictors of such violence. This can help policymakers make informed decisions and can also help governments and NGOs better allocate security and humanitarian resources, which could prove beneficial in tackling this problem. Using a variety of political event aggregations from the ICEWS database alongside other textual and demographic features, we trained various classical machine learning algorithms, including but not limited
to Logistic Regression, Random Forest, XGBoost, and a Voting classifier. The development of this reseearch was a stepwise process in three phases where the following phase was built upon the shortcomings of the previous phases. In the very first phase, we considered a mix of CNN + Long Short Term Memory (LSTM) networks to decode the spatial and temporal relationship in the data. The performance of all the black box deep learning models was not at par with the classical machine learning models. The second phase deals with the analysis of the temporal relationships in the data to identify the dependency of the conflicts over time and its lagged relationship. This also serves as a method to reduce feature dimension space by removing variables not covered with the cutoff lag. The third phase talks about the general variable selection methodologies used to further reduce the feature space along with identifying the important predictors that fuel anti-government violence along with their directional effect using Shapley additive values. The voting classifier, utilizing a subset of features derived from LASSO
across 100 simulations, consistently surpasses alternative models in performance and demonstrates efficacy in accurately classifying future anti-government conflicts. Notably, Random Forest feature importance indicates that some features, including but not limited to homicides, accidents, material conflicts, and positive worded citizen information sentiments emerge as pivotal predictors in the classification of anti-government conflicts. Finally, in the fourth phase, we conclude the
research by analysing the spatial structure of the data using Moran’s I index extended version for spatiotemporal data to identify the global spatial dependency and local clusters followed by modelling the data spatially and evaluating the same using Gaussian Process Boost(GPBoost). The global spatial autocorrelation is minimal, characterized by localized conflicts cluster within the region. Furthermore, the Voting Classifier demonstrates superior performance over GPBoost, leading to the inference that no substantial spatial dependency exists among the various locations.

In classification problems when working in high dimensions with a large number of classes and few observations per class, linear discriminant analysis (LDA) requires the strong assumptions of a shared covariance matrix between all classes and quadratic discriminant analysis leads to singular or unstable covariance matrix estimates. Both of these can lead to lower than desired classification performance. We introduce a novel, model-based clustering method which can relax the shared covariance assumptions of LDA by clustering sample covariance matrices, either singular or non-singular. This will lead to covariance matrix estimates which are pooled within each cluster. We show using simulated and real data that our method for classification tends to yield better discrimination compared to other methods.

Cyber Test and Evaluation serves a critical role in the procurement process of Red Team tools; however, once a tool is vetted and approved for use at the Red Team level, it is generally incorporated into their steady state operations without additional concern with regards to testing or maintenance of the tool. As a result, approved tools may not undergo routine in-depth T&E as new versions are released. This presents a major concern for the Red Team community as new versions can change the Operational Security of those tools. Similarly, cyber defenders - either through lack of training or limited resources - have been known to upload Red Team tools to commercial malware analysis platforms, which inadvertently releases potentially sensitive information about Red Team operations. The DOT&E Advanced Cyber Operations team, as part of the Cyber Assessment Program, performed in-depth analysis into Cobalt Strike, versions 4.8 and newer, an adversary simulation software widely used across the Department of Defense and the United States Government. Advanced Cyber Operations identified several operational security concerns that could disclose sensitive information to an adversary with access to payloads generated by Cobalt Strike. This highlights the need to improve the test and evaluation of cyber tooling, at a minimum, for major releases of tools utilized by Red Teams. Advanced Cyber Operations recommends in-depth, continuous test and evaluation of offensive operations tools and continued evaluation to mitigate potential operational security concerns.

The Army's Human Resources Command (HRC) annually takes on the critical task of the Centralized Selection List (CSL) process, where approximately 400 officers are assigned to key battalion command roles. This slating process is a cornerstone of the Army's broader talent management strategy, involving collaborative input from branch proponent officers and culminating in the approval of the Army Chief of Staff. The study addresses crucial shortcomings in the existing process for officer assignments, focusing on the biases and inconsistent weighting that affect slate selection outcomes. It examines the effects of incorporating specific criteria like Skill Experience Match (SEM), Knowledge, Skills, Behaviors (KSB), Order of Merit List (OML), and Officer Preferences (PREF) into the selection process of a pilot program. Our research specifically addresses the terms efficiency, strength, and weakness within the context of the pairing process. Our objective is to illuminate the potential advantages of a more comprehensive approach to decision-making in officer-job assignments, ultimately enhancing the effectiveness of placing the most suitable officer in the most fitting role.

In the early phases of project formulation, mission integration and test (I&T) costs are typically estimated via a wrap factor approach, analogies to similar missions adjusted for mission specifics, or a Bottom Up Estimate (BUE). The wrap factor approach estimates mission I&T costs as a percentage of payload and spacecraft hardware costs. This percentage is based on data from historical missions, with the assumption that the project being estimated shares similar characteristics with the underlying data set used to develop the wrap factor. This technique has worked well for traditional spacecraft builds since typically as hardware costs grow, I&T test costs do as well. However, with the emergence of CubeSats and nanosatellites, the cost basis of hardware is just not large enough to use the same approach. This suggests that there is a cost “floor” that covers basic I&T tasks, such as a baseline of labor and testing.

This paper begins the process of developing a cost estimating relationship (CER) for estimating Small Satellite (SmallSat) Integration & Test (I&T) costs. CERs are a result of a cost estimating methodology using statistical relationships between historical costs and other program variables. The objective in generating a CER equation is to show a relationship between the dependent variable, cost, to one or more independent variables. The results of this analysis can be used to better predict SmallSat I&T costs.

In the early phases of project formulation, mission integration and test (I&T) costs are typically estimated via a wrap factor approach, analogies to similar missions adjusted for mission specifics, or a Bottom Up Estimate (BUE). The wrap factor approach estimates mission I&T costs as a percentage of payload and spacecraft hardware costs. This percentage is based on data from historical missions, with the assumption that the project being estimated shares similar characteristics with the underlying data set used to develop the wrap factor. This technique has worked well for traditional spacecraft builds since typically as hardware costs grow, I&T test costs do as well. However, with the emergence of CubeSats and nanosatellites, the cost basis of hardware is just not large enough to use the same approach. This suggests that there is a cost “floor” that covers basic I&T tasks, such as a baseline of labor and testing.

This paper begins the process of developing a cost estimating relationship (CER) for estimating Small Satellite (SmallSat) Integration & Test (I&T) costs. CERs are a result of a cost estimating methodology using statistical relationships between historical costs and other program variables. The objective in generating a CER equation is to show a relationship between the dependent variable, cost, to one or more independent variables. The results of this analysis can be used to better predict SmallSat I&T costs.

The US Government has a specific need for tools that intelligence analysts can use to search and filter data effectively. Artificial Intelligence (AI), through the application of Deep Neural Networks (DNNs) can assist in a multitude of military applications, requiring a constant supply of relevant data sets to keep up with the always-evolving battlefield. Existing imagery does not adequately represent the evolving nature of modern warfare; therefore, finding a way to simulate images of future conflicts could give us a strategic advantage against our adversaries. Additionally, using physical cameras to capture sufficient various lighting and environmental conditions is nearly impossible. The technical challenge in this area is to create software tools for edge computing devices integrated with cameras to process the video feed locally without having to send the video data through bandwidth-constrained networks to servers in data centers. The ability to collect and process data locally, often in austere environments, can accelerate decision making and action taken in response to emergency situations. An important part of this challenge is to create labeled datasets that are relevant to the problem and are needed for training the edge-efficient AI. Teams from Fayetteville State University (FSU) and The United States Military Academy (USMA) will present their proposed workflows that will enable accurate detection of various threats using Unreal Engine (UE) to generate synthetic training data. In principle, production of synthetic data is unlimited and can be customized to location, various environmental variables, and human and crowd characteristics. Together, both teams address the challenges of realism and fidelity; diversity and variability; and integration with real data.
The focus of the FSU team is on creating semi-automated workflows to create simulated human-crowd behaviors and the ability to detect anomalous behaviors. It will provide methods of specifying collective behaviors to create crowd simulations of many human agents, and for selecting a few of those agents to exhibit behaviors that are outside of the defined range of normality. The analysis is needed for rapid detection of anomalous activities that can pose security threats and cost human lives.
The focus of the USMA team will be in creating semi-autonomous workflows that evaluate the ability of DNNs to identify key military assets under various environmental conditions, specifically armored vehicles and personnel. We aim to vary environmental parameters to simulate varying light conditions and introduce obscuration experiments using artificial means like smoke and natural phenomena like fog to add complexity to the scenarios. Additionally, the USMA team will explore a variety of camouflage patterns and various levels of defilade.
The outcome of both teams is to provide workflow solutions that maximize the use of UE to provide realistic datasets that simulate future battlefields and emergency scenarios for evaluating and training existing models. These studies pave the way for creating advanced models trained specifically for military application. Creating adaptive models that can keep up with today’s evolving battlefield will give the military a great advantage in the race for artificial intelligence applications.

Data – collection, preparation, and curation - is a crucial need in the AI lifecycle. Ensuring that the data are consistent, correct, and representative for the intended use is critical to ensuring the efficacy of an AI enabled system. Data verification, validation, and accreditation (VV&A) is meant to address this need. The dramatic increase in the prevalence of AI-enabled capabilities and analytic tools across the DoD has emphasized the need for a unified understanding of data VV&A, as quality data forms the foundation of AI models. In practice, data VV&A and associated activities are often used in an ad-hoc manner that may limit the ability to support development and testing of AI enabled capabilities. However, existing DOD frameworks for data VV&A are applicable to the AI lifecycle and embody important supporting activities for T&E of AI enabled systems. We highlight the importance of data VV&A, relying on established definitions, and outline some concerns and best practices.

This paper applies a Bayesian Optimization approach to the design of a wing subject to stress and aeroelastic constraints. The parameters of these constraints, which correspond to various flight conditions and uncertain parameters, are prescribed by a finite number of scenarios. Chance-constrained optimization is used to seek a wing design that is robust to the parameter variation prescribed by such scenarios. This framework enables computing designs with varying degrees of robustness. For instance, we can deliberately eliminate a given number of scenarios in order to obtain a lighter wing that is more likely to violate a requirement, or might seek a conservative wing design that satisfies the constraints for as many scenarios as possible.

In the field of Artificial Intelligence and Machine Learning (AI/ML), the literature can be filled with technical language and/or buzzwords, making it challenging for readers to understand the content. It will be a pedagogical talk focusing on demystifying "Artificial Intelligence" by providing a mathematical, but most importantly, an intuitive understanding of how deep learning really works. I will provide some existing tools and practical steps for how one can train their own neutral networks using an example of automatically identifying aircrafts and their attributes (e.g., civil vs. military, engine types, and size) from satellite images. Audience members with some knowledge of linear regression and coding will be armed with an increased understanding, confidence, and practical tools to develop their own AI applications.

This is the short course for you if you are familiar with the fundamental techniques in the science of test and want to learn useful, real-world, and advanced methods applicable in the DoD/NASA test community. The focus will be on use cases not typically covered in most short courses. JMP software will primarily be used, and datasets will be provided for you to follow along many of the hands-on demonstrations of practical case studies. Design topics will include custom design of experiments tips, choosing optimality criteria, creating designs from existing runs, augmenting adaptively in high gradient regions, creating designs with constraints, repairing broken designs, mixture design intricacies, modern screening designs, designs for computer simulation, accelerated life test, and measurement system testing. Analysis topics will include ordinary least squares, stepwise, and logistic regression, generalized regression (LASSO, ridge, elastic net), model averaging (to include Self-Validated Ensemble Models), random effects (split-plot, repeated measures), comparability/equivalence, functional data analysis (think your data is a curve), nonlinear approaches and multiple response optimization and trade-space analysis. The day will finish with an hour-long Q&A session to help solve your specific T&E challenges.

This is the short course for you if you are familiar with the fundamental techniques in the science of test and want to learn useful, real-world, and advanced methods applicable in the DoD/NASA test community. The focus will be on use cases not typically covered in most short courses. JMP software will primarily be used, and datasets will be provided for you to follow along many of the hands-on demonstrations of practical case studies. Design topics will include custom design of experiments tips, choosing optimality criteria, creating designs from existing runs, augmenting adaptively in high gradient regions, creating designs with constraints, repairing broken designs, mixture design intricacies, modern screening designs, designs for computer simulation, accelerated life test, and measurement system testing. Analysis topics will include ordinary least squares, stepwise, and logistic regression, generalized regression (LASSO, ridge, elastic net), model averaging (to include Self-Validated Ensemble Models), random effects (split-plot, repeated measures), comparability/equivalence, functional data analysis (think your data is a curve), nonlinear approaches and multiple response optimization and trade-space analysis. The day will finish with an hour-long Q&A session to help solve your specific T&E challenges.

Cyber resilience technologies are critical to ensuring the survival of mission critical assets for space systems. Such emerging cyber resilience technologies ultimately need to be proven out through in-flight experimentation. However, there are significant technical challenges for proving that new technologies actually enhance resilience of spacecraft. In particular, in-flight experimentation suffers from a “low data” problem due to many factors including 1) no physical access limits what types of data can be collected, 2) even if data can be collected, size, weight, and power (SWaP) constraints of the spacecraft make it difficult to store large amounts of data, 3) even if data can be stored, bandwidth constraints limit the transfer of data to the ground in a timely manner, and 4) only a limited number of trials can be performed due to spacecraft scheduling and politics. This talk will discuss a framework developed for design and execution of in-flight cyber experimentation as well as statistical techniques appropriate for analyzing the data. More specifically, we will discuss how data from ground-based test beds can be used to augment the results of in-flight experiments. The discussed framework and statistical techniques will be demonstrated on a use case.

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND2023-14847A.

The Director, Operational Test and Evaluation (DOT&E) and the Institute for Defense Analyses (IDA) are developing recommendations for how to account for trust and trustworthiness in AI-enabled systems during Department of Defense (DoD) Operational Testing (OT). Trust and trustworthiness have critical roles in system adoption, system use and misuse, and performance of human-machine teams. The goal, however, is not to maximize trust, but to calibrate the human’s trust to the system’s trustworthiness. Trusting more than a system warrants can result in shattered expectations, disillusionment, and remorse. Conversely, under trusting implies that humans are not making the most of available resources.
Trusted and trustworthy systems are commonly referenced as essential for the deployment of AI by political and defense leaders and thinkers. Executive Order 14110 requires “safe, secure, and trustworthy development and use” of AI. Furthermore, the desired end state of the Department of Defense Responsible AI Strategy is trust. These terms are not well characterized and there is no standard, accepted model for understanding, or method for quantifying, trust or trustworthiness for test and evaluation (T&E). This has resulted in trust and trust calibration rarely being assessed in T&E. This is, in part, due to the contextual and relational nature of trustworthiness. For instance, the developmental tester requires a different level of algorithmic transparency than the operational tester or the operator; whereas the operator may need more understandability than transparency. This means that to successfully operationally test AI-enabled systems, such testing must be done at the right level, with the actual operators and commanders and up-to-date CONOPS as well as sufficient time for training and experience for trust to evolve. The need for testing over time is further amplified by particular features of AI, wherein machine behaviors are no longer as predictable or static as traditional systems but may continue to be updated and adaptive. Thus, testing for trust and trustworthiness cannot be one and done.
It is critical to ensure that those who work within AI – in its design, development, and testing – understand exactly what trust actually means, why it is important, and how to operationalize and measure it. This session will empower testers by:
• Establishing a common foundation for understanding what trust and trustworthiness are.
• Defining key terms related to trust, enabling testers to think about trust more effectively.
• Demonstrating the importance of trust calibration for system acceptance and use and the risks of poor calibration.
• Decomposing the factors within trust to better elucidate how trust functions and what factors and antecedents have been shown to effect trust in human-machine interaction.
• Introducing concepts on how to design AI-enabled systems for better trust calibration, assurance, and safety.
• Proposing validated and reliable survey measures for trust.
• Discussing common cognitive biases implicated in trust and AI and both the positive and negative roles biases play.
• Addressing common myths around trust in AI, including that trust or its measurement doesn’t matter, or that trust in AI can be “solved” with ever more transparency, understandability, and fairness.

The Department of Defense (DoD) is undergoing a digital engineering transformation in every process of the systems engineering lifecycle. This transformation provides the requirement that DoD Test and Evaluation (T&E) processes begin to implement and begin executing model-based testing methodologies. This paper describes and assesses a grey box model-driven test design (MDTD) approach to create flight test scenarios based on model-based systems engineering artifacts. To illustrate the methodology and evaluate the expected outcomes of the process in practice, a case study using a model representation of a training system utilized to train new Air Force Operational Test and Evaluation Center (AFOTEC) members in conducting operational test and evaluation (OT&E) is presented. The results of the grey box MDTD process are a set of activity diagrams that are validated to generate the same test scenario cases as the traditional document-centric approach. Using artifacts represented in System Modeling Language (SysML), this paper will discuss key comparisons between the traditional and MDTD processes. This paper demonstrates the costs and benefits of model-based testing and their relevance in the context of operational flight testing.

As data becomes more commoditized across all echelons of the DoD, developing Artificial Intelligence (AI ) solutions, even at small scales, offer incredible opportunity for advanced data analysis and processing. However, these solutions require intimate knowledge of the data in question, as well as robust Test and Evaluation (T&E) procedures to ensure performance and trustworthiness. This paper presents a case study and recommendations for developing and evaluating small-scale AI solutions. The model automates an acoustic trilateration system. First, the system accurately identifies the precise times of acoustic events across a variable number of sensors using a neural network. It then corresponds the events across the sensors through a heuristic matching process. Finally, using the correspondences and difference of times, the system triangulates a physical location. We find that even a relatively simple dataset requires extensive understanding at all phases of the process. Techniques like data augmentation and data synthesis, which must capture the unique attributes of the real data, were necessary both for improved performance, as well as robust T&E. The T&E metrics and pipeline required unique approaches to account for the AI solution, which lacked traceability and explainability. As leaders leverage the growing availability of AI tools to solve problems within their organizations, strong data analysis skills must remain at the core of process.

The Dynamo paradigm for T&E compares a set of test options for a system by computing which of them provides the greatest expected operational benefit relative to the cost of testing. This paradigm will be described and demonstrated for simple, realistic cases. Dyanmo stands for DYNAmic Knowledge + MOneyball. These two halves of Dynamo are its modeling framework and its chief evaluation criterion, respectively.

A modeling framework for T&E is what allows test results (and domain knowledge) to be leveraged to predict operational system performance. Without a model, one can only predict, qualitatively, that operational performance will be similar to test performance in similar environments. For quantitative predictions one can formulate a model that inputs a representation of an operational environment and outputs the probabilities of the various possible outcomes of using the system there. Such models are typically parametric: they have a set of unknown parameters to be calibrated during test. The more knowledge one has about a suitable model’s parameters, the better predictions one can make about the modeled system’s operational performance. The Bayesian approach to T&E encodes this knowledge as a probability distribution over the model parameters. This knowledge is initialized with data from previous testing and with subject matter expertise, and it is “dynamic” because it is updated whenever new test results arrive.

An evaluation criterion is a metric for the operational predictions provided by the modeling framework. One type of metric is about whether test results indicate a system meets requirements: this question can be addressed with increasing nuance as one employs more sophisticated modeling frameworks. Another type of metric is how well a test design will tighten knowledge about model parameters, regardless of what the test results themselves are. The Dynamo paradigm can leverage either, but it uses a “Moneyball” metric for recommending test decisions. A Moneyball metric quantifies the expected value of the knowledge one would gain from testing (whether from an entire test event, or from just a handful of trials) in terms of the operational value this knowledge would provide. It requires a Bayesian modeling framework so that incremental gains in knowledge can be represented and measured. A Moneyball metric quantifies stakeholder preferences in the same units as testing costs, which enables a principled cost/benefit analysis not only of which tests to perform, but of whether to conduct further testing at all.

The essence of Dynamo is that it applies Bayesian Decision Theory to T&E to maintain and visualize the state of knowledge about a system under test at all times, and that it can make recommendations at any time about which test options to conduct to provide the greatest expected benefit to stakeholders relative to the cost of testing. This talk will discuss the progress to date developing Dynamo and some of the future work remaining to make it more easily adaptable to testing specific systems.

The ability to detect change in flight patterns using air traffic control (ATC) communication can better inform battlefield intelligence. ADS-B (Automatic Dependent Surveillance Broadcast) technology has this capability to capture movement of both military and civilian aircraft over conflict zones. Leveraging the inclusivity of ADS-B in flight tracking and its widespread global availability, we focus on its application in understanding changes leading up to conflicts, with a specific case study on Ukraine.
In this presentation we analyze days leading up to Russia’s February 24 invasion to understand how ADS-B technology can indicate change in Russo-Ukrainian military movements. The proposed detection algorithm encourages the use of ADS-B technology in future intelligence efforts. The potential for fusion with GICB (Ground-initiated Comm-B) ATC communication and other modes of data is also explored.
This is a submission for the Student Poster Competition

Resilience engineering involves creating and maintaining systems capable of efficiently managing disruptive incidents. Past research in this field has employed various statistical techniques to track and forecast the system's recovery process within the resilience curve. However, many of these techniques fall short in terms of flexibility, struggling to accurately capture the details of shocks. Moreover, most of them are not able to predict long-term dependencies. To address these limitations, this paper introduces an advanced statistical method, the transfer function, which effectively tracks and predicts changes in system performance when subjected to multiple shocks and stresses of varying intensity and duration. This approach offers a structured methodology for planning resilience assessment tests tailored to specific shocks and stresses and guides the necessary data collection to ensure efficient test execution. Although resilience engineering is domain-specific, the transfer function is a versatile approach, making it suitable for various domains. To assess the effectiveness of the transfer function model, we conduct a comparative analysis with the interaction regression model, using historical data on job losses during the 1980 recessions in the United States. This comparison not only underscores the strengths of the transfer function in handling complex temporal data but also reaffirms its competitiveness compared to existing methods. Our numerical results using goodness of fit measures provide compelling evidence of the transfer function model's enhanced predictive power, offering an alternative for advancing resilience prediction in time series analysis.

The prevalence of unmanned aerial systems (UAS) and remote sensing technology on the modern battlefield has laid the foundation for an automated targeting system. However, no computer vision model has been trained to support such a system. Difficulties arise in creating these models due to a lack of available battlefield training data. This work aims to investigate the use of synthetic images generated in Unreal Engine as supplementary training data for a battlefield image classifier. We test state-of-the-art computer vision models to determine their performance on drone images of modern battlefields and the suitability of synthetic images as training data. Our results suggest that synthetic training images can improve the performance of state-of-the-art models in battlefield computer vision tasks.

This is an abstract for a student poster.

When conducting probabilistic cost analysis, correlation assumptions are key assumptions and often a driver for the total output or point estimate of a cost model. Although the National Aeronautics and Space Administration (NASA) has an entire community dedicated to the development of statistical cost estimating tools and techniques to manage program and project performance, the application of accurate and data-driven correlation coefficients within these models is often overlooked. Due to the uncertain nature of correlation between random variables, NASA has had difficulty quantifying the relationships between spacecraft subsystems with specific, data-driven correlation matrices. Previously, the NASA cost analysis community has addressed this challenge by either selecting a blanket correlation value to address uncertainty within the model or opting out of using any correlation value altogether. One hypothesized method of improving NASA cost estimates involves deriving subsystem correlation coefficients from the residuals of the regression equations for the cost estimating relationships (CERs) of various spacecraft subsystems and support functions. This study investigates the feasibility of this methodology using the CERs from NASA's Project Cost Estimating Capability (PCEC) model. The correlation coefficients for each subsystem of the NASA Work Breakdown Structure were determined by correlating the residuals of PCEC's subsystem CERs. These correlation coefficients were then compiled into a 20x20 correlation matrix and were implemented into PCEC as an uncertainty factor influencing the model's pre-existing cost distributions. Once this correlation matrix was implemented into the cost distributions of PCEC, the Latin Hypercube Sampling function of the Microsoft Excel add-in Argo was used to simulate PCEC results for 40 missions within the PCEC database. These steps were repeated three additional times using the following correlation matrices: (1) a correlation matrix assuming the correlation between each subsystem is zero, (2) a correlation matrix assuming the correlation between each subsystem is 1, and (3) a correlation matrix using a blanket value of 0.3. The results of these simulations showed that the correlation matrix derived from the residuals of the subsystem CERs significantly reduced bias and error within PCEC's estimating capability. The results also indicated that the probability density function and cumulative distribution function of each mission in the PCEC database were altered significantly by the correlation matrices that were implemented into the model. This research produced (1) a standard subsystem correlation matrix that has been proven to improve estimating accuracy within PCEC and (2) a replicable methodology for creating this correlation matrix that can be used in future cost estimating models. This information can help the NASA cost analysis community understand the effects of applying uncertainty within cost models and perform sensitivity analyses on project cost estimates. This is significant because NASA has been frequently critiqued for underestimating project costs and this methodology has shown promise in improving NASA's future cost estimates and painting a more realistic picture of the total possible range of spacecraft development costs.

Large language models (LLMs) are poised to dramatically impact the process of composing, analyzing, and editing documents, including within DoD and IC communities. However, there have been few studies that focus on understanding human interactions and perceptions of LLM outputs, and even fewer still when one considers only those relevant to a government context. Furthermore, there is a paucity of benchmark datasets and standardized data collection schemes necessary for assessing the usability of LLMs in complex tasks, such as summarization, across different organizations and mission use cases. Such usability studies require an understanding beyond the literal content of the document; the needs and interests of the reader must be considered, necessitating an intimate understanding of the operational context. Thus, adequately measuring the effectiveness and suitability of LLMs requires usability testing to be incorporated into the testing and evaluation process.

However, measures of usability are stymied by three challenges. First, there is an unsatisfied need for mission-relevant data that can be used for assessment, a critical first step. Agencies must provide data for assessment of LLM usage, such as report summarization, to best evaluate the effectiveness of LLMs. Current widely available datasets for assessing LLMs consist primarily of ad hoc exams ranging from the LSAT to sommelier exams. High performance on these exams offers little insight into LLM performance on mission tasks, which possess a unique lexicon, set of high-stakes mission applications, and DoD and IC userbase. Notably, our prior work indicates that currently available curated datasets are unsuitable proxies for government reporting. Our search for proxy data for intelligence reports led us on a path to create our own dataset in order to evaluate LLMs within mission contexts.

Second, a range of experimental design techniques exists for collecting human-centric measures of LLM usability, each with their own benefits and disadvantages. Navigating the tradeoffs between these different techniques is challenging, and the lack of standardization across different groups inhibits comparison between groups. A discussion is provided on the potential usage of commonly conducted usability studies, including heuristic evaluations, observational and user experience studies, and tool instrumentation focusing on LLMs used in summarization. We will describe the pros and cons of each study, crafting guidance against their approximate required resources in terms of time (planning, participant recruitment, study, and analysis), compute, and data. We will demonstrate how our data collection prototype for summarization tasks can be used to streamline the above.

The final challenge involves associating human-centric measures, such as ratings of fluency, to other more quantitative and mission-level metrics. We will provide an overview of measures for summarization quality, including ratings for accuracy, concision, fluency, and completeness, and discuss current efforts and existing challenges in associating those measures to quantitative and qualitative metrics. We will also discuss the value of such efforts in building a more comprehensive assessment of LLMs, as well as the relevance of these efforts to document summarization.

For a parallel system, when one component fails, the failure distribution of the remaining components will have an increased failure rate. This research takes a novel approach to finding the associated failure distribution of the full system using ordinal statistic distributions for correlated Weibull components, allowing for unknown correlations between the dependent components. A Taylor series approximation is presented for two components; system failure time distributions are also derived for two failures in a two component system, two failures in an n component system, three failures in a three component system, and k failures in an component system. Additionally, a case study is presented on aircraft turnbuckles. Simulated data is used to illustrate how the derived formulas can be used to create a maintenance plan for the second turnbuckle in the two component system.

As a research organization, the Institute for Defense Analyses (IDA) produces a variety of deliverables like reports, memoranda, slides, and other formats for our sponsors. Due to their length and volume, summarizing these products quickly for efficient retrieval of information on specific research topics poses a challenge. IDA has led numerous initiatives for historical tagging of documents, but this is a manual and time-consuming process, and must be led periodically to tag newer products. To address this challenge, we have developed a Python-based automated product tagging pipeline using natural language processing (NLP) techniques.

This pipeline utilizes NLP keyword extraction techniques to identify descriptive keywords within the content. Filtering these keywords with IDA's research taxonomy terms produces a set of product tags, serving as metadata. This process also enables standardized tagging of products, compared to the manual tagging process, which introduces variability in tagging quality across project leaders, authors, and divisions. Instead, the tags produced through this pipeline are consistent and descriptive of the contents. This product-tagging pipeline facilitates an automated and standardized process for streamlined topic summarization of IDA's research products, and has many applications for quantifying and analyzing IDA's research in terms of these product tags.

Functional data are an ordered series of data collected over a continuous scale, such as time or distance. The data are collected in ordered x,y pairs and can be viewed as a smoothed line with an underlying function. The Army Evaluation Center (AEC) has identified multiple instances where functional data analysis could have been applied, but instead evaluators used more traditional and/or less statistically rigorous methods to evaluate the data. One of these instances is radar tracking data.

This poster highlights historical shortcomings of how AEC currently analyzes functional data, such as radar tracking data and our vision for future applications. Using a notional data from a real radar example, the response of 3D track error is plotted against distance, where each function represents a unique run number and additional factors held constant throughout a given run. The example includes the selected model, functional principle components, the resulting significant factors, and summary graphics used for report. Additionally, the poster will highlight historical analysis methods and the improvements the functional data analysis method brings. The analysis and output from this poster will utilize JMP's functional data analysis platform.

Generative artificial intelligence (AI) is a rapidly advancing field and transformative technology that involves the creation of new content. Generative AI encompasses AI models that produce novel data, information, or documents in response to the prompts. This technology has gained significant attention due to the emergence of models like DALL-E , Imagen, and ChatGPT. Generative AI excels in generating content across various domains. The versatility of Generative AI extends to generating text, software code, images, videos, and music by statistically analyzing patterns in training data.
One of the most prominent applications of Generative AI is ChatGPT, developed by OpenAI. ChatGPT is a sophisticated language model trained on vast amounts of text data from diverse sources. It can engage in conversations, answer questions, write essays, generate code snippets, and more. Generative AI's strengths lie in its ability to produce diverse and seemingly original outputs quickly.

Large Language Models (LLMs) are advanced deep learning algorithms that can understand, summarize, translate, predict, and generate content using extensive datasets. These models work by being trained on massive amounts of data, deriving relationships between words and concepts, and then using transformer neural network processes to understand and generate responses.

LLMs are widely used for tasks like text generation, translation, content summarization, rewriting content, classification, and categorization. They are trained on huge datasets to understand language better and provide accurate responses when given prompts or queries. The key algorithms used in LLMs include:
• Word Embedding: This algorithm represents the meaning of words in a numerical format, enabling the AI model to process and analyze text data efficiently.
• Attention Mechanisms: These algorithms allow the AI to focus on specific parts of input text, such as sentiment-related words, when generating an output, leading to more accurate responses.
• Transformers: Transformers are a type of neural network architecture designed to solve sequence-to-sequence tasks efficiently by using self-attention mechanisms. They excel at handling long-range dependencies in data sequences. They learn context and meaning by tracking relationships between elements in a sequence.
This presentation will focus on basics of large language models, algorithms, and applications to nuclear decommissioning knowledge management.

This work describes sensitivity analyses performed on complex black-box models used to support experimental test planning under limited resources in the context of the Mars Sample Return program, which aims at bringing to Earth rock and atmospheric samples from Mars. We develop a systematic workflow that allows the analysts to simultaneously obtain quantitative insights on key drivers of uncertainty, on the direction of impact, and the presence of interactions. We apply novel optimal transport-based global sensitivity measures to tackle the multivariate nature of the output. On the modeling side, we apply multi-fidelity techniques that leverage low-fidelity models to speed up the calculations and make up for the limited amount of high-fidelity samples, while keeping these in the loop for accuracy guarantees. The sensitivity analysis reveals insights useful for the analysts to understand the model's behavior and identify the factors to focus on during testing in order to maximize the value of information extracted from them to ensure mission success when limited resources are available.

The topic of synthetic data in test and evaluation is steeped in controversy – and rightfully so. Generative AI techniques can be erratic, producing non-credible results that should give evaluators pause. At the same time, there are mission domains that are difficult to test, and these rely on modeling and simulation to generate insights for evaluation. High fidelity modeling and simulation can be slow, computationally intensive, and burdened by large volumes of data – challenges which become prohibitive as test complexity grows.

To mitigate these challenges, we posit a defensible, physically valid generative AI approach to creating fast-running synthetic data for M&S studies of hard-to-test scenarios. Characterized as a “Narrow Digital Twin,” we create an exemplar Generative AI model of high-fidelity Hypersonic Glide Vehicle trajectories. The model produces a set of trajectories that meets user-specified criteria (particularly as directed by a Design of Experiments) and that can be validated against the equations of motion that govern these trajectories. This presentation will identify the characteristics of the model that make it suitable for generating synthetic data and propose easy-to-measure acceptability criteria. We hope to advance a conversation about appropriate and rigorous uses of synthetic data within T&E.

Poster Abstract

Shiny is an R package and framework used to build applications in a web browser without needing any programming experience. This capability allows the power and functionality of R to be accessible to audiences that would typically not utilize R because of the roadblocks with learning a programming language.

Army Evaluation Center (AEC) has rapidly increased their utilization of R shiny to develop apps for use in Test and Evaluation. These apps have allowed evaluators to upload data, perform calculations, and then create data visualizations which can be customized for their reporting needs. The apps have streamlined and standardized the analysis process.

The poster outlines the before and after of three data visualizations utilizing the shiny apps. The first displays survey data using the likert package. The second graphs a cyberattack cycle timeline. And the third displays individual qualification course results and calculates qualification scores. The apps are hosted in a cloud environment and their usage is tracked with an additional shiny app.

Uncertainty quantification (UQ) sits at the confluence of data, computers, basic science, and operation. It has emerged with the need to inform risk assessment with rapidly evolving science and to bring the full power of sensing and computing to bear on its management.
With this role, UQ must provide analytical insight into several disparate disciplines, a task that may seem daunting and highly technical. But not necessarily so.
In this mini-tutorial, I will present foundational concepts of UQ, showing how it is the simplicity of the underlying ideas that allows them to straddle multiple disciplines. I will also describe how operational imperatives have helped shape the evolution of UQ and discuss how current research at the forefront of UQ can in turn affect these operations.

First introduced in 1987, modern design thinking was popularized by the Stanford Design School and the global design and innovation company, IDEO. Design thinking is now recognized as a “way of thinking which leads to transformation, evolution and innovation” and has been so widely accepted across industry and within the DoD, that universities offer graduate degrees in the discipline. Relying on the design thinking foundation, the Decision Science Division (DSD) of Virginia Tech Applied Research Corporation (VT-ARC) human centered design facilitation technique integrates related methodologies including liberating structures and open thinking. Liberating structures, are “simple and concrete tools that can enhance group performance in diverse organizational settings.” Open thinking, popularized by Dan Pontefract, provides a comprehensive approach to decision-making that incorporates critical and creative thinking techniques. The combination of these methodologies enables tailored problem framing, innovative solution discovery, and creative adaptability to harness collaborative analytic potential, overcome the limitations of cognitive biases, and lead change. DSD VT-ARC applies this approach to complex and wicked challenges to deliver solutions that address implementation challenges and diverse stakeholder requirements. Operating under the guiding principle that collaboration is key to success, DSD regularly partners with other research organizations, such as Virginia Tech National Security Institute (VT NSI), in human centered design activities to help further the understanding, use, and benefits of the approach. This experiential session will provide attendees with some basic human centered design facilitation tools and an understanding of how these techniques might be applied across a multitude of technical and non-technical projects.

First introduced in 1987, modern design thinking was popularized by the Stanford Design School and the global design and innovation company, IDEO. Design thinking is now recognized as a “way of thinking which leads to transformation, evolution and innovation” and has been so widely accepted across industry and within the DoD, that universities offer graduate degrees in the discipline. Relying on the design thinking foundation, the Decision Science Division (DSD) of Virginia Tech Applied Research Corporation (VT-ARC) human centered design facilitation technique integrates related methodologies including liberating structures and open thinking. Liberating structures, are “simple and concrete tools that can enhance group performance in diverse organizational settings.” Open thinking, popularized by Dan Pontefract, provides a comprehensive approach to decision-making that incorporates critical and creative thinking techniques. The combination of these methodologies enables tailored problem framing, innovative solution discovery, and creative adaptability to harness collaborative analytic potential, overcome the limitations of cognitive biases, and lead change. DSD VT-ARC applies this approach to complex and wicked challenges to deliver solutions that address implementation challenges and diverse stakeholder requirements. Operating under the guiding principle that collaboration is key to success, DSD regularly partners with other research organizations, such as Virginia Tech National Security Institute (VT NSI), in human centered design activities to help further the understanding, use, and benefits of the approach. This experiential session will provide attendees with some basic human centered design facilitation tools and an understanding of how these techniques might be applied across a multitude of technical and non-technical projects.

Over the past decade, the number and severity of cyber-attacks to critical infrastructure has continued to increase, necessitating a deeper understanding of these systems and potential threats. Recent advancements for high-fidelity system modeling, also called emulation, have enabled quantitative cyber experimentation to support analyses of system design, planning decisions, and threat characterization. However, much remains to be done to establish scientific methodologies for performing these cyber analyses more rigorously.
Without a rigorous approach to cyber experimentation, it is difficult for analysts to fully characterize their confidence in the results of an experiment, degrading the ability to make decisions based upon analysis results, and often defeating the purpose of performing the analysis. This issue is particularly salient when analyzing critical infrastructures or similarly impactful systems, where confident, well-informed decision making is imperative. Thus, the integration of tools for rigorous scientific analysis with platforms for emulation-driven experimentation is crucial.
This work discusses one such effort to integrate the tools necessary to perform uncertainty quantification (UQ) on an emulated model, motivated by a study on a notional critical infrastructure use case. The goal of the study was to determine how variations in the aggressiveness of the given threat affected how resilient the system was to the attacker. Resilience was measured using a series of metrics which were designed to capture the system’s ability to perform its mission in the presence of the attack. One reason for the selection of this use case was that the threat and system models were believed to be fairly deterministic and well-understood. The expectation was that results would show a linear correlation between the aggressiveness of the attacker and the resilience of the system. Surprisingly, this hypothesis was not supported by the data.
The initial results showed no correlation, and they were deemed inconclusive. These findings spurred a series of mini analyses, leading to extensive evaluation of the data, methodology, and model to identify the cause of these results. Significant quantities of data collected as part of the initial UQ study enabled closer inspection of data sources and metrics calculation. In addition, tools developed during this work facilitated supplemental statistical analyses, including a noise study. These studies all supported the conclusion that the system model and threat model chosen were far less deterministic than initially assumed, highlighting key lessons learned for approaching similar analyses in the future.
Although this work is discussed in the context of a specific use case, the authors believe that the lessons learned are generally applicable to similar studies applying statistical testing to complex, high-fidelity system models. Insights include the importance of deeply understanding potential sources of stochasticity in a model, planning how to handle or otherwise account for such stochasticity, and performing multiple experiments and looking at multiple metrics to gain a more holistic understanding of a modeled scenario. These results highlight the criticality of approaching system experimentation with a rigorous scientific mindset.

This mini-tutorial will outline approaches to apply Bayesian methods to the test and evaluation process, from development of tests to interpretation of test results to translating that understanding into decision-making. We will begin by outlining the basic concepts that underlie the Bayesian approach to statistics and the potential benefits of applying that approach to test and evaluation. We will then walk through application to an example (notional) program, setting up data models and priors on the associated parameters, and interpreting the results. From there, techniques for integrating results from multiple stages of tests will be discussed, building understanding of system behavior as evidence accumulates. Finally, we will conclude by describing how Bayesian thinking can be used to translate information from test outcomes into requirements and decision-making. The mini-tutorial will assume some background in statistics but the audience need not have prior exposure to Bayesian methods.

This mini-tutorial will outline approaches to apply Bayesian methods to the test and evaluation process, from development of tests to interpretation of test results to translating that understanding into decision-making. We will begin by outlining the basic concepts that underlie the Bayesian approach to statistics and the potential benefits of applying that approach to test and evaluation. We will then walk through application to an example (notional) program, setting up data models and priors on the associated parameters, and interpreting the results. From there, techniques for integrating results from multiple stages of tests will be discussed, building understanding of system behavior as evidence accumulates. Finally, we will conclude by describing how Bayesian thinking can be used to translate information from test outcomes into requirements and decision-making. The mini-tutorial will assume some background in statistics but the audience need not have prior exposure to Bayesian methods.

The goal of this project was to develop an algorithm that automatically detects and predicts the future position of boats in a maritime environment. We integrated these algorithms into an intelligent combat system developed by the Naval Surface Warfare Center Dahlgren Division (NSWCDD). This algorithm used YOLOv8 for computer vision detection and a linear Kalman filter for prediction. The data used underwent extensive augmentation and third party integration. It was tested at a Live, Virtual, and Constructive (LVC) event held at NSWCDD this past fall (October 2023).

The initial models faced challenges of overfitting. However, through processes such as data augmentation, incorporation of third-party data, and layer freezing techniques, we were able to develop a more robust model. Various datasets were processed by tools to improve data robustness. By further labeling the data, we were able to obtain ground truth data to evaluate the Kalman filter. The Kalman filter was chosen for its versatility and predictive tracking capabilities. Qualitative and quantitative analysis were performed for both the YOLO and Kalman filter models.

Much of the project's contribution lay in its ability to adapt to a variety of data. YOLO displayed effectiveness across various maritime scenarios, and the Kalman Filter excelled in predicting boat movements across difficult situations such as abrupt camera movements.

In preparation for the live fire test event, our algorithm was integrated into the NSWCDD system and code was written to produce the expected output files.

In summary, this project successfully developed an algorithm for detecting and predicting boats in a maritime environment. This project demonstrated the potential of the intersection of machine learning, rapid integration of technology, and maritime security.

Many Department of Defense (DoD) systems aim to increase or maintain Situational Awareness (SA) at the individual or group level. In some cases, maintenance or enhancement of SA is listed as a primary function or requirement of the system. However, during test and evaluation SA is examined inconsistently or is not measured at all. Situational Awareness Linked Indicators Adapted to Novel Tasks (SALIANT) is an empirically-based methodology meant to measure SA at the team, or group, level. While research using the SALIANT model suggests that it effectively quantifies team SA, no study has examined the effectiveness of SALIANT across the entirety of the existing empirical research. The aim of the current work is to conduct a meta-analysis of previous research to examine the overall reliability of SALIANT as an SA measurement tool. This meta-analysis will assess when and how SALIANT can serve as a reliable indicator of performance at testing. Additional applications of SALIANT in non-traditional operational testing domains will also be discussed.

The US Department of Defense (DoD) has expanded their emphasis on the application of systems engineering approaches to ‘missions’. As originally defined in the Defense Acquisition Guidebook, Mission Engineering (ME) is “the deliberate planning, analyzing, organizing, and integrating of current and emerging operational and system capabilities to achieve desired operational mission effects”. Based on experience to date, the new definition reflects ME as an “an interdisciplinary approach and process encompassing the entire technical effort to analyze, design, and integrate current and emerging operational needs and capabilities to achieve desired mission outcomes”. This presentation presents the current mission engineering methodology, describes how it is currently being applied, and explores the role of T&E in the ME process. Mission engineering is applying systems engineering to missions – that is, engineering a system of systems, (including organizations, people and technical systems) to provide desired impact on mission or capability outcomes. Traditionally, systems of systems engineering focused on designing systems or systems of systems to achieve specified technical performance. Mission engineering goes one step further to assess whether the system of systems, when deployed in a realistic user environment, achieves the user mission or capability objectives. Mission engineering applies digital model-based engineering approaches to describe the sets of activities in the form of ‘mission threads’ (or activity models) needed to execute the mission and then adds information on players and systems used to implement these activities in the form of ‘mission engineering threads.’ These digital ‘mission models’ are then implemented in operational simulations to assess how well they achieve user capability objectives. Gaps are identified and models are updated to reflect proposed changes, including reorientation of systems and insertion of new candidate solutions, and which are assessed relative to changes in overall mission effectiveness.

Many organizations seek to ensure that machine learning (ML) and artificial intelligence (AI) systems work as intended in production but currently do not have a cohesive methodology in place to do so. To fill this gap, we built MLTE (Machine Learning Test and Evaluation, colloquially referred to as “melt”), a framework and implementation to evaluate ML models and systems. The framework compiles state-of-the-art evaluation techniques into an organizational process for interdisciplinary teams, including model developers, software engineers, system owners, and other stakeholders. MLTE tooling, a Python package, supports this process by providing a domain-specific language that teams can use to express model requirements, an infrastructure to define, generate, and collect ML evaluation metrics, and the means to communicate results.

In this presentation, we will discuss current MLTE details as well as future plans to support developmental testing (DT) and operational testing (OT) organizations and teams. A problem in the Department of Defense (DoD) is that test and evaluation (T&E) organizations are segregated: OT organizations work independently from DT organizations, which leads to inefficiencies. Model developers doing contractor testing (CT) may not have access to mission and system requirements and therefore fail to adequately address the real-world operational environment. Motivation to solve these two problems has generated a push for Integrated T&E — or T&E as a Continuum — in which testing is iteratively updated and refined based on previous test outcomes, and is informed by mission and system requirements. MLTE helps teams to better negotiate, evaluate, and document ML model and system qualities, and will aid in the facilitation of this iterative testing approach. As MLTE matures, it can be extended to further support Integrated T&E by (1) providing test data and artifacts that OT can use as evidence to make risk-based assessments regarding the appropriate level of OT and (2) ensuring that CT and DT testing of ML models accurately reflects the challenges and constraints of real-world operational environments.

Many organizations seek to ensure that machine learning (ML) and artificial intelligence (AI) systems work as intended in production but currently do not have a cohesive methodology in place to do so. To fill this gap, we built MLTE (Machine Learning Test and Evaluation, colloquially referred to as “melt”), a framework and implementation to evaluate ML models and systems. The framework compiles state-of-the-art evaluation techniques into an organizational process for interdisciplinary teams, including model developers, software engineers, system owners, and other stakeholders. MLTE tooling, a Python package, supports this process by providing a domain-specific language that teams can use to express model requirements, an infrastructure to define, generate, and collect ML evaluation metrics, and the means to communicate results.

In this presentation, we will discuss current MLTE details as well as future plans to support developmental testing (DT) and operational testing (OT) organizations and teams. A problem in the Department of Defense (DoD) is that test and evaluation (T&E) organizations are segregated: OT organizations work independently from DT organizations, which leads to inefficiencies. Model developers doing contractor testing (CT) may not have access to mission and system requirements and therefore fail to adequately address the real-world operational environment. Motivation to solve these two problems has generated a push for Integrated T&E — or T&E as a Continuum — in which testing is iteratively updated and refined based on previous test outcomes, and is informed by mission and system requirements. MLTE helps teams to better negotiate, evaluate, and document ML model and system qualities, and will aid in the facilitation of this iterative testing approach. As MLTE matures, it can be extended to further support Integrated T&E by (1) providing test data and artifacts that OT can use as evidence to make risk-based assessments regarding the appropriate level of OT and (2) ensuring that CT and DT testing of ML models accurately reflects the challenges and constraints of real-world operational environments.

Space systems provide many critical functions to the military, federal agencies, and infrastructure networks. In particular, MIL-STD-1553 serves as a common command and control network for space systems, nuclear weapons, and DoD weapon systems. Nation-state adversaries have shown the ability to disrupt critical infrastructure through cyber-attacks targeting systems of networked, embedded computers. Moving target defenses (MTDs) have been proposed as a means for defending various networks and systems against potential cyber-attacks. In addition, MTDs could be employed as an ‘operate through’ mitigation for improving cyber resilience.

We devised a MTD algorithm and tested its application to a MIL-STD-1553 network. We demonstrated and analyzed four aspects of the MTD algorithm usage: 1) characterized the performance, unpredictability, and randomness of the core algorithm, 2) demonstrated feasibility by conducting experiments on actual commercial hardware, 3) conducted an exfiltration experiment where the reduction in adversarial knowledge was 97%, and 4) employed the LSTM machine learning model to see if it could defeat the algorithm and glean information about the algorithm’s resistance to machine learning attacks. Given the above analysis, we show that the algorithm has the ability to be used in real-time bus networks as well as other (non-address) applications.

In rapidly evolving threat scenarios, the accurate and timely identification of hostile enemies armed with weapons is crucial to strategic advantage and personnel safety. This study aims to develop both a timely and accurate model utilizing YOLOv5 for the detection of weapons and persons in real-time drone footage and generate an alert containing the count of weapons and persons detected. Existing methods in this field often focus on either minimizing type I/type II errors or the speed at which the model runs. In our current work, we have focused on two main points of emphasis throughout training our model. The minimization of type II error (minimizing instances of weapons of persons present but not detected) and keeping accuracy and precision consistent while increasing the speed of our model to keep up with real-time footage. Various parameters were adjusted within our model, including but not limited to speed, freezing layers, and image size. Going from our first to the final adjusted model, overall precision and recall went from 71.9% to 89.2% and 63.7% to 77.5%, respectively. The occurrences of misidentification produced from our model decreased dramatically, from 27% of persons misidentified as either weapon or background noise to 14%, and the misidentification of weapons from 50% to 34%. An important consideration for future work is mitigating overfitting the model to a particular dataset when training. In real-world implementation, our model needs to perform well across a variety of conditions and angles not all of which were introduced in the training data set.

Modeling thermal states for complex space missions, such as the surface exploration of airless bodies, requires high computation, whether used in ground-based analysis for spacecraft design or during onboard reasoning for autonomous operations. For example, a finite-element-method (FEM) thermal model with hundreds of elements can take significant time to simulate on a typical workstation, which makes it unsuitable for onboard reasoning during time-sensitive scenarios such as descent and landing, proximity operations, or in-space assembly. Further, the lack of fast and accurate thermal modeling drives thermal designs to be more conservative and leads to spacecraft with larger mass and higher power budgets.
The emerging paradigm of physics-informed machine learning (PIML) presents a class of hybrid modeling architectures that address this challenge by combining simplified physics models (e.g., analytical, reduced-order, and coarse mesh models) with sample-based machine learning (ML) models (e.g., deep neural networks and Gaussian processes) resulting in models which maintain both interpretability and robustness. Such techniques enable designs with reduced mass and power through onboard thermal-state estimation and control and may lead to improved onboard handling of off-nominal states, including unplanned down-time (e.g. GOES-7 cite{bedingfield1996spacecraft}, and M2020)
The PIML model or hybrid model presented here consists of a neural network which predicts reduced nodalizations (coarse mesh size) given on-orbit thermal load conditions, and subsequently a (relatively coarse) finite-difference model operates on this mesh to predict thermal states. We compare the computational performance and accuracy of the hybrid model to a purely data-driven model, and a high-fidelity finite-difference model (on a fine mesh) of a prototype Earth-orbiting small spacecraft. This hybrid thermal model promises to achieve 1) faster design iterations, 2) reduction in mission costs by circumventing worst-case-based conservative planning, and 3) safer thermal-aware navigation and exploration.

AI will play an important role in future military systems. However, large questions remain about how to test AI systems, especially in operational settings. Here, we discuss an approach for the operational test and evaluation (OT&E) of AI-supported data integration, fusion, and analysis systems. We highlight new challenges posed by AI-supported systems and we discuss new and existing OT&E methods for overcoming them. We demonstrate how to apply these OT&E methods via a notional test concept that focuses on evaluating an AI-supported data integration system in terms of its technical performance (how accurate is the AI output?) and human systems interaction (how does the AI affect users?).

This talk discusses the ARINC 429 standard and its inherent lack of security, demonstrates proven mission effects in a hardware-in-the-loop (HITL) simulator, and presents a data set collected from real avionics.
ARINC 429 is a ubiquitous data bus for civil avionics, enabling safe and reliable communication between devices from disparate manufacturers. However, ARINC 429 lacks any form of encryption or authentication, making it an inherently insecure communication protocol and rendering any connected avionics vulnerable to a range of attacks.
We constructed a HITL simulator with ARINC 429 buses to explore these vulnerabilities, and to identify potential mission effects. The HITL simulator includes commercial off-the-shelf avionics hardware including a multi-function display, an Enhanced Ground Proximity Warning System, as well as a realistic flight simulator.
We performed a denial-of-service attack against the multi-function display via a compromised transmit node on an ARINC 429 bus, using commercially available tools, which succeeded in disabling important navigational aids. This simple replay attack demonstrates how effectively a “leave-behind” device can cause serious mission effects.
This proven adversarial effect on physical avionics illustrates the risk inherent in ARINC 429 and the need for the ability to detect, mitigate, and recover from these attacks. One potential solution is an intrusion detection system (IDS) trained using data collected from the electrical properties of the physical bus. Although previous research has demonstrated the feasibility of an IDS on an ARINC 429 bus, none have been trained on data generated by actual avionics hardware.

This poster session considers the ARINC 429 standard and its inherent lack of security by using a hardware-in-the-loop (HITL) simulator to demonstrate possible mission effects from a cyber compromise. ARINC 429 is a ubiquitous data bus for civil avionics, enabling safe and reliable communication between devices from disparate manufacturers. However, ARINC 429 lacks any form of encryption or authentication, making it an inherently insecure communication protocol and rendering any connected avionics vulnerable to a range of attacks.

This poster session includes a hands-on demonstration of possible mission effects due to a cyber compromise of the ARINC 429 data bus by putting the audience at the control of the HITL flight simulator with ARINC 429 buses. The HITL simulator uses commercial off-the-shelf avionics hardware including a multi-function display, and an Enhanced Ground Proximity Warning System to generate operationally realistic ARINC 429 messages. Realistic flight controls and flight simulation software are used to further increase the simulator’s fidelity. The cyberattack is based on a system with a malicious device physically-connected to the ARINC 429 bus network. The cyberattack degrades the multi-function display through a denial-of-service attack which disables important navigational aids. The poster also describes how testers can plan to test similar buses found on vehicles and can observe and document data from this type of testing event.

This study delves into the application of influence diagrams in mission concept development at the Jet Propulsion Laboratory, emphasizing the importance of how technical variables influence mission costs. Concept development is an early stage in the design process that requires extensive decision-making, in which the tradeoffs between time, money, and scientific goals are explored to generate a wide range of project ideas from which new missions can be selected. Utilizing influence diagrams is one strategy for optimizing decision making. An influence diagram represents decision scenarios in a graphical and mathematical manner, providing an intuitive interpretation of the relationships between input variables (which are functions of the decisions made in a trade space) and outcomes. These input-to-outcome relationships may be mediated by intermediate variables, and the influence diagram provides a convenient way to encode the hypothesized “trickle-down” structure of the system. In the context of mission design and concept development, influence diagrams can inform analysis and decision-making under uncertainty to encourage the design of realistic projects within imposed cost limitations, and better understand the impacts of trade space decisions on outcomes like cost.

This project addresses this initiative by focusing on the analysis of an influence diagram framed as a Directed Acyclic Graph (DAG), a graphical structure where vertices are connected by directed edges that do not form loops. Edge weights in the DAG represent the strength and direction of relationships between variables. The DAG aims to model the trickle-down effects of mission technical parameters, such as payload mass, payload power, delta V, and data volume, on mission cost elements. A Bayesian multilevel regression model with random effects is used for estimating the edge weights in a specific DAG (constructed according to expert opinion) that is meant to represent a hypothesized trickle-down structure from technical parameters to cost. This Bayesian approach provides a flexible and robust framework, allowing us to incorporate prior knowledge, handle small datasets effectively, and leverage its capacity to capture the inherent uncertainty in our data.

In its mission to expand knowledge and improve aviation, NASA conducts research to address sonic boom noise, the prime barrier to overland supersonic flight. NASA is currently preparing for a community survey campaign to assess response to noise from the new X-59 aircraft. During each community survey, a substantial number of observations must be collected over a limited timeframe to generate a dose response relationship. A sample of residents will be recruited in advance to fill out a brief survey each time X-59 flies over, approximately 80 times throughout a month. In preparation NASA conducted a month-long test of survey methods in 2023. A sample of 800 residents was recruited from a simulated fly-over area. Because there were no actual X-59 fly-overs, respondents were asked about their reactions to noise from normal aircraft operations. The respondents chose whether to fill out the survey on the web or via a smartphone application. Evaluating response rates and how they evolved over time was a specific focus of the test. Also, a graduated incentive structure was implemented to keep respondents engaged. Finally, location data was collected from respondents since it will be needed to estimate individual noise exposure from X-59. The results of this survey test will help determine the design of the community survey campaign. This is an overview presentation that will cover the key goals, results, and lessons learned from the survey test.

Defense and aerospace testing commonly involves binary responses to changing levels of a system configuration or an explanatory variable. Examples of binary responses are hit or miss, detect or not detect, and success or fail, and they are a special case of categorical responses with multiple discreet levels. The test objective is typically to estimate a statistical model that predicts the probability of occurrence of the binary response as a function of the explanatory variable(s). Statistical approaches are readily available for modeling binary responses; however, they often assume that the design features large sample sizes that provide responses distributed across the range of the explanatory variable. In practice, these assumptions are often challenged by small sample sizes and response levels focused over a limited range of the explanatory variable(s). These practical restrictions are due to experimentation cost, operational constraints, and a primary interest in one response level, e.g., testing may be more focused on hits compared to a misses. This presentation provides strategies to address these challenges with an emphasis on collaboration techniques to develop experimental design approaches under practical constraints. Case studies are presented to illustrate these strategies from estimating human annoyance to low noise supersonic overflights in NASA’s Quesst mission and evaluating detection capability of nondestructive evaluation methods for fracture-critical human-spaceflight components. This presentation offers practical guidance on experimental design strategies for binary responses under operational constraints.

Advances in machine learning (ML) have led to applications in safety-critical domains, including security, defense, and healthcare. These ML models are confronted with dynamically changing and actively hostile conditions characteristic of real-world applications, requiring systems incorporating ML to be reliable and resilient. Many studies propose techniques to improve the robustness of ML algorithms. However, fewer consider quantitative methods to assess the reliability and resilience of these systems. To address this gap, this study demonstrates how to collect relevant data during the training and testing of ML suitable for applying software reliability, with and without covariates, and resilience models, and the subsequent interpretation of these analyses. The proposed approach promotes quantitative risk assessment of machine learning technologies, providing the ability to track and predict degradation and improvement in the ML model performance and assisting ML and system engineers with an objective approach to compare the relative effectiveness of alternative training and testing methods. The approach is illustrated in the context of an image recognition model subjected to two generative adversarial attacks and then iteratively retrained to improve the system's performance. Our results indicate that software reliability models incorporating covariates characterized the misclassification discovery process more accurately than models without covariates. Moreover, the resilience model based on multiple linear regression incorporating interactions between covariates tracked and predicted degradation and recovery of performance best. Thus, software reliability and resilience models offer rigorous quantitative assurance methods for ML-enabled systems and processes.

The Carpentries more introductory R lesson. In addition to their standard content, this workshop covers data analysis and visualization in R, focusing on working with tabular data and other core data structures, using conditionals and loops, writing custom functions, and creating publication-quality graphics. As their more introductory R offering, this workshop also introduces learners to RStudio and strategies for getting help. This workshop is appropriate for learners with no previous programming experience.