Session Title | Speaker | Type | Recording | Materials | Year |
---|---|---|---|---|---|
Breakout A Great Test Requires a Great Plan (Abstract)
The Scientific Test and Analysis Techniques (STAT) process is designed to provide structure for a test team to progress from a requirement to decision quality information. The four phases of the STAT process are Plan, Design, Execute, and Analyze. Within the Test and Evaluation (T&E) community we tend to focus on the quantifiable metrics and the hard science of testing, which are the Design and the Analyze phases. At the STAT Center of Excellence (COE) we have emphasized an increased focus on the planning phase and in this presentation we focus on the elements necessary for a comprehensive planning session. In order to efficiently and effectively test a system it is vital that the test team understand the requirements, the System Under Test (SUT) to include any subsystems that will be tested, and the test facility. To accomplish this the right team members with the necessary knowledge must be in the room and prepared to present their information and have an educated discussion to arrive at a comprehensive agreement about the desired end stated of the test. Our recommendations for the initial planning meeting are based on a thorough study of the STAT process and on lessons learned from actual planning meetings. |
Aaron Ramert STAT Analyst Scientific Test and Analytics Techniques Center of Excellence (STAT COE) ![]() (bio)
Mr. Ramert is a graduate of the US Naval Academy and the Naval Postgraduate School and a 20 year veteran of the Marine Corps. During his career in the Marines he served tours in operational air and ground units as well as academic assignments. He joined the Scientific Test and Analysis Techniques (STAT) Center of Excellence (COE) in 2016 where he works with major acquisition programs with the Department of Defense to apply rigor and efficiency to their test and evaluation methodology through the application of the STAT process. |
Breakout | Session Recording |
| 2021 |
Breakout The 80/20 rule, can and should we break it using efficient data management tools? (Abstract)
Abstract: Data scientists spend approximately 80% of their time preparing, cleaning, and feature engineering data sets. In this talk I will share use cases that show why this is important and why we need to do it. I will also describe the Earth System Grid Federation (ESGF) which is an open source effort providing a robust, distributed data and computation platform, enabling world wide access to Peta/Exa-scale scientific data. ESGF will help reduce the amount of effort needed for climate data preprocessing by integrating the necessary analysis and data sharing tools. |
Ghaleb Abdulla | Breakout |
| 2019 |
|
Breakout Using Sequential Testing to Address Complex Full-Scale Live Fire Test and Evaluation (Abstract)
Co-authors: Dr. Darryl Ahner, Director, STAT COE Dr. Lenny Truett, STAT COE Mr. Scott Wacker, 96 TG/OL-ACS. This presentation will present the benefits of sequential testing and demonstrate how sequential testing can be used to address complex test conditions by developing well controlled early experiments to explore basic questions before proceeding to full-scale testing. This approach can result in increased knowledge and decreased cost. As of FY13 the Air Force had spent an estimated $47M on dry bay fire testing making fire the largest cost contributor for Live Fire Test and Evaluation (LFT&E) programs. There is currently an estimated 60% uncertainty in total platform vulnerable area (Av) driven by probability of kill (PK) due to ballistically ignited fires. A large part of this uncertain comes from the fact that current spurt modeling does not predict fuel spurt delay with reasonable accuracy despite a large amount of test data. A low-cost sequential approach was developed to improve spurts models. Initial testing used a spherical projectile to test 10 different factors in a definitive screening design. Once the list of factors was refined, a second phase of testing determined if a suitable methodology could be developed to scaled results using water as a surrogate for JP-8 fuel. Finally testing was performed with cubical projectiles to evaluate the effect of fragment orientation. The entire cost for this effort was less than one or two typical full-scale live fire tests. |
Darryl Ahner AFIT, STAT COE |
Breakout | 2016 |
||
Breakout Testing Autonomous Systems (Abstract)
Autonomous robotic systems (hereafter referred to simply as autonomous systems) have attracted interest in recent years as capabilities improve to operate in unstructured, dynamic environments without continuous human guidance. Acquisition of autonomous systems potentially decrease personnel costs and provide a capability to operate in dirty, dull, or dangerous mission segments or achieve greater operational performance. Autonomy enables a particular action of a system to be automatic or, within programmed boundaries, self-governing. For our purposes, autonomy is defined as the system having a set of intelligence-based capabilities (i.e., learned behaviors) that allows it to respond to situations that were not pre-programmed or anticipated (i.e. learning-based responses) prior to system deployment. Autonomous systems have a degree of self-governance and self-directed behavior, possibly with a human’s proxy for decisions. Because of these intelligence-based capabilities, autonomous systems pose new challenges in conducting test and evaluation that assures adequate performance, safety, and cybersecurity outcomes. We propose an autonomous systems architecture concept and map the elements of a decision theoretic view of a generic decision problem to the components of this architecture. These models offer a foundation for developing a decision-based, common framework for autonomous systems. We also identify some of the various challenges faced by the Department of Defense (DoD) test and evaluation community in assuring the behavior of autonomous systems as well as test and evaluation requirements, processes, and methods needed to address these challenges. |
Darryl Ahner Director AFIT |
Breakout | Materials | 2018 |
|
Breakout Forecasting with Machine Learning (Abstract)
The Department of Defense (DoD) has a considerable interest in forecasting key quantities of interest including demand signals, personnel flows, and equipment failure. Many forecasting tools exist to aid in predicting future outcomes, and there are many methods to evaluate the quality and uncertainty in those forecasts. When used appropriately, these methods can facilitate planning and lead to dramatic reductions in costs. This talk explores the application of machine learning algorithms, specifically gradient-boosted tree models, to forecasting and presents some of the various advantages and pitfalls of this approach. We conclude with an example where we use gradient-boosted trees to forecast Air National Guard personnel retention. |
Akshay Jain Data Science Fellow IDA (bio)
Akshay earned his Bachelor of Arts in Math, Political Science, Mathematical Methods in the Social Sciences (MMSS) from Northwestern University. He is currently a Data Science Fellow in the Strategy, Forces, and Resources Division at the Institute for Defense Analyses. |
Breakout | Session Recording |
| 2022 |
Breakout Legal, Moral, and Ethical Implications of Machine Learning (Abstract)
Machine learning algorithms can help to distill vast quantities of information to support decision making. However, machine learning also presents unique legal, moral, and ethical concerns – ranging from potential discrimination in personnel applications to misclassifying targets on the battlefield. Building on foundational principles in ethical philosophy, this presentation summarizes key legal, moral, and ethical criteria applicable to machine learning and provides pragmatic considerations and recommendations. |
Alan B. Gelder Research Staff Member IDA (bio)
Alan earned his PhD in Economics from the University of Iowa in 2014 and currently leads the Human Capital Group in the Strategy, Forces, and Resources Division at the Institute for Defense Analyses. He specializes in microeconomics, game theory, experimental and behavioral economics, and machine learning, and his research focuses on personnel attrition and related questions for the DOD. |
Breakout | Session Recording |
| 2022 |
Breakout Search for Extended Test Design Methods for Complex Systems of Systems |
Alex Alaniz AFOTEC |
Breakout | Materials | 2017 |
|
Closing Remarks |
Alyson Wilson NCSU ![]() (bio)
Dr. Alyson Wilson is the Associate Vice Chancellor for National Security and Special Research Initiatives at North Carolina State University. She is also a professor in the Department of Statistics and Principal Investigator for the Laboratory for Analytic Sciences. Her areas of expertise include statistical reliability, Bayesian methods, and the application of statistics to problems in defense and national security. Dr. Wilson is a leader in developing transformative models for rapid innovation in defense and intelligence. Prior to joining NC State, Dr. Wilson was a jointly appointed research staff member at the IDA Science and Technology Policy Institute and Systems and Analyses Center (2011-2013); associate professor in the Department of Statistics at Iowa State University (2008-2011); Scientist 5 and technical lead for Department of Defense Programs in the Statistical Sciences Group at Los Alamos National Laboratory (1999-2008); and senior statistician and operations research analyst with Cowboy Programming Resources (1995-1999). She is currently serving on the National Academy of Sciences Committee on Applied and Theoretical Statistics and on the Board of Trustees for the National Institute of Statistical Sciences. Dr. Wilson is a Fellow of the American Statistical Association, the American Association for the Advancement of Science, and an elected member of the International Statistics Institute. |
2022 |
|||
Breakout Prior Formulation in a Bayesian Analysis of Biomechanical Data (Abstract)
Biomechanical experiments investigating the failure modes of biological tissue require a significant investment of time and money due to the complexity of procuring, preparing, and testing tissue. Furthermore, the potentially destructive nature of these tests makes repeated testing infeasible. This leads to experiments with notably small sample sizes in light of the high variance common to biological material. When the goal is to estimate parameters for an analytic artifact such as an injury risk curve (IRC), which relates an input quantity to a probability of injury, small sample sizes result in undesirable uncertainty. One way to ameliorate this effect is through a Bayesian approach, incorporating expert opinion and previous experimental data into a prior distribution. This has the advantage of leveraging the information contained in expert opinion and related experimental data to obtain faster convergence to an appropriate parameter estimation with a desired certainty threshold. We explore several ways of implementing Bayesian methods in a biomechanical setting, including permutations on the use of expert knowledge and prior experimental data. Specifically, we begin with a set of experimental data from which we generate a reference IRC. We then elicit expert predictions of the 10th and 90th quantiles of injury, and use them to formulate both uniform and normal prior distributions. We also generate priors from qualitatively similar experimental data, both directly on the IRC parameters and on the injury quantiles, and explore the use of weighting schemes to assign more influence to better datasets. By adjusting the standard deviation and shifting the mean, we can create priors of variable quality. Using a subset of the experimental data in conjunction with our derived priors, we then re-fit the IRC and compare it to the reference curve. For all methods we will measure the certainty, speed of convergence, and accuracy relative to the reference IRC, with the aim of recommending a best practices approach for the application of Bayesian methods in this setting. Ultimately an optimized approach for handling small samples sizes with Bayesian methods has the potential to increase the information content of individual biomechanical experiments by integrating them into the context of expert knowledge and prior experimentation. |
Amanda French Data Scientist Johns Hopkins University Applied Physics Laboratory ![]() (bio)
Amanda French is a data scientist at Johns Hopkins University Applied Physics Laboratory. She obtained her PhD in mathematics from UNC Chapel Hill and went on to perform data science for a variety of government agencies, including the Department of State, Military Health System, and Department of Defense. Her expertise includes statistics, machine learning, and experimental design. |
Breakout | Session Recording |
| 2021 |
Breakout Design and Analysis of Experiments for Europa Clipper’s Science Sensitivity Model (Abstract)
The Europa Clipper Science Sensitivity Model (SSM) can be thought of as a graph in which the nodes are mission requirements at ten levels in a hierarchy, and edges represent how requirements at one level of the hierarchy depend on those at lower levels. At the top of the hierarchy, there are ten nodes representing ten, Level 1 science requirements for the mission. At the bottom of the hierarchy, there are 100 or so nodes representing instrument-specific science requirements. In between, nodes represent intermediate science requirements with complex interdependencies. Meeting, or failing to meet, bottom-level requirements depends on the frequency of faults and the lengths of recovery times on the nine Europa Clipper instruments and the spacecraft. Our task was to design and analyze the results of a Monte Carlo experiment to estimate the probabilities of meeting the Level 1 science requirements based on parameters of the distributions of time between failures and of recovery times. We simulated an ensemble of synthetic missions in which failures and recoveries were random realizations from those distributions. The pass-fail status of the bottom-level instrument-specific requirements were propagated up the graph for each of the synthetic missions. Aggregating over the collection of synthetic missions produced estimates of the pass-fail probabilities for the Level 1 requirements. We constructed a definitive screening design and supplemented it with additional space-filing runs, using JMP 14 software. Finally, we used the vectors of failure and recovery parameters as predictors, and the pass-fail probabilities of the high-level requirements as responses, and built statistical models to predict the latter from the former. In this talk, we will describe the design considerations and review the fitted models and their implications for mission success. |
Amy Braverman | Breakout |
| 2019 |
|
Tutorial Tutorial: Combinatorial Methods for Testing and Analysis of Critical Software and Security Systems (Abstract)
Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost, but when are these methods practical and cost-effective? This tutorial includes two sections on the basis and application of combinatorial test methods: The first section explains the background, process, and tools available for combinatorial testing, with illustrations from industry experience with the method. The focus is on practical applications, including an industrial example of testing to meet FAA-required standards for life-critical software for commercial aviation. Other example applications include modeling and simulation, mobile devices, network configuration, and testing for a NASA spacecraft. The discussion will also include examples of measured resource and cost reduction in case studies from a variety of application domains. The second part explains combinatorial testing-based techniques for effective security testing of software components and large-scale software systems. It will develop quality assurance and effective re-verification for security testing of web applications and testing of operating systems. It will further address how combinatorial testing can be applied to ensure proper error-handling of network security protocols and provide the theoretical guarantees for detecting Trojans injected in cryptographic hardware. Procedures and techniques, as well as workarounds will be presented and captured as guidelines for a broader audience. |
Rick Kuhn, Dimitris Simos, and Raghu Kacker National Institute of Standards & Technology |
Tutorial |
| 2019 |
|
Keynote STAT Engineering Keynote-Wednesday AM |
Christine Anderson Cook Statistics Los Alamos National Lab ![]() (bio)
s in the areas of complex system reliability, non-proliferation, malware detection and statistical process control. Before joining LANL, she was a faculty member in the Department of Statistics at Virginia Tech for 8 years. Her research areas include response surface methodology, design of experiments, reliability, multiple criterion optimization and graphical methods. She has authored more than 130 articles in statistics and quality peerreviewed journals, and has been a long time contributor to the Quality Progress Statistics Roundtable column. In 2012, she edited a special issue in Quality Engineering on Statistical Engineering with Lu Lu. She is an elected fellow of the American Statistical Association and the American Society for Quality. In 2012 she was honored with the ASQ Statistics Division William G. Hunter Award. In 2011 she received the 26th Annual Governor’s Award for Outsta |
Keynote | 2016 |
||
Breakout XPCA: A Copula-based Generalization of PCA for Ordinal Data (Abstract)
Principal Component Analysis is a standard tool in an analyst’s toolbox. The standard practice of rescaling each column can be reframed as a copula-based decomposition in which the marginal distributions are fit with a univariate Gaussian distribution and the joint distribution is modeled with a Gaussian copula. In this light, we present an alternative to traditional PCA we call XPCA by relaxing the marginal Gaussian assumption and instead fit each marginal distribution with the empirical distribution function. Interval-censoring methods are used to account for the discrete nature of the empirical distribution function when fitting the Gaussian copula model. In this talk, we derive the XPCA estimator and inspect the differences in fits on both simulated and real data applications. |
Cliff Anderson-Bergman Sandia National Laboratories |
Breakout | Materials | 2018 |
|
Panel The Keys to Successful Collaborations during Test and Evaluation: Moderator (Abstract)
The defense industry faces increasingly complex systems in test and evaluation (T&E) that require interdisciplinary teams to successfully plan testing. A critical aspect in test planning is a successful collaboration between T&E experts, subject matter experts, program leadership, statisticians, and others. This panel, based on their own experiences as consulting statisticians, will discuss elements that lead to successful collaborations, barriers during collaboration, and recommendations to improve collaborations during T&E planning. |
Christine Anderson-Cook Los Alamos National Lab ![]() |
Panel | Session Recording | 2021 |
|
Breakout M&S approach for quantifying readiness impact of sustainment investment scenarios (Abstract)
Sustainment for weapon systems involves multiple components that influence readiness outcomes through a complex array of interactions. While military leadership can use simple analytical approaches to yield insights into current metrics (e.g., dashboard for top downtime drivers) or historical trends of a given sustainment structure (e.g., correlative studies between stock sizes and backorders), they are inadequate tools for guiding decision-making due to their inability to quantify the impact on readiness. In this talk, we discuss the power of IDA’s end-to-end modeling and simulation (M&S) approach that estimates time-varying readiness outcomes based on real-world data on operations, supply, and maintenance. These models are designed to faithfully emulate fleet operations at the level of individual components and operational units, as well as to incorporate the multi-echelon inventory system used in military sustainment. We showcase a notional example in which our M&S approach produces a set of recommended component-level investments and divestments in wholesale supply that would improve the readiness of a weapon system. We argue for the urgency of increased end-to-end M&S efforts across the Department of Defense to guide the senior leadership in its data-driven decision-making for readiness initiatives. |
Andrew C. Flack, Han G. Yi Research Staff Member IDA (OED) (bio)
Han Yi is a Research Staff Member in the Operational Evaluation division at IDA. His work focuses on weapons system sustainment and readiness modeling. Prior to joining IDA in 2020, he completed his PhD in Communication Sciences and Disorders at The University of Texas at Austin and served as a Postdoctoral Scholar at the University of California, San Francisco. Andrew Flack is a Research Staff Member in the Operational Evaluation division at IDA. His work focuses on weapons system sustainment and readiness modeling. Prior to joining IDA in 2016, Andrew was an analyst at the Defense Threat Reduction Agency (DTRA) studying M&S tools for chemical and biological defense. |
Breakout | Session Recording |
| 2022 |
Breakout Challenges in Verification and Validation of CFD for Industrial Aerospace Applications (Abstract)
Verification and validation represent important steps for appropriate use of CFD codes and it is presently considered the user’s responsibility to ensure that these steps are completed. Inconsistent definitions and use of these terms in aerospace complicate the effort. For industrial-use CFD codes, there are a number of challenges that can further confound these efforts including varying grid topology, non-linearities in the solution, challenges in isolating individual components, and difficulties in finding validation experiments. In this presentation, a number of these challenges will be reviewed with some specific examples that demonstrate why verification is much more involved and challenging than typically implied in numerical method courses, but remains an important exercise. Some of the challenges associated with validation will also be highlighted using a range of different cases, from canonical flow elements to complete aircraft models. Benchmarking is often used to develop confidence in CFD solutions for engineering purposes, but falls short of validation in the absence of being able to predict bounds on the simulation error. The key considerations in performing benchmarking and validation will be highlighted and some current shortcomings in practice will be presented, leading to recommendations for conducting validation exercises. CFD workshops have considerably improved in their application of these practices, but there continues to be need for additional steps. |
Andrew Cary Technical Fellow Boeing Research and Technology ![]() (bio)
Andrew Cary is a technical fellow of the Boeing Company in CFD and is the focal for the BCFD solver. In this capacity, he has a strong focus on supporting users of the code across the Boeing enterprise as well as leading the development team. These responsibilities align with his interests in verification, validation, and uncertainty quantification as an approach to ensure reliable results as well as in algorithm development, CFD-based shape optimization, and unsteady fluid dynamics. Since hiring into the CFD team in 1996, he has led CFD application efforts across a full range of Boeing products as well as working in grid generation methods, flow solver algorithms, post-processing approaches, and process automation. These assignments have given him the opportunity to work with teams around the world, both inside and outside Boeing. Andrew has been an active member of the American Institute of Aeronautics and Astronautics, serving in multiple technical committees, including his present role on the CFD Vision 2030 Integration Committee. Andrew has also been an adjunct professor at Washington University since 1999, teaching graduate classes in CFD and fluid dynamics. Andrew received a Ph.D. (97) in Aerospace Engineering from the University of Michigan and a B.S. (92) and M.S. (97) in Aeronautical and Astronautical Engineering from the University of Illinois Urbana-Champaign. |
Breakout |
| 2021 |
|
Breakout Using the R ecosystem to produce a reproducible data analysis pipeline (Abstract)
Advances in open-source software have brought powerful machine learning and data analysis tools requiring little more than a few coding basics. Unfortunately, the very nature of rapidly changing software can contribute to legitimate concerns surrounding the reproducibility of research and analysis. Borrowing from current practices in data science and software engineering fields, a more robust process using the R ecosystem to produce a version-controlled data analysis pipeline is proposed. By integrating the data cleaning, model generation, manuscript writing, and presentation scripts, a researcher or data analyst can ensure small changes at any step will automatically be reflected throughout using the Rmarkdown, targets, renv, and xaringan R packages. |
Andrew Farina Assistant Professor- Department of Behavioral Sciences and Leadership United States Military Academy ![]() (bio)
Andrew G. Farina is an Assistant Professor at the United States Military Academy, Department of Behavioral Sciences and Leadership. He has ten combat deployments, serving with both conventional and special operations units. His research interests include leadership, character development, and risk-taking propensity. |
Breakout |
| 2022 |
|
Tutorial Tutorial: Reproducible Research (Abstract)
Analyses are “reproducible” if the same methods applied to the same data produce identical results when run again by another researcher (or you in the future). Reproducible analyses are transparent and easy for reviewers to verify, as results and figures can be traced directly to the data and methods that produced them. There are also direct benefits to the researcher. Real-world analysis workflows inevitably require changes to incorporate new or additional data, or to address feedback from collaborators, reviewers, or sponsors. These changes are easier to make when reproducible research best practices have been considered from the start. Poor reproducibility habits result in analyses that are difficult or impossible to review, are prone to compounded mistakes, and are inefficient to re-run in the future. They can lead to duplication of effort or even loss of accumulated knowledge when a researcher leaves your organization. With larger and more complex datasets, along with more complex analysis techniques, reproducibility is more important than ever. Although reproducibility is critical, it is often not prioritized either due to a lack of time or an incomplete understanding of end-to-end opportunities to improve reproducibility. This tutorial will discuss the benefits of reproducible research and will demonstrate ways that analysts can introduce reproducible research practices during each phase of the analysis workflow: preparing for an analysis, performing the analysis, and presenting results. A motivating example will be carried throughout to demonstrate specific techniques, useful tools, and other tips and tricks where appropriate. The discussion of specific techniques and tools is non-exhaustive; we focus on things that are accessible and immediately useful for someone new to reproducible research. The methods will focus mainly on work performed using R, but the general concepts underlying reproducible research techniques can be implemented in other analysis environments, such as JMP and Excel, and are briefly discussed. By implementing the approaches and concepts discussed during this tutorial, analysts in defense and aerospace will be equipped to produce more credible and defensible analyses of T&E data. |
Andrew Flack, Kevin Kirshenbaum, and John Haman IDA |
Tutorial |
| 2019 |
|
Sparse Models for Detecting Malicious Behavior in OpTC (Abstract)
Host-based sensors are standard tools for generating event data to detect malicious activity on a network. There is often interest in detecting activity using as few event classes as possible in order to minimize host processing slowdowns. Using DARPA’s Operationally Transparent Cyber (OpTC) Data Release, we consider the problem of detecting malicious activity using event counts aggregated over five-minute windows. Event counts are categorized by eleven features according to MITRE CAR data model objects. In the supervised setting, we use regression trees with all features to show that malicious activity can be detected at above a 90% true positive rate with a negligible false positive rate. Using forward and exhaustive search techniques, we show the same performance can be obtained using a sparse model with only three features. In the unsupervised setting, we show that the isolation forest algorithm is somewhat successful at detecting malicious activity, and that a sparse three-feature model performs comparably. Finally, we consider various search criteria for identifying sparse models and demonstrate that the RMSE criteria is generally optimal. |
Andrew Mastin Operations Research Scientist Lawrence Livermore National Laboratory ![]() (bio)
Andrew Mastin is an Operations Research Scientist at Lawrence Livermore National Laboratory. He holds a Ph.D. in Electrical Engineering and Computer Science from the Massachusetts Institute of Technology. His current research interests include cybersecurity, network interdiction, dynamic optimization, and game theory. |
| 2022 |
||
Breakout Test & Evaluation of ML Models (Abstract)
Machine Learning models have been incredibly impactful over the past decade; however, testing those models and comparing their performance has remained challenging and complex. In this presentation, I will demonstrate novel methods for measuring the performance of computer vision object detection models, including running those models against still imagery and video. The presentation will start with an introduction to the pros and cons of various metrics, including traditional metrics like precision, recall, average precision, mean average precision, F1, and F-beta. The talk will then discuss more complex topics such as tracking metrics, handling multiple object classes, visualizing multi-dimensional metrics, and linking metrics to operational impact. Anecdotes will be shared discussing different types of metrics that are appropriate for different types of stakeholders, how system testing fits in, best practices for model integration, best practices for data splitting, and cloud vs on-prem compute lessons learned. The presentation will conclude by discussing what software libraries are available to calculate these metrics, including the MORSE-developed library Charybdis. |
Anna Rubinstein Director of Test and Evaluation MORSE Corporation ![]() (bio)
Dr. Anna Rubinstein serves as the Director of Test and Evaluation for a Department of Defense (DoD) Artificial Intelligence (AI) program. She directs testing for AI models spanning the fields of computer vision, natural language processing, and other forms of machine perception. She leads teams developing metrics and assessing capabilities at the algorithm, system, and operational level, with a particular interest in human-machine teaming. Dr. Rubinstein has spent the last five years supporting national defense as a contractor, largely focusing on model and system evaluation. In her previous role as a Science Advisor in the Defense Advanced Research Projects Agency’s (DARPA) Information Innovation Office (I2O), she provided technical insight to research programs modeling cyber operations in the information domain and building secure software-reliant systems. Before that, Dr. Rubinstein served as a Research Staff Member at the Institute for Defense Analyses (IDA), leading efforts to provide verification and validation of nuclear weapons effects modeling codes in support of the Defense Threat Reduction Agency (DTRA). Dr. Rubinstein also has several years of experience developing algorithms for atmospheric forecasting, autonomous data fusion, social network mapping, anomaly detection, and pattern optimization. Dr. Rubinstein holds an M.A. in Chemical Engineering and a Ph.D. in Chemical Engineering and Materials Science from Princeton University, where she was a National Science Foundation Graduate Research Fellow. She also received a B.S. in Chemical Engineering, a B.A. in Chemistry, and a B.A. in Chinese, all from the University of Mississippi, where she was a Barry M. Goldwater Scholar. |
Breakout | Session Recording |
| 2022 |
Utilizing Machine Learning Models to Predict Success in Special Operations Assessment (Abstract)
The 75th Ranger Regiment is an elite Army Unit responsible for some of the most physically and mentally challenging missions. Entry to the unit is based on an assessment process called Ranger Regiment Assessment and Selection (RASP), which consists of a variety of tests and challenges of strength, intellect, and grit. This study explores the psychological and physical profiles of candidates who attempt to pass RASP. Using a Random Forest Artificial Intelligence model, and a penalized logistic regression model, we identify initial entry characteristics that are predictive of success in RASP. We focus on the differences between racial sub-groups and military occupational specialties (MOS) sub-groups to provide information for recruiters to identify underrepresented groups who are likely to succeed into the selection process. |
Anna Vinnedge Student United States Military Academy ![]() (bio)
Anna Vinnedge is a fourth year cadet at the United States Military Academy. She is a mathematics major from originally from Seattle, WA and the captain of the Varsity Women’s Crew team. After graduation, she will serve as a cyber officer in the Army. She has conducted previous research in quantum and DNA based cryptography and coding theory, and is currently focusing her time on machine learning research for statistical analysis. |
Session Recording |
| 2022 |
|
Breakout Debunking Stress Rupture Theories Using Weibull Regression Plots (Abstract)
As statisticians, we are always working on new ways to explain statistical methodologies to non-statisticians. It is in this realm that we never underestimate the value of graphics and patience! In this presentation, we present a case study that involves stress rupture data where a Weibull regression is needed to estimate the parameters. The context of the case study results from a multi-stage project supported by NASA’s Engineering Safety Center (NESC) where the objective was to assess the safety of composite overwrapped pressure vessels (COPVs). The analytical team was tasked with devising a test plan to model stress rupture failure risk in carbon fiber strands that encase the COPVs with the goal of understanding the reliability of the strands at use conditions for the expected mission life. While analyzing the data, we found that the proper analysis contradicts accepted theories about the stress rupture phenomena. In this talk, we will introduce ways to graph the stress rupture data to better explain the proper analysis and also explore assumptions. |
Anne Driscoll Associate Collegiate Professor Virginia Tech ![]() (bio)
Anne Ryan Driscoll is an Associate Collegiate Professor in the Department of Statistics at Virginia Tech. She received her PhD in Statistics from Virginia Tech. Her research interests include statistical process control, design of experiments, and statistics education. She is a member of ASQ and ASA. |
Breakout |
| 2021 |
|
Breakout Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation (Abstract)
Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices. |
Martin Annett Aerospace Corporation |
Breakout | Materials | 2017 |
|
Building Bridges: a Case Study of Assisting a Program from the Outside (Abstract)
STAT practitioners often find ourselves outsiders to the programs we assist. This session presents a case study that demonstrates some of the obstacles in communication of capabilities, purpose, and expectations that may arise due to approaching the project externally. Incremental value may open the door to greater collaboration in the future, and this presentation discusses potential solutions to provide greater benefit to testing programs in the face of obstacles that arise due to coming from outside the program team. DISTRIBUTION STATEMENT A. Approved for public release; distribution is unlimited. CLEARED on 5 Jan 2022. Case Number: 88ABW-2022-0002 |
Anthony Sgambellone Huntington Ingalls Industries ![]() (bio)
Dr. Tony Sgambellone is a STAT Expert (Huntington Ingalls Industries contractor) at the Scientific Test and Analysis Techniques (STAT) Center of Excellence (COE) at the Air Force Institute of Technology (AFIT). The STAT COE provides independent STAT consultation to designated acquisition programs and special projects to improve Test & Evaluation (T&E) rigor, effectiveness, and efficiency,. Dr. Sgambellone holds a Ph.D. in Statistics, a graduate minor in College and University Teaching, and has a decade of experience spanning the fields of finance, software, and test and development. His current interests include artificial neural networks and the application of machine learning. |
Session Recording |
| 2022 |
|
Tutorial Evolving Statistical Tools (Abstract)
In this session, researchers from the Institute for Defense Analyses (IDA) present a collection of statistical tools designed to meet ongoing and emerging needs for planning, designing, and evaluating operational tests. We first present a suite of interactive applications hosted on test.testscience.testscience.org that are designed to address common analytic needs in the operational test community. These freely available resources include tools for constructing confidence intervals, computing statistical power, comparing distributions, and computing Bayesian reliability. Next, we discuss four dedicated software tools: JEDIS – a JMP Add-In for automating power calculations for designed experiments skpr – an R package for generating optimal experimental designs and easily evaluating power for normal and non-normal response variables ciTools – an R package for quickly and simply generating confidence intervals and quantifying uncertainty for simple and complex linear models nautilus – an R package for visualizing and analyzing aspects of sensor performance, such as detection range and track completeness |
Benjamin Ashwell Research Staff Member IDA |
Tutorial | Materials | 2018 |
Session Title | Speaker | Type | Recording | Materials | Year |
---|---|---|---|---|---|
Breakout A Great Test Requires a Great Plan |
Aaron Ramert STAT Analyst Scientific Test and Analytics Techniques Center of Excellence (STAT COE) ![]() |
Breakout | Session Recording |
| 2021 |
Breakout The 80/20 rule, can and should we break it using efficient data management tools? |
Ghaleb Abdulla | Breakout |
| 2019 |
|
Breakout Using Sequential Testing to Address Complex Full-Scale Live Fire Test and Evaluation |
Darryl Ahner AFIT, STAT COE |
Breakout | 2016 |
||
Breakout Testing Autonomous Systems |
Darryl Ahner Director AFIT |
Breakout | Materials | 2018 |
|
Breakout Forecasting with Machine Learning |
Akshay Jain Data Science Fellow IDA |
Breakout | Session Recording |
| 2022 |
Breakout Legal, Moral, and Ethical Implications of Machine Learning |
Alan B. Gelder Research Staff Member IDA |
Breakout | Session Recording |
| 2022 |
Breakout Search for Extended Test Design Methods for Complex Systems of Systems |
Alex Alaniz AFOTEC |
Breakout | Materials | 2017 |
|
Closing Remarks |
Alyson Wilson NCSU ![]() |
2022 |
|||
Breakout Prior Formulation in a Bayesian Analysis of Biomechanical Data |
Amanda French Data Scientist Johns Hopkins University Applied Physics Laboratory ![]() |
Breakout | Session Recording |
| 2021 |
Breakout Design and Analysis of Experiments for Europa Clipper’s Science Sensitivity Model |
Amy Braverman | Breakout |
| 2019 |
|
Tutorial Tutorial: Combinatorial Methods for Testing and Analysis of Critical Software and Security Systems |
Rick Kuhn, Dimitris Simos, and Raghu Kacker National Institute of Standards & Technology |
Tutorial |
| 2019 |
|
Keynote STAT Engineering Keynote-Wednesday AM |
Christine Anderson Cook Statistics Los Alamos National Lab ![]() |
Keynote | 2016 |
||
Breakout XPCA: A Copula-based Generalization of PCA for Ordinal Data |
Cliff Anderson-Bergman Sandia National Laboratories |
Breakout | Materials | 2018 |
|
Panel The Keys to Successful Collaborations during Test and Evaluation: Moderator |
Christine Anderson-Cook Los Alamos National Lab ![]() |
Panel | Session Recording | 2021 |
|
Breakout M&S approach for quantifying readiness impact of sustainment investment scenarios |
Andrew C. Flack, Han G. Yi Research Staff Member IDA (OED) |
Breakout | Session Recording |
| 2022 |
Breakout Challenges in Verification and Validation of CFD for Industrial Aerospace Applications |
Andrew Cary Technical Fellow Boeing Research and Technology ![]() |
Breakout |
| 2021 |
|
Breakout Using the R ecosystem to produce a reproducible data analysis pipeline |
Andrew Farina Assistant Professor- Department of Behavioral Sciences and Leadership United States Military Academy ![]() |
Breakout |
| 2022 |
|
Tutorial Tutorial: Reproducible Research |
Andrew Flack, Kevin Kirshenbaum, and John Haman IDA |
Tutorial |
| 2019 |
|
Sparse Models for Detecting Malicious Behavior in OpTC |
Andrew Mastin Operations Research Scientist Lawrence Livermore National Laboratory ![]() |
| 2022 |
||
Breakout Test & Evaluation of ML Models |
Anna Rubinstein Director of Test and Evaluation MORSE Corporation ![]() |
Breakout | Session Recording |
| 2022 |
Utilizing Machine Learning Models to Predict Success in Special Operations Assessment |
Anna Vinnedge Student United States Military Academy ![]() |
Session Recording |
| 2022 |
|
Breakout Debunking Stress Rupture Theories Using Weibull Regression Plots |
Anne Driscoll Associate Collegiate Professor Virginia Tech ![]() |
Breakout |
| 2021 |
|
Breakout Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation |
Martin Annett Aerospace Corporation |
Breakout | Materials | 2017 |
|
Building Bridges: a Case Study of Assisting a Program from the Outside |
Anthony Sgambellone Huntington Ingalls Industries ![]() |
Session Recording |
| 2022 |
|
Tutorial Evolving Statistical Tools |
Benjamin Ashwell Research Staff Member IDA |
Tutorial | Materials | 2018 |