| Session Title | Speaker | Type | Materials | Year |
|---|---|---|---|---|
Short Course Overview of Design of Experiments (Abstract)
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.The course outcomes are: • Ability to plan and execute experiments. • Ability to collect data and analyze and interpret these data to provide the knowledge required for business success. • Knowledge of a wide range of modern experimental tools that enable practitioners to customize their experiment to meet practical resource constraintsThe topics covered during the course are: • Fundamentals of DOX – randomization, replication, and blocking. • Planning for a designed experiment – type and size of design, factor selection, levels and ranges, response measurement, sample sizes. • Graphical and statistical approaches to DOX analysis. • Blocking to eliminate the impact of nuisance factors on experimental results. • Factorial experiments and interactions. • Fractional factorials – efficient and effective use of experimental resources. • Optimal designs. • Response surface methods. • A demonstration illustrating and comparing the effectiveness of different experimental design strategies. This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course. |
Bradley Jones Distinguished Research Fellow JMP Division/SAS |
Short Course | Materials | 2018 |
Short Course Overview of Design of Experiments (Abstract)
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.The course outcomes are: • Ability to plan and execute experiments. • Ability to collect data and analyze and interpret these data to provide the knowledge required for business success. • Knowledge of a wide range of modern experimental tools that enable practitioners to customize their experiment to meet practical resource constraintsThe topics covered during the course are: • Fundamentals of DOX – randomization, replication, and blocking. • Planning for a designed experiment – type and size of design, factor selection, levels and ranges, response measurement, sample sizes. • Graphical and statistical approaches to DOX analysis. • Blocking to eliminate the impact of nuisance factors on experimental results. • Factorial experiments and interactions. • Fractional factorials – efficient and effective use of experimental resources. • Optimal designs. • Response surface methods. • A demonstration illustrating and comparing the effectiveness of different experimental design strategies. This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course. |
Doug Motgomery Regents’ Professor of Industrial Engineering and Statistics ASU Foundation Professor of Engineering Arizona State University |
Short Course | Materials | 2018 |
Short Course Introduction to R (Abstract)
This course is designed to introduce participants to the R programming language and the R studio editor. R is a free and open-source software for summarizing data, creating visuals of data, and conducting statistical analyses. R can offer many advantages over programs such as Excel including faster computation, customized analyses, access to the latest statistical techniques, automation of tasks, and the ability to easily reproduce research. After completing this course, a new user should be able to: • Import/export data from/to external files. • Create and manipulate new variables. • Conduct basic statistical analyses (such as t-tests and linear regression). • Create basic graphs. • Install and use R packages Participants should bring a laptop for the interactive components of the course. |
Justin Post North Carlina State Univeristy |
Short Course | Materials | 2018 |
Short Course Survey Construction and Analysis (Abstract)
In this course, we introduce the main concepts of the survey methodology process – from survey sampling design to analyzing the data obtained from complex survey designs. The course topics include: 1. Introduction to the Survey Process 2. R Tools 3. Sampling Designs – Simple Random Sampling, Cluster Sampling, Stratified Sampling, and more 4. Weighting and Variance Estimation 5. Exploratory Data Analysis 6. Complex Survey Analysis We use a combination of lectures and hands-on exercises using R. Students are expected to have R and associated packages installed on their computers. We will send a list of required packages before the course. We also use data from Department of Defense surveys, where appropriate. |
Wendy Matrinez Bureau of Labor and Statistics |
Short Course | Materials | 2018 |
Short Course Survey Construction and Analysis (Abstract)
In this course, we introduce the main concepts of the survey methodology process – from survey sampling design to analyzing the data obtained from complex survey designs. The course topics include: 1. Introduction to the Survey Process 2. R Tools 3. Sampling Designs – Simple Random Sampling, Cluster Sampling, Stratified Sampling, and more 4. Weighting and Variance Estimation 5. Exploratory Data Analysis 6. Complex Survey Analysis We use a combination of lectures and hands-on exercises using R. Students are expected to have R and associated packages installed on their computers. We will send a list of required packages before the course. We also use data from Department of Defense surveys, where appropriate. |
MoonJung Cho Bureau of Labor and Statistics |
Short Course | Materials | 2018 |
Short Course Uncertainty Quantification (Abstract)
We increasingly rely on mathematical and statistical models to predict phenomena ranging from nuclear power plant design to profits made in financial markets. When assessing the feasibility of these predictions, it is critical to quantify uncertainties associated with the models, inputs to the models, and data used to calibrate the models. The synthesis of statistical and mathematical techniques, which can be used to quantify input and response uncertainties for simulation codes that can take hours to days to run, comprises the evolving field of uncertainty quantification. The use of data, to improve the predictive accuracy of models, is central to uncertainty quantification so we will begin by providing an overview of how Bayesian techniques can be used to construct distributions for model inputs. We will subsequently describe the computational issues associated with propagating these distributions through complex models to construct prediction intervals for statistical quantities of interest such as expected profits or maximal reactor temperatures. Finally, we will describe the use of sensitivity analysis to isolate critical model inputs and surrogate model construction for simulation codes that are too complex for direct statistical analysis. All topics will be motivated by examples arising in engineering, biology, and economics. |
Ralph Smith North Carlina State Univeristy |
Short Course | Materials | 2018 |
Short Course Combinatorial Interaction Testing (Abstract)
This mini-tutorial provides an introduction to combinatorial interaction testing (CIT). The main idea behind CIT is to pseudo-exhaustively test software and hardware systems by covering combinations of components in order to detect faults. In 90 minutes, we provide an overview of this domain that includes the following topics: the role of CIT in software and hardware testing, how it complements and differs from design of experiments, considerations such as variable strength and constraints, the typical combinatorial arrays used for constructing test suites, and existing tools for test suite construction. Last, defense systems are increasingly relying on software with embedded machine learning (ML), yet ML poses unique challenges to applying conventional software testing due to characteristics such as the large input space, effort required for white box testing, and emergent behaviors apparent only at integration or system levels. As a well-studied black box approach to testing integrated systems with a pseudo-exhaustive strategy for handling large input spaces, CIT provides a good foundation for testing ML. In closing, we present recent research adapting concepts of combinatorial coverage to test design for ML. |
Erin Lanus Research Assistant Professor Virginia Tech 
(bio)
Erin Lanus is a Research Assistant Professor at the Hume Center for National Security and Technology at Virginia Tech. She has a Ph.D. in Computer Science with a concentration in cybersecurity from Arizona State University. Her experience includes work as a Research Fellow at University of Maryland Baltimore County and as a High Confidence Software and Systems Researcher with the Department of Defense. Her current interests are software and combinatorial testing, machine learning in cybersecurity, and artificial intelligence assurance. |
Short Course |
Materials
Recording | 2021 |
Short Course Introduction to Neural Networks for Deep Learning with Tensorflow (Abstract)
This mini-tutorial session discusses the practical application of neural networks from a lay person’s perspective and will walk through a hands-on case study in which we build, train, and analyze a few neural network models using TensorFlow. The course will review the basics of neural networks and touch on more complex neural network architecture variants for deep learning applications. Deep learning techniques are becoming more prevalent throughout the development of autonomous and AI-enabled systems, and this session will provide students with the foundational intuition needed to understand these systems. |
Roshan Patel Data Scientist US Army CCDC Armaments Center 
(bio)
Mr. Roshan Patel is a systems engineer and data scientist working at CCDC Armament Center. His role focuses on systems engineering infrastructure, statistical modeling, and the analysis of weapon systems. He holds a Masters of Computer Science from Rutgers University, where he specialized in operating systems programming and machine learning. At Rutgers, Mr. Patel was a part-time lecturer for systems programming and data science seminars. Mr. Patel is the current AI lead for the Systems Engineering Directorate at CCDC Armaments Center. |
Short Course |
Materials
Recording | 2021 |
Roundtable The Role of the Statistics Profession in the DoD’s Current AI Initiative (Abstract)
In 2019, the DoD unveiled comprehensive strategies related to Artificial Intelligence, Digital Modernization, and Enterprise Data Analytics. Recognizing that data science and analytics are fundamental to these strategies, in October 2020 the DoD has issued a comprehensive Data Strategy for national security and defense. For over a hundred years, statistical sciences have played a pivotal role in our national defense, from quality assurance and reliability analysis of munitions fielded in WWII, to operational analyses defining battlefield force structure and tactics, to helping optimize the engineering design of complex products, to rigorous testing and evaluating of Warfighter systems. The American Statistical Association (ASA) in 2015 recognized in its statement on The Role of Statistics in Data Science that “statistics is foundational to data science… and its use in this emerging field empowers researchers to extract knowledge and obtain better results from Big Data and other analytics projects.” It is clearly recognized that data as information is a key asset to the DoD. The challenge we face is how to transform existing talent to add value where it counts. |
Laura Freeman Research Associate Professor of Statistics and Director of the Intelligent Systems Lab Virginia Tech 
(bio)
Dr. Laura Freeman is a Research Associate Professor of Statistics and the Director of the Intelligent Systems Lab at the Virginia Tech Hume Center. Her research leverages experimental methods for conducting research that brings together cyber-physical systems, data science, artificial intelligence (AI), and machine learning to address critical challenges in national security. She is also a hub faculty member in the Commonwealth Cyber Initiative and leads research in AI Assurance. She develops new methods for test and evaluation focusing on emerging system technology. She is also the Assistant Dean for Research in the National Capital Region, in that capacity she works to shape research directions and collaborations in across the College of Science in the National Capital Region. Previously, Dr. Freeman was the Assistant Director of the Operational Evaluation Division at the Institute for Defense Analyses. In that position, she established and developed an interdisciplinary analytical team of statisticians, psychologists, and engineers to advance scientific approaches to DoD test and evaluation. During 2018, Dr. Freeman served as that acting Senior Technical Advisor for Director Operational Test and Evaluation (DOT&E). As the Senior Technical Advisor, Dr. Freeman provided leadership, advice, and counsel to all personnel on technical aspects of testing military systems. She reviewed test strategies, plans, and reports from all systems on DOT&E oversight. Dr. Freeman has a B.S. in Aerospace Engineering, a M.S. in Statistics and a Ph.D. in Statistics, all from Virginia Tech. Her Ph.D. research was on design and analysis of experiments for reliability data. |
Roundtable | 2021 |
|
Roundtable Test Design and Analysis for Modeling & Simulation Validation (Abstract)
System evaluations increasingly rely on modeling and simulation (M&S) to supplement live testing. It is thus crucial to thoroughly validate these M&S tools using rigorous data collection and analysis strategies. At this roundtable, we will identify and discuss some of the core challenges currently associated with implementing M&S validation for T&E. First, appropriate design of experiments (DOE) for M&S is not universally adopted across the T&E community. This arises in part due to limited knowledge of gold standard techniques from academic research (e.g., space filing designs; Gaussian Process emulators) as well as lack of expertise with the requisite software tools. Second, T&E poses unique demands in testing, such as extreme constraints in live testing conditions and reliance on binary outcomes. There is no consensus on how to incorporate these needs into the existing academic framework for M&S. Finally, some practical considerations lack clear solutions yet have direct consequences on design choice. In particular, we may discuss the following: (1) sample size determination when calculating power and confidence is not applicable, and (2) non-deterministic M&S output with high levels of noise, which may benefit from replication samples as in classical DOE. |
Kelly Avery Research Staff Member Institute for Defense Analyses 
(bio)
Kelly M. Avery is a Research Staff Member at the Institute for Defense Analyses. She supports the Director, Operational Test and Evaluation (DOT&E) on the use of statistics in test & evaluation and modeling & simulation, and has designed tests and conducted statistical analyses for several major defense programs including tactical aircraft, missile systems, radars, satellite systems, and computer-based intelligence systems. Her areas of expertise include statistical modeling, design of experiments, modeling & simulation validation, and statistical process control. Dr. Avery has a B.S. in Statistics, a M.S. in Applied Statistics, and a Ph.D. in Statistics, all from Florida State University. |
Roundtable | 2021 |
|
Roundtable Organizing and Sharing Data within the T&E Community (Abstract)
Effective data sharing requires alignment of personnel, systems, and policies. Data are costly and precious, and to get the most value out of the data we collect, it is important that we share and reuse it whenever possible and appropriate. Data are typically collected and organized with a single specific use or goal in mind, and after that goal has been achieved (e.g., the report is published), the data are no longer viewed as important or useful. This process is self-fulfilling, as future analysts who might want to use these data will not be able to find them or will be unable to understand the data sufficiently due to lack of documentation and metadata. Implementing data standards and facilitating sharing are challenging in the national security environment. There are many data repositories within the DoD, but most of them are specific to certain organizations and are accessible by a limited number of people. Valid concerns about security make the process of sharing particular data sets challenging, and the opacity of data ownership often complicates the issue. This roundtable will facilitate discussion of these issues. Participants will have opportunities to share their experiences trying to share data and make use of data from previous testing. We hope to identify useful lessons learned and find ways to encourage data sharing within the community. |
Matthew Avery Research Staff Memeber Institute for Defense Analyses 
(bio)
Dr. Matthew Avery is a Research Staff Member at the Institute for Defense Analyses in the Operational Evaluation Division. As the Tactical Aerial Reconnaissance Systems Project Leader, he provides input to the DoD on test plans and reports for Army, Marine Corps, and Navy tactical unmanned aircraft systems. Dr. Avery co-chairs IDA’s Data Governance Committee and leads the Data Management Group within IDA’s Test Science team, where he helps develop internal policies and trainings on data usage, access and storage. From 2017 to 2018, Dr. Avery worked as an embedded analyst at the Department of Defense Office of Cost Analysis and Program Evaluation (CAPE), focusing on Space Control and Mobilization. Dr. Avery received his PhD in Statistics from North Carolina State University in 2012. |
Roundtable | 2021 |
|
Roundtable Opportunities and Challenges for Openly Publishing Statistics Research for National Defense (Abstract)
Openly publishing on statistics for defense and national security poses certain challenges and is often not straightforward, but research in this area is important to share with the open community to advance the field. Since statistical research for national defense applications is rather niche, target journals and audiences are challenging to identify. Adding an additional hurdle, much of the data for practical implementation is sensitive and surrogate datasets must be relied upon for publication. Lastly, many statisticians in these areas do not face the same expectations to openly publish as their colleagues. This roundtable is an opportunity for statisticians in the defense and national security community to come together and discuss the importance and challenges for publishing within this space. Participants will be asked to share challenges and successes related to publishing research for national defense applications. A handout summarizing common challenges and tips for succeeding will be provided. Specific topics for discussion will include: What expectations exist for statisticians to publish in this community? Are these expectations reasonable? Are you encouraged and supported by your funding institution to openly publish? Are there opportunities for collaborative work across institutions that might further encourage publications? |
Lyndsay Shand R&D Statistician Sandia National Laboratories 
(bio)
Lyndsay Shand received her PhD in Statistics from University of Illinois Urbana-Champaign in 2017 with a focus on spatio-temporal analysis. At Sandia, Lyndsay continues to conduct, lead and publish on research centered around space-time statistics with applications to material science, climate science and disease modeling to name a few. Specific research topics have included accounting for missing data in space-time point processes, Bayesian functional data registration, hierarchical disease modeling, spatial extremes and design of experiments for computer model. Lyndsay currently leads efforts to quantify the effects of ship-emissions on clouds to reduce uncertainty of aerosol processes in climate models and serves as programmatic lead and statistical subject matter expert of a team developing experimental design and surrogate modeling approaches for material modeling efforts at Sandia. Additionally, Lyndsay serves as a statistical consultant to many projects across the lab and was nominated for an employee recognition award for her leadership in 2019. Outside of research, Lyndsay is actively engaged in joint research with university partners including the University of Illinois, Brigham Young University, and the University of Washington. She is also a member and of the International Society for Bayesian Analysis (ISBA), the American Statistical Association, and the American Geophysical Union, serving as co-web editor for ISBA and treasurer for the environmental section of ISBA. |
Roundtable |
| 2021 |
Roundtable Identifying Challenges and Solutions to T&E of Non-IP Networks (Abstract)
Many systems within the Department of Defense (DoD) contain networks that use both Internet Protocol (IP) and non-IP forms of information exchange. While IP communication is widely understood among the cybersecurity community, expertise and available test tools for non-IP protocols such as Controller Area Network (CAN), MIL-STD-1553, and SCADA are not as commonplace. Over the past decade, the DoD has repeatedly identified gaps in data collection and analysis when assessing the cybersecurity of non-IP buses. This roundtable is intended to open a discussion among testers and evaluators on the existing measurement and analysis tools for non-IP buses used across the community and also propose solutions to recurring roadblocks experienced when performing operational testing on non-IP components. Specific topics of discussion will include: What tools do you or your supporting teams use during cybersecurity events to attack, scan, and monitor non-IP communications? What raw quantitative data do you collect that captures the adversarial activity and/or system response from cyber aggression to non-IP components? Please provide examples of test instrumentation and data collection methods. What data analysis tools do you use to draw conclusions from measured data? What types of non-IP buses, including components on those buses, have you personally been able to test? What components were you not able to test? Why were you not able to test them? Was it due to safety concerns, lack of permission, lack of available tools and expertise, or other? Had you been given authority to test those components, do you think it would have improved the quality of test and comprehensiveness of the assessment? |
Peter Mancini Research Staff Member Institute for Defense Analyses 
(bio)
Peter Mancini works at the Institute for Defense Analyses, supporting the Director, Operational Test and Evaluation (DOT&E) as a Cybersecurity OT&E analyst. |
Roundtable | 2021 |
|
Roundtable Overcoming Challenges and Applying Sequential Procedures to T&E (Abstract)
The majority of statistical analyses involves observing a fixed set of data and analyzing those data after the final observation has been collected to draw some inference about the population from which they came. Unlike these traditional methods, sequential analysis is concerned with situations for which the number, pattern, or composition of the data is not determined at the start of the investigation but instead depends upon the information acquired throughout the course of the investigation. Expanding the use of sequential analysis in DoD testing has the potential to save substantial test dollars and decrease test time. However, switching from traditional to sequential planning will likely induce unique challenges. The goal of this round table is to provide an open forum for topics related to sequential analyses. We aim to discuss potential challenges, identify potential ways to overcome them, and talk about successful stories of sequential analyses implementation and lessons learned. Specific questions for discussion will be provided to participants prior to the event. |
Rebecca Medlin Research Staff Member Institute for Defense Analyses 
(bio)
Dr. Rebecca Medlin is a Research Staff Member at the Institute for Defense Analyses. She supports the Director, Operational Test and Evaluation (DOT&E) on the use of statistics in test & evaluation and has designed tests and conducted statistical analyses for several major defense programs including tactical vehicles, mobility aircraft, radars, and electronic warfare systems. Her areas of expertise include design of experiments, statistical modeling, and reliability. She has a Ph.D. in Statistics from Virginia Tech. |
Roundtable | 2021 |
|
Panel Army’s Open Experimentation Test Range for Internet of Battlefield Things: MSA-DPG (Abstract)
One key feature of future Multi-Domain Operations (MDO) is expected to be the ubiquity of devices providing information connected in an Internet of Battlefield Things (IoBT). To this end, U.S. Army aims to advance the underlying science of pervasive and heterogeneous IoBT sensing, networking, and actuation. In this effort, IoBT experimentation testbed is an integral part of the capability development, which evaluates and validates the scientific theories, algorithms, and technologies integrated with C2 systems under the military scenarios. Originally conceived for this purpose, Multi-Purpose Sensing Area Distributed Proving Ground (MSA-DPG) is an open-range test bed developed by the Army Research Laboratory (ARL). We discuss the vision and the development of MSA-DPG and its fundamental roles of MSA-DPG in research serving the communities of Military Sciences. |
Jade Freeman Research Scientist U.S. Army DEVCOM Army Research Laboratory 
(bio)
Dr. Jade Freeman currently serves as the Associate Branch Chief and a Team Lead at Battlefield Information Systems Branch. In this capacity, Dr. Freeman oversees information systems and engineering research projects and analyses. Prior to joining ARL, Dr. Freeman served as the Senior Statistician at the Office of Cybersecurity and Communications, Department of Homeland Security. Throughout the career, her work in operations and research includes cyber threat analyses, large survey design and analyses, experimental design, survival analysis, and missing data imputation methods. Dr. Freeman is also a PMP certified project manager, experienced in leading and managing IT development projects. Dr. Freeman obtained a Ph. D. in Statistics from the George Washington University. |
Panel |
| 2021 |
Panel Multi-Agent Adaptive Coordinated Autonomy in Contested Battlefields (Abstract)
Autonomous multi-robot systems have the potential to augment the future force with enhanced capability while reducing the risk to human personnel in multi-domain operations (MDO). Mobile robots can constitute nodes in a heterogeneous Internet of Battlefield Things (IoBT); they can offer additional capability in the form of mobility to effectively make observations useful for planning and executing military operations against adversaries. In this talk, I will present the result of a series of field experiments where robots are tasked to perform military-relevant missions in realistic environments, in addition to describing the integration of mobile robot assets in the Multi-Purpose Sensing Array Distributed Proving Ground (MSA-DPG) for the purpose of augmenting IoBT systems. |
John Rogers Senior Research Scientist U.S. Army DEVCOM Army Research Laboratory 
(bio)
John Rogers is a research scientist specializing in autonomous mobile robotics at the Army Research Laboratory’s Intelligent Robotics Branch of the Computational and Information Sciences Directorate (CISD). John’s research has focused on autonomy for multi-robot teams as well as distributed multi-robot state estimation. John is currently leading the Tactical Behaviors group in the AI for mobility and maneuver essential research program, which is focused on developing deep learning and game theoretic maneuver for ground robots against adversaries. Prior to this, John led the multi-robot mapping research on the Autonomy Research Pilot Initiative (ARPI) on “autonomous collective defeat of hard and deeply buried targets” in collaboration with his colleagues at the Air Force Research Laboratory. John has also partnered with DCIST, MAST and Robotics CTA partners to extend funded research programs within-house collaborative projects. John completed his Ph.D. degree at the Georgia Institute of Technology in2012 with his advisor, Prof. Henrik Christensen from the Robotics and Intelligent Machines center. While at Georgia Tech, John participated in a variety of sponsored research projects including the Micro Autonomous Systems and Technology (MAST) project from the Army Research Laboratory, a counter-terrorism “red team” project for the Naval Research Laboratory in addition to his thesis research on semantic mapping and reasoning culminating in his thesis “Life-long mapping of objects and places for domestic robots”. Prior to attending Georgia Tech, John completed a M.S. degree in Computer Science at Stanford (2006) while working with Prof. Sebastian Thrun and Prof. Andrew Ng on the DARPA Learning Applied to Ground Robots (LAGR) project. John also holds M.S. and B.S. degrees in Electrical and Computer Engineering from Carnegie Mellon University (2002). John has authored or co-authored over 50 scientific publications in Robotics and Computer Vision journals and conferences. John’s current research interests are automatic exploration and mapping of large-scale indoor, outdoor, and subterranean environments, place recognition in austere locations, semantic scene understanding, and probabilistic reasoning for autonomous mobile robots. CV: Google Scholar: https://scholar.google.com/citations?user=uH_LDocAAAAJ&hl=en |
Panel |
| 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
Sarah Burke STAT Expert STAT Center of Excellence 
(bio)
Dr. Sarah Burke is a scientific test and analysis techniques (STAT) Expert for the STAT Center of Excellence. She works with acquisition programs in the Air Force, Army, and Navy to improve test efficiency, plan tests effectively, and analyze the resulting test data to inform decisions on system development. She received her M.S. in Statistics and Ph.D. in Industrial Engineering from Arizona State University. |
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
John Haman RSM Institute for Defense Analyses 
(bio)
Dr. John Haman is a statistician at the Institute for Defense Analyses, where he develops methods and tools for analyzing test data. He has worked with a variety of Army, Navy, and Air Force systems, including counter-UAS and electronic warfare systems. Currently, John is supporting the Joint Artificial Intelligence Center. |
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
Peter Parker Team Lead for Advanced Measurement Systems NASA Langley 
(bio)
Dr. Peter Parker is Team Lead for Advanced Measurement Systems at the National Aeronautics and Space Administration’s Langley Research Center in Hampton, Virginia. He serves an Agency-wide statistical expert across all of NASA’s mission directorates of Exploration, Aeronautics, and Science to infuse statistical thinking, engineering, and methods. His expertise is in collaboratively integrating research objectives, measurement sciences, modeling and simulation, and test design to produce actionable knowledge that supports rigorous decision-making for aerospace research and development. |
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
Willis Jensen HR Analyst W.L. Gore & Associates 
(bio)
Dr. Willis Jensen is a member of the HR Analytics team at W.L. Gore & Associates, where he supports people related analytics work across the company. At Gore, he previously spent 12 years as a statistician and as the Global Statistics Team Leader where he led a team of statisticians that provided statistical support and training across the globe. He holds degrees in Statistics from Brigham Young University and a Ph.D. in Statistics from Virginia Tech. |
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Moderator (Abstract)
The defense industry faces increasingly complex systems in test and evaluation (T&E) that require interdisciplinary teams to successfully plan testing. A critical aspect in test planning is a successful collaboration between T&E experts, subject matter experts, program leadership, statisticians, and others. This panel, based on their own experiences as consulting statisticians, will discuss elements that lead to successful collaborations, barriers during collaboration, and recommendations to improve collaborations during T&E planning. |
Christine Anderson-Cook Los Alamos National Lab 
|
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: HSI | Trustworthy AI (Abstract)
Recent successes and shortcomings of AI implementations have highlighted the importance of understanding how to design and interpret trustworthiness. AI Assurance is becoming a popular objective for some stakeholders, however, assurance and trustworthiness are context-sensitive concepts that rely not only on software performance and cybersecurity, but also on human-centered design. This talk summarizes Cognitive Engineering principles in the context of resilient AI engineering. It also introduces approaches for successful Human-Machine Teaming in high risk work domains. |
Stoney Trent Research Professor and Principal Advisor for Research and Innovation; Founder Virginia Tech; The Bulls Run Group, LLC 
(bio)
Stoney Trent, Ph.D. Research Professor and Principal Advisor for Research and Innovation, Virginia Tech; Founder, The Bulls Run Group, LLC Stoney is a Cognitive Engineer and Military Intelligence and Cyber Warfare veteran, who specializes in human-centered innovation. As an Army officer, Stoney designed and secured over $350M to stand up the Joint Artificial Intelligence Center (JAIC) for the Department of Defense. As the Chief of Missions in the JAIC, Stoney established product lines to deliver human-centered AI to improve warfighting and business functions in the world’s largest bureaucracy. Previously, he established and directed U.S. Cyber Command’s $50M applied research lab, which develops and assesses products for the Cyber Mission Force. Stoney has served as a Strategic Policy Research Fellow with the RAND Arroyo Center and is a former Assistant Professor in the Department of Behavioral Science and Leadership at the United States Military Academy. He has served in combat and stability operations in Iraq, Kosovo, Germany, and Korea. Stoney is a graduate of the Army War College and former Cyber Fellow at the National Security Agency. |
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: Evaluating Warfighters’ Ability to Employ AI Capabilities (Abstract)
Although artificial intelligence may take over tasks traditionally performed by humans or power systems that act autonomously, humans will still interact with these systems in some way. The need to ensure these interactions are fluid and effective does not disappear—if anything, this need only grows with AI-enabled capabilities. These technologies introduce multiple new hazards for achieving high quality human-system integration. Testers will need to evaluate both traditional HSI issues as well as these novel concerns in order to establish the trustworthiness of a system for activity in the field, and we will need to develop new T&E methods in order to do this. In this session, we will hear how three national security organizations are preparing for these HSI challenges, followed by a broader panel discussion on which of these problems is most pressing and which is most promising for DoD research investments. |
Dan Porter Research Staff Member Institute for Defense Analyses 
|
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: Evaluating HSI with AI-Enabled Systems: What should you consider in a TEMP? |
Jane Pinelis Chief of the Test, Evaluation, and Assessment branch Department of Defense Joint Artificial Intelligence Center (JAIC) 
(bio)
Dr. Jane Pinelis is the Chief of the Test, Evaluation, and Assessment branch at the Department of Defense Joint Artificial Intelligence Center (JAIC). She leads a diverse team of testers and analysts in rigorous test and evaluation (T&E) for JAIC capabilities, as well as development of T&E-specific products and standards that will support testing of AI-enabled systems across the DoD. Prior to joining the JAIC, Dr. Pinelis served as the Director of Test and Evaluation for USDI’s Algorithmic Warfare Cross-Functional Team, better known as Project Maven. She directed the developmental testing for the AI models, including computer vision, machine translation, facial recognition and natural language processing. Her team developed metrics at various levels of testing for AI capabilities and provided leadership empirically-based recommendations for model fielding. Additionally, she oversaw operational and human-machine teaming testing, and conducted research and outreach to establish standards in T&E of systems using artificial intelligence. Dr. Pinelis has spent over 10 years working predominantly in the area of defense and national security. She has largely focused on operational test and evaluation, both in support of the service operational testing commands and also at the OSD level. In her previous job as the Test Science Lead at the Institute of Defense Analyses, she managed an interdisciplinary team of scientists supporting the Director and the Chief Scientist of the Department of Operational Test and Evaluation on integration of statistical test design and analysis and data-driven assessments into test and evaluation practice. Before, that, in her assignment at the Marine Corps Operational Test and Evaluation Activity, Dr. Pinelis led the design and analysis of the widely publicized study on the effects of integrating women into combat roles in the Marine Corps. Based on this experience, she co-authored a book, titled “The Experiment of a Lifetime: Doing Science in the Wild for the United States Marine Corps.” In addition to T&E, Dr. Pinelis has several years of experience leading analyses for the DoD in the areas of wargaming, precision medicine, warfighter mental health, nuclear non-proliferation, and military recruiting and manpower planning. Her areas of statistical expertise include design and analysis of experiments, quasi-experiments, and observational studies, causal inference, and propensity score methods. Dr. Pinelis holds a BS in Statistics, Economics, and Mathematics, an MA in Statistics, and a PhD in Statistics, all from the University of Michigan, Ann Arbor. |
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: Panelist |
Rachel Haga Research Associate Institute for Defense Analyses 
(bio)
Rachel is a Research Associate at the Institute for Defense Analyses where she applies rigorous statistics and study design to evaluate, test, and report on various programs. She specializes in human system integration. |
Panel |
Recording | 2021 |
| Session Title | Speaker | Type | Materials | Year |
|---|---|---|---|---|
Short Course Overview of Design of Experiments |
Bradley Jones Distinguished Research Fellow JMP Division/SAS |
Short Course | Materials | 2018 |
Short Course Overview of Design of Experiments |
Doug Motgomery Regents’ Professor of Industrial Engineering and Statistics ASU Foundation Professor of Engineering Arizona State University |
Short Course | Materials | 2018 |
Short Course Introduction to R |
Justin Post North Carlina State Univeristy |
Short Course | Materials | 2018 |
Short Course Survey Construction and Analysis |
Wendy Matrinez Bureau of Labor and Statistics |
Short Course | Materials | 2018 |
Short Course Survey Construction and Analysis |
MoonJung Cho Bureau of Labor and Statistics |
Short Course | Materials | 2018 |
Short Course Uncertainty Quantification |
Ralph Smith North Carlina State Univeristy |
Short Course | Materials | 2018 |
Short Course Combinatorial Interaction Testing |
Erin Lanus Research Assistant Professor Virginia Tech
|
Short Course |
Materials
Recording | 2021 |
Short Course Introduction to Neural Networks for Deep Learning with Tensorflow |
Roshan Patel Data Scientist US Army CCDC Armaments Center
|
Short Course |
Materials
Recording | 2021 |
Roundtable The Role of the Statistics Profession in the DoD’s Current AI Initiative |
Laura Freeman Research Associate Professor of Statistics and Director of the Intelligent Systems Lab Virginia Tech
|
Roundtable | 2021 |
|
Roundtable Test Design and Analysis for Modeling & Simulation Validation |
Kelly Avery Research Staff Member Institute for Defense Analyses
|
Roundtable | 2021 |
|
Roundtable Organizing and Sharing Data within the T&E Community |
Matthew Avery Research Staff Memeber Institute for Defense Analyses
|
Roundtable | 2021 |
|
Roundtable Opportunities and Challenges for Openly Publishing Statistics Research for National Defense |
Lyndsay Shand R&D Statistician Sandia National Laboratories
|
Roundtable |
| 2021 |
Roundtable Identifying Challenges and Solutions to T&E of Non-IP Networks |
Peter Mancini Research Staff Member Institute for Defense Analyses
|
Roundtable | 2021 |
|
Roundtable Overcoming Challenges and Applying Sequential Procedures to T&E |
Rebecca Medlin Research Staff Member Institute for Defense Analyses
|
Roundtable | 2021 |
|
Panel Army’s Open Experimentation Test Range for Internet of Battlefield Things: MSA-DPG |
Jade Freeman Research Scientist U.S. Army DEVCOM Army Research Laboratory
|
Panel |
| 2021 |
Panel Multi-Agent Adaptive Coordinated Autonomy in Contested Battlefields |
John Rogers Senior Research Scientist U.S. Army DEVCOM Army Research Laboratory
|
Panel |
| 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
Sarah Burke STAT Expert STAT Center of Excellence
|
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
John Haman RSM Institute for Defense Analyses
|
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
Peter Parker Team Lead for Advanced Measurement Systems NASA Langley
|
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Panelist |
Willis Jensen HR Analyst W.L. Gore & Associates
|
Panel |
Recording | 2021 |
Panel The Keys to Successful Collaborations during Test and Evaluation: Moderator |
Christine Anderson-Cook Los Alamos National Lab
|
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: HSI | Trustworthy AI |
Stoney Trent Research Professor and Principal Advisor for Research and Innovation; Founder Virginia Tech; The Bulls Run Group, LLC
|
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: Evaluating Warfighters’ Ability to Employ AI Capabilities |
Dan Porter Research Staff Member Institute for Defense Analyses
|
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: Evaluating HSI with AI-Enabled Systems: What should you consider in a TEMP? |
Jane Pinelis Chief of the Test, Evaluation, and Assessment branch Department of Defense Joint Artificial Intelligence Center (JAIC)
|
Panel |
Recording | 2021 |
Panel Finding the Human in the Loop: Panelist |
Rachel Haga Research Associate Institute for Defense Analyses
|
Panel |
Recording | 2021 |
