| Session Title | Speaker | Type | Materials | Yr |
|---|---|---|---|---|
| Open Architecture Tradeoffs (OAT): A simple, computational game engine for rapidly exploring hypotheses in Battle Management Command and Control (BMC2)
Abstract
We created the Open Architectures Tradeoff (OAT) tool, a simple, computational game engine for rapidly exploring hypotheses about mission effectiveness in Battle Management Command and Control (BMC2). Each run of an OAT game simulates a military mission in contested airspace. Game objects represent U.S., adversary, and allied assets, each of which moves through the simulated airspace. Each U.S. asset has a Command and Control (C2) package the controls its actions—currently, neural networks form the basis of each U.S. asset’s C2 package. The weights of the neural network are randomized at the beginning of each game and are updated over the course of the game as the U.S. asset learns which of its actions lead to rewards, e.g., intercepting an adversary. Weights are updated via a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) altered to accommodate a Reinforcement Learning paradigm.
OAT allows a user to winnow down the trade space that should be considered when setting up more expensive and time-consuming campaign models. OAT could be used to weed out bad ideas for “fast failure”, thus avoiding waste of campaign modeling resources. Questions can be explored via OAT such as: Which combination of system capabilities is likely to be more or less effective in a particular military mission?
For example, in an early analysis, OAT was used to test the hypothesis that increases in U.S. assets’ sensor range always lead to increases in mission effectiveness, quantified as the percent of adversaries intercepted. We ran over 2500 OAT games, each time varying the sensor range of U.S. assets and the density of adversary assets. Results show that increasing sensor range did lead to an increase in military effectiveness—but only up to a certain point. Once the sensor range surpassed approximately 10-15% of the simulated airspace size, no further gains were made in the percent of adversaries intercepted. Thus, campaign modelers should hesitate to devote resources to exploring sensor range in isolation.
More recent OAT analyses are exploring more complex hypotheses regarding the trade space between sensor range and communications range.
|
Shelley Cazares |
Breakout |
| 2019 |
| Comparison of Methods for Testing Uniformity to Support the Validation of Simulation Models used for Live-Fire Testing
Abstract
Goodness-of-fit (GOF) testing is used in many applications, including statistical hypothesis testing to determine if a set of data come from a hypothesized distribution. In addition, combined probability tests are extensively used in meta-analysis to combine results from several independent tests to asses an overall null hypothesis. This paper summarizes a study conducted to determine which GOF and/or combined probability test(s) can be used to determine if a set of data with relative small sample size comes from the standard uniform distribution, U(0,1). The power against different alternative hypothesis of several GOF tests and combined probability methods were examined. The GOF methods included: Anderson-Darling, Chi-Square, Kolmogorov-Smirnov, Cramér-Von Mises, Neyman-Barton, Dudewicz-van der Meulen, Sherman, Quesenberry-Miller, Frosini, and Hegazy-Green; while thecombined probability test methods included: Fisher’s Combined Probability Test, Mean Z, Mean P, Maximum P, Minimum P, Logit P, and Sum Z. While no one method was determined to provide the best power in all situations, several useful methods to support model validation were identified.
|
Shannon Shelburne |
Breakout |
| 2019 |
| Valuing Human Systems Integration: A Test and Data Perspective
Abstract
Technology advances are accelerating at a rapid pace, with the potential to enable greater capability and power to the Warfighter. However, if human capabilities and limitations are not central to concepts, requirements, design, and development then new/upgraded weapons and systems will be difficult to train, operate, and maintain, may not result in the skills, job, grade, and manpower mix as projected, and may result in serious human error, injury or Soldier loss.
The Army Human Systems Integration (HSI) program seeks to overcome these challenges by ensuring appropriate consideration and integration of seven technical domains: Human Factors Engineering (e.g., usability), Manpower, Personnel, Training, Safety and Occupational Health, Habitability, Force Protection and Survivability. The tradeoffs, constraints, and limitations occurring among and between these technical domains allows HSI to execute a coordinated, systematic process for putting the warfighter at the center of the design process – equipping the warfighter rather than manning equipment. To that end, the Army HSI Headquarters, currently as a directorate within the Army Headquarters Deputy Chief of Staff (DCS), G-1 develops strategies and ensures human systems factors are early key drivers in concepts, strategy, and requirements, and are fully integrated throughout system design, development, testing and evaluation, and sustainment
The need to consider HSI factors early in the development cycle is critical. Too often, man-machine interface issues are not addressed until late in the development cycle (i.e. production and deployment phase) after the configuration of a particular weapon or system has been set. What results is a degraded combat capability, suboptimal system and system-of-systems integration, increased training and sustainment requirements, or fielded systems not in use.
Acquisition test data are also good sources to glean HSI return on investment (ROI) metrics. Defense acquisition reports such as test and evaluation operational assessments identifies HSI factors as root causes when Army programs experience increase cost, schedule overruns, or low performance. This is identifiable by the number and type of systems that require follow-on test and evaluation (FOT&E), over reliance on field service representatives (FSRs), costly and time consuming engineering change requests (ECRs), or failures in achieving reliability, availability, and maintainability (RAM) key performance parameters (KPPs) and key system attributes (KSAs).
In this presentation, we will present these data and submit several return on investment (ROI) metrics, closely aligned to the defense acquisition process, to emphasize and illustrate the value of HSI. Optimizing Warfighter-System performance and reducing human errors, minimizing risk of Soldier loss or injury, and reducing personnel and materiel life cycle costs produces data that are inextricably linked to early, iterative, and measurable HSI processes within the defense acquisition system.
|
Jeffrey Thomas |
Breakout |
| 2019 |
| Air Force Human Systems Integration Program
Abstract
The Air Force (AF) Human Systems Integration (HSI) program is led by the 711th Human Performance Wing’s Human Systems Integration Directorate (711 HPW/HP). 711 HPW HP provides direct support to system program offices and AF Major Commands (MAJCOMs) across the acquisition lifecycle from requirements development to fielding and sustainment in addition to providing home office support. With an ever-increasing demand signal for support, HSI practitioners within 711 HPW/HP assess HSI domain areas for human-centered risks and strive to ensure systems are designed and developed to safely, effectively, and affordably integrate with human capabilities and limitations. In addition to system program offices and MAJCOMs, 711 HPW/HP provides HSI support to AF Centers (e.g., AF Sustainment Center, AF Test Center), the AF Medical Service, and special cases as needed. The AF Global Strike Command (AFGSC) is the largest MAJCOM with several Programs of Record (POR), such as the B-1, B-2, and B-52 bombers, Intercontinental Ballistic Missiles (ICBM), Ground-Based Strategic Deterrent (GBSD), Airborne Launch Control System (ALCS), and other support programs/vehicles like the UH-1N. Mr. Anthony Thomas (711 HPW/HP), the AFGSC HSI representative, will discuss how 711 HPW/HP supports these programs at the MAJCOM headquarters level and in the system program offices.
|
Anthony Thomas |
Breakout |
| 2019 |
| Toward Real-Time Decision Making in Experimental Settings
Abstract
Materials scientists, computer scientists and statisticians at LANL have teamed up to investigate how to make near real time decisions during fast-paced experiments. For instance, a materials scientist at a beamline typically has a short window in which to perform a number of experiments, after which they analyze the experimental data, determine interesting new experiments and repeat. In typical circumstances, that cycle could take a year. The goal of this research and development project is to accelerate that cycle so that interesting leads are followed during the short window for experiments, rather than in years to come. We detail some of our UQ work in materials science, including emulation, sensitivity analysis, and solving inverse problems, with an eye toward real-time decision making in experimental settings.
|
Devin Francom |
Breakout | 2019 | |
| Area Validation for Applications with Mixed Uncertainty
Abstract
Model validation is a process for determining how accurate a model is
when compared to a true value. The methodology uses uncertainty analysis
in order to assess the discrepancy between a measured and predicted value.
In the literature, there have been several area metrics introduced to handle
these type of discrepancies. These area metrics were applied to problems that
include aleatory uncertainty, epistemic uncertainty, and mixed uncertainty.
However, these methodologies lack the ability to fully characterize the true
dierences between the experimental and prediction data when mixed un-
certainty exists in the measurements and/or in the predictions. This work
will introduce a new area metric validation approach which aims to com-
pensate for the shortcomings in current techniques. The approach will be
described in detail and comparisons between existing metrics will be shown.
To demonstrate its applicability the new area metric will be applied to a
stagnation point calibration probe’s surface predictions for a low-enthalpy
conditions. For this application, testing was preformed in the Hypersonic
Materials Environmental Test System (HYMETS) facility located at NASA
Langley Research Center.
|
Laura White |
Breakout |
| 2019 |
| A 2nd-Order Uncertainty Quantification Framework Applied to a Turbulence Model Validation Effort
Abstract
Computational fluid dynamics is now considered to be an indispensable tool for the design and development of scramjet engine components. Unfortunately, the quantification of uncertainties is rarely addressed with anything other than sensitivity studies, so the degree of confidence associated with the numerical results remains exclusively with the subject matter expert that generated them. This practice must be replaced with a formal uncertainty quantification process for computational fluid dynamics to play an expanded role in the system design, development, and flight certification process. Given the limitations of current hypersonic ground test facilities, this expanded role is believed to be a requirement by some in the hypersonics community if scramjet engines are to be given serious consideration as a viable propulsion system. The present effort describes a simple, relatively low cost, nonintrusive approach to uncertainty quantification that includes the basic ingredients required to handle both aleatoric (random) and epistemic (lack of knowledge) sources of uncertainty. The nonintrusive nature of the approach allows the computational fluid dynamicist to perform the uncertainty quantification with the flow solver treated as a “black box”. Moreover, a large fraction of the process can be automated, allowing the uncertainty assessment to be readily adapted into the engineering design and development workflow. In the present work, the approach is applied to a model scramjet isolator problem where the desire is to validate turbulence closure models in the presence of uncertainty. In this context, the relevant uncertainty sources are determined and accounted for to allow the analyst to delineate turbulence model-form errors from other sources of uncertainty associated with the simulation of the facility flow.
|
Robert Baurle |
Breakout |
| 2019 |
| Sources of Error and Bias in Experiments with Human Subjects
Abstract
No set of experimental data is perfect and researchers are aware that data from experimental studies invariably contain some margin of error. This is particularly true of studies with human subjects since human behavior is vulnerable to a range of intrinsic and extrinsic influences beyond the variables being manipulated in a controlled experimental setting. Potential sources of error may lead to wide variations in the interpretation of results and the formulation of subsequent implications. This talk will discuss specific sources of error and bias in the design of experiments and present systematic ways to overcome these effects. First, some of the basic errors in general experimental design will be discussed, including human errors, systematic errors and random errors. Second, we will explore specific types of experimental error that appear in human subjects research. Lastly, we will discuss the role of bias in experiments with human subjects. Bias is a type of systematic error that is introduced into the sampling or testing phase and encourages one outcome over another. Often, bias is the result of the intentional or unintentional influence that an experimenter may exert on the outcomes of a study. We will discuss some common sources of bias in research with human subjects, including biases in sampling, selection, response, performance execution, and measurement. The talk will conclude with a discussion of how errors and bias influence the validity of human subjects research and will explore some strategies for controlling these errors and biases
|
Poornima Madhavan |
Breakout | 2019 | |
| Deep Reinforcement Learning
Abstract
An overview of Deep Reinforcement Learning and it’s recent successes in creating high performing agents. Covering it’s application in “easy” environments up to massively complex multi-agent strategic environments. Will analyze the behaviors learned, discuss research challenges, and imagine future possibilities.
|
Benjamin Bell |
Breakout |
| 2019 |
| Decentralized Signal Processing and Distributed Control for Collaborative Autonomous Sensor Networks
Abstract
Collaborative autonomous sensor networks have recently been used in many applications including inspection, law enforcement, search and rescue, and national security. They offer scalable, low cost solutions which are robust to the loss of multiple sensors in hostile or dangerous environments. While often comprised of less capable sensors, the performance of a large network can approach the performance of far more capable and expensive platforms if nodes are effectively coordinating their sensing actions and data processing. This talk will summarize work to date at LLNL on distributed signal processing and decentralized optimization algorithms for collaborative autonomous sensor networks, focusing on ADMM-based solutions for detection/estimation problems and sequential greedy optimization solutions which maximize submodular functions, e.g. mutual information.
|
Ryan Goldhahn |
Breakout | 2019 | |
| An Overview of Uncertainty-Tolerant Decision Support Modeling for Cybersecurity
Abstract
Cyber system defenders face the challenging task of continually protecting critical assets and information from a variety of malicious attackers. Defenders typically function within resource constraints, while attackers operate at relatively low costs. As a result, design and development of resilient cyber systems that support mission goals under attack, while accounting for the dynamics between attackers and defenders, is an important research problem. This talk will highlight decision support modeling challenges under uncertainty within non-cooperative cybersecurity settings. Multiple attacker-defender game formulations under uncertainty are discussed with steps for further research.
|
Samrat Chatterjee |
Breakout | 2019 | |
| Software Reliability and Security Assessment: Automation and Frameworks
Abstract
Software reliability models enable several quantitative predictions such as the number of faults remaining, failure rate, and reliability (probability of failure free operation for a specified period of time in a specified environment). This talk will describe recent efforts in collaboration with NASA, including (1) the development of an automated script for the SFRAT (Software Failure and Reliability Assessment Tool) to streamline application of software reliability methods to ongoing programs, (2) application to a NASA program, (3) lessons learned, (4) and future directions for model and tool development to support the practical needs of the software reliability and security assessment frameworks.
|
Lance Fiondella |
Breakout |
| 2019 |
| AI & ML in Complex Environment
Abstract
The U.S. Army Research Laboratory’s (ARL) Essential Research Program (ERP) on Artificial Intelligence & Machine Learning (AI & ML) seeks to research, develop and employ a suite of AI-inspired and ML techniques and systems to assist teams of soldiers and autonomous agents in dynamic, uncertain, complex operational conditions. Systems will be robust, scalable, and capable of learning and acting with varying levels of autonomy, to become integral components of networked sensors, knowledge bases, autonomous agents, and human teams. Three specific research gaps will be examined: (i) Learning in Complex Data Environments, (ii) Resource-constrained AI Processing at the Point-of-Need and (iii) Generalizable & Predictable AI. The talk will highlight ARL’s internal research efforts over the next 3-5 years that are connected, cumulative and converging to produce tactically-sensible AI-enabled capabilities for decision making at the tactical edge, specifically addressing topics in: (1) adversarial distributed machine learning, (2) robust inference & machine learning over heterogeneous sources, (3) adversarial reasoning integrating learned information, (4) adaptive online learning and (5) resource-constrained adaptive computing. The talk will also highlight collaborative research opportunities in AI & ML via ARL’s Army AI Innovation Institute (A2I2) which will harness the distributed research enterprise via the ARL Open Campus & Regional Campus initiatives.
|
Tien Pham |
Breakout | 2019 | |
| Waste Not, Want Not: A Methodological Illustration of Quantitative Text Analysis
Abstract
“The wise use of one’s resources will keep one from poverty.” This is the definition of the proverbial saying “waste not, want not” according to www.dictionary.com. Indeed, one of the most common resources analysts encounter is text in free-form. This text might come from survey comments, feedback, websites, transcriptions of interviews, videos, etcetera. Notably, researchers have used wisely the information conveyed in text for many years. However, in many instances, the qualitative methods employed require numerous hours of reading, training, coding, and validating, among others.
As technology continues to evolve, simple access to text data is blooming. For example, analysts conducting online studies can have thousands of text entries from participants’ comments. Even without recent advances in technology analysts have had access to text in books, letters, and other archival data for centuries. One important challenge, however, is figuring out how to make sense of text data without investing a large number of resources, time, and the effort involved in qualitative methodology or “old-school” quantitative approaches (such as reading a collection of 200 books and counting the occurrence of important terms in the text). This challenge has been solved in the information retrieval field –a branch of computer science—with the implementation of a technique called latent semantic analysis (LSA; Manning, Raghavan, & Schütze, 2008) and a closely related technique called topic analysis (TA; SAS Institute Inc., 2018). Undoubtedly, other quantitative methods for text analysis, such as latent Dirichlet analysis (Blei, Ng, & Jordan, 2003), are also apt for the task of unveiling knowledge from text data, but we restrict the discussion in this presentation to LSA and TA because these exclusively deal with the underlying structure of the text rather than identifying clusters.
In this presentation, we aim to make quantitative text analysis –specifically LSA and TA– accessible to researchers and analysts from a variety of disciplines. We do this by leveraging understanding of a popular multivariate technique: principal components analysis (PCA). We start by describing LSA and TA by drawing comparisons and equivalencies to PCA. We make these comparisons in an intuitive, user-friendly manner and then through a technical description of mathematical statements, which rely on the singular value decomposition of a document-term matrix. Moreover, we explain the implementation of LSA and TA using statistical software to enable simple application of these techniques. Finally, we show a practical application of LSA and TA with empirical data of aircraft incidents.
|
Laura Castro-Schilo |
Breakout |
| 2019 |
| Machine Learning Prediction With Streamed Sensor Data: Fitting Neural Networks using Functional Principal Components
Abstract
Sensors that record sequences of measurements are now embedded in many products from wearable exercise watches to chemical and semiconductor manufacturing equipment. There is information in the shapes of the sensor stream curves that is highly predictive of a variety of outcomes such as the likelihood of a product failure event or batch yield. Despite this data now being common and readily available, it is often being used either inefficiently or not at all due to lack of knowledge and tools for how to properly leverage it. In this presentation, we will propose fitting splines to sensor streams and extracting features called functional principal component scores that offer a highly efficient low dimensional compression of the signal data. Then, we use these features as inputs into machine learning models like neural networks and LASSO regression models. Once one sees sensor data in this light, answering a wide variety of applied questions becomes a straightforward two stage process of data cleanup/functional feature extraction followed by modeling using those features as inputs.
|
Chris Gotwalt |
Breakout |
| 2019 |
| Screening Designs for Resource Constrained Deterministic M&S Experiments: A Munitions Case Study
Abstract
Abstract: In applications where modeling and simulation runs are quick and cheap, space filling designs will give the tester all the information they need to make decisions about their system. In some applications however, this luxury does not exist, and each M&S run can be time consuming and expensive. In these scenarios, a sequential test approach provides an efficient solution where an initial screening is conducted, followed by an augmentation to fit specified models of interest. Until this point, no dedicated screening designs for UQ applications in resource constrained situations existed. Due to the Army’s frequent exposure to this type of situation, the need sparked a collaboration between Picatinny’s Statistical Methods and Analysis group and Professor V. Roshan Joseph of Georgie Tech, where a new type of UQ screening design was created. This paper provides a brief introduction to the design, its intended use, and a case study in which this new methodology was applied.
|
Christopher Drake |
Breakout |
| 2019 |
| The Isle of Misfit Designs: A Guided Tour of Optimal Designs That Break the Mold
Abstract
Whether it was in a Design of Experiments course or through your own work, you’ve no doubt seen and become well acquainted with the standard experimental design. You know the features: they’re “orthogonal” (no messy correlations to deal with), their correlation matrices are nice pretty diagonals, and they can only happen with run sizes of 4, 8, 12, 16, and so on. Well what if I told you that there existed optimal designs that defied convention. What if I told you that, yes, you can run an optimal design with, say, 5 factors in 9 runs. Or 10. Or even 11 runs! Join me as I show you a strange new world of optimal designs that are the best at what they do, even though they might not look very nice.
|
Caleb King |
Breakout |
| 2019 |
| Adapting Operational Test to Rapid-Acquisition Programs
Abstract
During the past several years, the DoD has begun applying rapid prototyping and fielding authorities—granted by Congress in the FY2016-FY2018 National Defense Authorization Acts (NDAA)—to many acquisition programs. Other programs have implemented an agile acquisition strategy where incremental capability is delivered in iterative cycles. As a result, Operational Test Agencies (OTA) have had to adjust their test processes to accommodate shorter test timelines and periodic delivery of capability to the warfighter. In this session, representatives from the Service OTAs will brief examples where they have implemented new practices and processes for conducting Operational Test on acquisition programs categorized as agile, DevOps, and/or Section 804 rapid-acquisition efforts. During the final 30 minutes of the session, a panel of OTA representatives will field questions from the audience concerning the challenges and opportunities related to test design, data collection, and analysis, that rapid-acquisition programs present.
|
Panel Discussion |
Breakout |
| 2019 |
| Sample Size Calculations for Quiet Sonic Boom Community Surveys
Abstract
NASA is investigating the dose-response relationship between quiet sonic boom exposure and community noise perceptions. This relationship is the key to possible future regulations that would replace the ban on commercial supersonic flights with a noise limit. We have built several Bayesian statistical models using pilot community study data. Using goodness of fit measures, we downselected to a subset of models which are the most appropriate for the data. From this subset of models we demonstrate how to calculate sample size requirements for a simplified example without any missing data. We also suggest how to modify the sample size calculation to account for missing data.
|
Jasme Lee |
Breakout |
| 2019 |
| Improved Surface Gunnery Analysis with Continuous Data
Abstract
Swarms of small, fast speedboats can challenge even the most capable modern warships, especially when they operate in or near crowded shipping lanes. As part of the Navy’s operational testing of new ships and systems, at-sea live-fire tests against remote-controlled targets allow us to test our capability against these threats. To ensure operational realism, these events are minimally scripted and allow the crew to respond in accordance with their training. This is a trade-off against designed experiments, which ensure statistically optimal sampling of data from across the factor space, but introduce many artificialities.
A recent test provided data on the effectiveness of naval gunnery. However, standard binomial (hit/miss) analyses fell short, as the number of misses was much larger than the number of hits. This prevented us from fitting more than a few factors and resulted in error bars so large as to be almost useless. In short, binomial analysis taught us nothing we did not already know. Recasting gunfire data from binomial (hit/miss) to continuous (time-to-kill) allowed us to draw statistical conclusions with tactical implications from these free-play, live-fire surface gunnery events. Using a censored-data analysis approach enabled us to make this switch and avoid the shortcomings of other statistical methods. Ultimately, our analysis provided the Navy with suggestions for improvements to its tactics and the employment of its weapons.
|
Benjamin Ashwell & V. Bram Lillard |
Breakout |
| 2019 |
| Human in the Loop Experiment Series Evaluating Synthetic Vision Displays for Enhanced Airplane State Awareness
Abstract
Recent data from Boeing’s Statistical Summary of Commercial Jet Airplane Accidents shows that Loss of Control – In Flight (LOC-I) is the leading cause of fatalities in commercial aviation accidents worldwide. The Commercial Aviation Safety Team (CAST), a joint government and industry effort tasked with reducing the rate of fatal accidents, requested that the National Aeronautics and Space Administration (NASA) conduct research on virtual day-visual meteorological conditions displays, such as synthetic vision, in order to combat LOC-I. NASA recently concluded a series of experiments using commercial pilots from various backgrounds to evaluate synthetic vision displays. This presentation will focus on the two most recent experiments: one conducted with the Navy’s Disorientation Research Device and one completed at NASA Langley Research Center that utilized the Microsoft HoloLens to display synthetic vision. Statistical analysis was done on aircraft performance data, pilot inputs, and a range of subjective questionnaires to assess the efficacy of the displays.
|
Kathryn Ballard |
Breakout |
| 2019 |
| Statistical Engineering and M&S in the Design and Development of DoD Systems
Abstract
This presentation will use a notional armament system case-study to illustrate the use of M&S DOE, surrogate modeling, sensitivity analysis, multi-objective optimization and model calibration during early lifecycle development and design activities in the context of a new armament system. In addition to focusing on the statistician’s, data scientist’s, or analyst’s role and the key statistical techniques in engineering DoD systems, this presentation will also emphasize the non-statistical / engineering domain-specific aspects in a multidisciplinary design and development process which make uses of these statistical approaches at the subcomponent and subsystem-level as well as the end-to-end system modeling. A statistical engineering methodology which emphasizes the use of ‘virtual’ DOE-based model emulators developed at the subsystem-level and integrated using a systems-engineering architecture framework can yield a more tractable engineering problem compared to traditional ‘design-build-test-fix’ cycles or direct simulation of computationally expensive models. This supports a more informed prototype design for physical experimentation while providing a greater variety of materiel solutions thereby reducing development and testing cycles and time to field complex systems.
|
Doug Ray & Melissa Jablonski |
Breakout | 2019 | |
| Uncertainty Quantification: Combining Large Scale Computational Models with Physical Data for Inference
Abstract
Combining physical measurements with computational models is key to many investigations involving validation and uncertainty quantification (UQ). This talk surveys some of the many approaches taken for validation and UQ, with large-scale computational models. Experience with such applications suggests classifications of different types of problems with common features (e.g. data size, amount of empiricism in the model, computational demands, availability of data, extent of extrapolation required, etc.). More recently, social and social-technical systems are being considered for similar analyses, bringing new challenges to this area. This talk will approaches for such problems and will highlight what might be new research directions for application and methodological development in UQ.
|
Dave Higdon |
Breakout |
| 2019 |
| A User-Centered Design Approach to Military Software Development
Abstract
This case study highlights activities performed during the front-end process of a software development effort undertaken by the Fire Support Command and Control Program Office. This program office provides the U.S. Army, Joint and coalition commanders with the capability to plan, execute and deliver both lethal and non-lethal fires. Recently, the program office has undertaken modernization of its primary field artillery command and control system that has been in use for over 30 years. The focus of this case study is on the user-centered design process and activities taken prior to and immediately following contract award.
A modified waterfall model comprised of three cyclic, yet overlapping phases (observation, visualization, and evaluation) provided structure for the iterative, user-centered design process. Gathering and analyzing data collected during focus groups, observational studies, and workflow process mapping, enabled the design team to identify 1) design patterns across the role/duty, unit and echelon matrix (a hierarchical organization structure), 2) opportunities to automate manual processes, 3) opportunities to increase efficiencies for fire mission processing, 4) bottlenecks and workarounds to be eliminated through design of the modernized system, 5) shortcuts that can be leveraged in design, 6) relevant and irrelevant content for each user population for streamlining access to functionality, 7) a usability baseline for later comparison (e.g., the number of steps and time taken to perform a task as captured in workflows for comparison to the same task in the modernized system), and provided the basis for creating visualizations using wireframes. Heuristic evaluations were conducted early to obtain initial feedback from users. In the next few months, usability studies will enable users to provide feedback based on actual interaction with the newly designed software.
Included in this case study are descriptions of the methods used to collect user-centered design data, how results were visualized/documented for use by the development team, and lessons learned from applying user-centered design techniques during software development of a military field artillery command and control system.
|
Pam Savage-Knepshield |
Breakout |
| 2019 |
| Engineering first, Statistics second: Deploying Statistical Test Optimization (STO) for Cyber
Abstract
Due to the immense potential use cases, configurations, and threat behaviors, thorough and efficient cyber testing is a significant challenge for the defense community. In this presentation, Phadke will present case studies where STO was successfully deployed for cyber testing, resulting in higher assurance, reduced schedule, and reduced testing cost. Phadke will also discuss importance first focusing on the engineering and science analysis, and only after that is complete, implementing statistical methods.
|
Kedar Phadke |
Breakout | 2019 | |
| Validation and Uncertainty Quantification of Complex Models
Abstract
Advances in high performance computing have enabled detailed simulations of real-world physical processes, and these simulations produce large datasets. Even as detailed as they are, these
simulations are only approximations of imperfect mathematical models, and furthermore, their outputs depend on inputs that are themselves uncertain. The main goal of a validation and uncertainty quantication methodology is to determine the uncertainty, that is, the relationship between the true value of a quantity of interest and its prediction by the simulation. The value of the computational results is limited unless the uncertainty can be quantied or bounded. Bayesian
calibration is a common method for estimating model parameters and quantifying their associated uncertainties; however, calibration becomes more complicated when the data arise from dierent
types of experiments. On an example from material science we will employ two types of data and demonstrate how one can obtain a set of material strength models that agree with both data sources.
|
Kassie Fronczyk |
Breakout |
| 2019 |
| Communicating Statistical Concepts and Results: Lessons Learned from the US Service Academies
Abstract
Communication is critical both for analysts and decision-makers who rely on analysis to inform their choices. The Service Academies are responsible for educating men and women who may serve in both roles over the course of their careers. Analysts must be able to summarize their results concisely and communicate them to the decision-maker in a way that is relevant and actionable. Decision-makers understand that analytical results may carry with them uncertainty and be able to incorporate this uncertainty properly when evaluating different options. This panel explores the role of the US Service Academies in preparing their students for both roles. Featuring representatives from the US Air Force Academy, the US Naval Academy, and the US Military Academy, this panel will cover how future US Officers are taught to use and communicate with data. Topics include developing and motivating numerical literacy, understandings of uncertainty, how analysts should frame uncertainty to decision-makers, and how decision-makers should understand information presented with uncertainty. Panelists will discuss what they think the academies do well and areas that are ripe for improvement.
|
Panel Discussion |
Breakout | 2019 | |
| Multivariate Density Estimation and Data-enclosing Sets Using Sliced-Normal Distributions
Abstract
This talk focuses on a means to characterize the variability in multivariate data. This characterization, given in terms of both probability distributions and closed sets, is instrumental in assessing and improving the robustness/reliability properties of system designs. To this end, we propose the Sliced-Normal (SN) class of distributions. The versatility of SNs enables modeling complex multivariate dependencies with minimal modeling effort. A polynomial mapping is defined which injects the physical space into a higher dimensional (so-called) feature space on which a suitable normal distribution is defined. Optimization-based strategies for the estimation of SNs from data in both physical and feature space are proposed. The formulations in physical space yield non-convex optimization programs whose solutions often outperform the solutions in feature space. However, the formulations in feature space yield either an analytical solution or a convex program thereby facilitating their application to problems in high dimensions. The superlevel sets of a SN density have a closed semi-algebraic form making them amenable to rigorous uncertainty quantification methods. Furthermore, we propose a chance-constrained optimization framework for identifying and eliminating the effects of outliers in the prescription of such regions. These strategies can be used to mitigate the conservatism intrinsic to many methods in system identification, fault detection, robustness/reliability analysis, and robust design caused by assuming parameter independence and by including outliers in the dataset.
|
Luis Crespo |
Breakout |
| 2019 |
| A Statistical Approach for Uncertainty Quantification with Missing Data
Abstract
Uncertainty quantification (UQ) has emerged as the science of quantitative characterization and reduction of uncertainties in simulation and testing. Stretching across applied mathematics, statistics, and engineering, UQ is a multidisciplinary field with broad applications. A popular UQ method to analyze the effects of input variability and uncertainty on the system responses is generalized Polynomial Chaos Expansion (gPCE). This method was developed using applied mathematics and does not require knowledge of a simulation’s physics. Thus, gPCE may be used across disparate industries and is applicable to both individual component and system level simulations.
The gPCE method can encounter problems when any of the input configurations fail to produce valid simulation results. gPCE requires that results be collected on a sparse grid Design of Experiment (DOE), which is generated based on probability distributions of the input variables. A failure to run the simulation at any one input configuration can result in a large decrease in the accuracy of a gPCE. In practice, simulation data sets with missing values are common because simulations regularly yield invalid results due to physical restrictions or numerical instability.
We propose a statistical approach to mitigating the cost of missing values. This approach yields accurate UQ results if simulation failure makes gPCE methods unreliable. The proposed approach addresses this missing data problem by introducing an iterative machine learning algorithm. This methodology allows gPCE modelling to handle missing values in the sparse grid DOE. The study will demonstrate the convergence characteristics of the methodology to reach steady state values for the missing points using a series of simulations and numerical results. Remarks about the convergence rate and the advantages and feasibility of the proposed methodology will be provided.
Several examples are used to demonstrate the proposed framework and its utility including a secondary air system example from the jet engine industry and several non-linear test functions. This is based on joint work with Dr. Mark Andrews at SmartUQ.
|
Mark Andrews |
Breakout | 2019 | |
| Exploring Problems in Shipboard Air Defense with Modeling
Abstract
One of the primary roles of navy surface combatants is defending high-value units against attack by anti-ship cruise missiles (ASCMs). They accomplish this either by launching their interceptor missiles and shooting the ASCMs down with rapid-firing guns (hard kill), or through the use of deceptive jamming, decoys, or other non-kinetic means (soft kill) to defeat the threat. The wide range of hostile ASCM capabilities and the different properties of friendly defenses, combined with the short time-scale for defeating these ASCMs, makes this a difficult problem to study.
IDA recently completed a study focusing on the extent to which friendly forces were vulnerable to massed ASCM attacks, and possible avenues for improvement. To do this we created a pair of complementary models with the combined flexibility to explore a wide range of questions. The first model employed a set of closed-form equations, and the second a time-dependent Monte Carlo simulation. This presentation discusses the thought processes behind the models and their relative strengths and weaknesses.
|
Ralph Donnelly & Benjamin Ashwell |
Breakout |
| 2019 |
| Sequential Testing for Fast Jet Life Support Systems
Abstract
The concept of sequential testing has many disparate meanings. Often, for statisticians it takes on a purely mathematical context while possibly meaning multiple disconnected test events for some practitioners. Here we present a pedagogical approach to creating test designs involving constrained factors using JMP software. Recent experiences testing one of the U.S. military’s fast jet life support systems (LSS) serves as a case study and back drop to support the presentation. The case study discusses several lessons learned during LSS testing, applicable to all practitioners of scientific test and analysis techniques (STAT) and design of experiments (DOE). We conduct a short analysis to specifically determine a test region with a set of factors pertinent to modeling human breathing and the use of breathing machines as part of the laboratory setup. A comparison of several government and industry laboratory test points and regions with governing documentation is made, along with the our proposal for determining a necessary and sufficient test region for tests involving human breathing as a factor.
|
Darryl Ahner Bio
Steven Thorsen, Sarah Burke &
|
Breakout |
| 2019 |
| Bayesian Component Reliability Estimation: F-35 Case Study
Abstract
A challenging aspect of a system reliability assessment is integrating
multiple sources of information, including component, subsystem, and
full-system data, previous test data, or subject matter expert opinion. A
powerful feature of Bayesian analyses is the ability to combine these
multiple sources of data and variability in an informed way to perform
statistical inference. This feature is particularly valuable in assessing
system reliability where testing is limited and only a small number (or no
failures at all) are observed.
The F-35 is DoD’s largest program; approximately one-third of the operations
and sustainment cost is attributed to the cost of spare parts and the
removal, replacement, and repair of components. The failure rate of those
components is the driving parameter for a significant portion of the
sustainment cost, and yet for many of these components, poor estimates of
the failure rate exist. For many programs, the contractor produces estimates
of component failure rates, based on engineering analysis and legacy systems
with similar parts. While these are useful, the actual removal rates can
provide a more accurate estimate of the removal and replacement rates the
program anticipates to experience in future years. In this presentation, we
show how we applied a Bayesian analysis to combine the engineering
reliability estimates with the actual failure data to overcome the problems
of cases where few data exist. Our technique is broadly applicable to any
program where multiple sources of reliability information need be combined
for the best estimation of component failure rates and ultimately
sustainment costs.
|
V. Bram Lillard & Rebecca Medlin |
Breakout |
| 2019 |
| Applying Functional Data Analysis throughout Aerospace Testing
Abstract
Sensors abound in aerospace testing and while many scientists look at the data from a physics perspective, the comparative statistics information is what drives decisions. A multi-company project was comparing launch data from the 1980’s to a current set of data that included 30 sensors. Each sensor was designed to gather 3000 data points during the 3-second launch event. The data included temperature, acceleration, and pressure information. This talk will compare the data analysis methods developed for this project as well as the use of the new Functional Data Analysis tool within JMP for its ability to discern in-family launch performances.
|
David Harrison |
Breakout |
| 2019 |
| Adopting Optimized Software Test Design Methods at Scale
Abstract
Using Combinatorial Test Design methods to select software test scenarios has repeatedly delivered large efficiency and thoroughness gains – which begs the questions:
• Why are these proven methods not used everywhere?
• Why do some efforts to promote adoption of new approaches stagnate?
• What steps can leaders take to introduce successfully introduce and spread new test design methods?
For more than a decade, Justin Hunter has helped large global organizations across six continents adopt new test design techniques at scale. Working in some environments, he has felt like Sisyphus, forever condemned to roll a boulder uphill only to watch it roll back down again. In other situations, things clicked; teams smoothly adopted new tools and techniques, and impressive results were quickly achieved. In this presentation, Justin will discuss several common challenges faced by large organizations, explain why adopting test design tools is more challenging than adopting other types of development and testing tools, and share actionable recommendations to consider when you roll out new test design approaches.
|
Justin Hunter |
Breakout |
| 2019 |
| A Quantitative Assessment of the Science Robustness of the Europa Clipper Mission
Abstract
Existing characterization of Europa’s environment is enabled by the Europa Clipper mission’s successful predecessors: Pioneer, Voyager, Galileo, and most recently, Juno. These missions reveal high intensity energetic particle fluxes at Europa’s orbit, requiring a multidimensional design challenge to ensure mission success (i.e. meeting Level 1 science requirements).
Risk averse JPL Design Principles and the Europa Environment Requirement Document (ERD) dictate practices and policy, which if masterfully followed, are designed to protect Clipper from failure or degradation due to radiation. However, even if workmanship is flawless and no waivers are assessed, modeling errors, shielding uncertainty, and natural variation in the Europa environment are cause for residual concern. While failure and part degradation are of paramount concern, the occurrence of temporary outages, causing loss or degradation of science observations, is also a critical mission risk, left largely unmanaged by documents like the ERD.
The referenced risk is monitored and assessed through a Project Systems Engineering-led mission robustness effort, which attempts balance the risk of science data loss with potential design cost and increased mission complexity required to mitigate such risk. The Science Sensitivity Model (SSM) was developed to assess mission and science robustness, with its primary goal being to ensure a high probability of achieving Level 1 (L1) science objectives by informing the design of a robust spacecraft, instruments, and mission design.
This discussion will provide an overview of the problem, the model, and solution strategies. Subsequent presentations discuss the experimental design used to understand the problem space and the graphics and visualization used to reveal important conclusions.
|
Kelli McCoy |
Breakout |
| 2019 |
| Identifying and Contextualizing Maximum Instrument Fault Rates and Minimum Instrument Recovery Times for Europa Clipper Science through Applied Statistics and Strategic Visualizations
Abstract
Using the right visualizations as part of broad system and statistical Monte Carlo analysis supports interpretation of key drivers and relationships between variables, provides context about the full system, and communicates to non-statistician stakeholders.
An experimental design was used to understand the relationships between instrument and spacecraft fault rate and recovery time in relation to the probability of achieving Europa Clipper science objectives during the Europa Clipper tour. Given spacecraft and instrument outages, requirement achievement checks were performed to determine the probability of meeting scientific objectives. Visualizations of the experimental design output enabled analysis of the full parameter set. Correlation between individual instruments and specific scientific objectives is not straight forward; some scientific objectives require a single instrument to be on at certain times and during varying conditions across the trajectory, while other science objectives require multiple instruments to function concurrently.
By examining the input conditions that meet scientific objectives with the highest probability, and comparing those to trials with the lowest probability of meeting scientific objectives, key relationships could be visualized, enabling valuable mission and engineering design insights. Key system drivers of scientific success were identified, such as fault rate tolerance and recovery time required for each instrument and the spacecraft.
Key steps, methodologies, difficulties and result-highlights are presented, along with a discussion of next steps and options for refinement and future analysis.
|
Thomas Youmans |
Breakout |
| 2019 |
| Design and Analysis of Experiments for Europa Clipper’s Science Sensitivity Model
Abstract
The Europa Clipper Science Sensitivity Model (SSM) can be thought of as a graph in which the nodes are mission requirements at ten levels in a hierarchy, and edges represent how requirements at one level of the hierarchy depend on those at lower levels. At the top of the hierarchy, there are ten nodes representing ten, Level 1 science requirements for the mission. At the bottom of the hierarchy, there are 100 or so nodes representing instrument-specific science requirements. In between, nodes represent intermediate science requirements with complex interdependencies. Meeting, or failing to meet, bottom-level requirements depends on the frequency of faults and the lengths of recovery times on the nine Europa Clipper instruments and the spacecraft.
Our task was to design and analyze the results of a Monte Carlo experiment to estimate the probabilities of meeting the Level 1 science requirements based on parameters of the distributions of time between failures and of recovery times. We simulated an ensemble of synthetic missions in which failures and recoveries were random realizations from those distributions. The pass-fail status of the bottom-level instrument-specific requirements were propagated up the graph for each of the synthetic missions. Aggregating over the collection of synthetic missions produced estimates of the pass-fail probabilities for the Level 1 requirements. We constructed a definitive screening design and supplemented it with additional space-filing runs, using JMP 14 software. Finally, we used the vectors of failure and recovery parameters as predictors, and the pass-fail probabilities of the high-level requirements as responses, and built statistical models to predict the latter from the former. In this talk, we will describe the design considerations and review the fitted models and their implications for mission success.
|
Amy Braverman |
Breakout |
| 2019 |
| SLS Structural Dynamics Sensor Optimization Study
Abstract
A crucial step in the design and development of a fight vehicle, such as NASA’s Space Launch System (SLS), is understanding its vibration behavior while in fight. Vehicle designers rely on low-cost finite element analysis (FEA) to predict the vibration behavior of the vehicle. During ground and flight tests, sensors are strategically placed at predefined locations that contribute the most vibration information under the assumption that FEA is accurate, producing points to validate the FEA models. This collaborative work focused on developing optimal sensor placement algorithms to validate FEA models against test data, and to characterize the vehicles vibration characteristics.
|
Ken Toro & Jon Stallrich |
Breakout | 2019 | |
| The 80/20 rule, can and should we break it using efficient data management tools?
Abstract
Abstract: Data scientists spend approximately 80% of their time preparing, cleaning, and feature engineering data sets. In this talk I will share use cases that show why this is important and why we need to do it. I will also describe the Earth System Grid Federation (ESGF) which is an open source effort providing a robust, distributed data and computation platform, enabling world wide access to Peta/Exa-scale scientific data. ESGF will help reduce the amount of effort needed for climate data preprocessing by integrating the necessary analysis and data sharing tools.
|
Ghaleb Abdulla |
Breakout |
| 2019 |
| Time Machine Learning: Getting Navy Maintenance Duration Right
Abstract
In support of the Navy’s effort to obtain improved outcomes through data-driven decision making, The Center for Naval Analyses’ Data Science Program (CNA/DSP) supports the performance-to plan(P2P) forum, which is co chaired by the Vice Chief of Naval Operations and the Assistant Secretary of the Navy (RD&A). The P2P forum provides senior Navy leadership forward looking performance forecasts, which are foundational to articulating Navy progress toward readiness and capability goals.
While providing analytical support for this forum, CNA/DSP leveraged machine learning techniques, including Random Forests and Artificial Neural Networks, to develop improved estimates of future maintenance durations for the Navy. When maintenance durations exceed their estimated timelines, these delays can affect training, manning, and deployments in support of operational commanders. Currently, the Navy creates maintenance estimates during numerous timeframes including the program objective memorandum (POM) process, the Presidential Budget (PB), and at contract award leading to evolving estimates over time. The limited historical accuracy for these estimates, especially with the POM and PB estimates, have persisted over the last decade. These errors have led to a gap between planned funding and actual costs in addition to changes in the assets available for operational commanders each year.
The CNA/DSP prediction model reduces the average error in forecasted maintenance duration days from 128 days to 31 days for POM estimates. Improvements in duration accuracy for the PB and contract award time frames were also achieved using similar ML processes. The data curation for these models involved numerous data sources of varying quality and required significant feature engineering to provide usable model inputs that could allow for forecasts over the Future Years Defense Program (FYDP) in order to support improved resource allocation and scheduling in support of the optimized fleet response training plan (OFRTP).
|
Tim Kao |
Breakout |
| 2019 |
| Behavioral Analytics: Paradigms and Performance Tools of Engagement in System Cybersecurity
Abstract
The application opportunities for behavioral analytics in the cybersecurity space are based upon simple realities.
1. The great majority of breaches across all cybersecurity venues is due to human choices and human error.
2. With communication and information technologies making for rapid availability of data, as well as behavioral strategies of bad actors getting cleverer, there is need for expanded perspectives in cybersecurity prevention.
3. Internally-focused paradigms must now be explored that place endogenous protection from security threats as an important focus and integral dimension of cybersecurity prevention.
The development of cybersecurity monitoring metrics and tools as well as the creation of intrusion prevention standards and policies should always include an understanding of the underlying drivers of human behavior. As temptation follows available paths, cyber-attacks follow technology, business models, and behavioral habits.
The human element will always be the most significant part in the anatomy of any final decision. Choice options – from input, to judgement, to prediction, to action – need to be better understood for their relevance to cybersecurity work. Behavioral Performance Indexes harness data about aggregate human participation in an active system, helping to capture some of the detail and nuances of this critically important dimension of cybersecurity.
|
Robert Gough |
Breakout |
| 2019 |
| 3D Mapping, Plotting, and Printing in R with Rayshader
Abstract
Is there ever a place for the third dimension in visualizing data? Is the use of 3D inherently bad, or can a 3D visualization be used as an effective tool to communicate results? In this talk, I will show you how you can create beautiful 2D and 3D maps and visualizations in R using the rayshader package. Additionally, I will talk about the value of 3D plotting and how good aesthetic choices can more clearly communicate results to stakeholders.
Rayshader is a free and open source package for transforming geospatial data into engaging visualizations using a simple, scriptable workflow. It provides utilities to interactively map, plot, and 3D print data from within R. It was nominated by Hadley Wickham to be one of 2018’s Data Visualizations of the Year for the online magazine Quartz.
|
Tyler Morgan-Wall |
Breakout | 2019 | |
| Functional Data Analysis for Design of Experiments
Abstract
With nearly continuous recording of sensor values now common, a new type of data called “functional data” has emerged. Rather than the individual readings being modeled, the shape of the stream of data over time is being modeled. As an example, one might model many historical vibration-over-time streams of a machine at start-up to identify functional data shapes associated with the onset of system failure. Functional Principal Components (FPC) analysis is a new and increasingly popular method for reducing the dimensionality of functional data so that only a few FPCs are needed to closely approximate any of a set of unique data streams. When combined with Design of Experiments (DoE) methods the response to be modeled in as fewest tests as possible is now the shape of a stream of data over time. Example analyses will be shown where the form of the curve is modeled as the function of several input variables allowing one to determine the input settings associated with shapes indicative of good or poor system performance. This allows the analyst to predict the shape of the curve as a function of the input variables.
|
Tom Donnelly |
Breakout |
| 2019 |
| Test and Evaluation of Emerging Technologies | Dr. Greg Zacharias, Chief Scientist Operational Test and Evaluation |
Breakout |
| 2019 |
| Challenges in Test and Evaluation of AI: DoD’s Project Maven
Abstract
The Algorithmic Warfare Cross Functional Team (AWCFT or Project Maven) organizes DoD stakeholders to enhance intelligence support to the warfighter through the use of automation and artificial intelligence. The AWCFT’s objective is to turn the enormous volume of data available to DoD into actionable intelligence and insights at speed. This requires consolidating and adapting existing algorithm-based technologies as well as overseeing the development of new solutions. This brief will describe some of the methodological challenges in test and evaluation that the Maven team is working through to facilitate speedy and agile acquisition of reliable and effective AI / ML capabilities.
|
Jane Pinelis |
Breakout | 2019 | |
| Demystifying the Black Box: A Test Strategy for Autonomy
Abstract
Systems with autonomy are beginning to permeate civilian, industrial, and military sectors. Though these technologies have the potential to revolutionize our world, they also bring a host of new challenges in evaluating whether these tools are safe, effective, and reliable. The Institute for Defense Analyses is developing methodologies to enable testing systems that can, to some extent, think for themselves. In this talk, we share how we think about this problem and how this framing can help you develop a test strategy for your own domain.
|
Dan Porter |
Breakout |
| 2019 |
| Satellite Affordability in LEO (SAL)
Abstract
The Satellite Affordability in LEO (SAL) model identifies the cheapest constellation capable of providing a desired level of performance within certain constraints. SAL achieves this using a combination of analytical models, statistical emulators, and geometric relationships. SAL is flexible and modular, allowing users to customize certain components while retaining default behavior in other cases. This is desirable if users wish to consider an alternative cost formulation or different types of payload. Uses for SAL include examining cost tradeoffs with respect to factors like constellation size and desired performance level, evaluating the sensitivity of constellation costs to different assumptions about cost behavior, and providing a first-pass look at what proliferated smallsats might be capable of. At this point, SAL is limited to Walker constellations with sun-synchronous, polar orbits.
|
Matthew Avery |
Breakout |
| 2019 |
| Statistical Process Control and Capability Study on the Water Content Measurements in NASA Glenn’s Icing Research Tunnel
Abstract
The Icing Research Tunnel (IRT) at NASA Glenn Research Center follows the recommended practice for icing tunnel calibration outlined in SAE’s ARP5905 document. The calibration team has followed the schedule of a full calibration every five years with a check calibration done every six months following. The liquid water content of the IRT has maintained stability within in the specifications presented to customers that the variation is within +/- 10% of the calibrated, target measurement. With recent measurements and instrumentation errors, a more thorough assessment of error source was desired. By constructing statistical process control charts, the ability to determine how the instrument varies in the short term, mid term, and long term was gained. The control charts offer a view of instrument error, facility error, or installation changes. It was discovered that there was a shift from target to mean baseline thus leading to the study of the overall capability indices of the liquid water content measuring instrument to perform within specifications defined in the IRT. This presentation describes data processing procedures for the Multi-Element Sensor in the IRT, including collision efficiency corrections, canonical correlation analysis, Chauvenet’s Criterion for rejection of data, distribution check of data, and mean, median and mode for construction of control charts. Further data is presented to describe the repeatability of the IRT with the Multi-Element Sensor and the ability to maintain a stable process for the defined calibration schedule.
|
Emily Timko |
Breakout | 2019 | |
| Reasoning about Uncertainty with the Stan Modeling Language
Abstract
This briefing discusses the practical advantages of using the probabilistic programming language (PPL) Stan to answer statistical questions, especially those related to the quantification of uncertainty. Stan is a relatively new statistical tool that allows users to specify probability models and reason about the processes that generate the data they encounter.
Stan has quickly become a popular language for writing statistical models because it allows one to specify rich (or sparse) Bayesian models using high level language. Further, Stan is fast, memory efficient, and robust.
Stan requires users be explicit about the model they wish to evaluate, which makes the process of statistical modeling more transparent to users and decision makers. This is valuable because it forces practitioners to consider assumptions at the beginning of the model building procedure, rather than at the end (or not at all). In this sense, Stan is the opposite of a “black box” modeling approach. This approach may be tedious and labor intensive at first, but the pay-offs are large. For example, once a model is set-up inferential tasks all essentially automatic, as changing the model does not change the how one analyzes the data. This is a generic approach to inference.
To illustrate these points, we use Stan to study a ballistic miss distance problem. In ballistic missile testing, the p-content circular error probable (CEP) in the circle that contains p percent of future shots fired, on average. Statistically, CEP is a bivariate prediction region, constrained by the model to be circular.
In Frequentist statistics, the determination of CEP is highly dependent on the model fit, and a different calculation of CEP must be produced for each plausible model. However, with Stan, we can approach the CEP calculation invariant of the model we use to fit the data.
We show how to use Stan to calculate CEP and uncertainty intervals for the parameters using summary statistics.
Statistical practitioners can access Stan from several programming languages, including R and Python.
|
John Haman |
Breakout |
| 2019 |
| Target Location Error Estimation Using Parametric Models | James Brownlow |
Breakout |
| 2019 |
| Anatomy of a Cyberattack: Standardizing Data Collection for Adversarial and Defensive Analyses
Abstract
Hardly a week goes by without news of a cybersecurity breach or an attack by cyber adversaries against a nation’s infrastructure. These incidents have wide-ranging effects, including reputational damage and lawsuits against corporations with poor data handling practices. Further, these attacks do not require the direction, support, or funding of technologically advanced nations; instead, significant damage can be – and has been – done with small teams, limited budgets, modest hardware, and open source software. Due to the significance of these threats, it is critical to analyze past events to predict trends and emerging threats.
In this document, we present an implementation of a cybersecurity taxonomy and a methodology to characterize and analyze all stages of a cyberattack. The chosen taxonomy, MITRE ATT&CK™, allows for detailed definitions of aggressor actions which can be communicated, referenced, and shared uniformly throughout the cybersecurity community. We translate several open source cyberattack descriptions into the analysis framework, thereby constructing cyberattack data sets.
These data sets (supplemented with notional defensive actions) illustrate example Red Team activities. The data collection procedure, when used during penetration testing and Red Teaming, provides valuable insights about the security posture of an organization, as well as the strengths and shortcomings of the network defenders. Further, these records can support past trends and future outlooks of the changing defensive capabilities of organizations.
From these data, we are able to gather statistics on the timing of actions, detection rates, and cyberattack tool usage. Through analysis, we are able to identify trends in the results and compare the findings to prior events, different organizations, and various adversaries.
|
Jason Schlup |
Breakout |
| 2019 |
| A Survey of Statistical Methods in Aeronautical Ground Testing | Drew Landman |
Breakout |
| 2019 |
| Your Mean May Not Mean What You Mean It to Mean
Abstract
The average and standard deviation of, say, strength or dimensional test data are basic engineering math, simple to calculate. What those resulting values actually mean, however, may not be simple, and can be surprisingly different from what a researcher wants to calculate and communicate. Mistakes can lead to overlarge estimates of spread, structures that are over- or under-designed and other challenges to understanding or communicating what your data is really telling you.
This talk will discuss some common errors and missed opportunities seen in engineering and scientific analyses along with mitigations that can be applied through smart and efficient test planning and analysis. It will cover when – and when not – to report a simple mean of a dataset based on the way the data was taken; why ignoring this often either hides or overstates risk; and a standard method for planning tests and analyses to avoid this problem. And it will cover what investigators can correctly (or incorrectly) say about means and standard deviations of data, including how and why to describe uncertainty and assumptions depending on what a value will be used for.
The presentation is geared toward the engineer, scientist or project manager charged with test planning, data analysis or understanding findings from tests and other analyses. Attenders’ basic understanding of quantitative data analysis is recommended; more-experienced participants will grasp correspondingly more nuance from the pitch. Some knowledge of statistics is helpful, but not required. Participants will be challenged to think about an average as not just “the average”, but a valuable number that can and must relate to the engineering problem to be solved, and must be firmly based in the data. Attenders will leave the talk with a more sophisticated understanding of this basic, ubiquitous but surprisingly nuanced statistic and greater appreciation of its power as an engineering tool.
|
Ken Johnson |
Breakout |
| 2019 |
| A Causal Perspective on Reliability Assessment
Abstract
Causality in an engineered system pertains to how a system output changes due to a controlled change or intervention on the system or system environment. Engineered systems designs reflect a causal theory regarding how a system will work, and predicting the reliability of such systems typically requires knowledge of this underlying causal structure. The aim of this work is to introduce causal modeling tools that inform reliability predictions based on biased data sources and illustrate how these tools can inform data integration in practice. We present a novel application of the popular structural causal modeling framework to reliability estimation in an engineering application, illustrating how this framework can inform whether reliability is estimable and how to estimate reliability using data integration given a set of assumptions about the subject matter and data generating mechanism. When data are insufficient for estimation, sensitivity studies based on problem-specific knowledge can inform how much reliability estimates can change due to biases in the data and what information should be collected next to provide the most additional information. We apply the approach to a pedagogical example related to a real, but proprietary, engineering application, considering how two types of biases in data can influence a reliability calculation.
|
Lauren Hund |
Breakout |
| 2019 |
| Hypergames for Control System Security
Abstract
The identification of the Stuxnet worm in 2010 provided a highly publicized example of a cyber attack on an industrial control system. This raised public awareness about the possibility of similar attacks against other industrial targets – including critical infrastructure. Here, we use hypergames to analyze how adversarial perturbations can be used to manipulate a system using optimal control. Hypergames form an extension of game theory that models strategic interactions between players with significantly different perceptions of the game(s) they are playing. Previous work on hypergames has been limited to simple interactions, where a small set of discrete choices are available to each player. However, we apply hypergames to larger systems with continuous variables. Our results highlight that manipulating constraints can be a more effective attacker strategy than directly manipulating objective function parameters. Moreover, the attacker need not influence the underlying system to carry out a successful attack – it may be sufficient to deceive the defender controlling the system. Finally, we identify several characteristics that will make our analysis amenable to higher-dimensional control systems.
|
Arnab Bhattacharya |
Breakout |
| 2019 |
| Probabilistic Data Synthesis to Provide a Defensible Risk Assessment for Army Munition
Abstract
Military grade energetics are, by design, required to operate under extreme conditions. As such, warheads in a munition must demonstrate a high level of structural integrity in order to ensure safe and reliable operation by the Warfighter. In this example which involved an artillery munition, a systematic analytics-driven approach was executed which synthesized physical test data results with probabilistic analysis, non-destructive evaluation, modeling and simulation, and comprehensive risk analysis tools in order to determine the probability of a catastrophic event. Once the severity, probability of detection, occurrence, were synthesized, a model was built to determine the risk of a catastrophic event during firing which then accounts for defect growth occurring as a result of rough-handling. This comprehensive analysis provided a defensible, credible, and dynamic snapshot of risk while allowing for a transparent assessment of contribution to risk of the various inputs through sensitivity analyses. This paper will illustrate intersection of product safety, reliability, systems-safety policy, and analytics, and highlight the impact of a holistic multidisciplinary approach. The benefits of this rigorous assessment included quantifying risk to the user, supporting effective decision-making, improving resultant safety and reliability of the munition, and supporting triage and prioritization of future Non-Destructive Evaluation (NDE) screening efforts by identifying at-risk subpopulations.
|
Kevin Singer |
Breakout |
| 2019 |
| Using Bayesian Neural Networks for Uncertainty Quantification of Hyperspectral Image Target Detection
Abstract
Target detection in hyperspectral images (HSI) has broad value in defense applications, and neural networks have recently begun to be applied for this problem. A common criticism of neural networks is they give a point estimate with no uncertainty quantification (UQ). In defense applications, UQ is imperative because the cost of a false positive or negative is high. Users desire high confidence in either “target” or “not target” predictions, and if high confidence cannot be achieved, more inspection is warranted. One possible solution is Bayesian neural networks (BNN). Compared to traditional neural networks which are constructed by choosing a loss function, BNN take a probabilistic approach and place a likelihood function on the data and prior distributions for all parameters (weights and biases),
which in turn implies a loss function. Training results in posterior predictive distributions, from which prediction intervals can be computed, rather than only point estimates. Heatmaps show where and how much uncertainty there is at any location and give insight into the physical area being imaged as well as possible improvements to the model. Using pytorch and pyro software, we test BNN on a simulated HSI scene produced using the Rochester Institute of Technology (RIT) Digital Imaging and Remote Sensing Image Generation (DIRSIG) model. The scene geometry used is also developed by RIT and is a detailed representation of a suburban neighborhood near Rochester, NY, named “MegaScene.” Target panels were inserted for this effort, using paint reflectance and bi-directional reflectance distribution function (BRDF) data acquired from the Nonconventional Exploitation Factors Database System (NEFDS). The target panels range in size from large to subpixel, with some targets only partially visible. Multiple renderings of this scene are created under different times of day and with different atmospheric conditions to assess model generalization. We explore the uncertainty heatmap for different times and
environments on MegaScene as well as individual target predictive distributions to gain insight into the power of BNN.
|
Daniel Ries |
Breakout |
| 2019 |
| Constructing Designs for Fault Location
Abstract
Abstract. While fault testing a system with many factors each appearing at some number of levels, it may not be possible to test all combinations of factor levels. Most faults are caused by interactions of only a few factors, so testing interactions up to size t will often find all faults in the system without executing an exhaustive test suite. Call an assignment of levels to t of the factors a t-way interaction. A covering array is a collection of tests that ensures that every t-way interaction is covered by at least one test in the test suite. Locating arrays extend covering arrays with the additional feature that they not only indicate the presence of faults but locate the faulty interactions when there are no more than d faults in the system. If an array is (d, t)-locating, for every pair of sets of t-way interactions of size d, the interactions do not appear in exactly the same tests. This ensures that the faulty interactions can be differentiated from non-faulty interactions
by the results of some test in which interactions from one set or the other but not both are tested. When the property holds for t-way interaction sets of size up to d, the notation (d, t ¯ ) is used. In addition to fault location, locating arrays have also been used to identify significant effects in
screening experiments.
Locating arrays are fairly new and few techniques have been explored for their construction. Most of the available work is limited to finding only one fault (d = 1). Known general methods require a covering array of strength t + d and produce many more tests than are needed. In this talk, we present Partitioned Search with Column Resampling (PSCR), a computational search
algorithm to verify if an array is (d, t ¯ )-locating by partitioning the search space to decrease the number of comparisons. If a candidate array is not locating, random resampling is performed until a locating array is constructed or an iteration limit is reached. Algorithmic parameters determine which factor columns to resample and when to add additional tests to the candidate array. We use a 5 × 5 × 3 × 2 × 2 full factorial design to analyze the performance of the algorithmic parameters and provide guidance on how to tune parameters to prioritize speed, accuracy, or a combination of both. Last, we compare our results to the number of tests in locating arrays constructed for the factors and levels of real-world systems produced by other methods.
|
Erin Lanus |
Breakout |
| 2019 |
| Accelerating Uncertainty Quantification for Complex Computational Models
Abstract
Scientific computing has undergone extraordinary growth in sophistication in recent years, enabling the simulation of a wide range of complex multiphysics and multiscale phenomena. Along with this increase in computational capability is the growing recognition that uncertainty quantification (UQ) must go hand-in-hand with numerical simulation in
order to generate meaningful and reliable predictions for engineering applications. If not rigorously considered, uncertainties due to manufacturing defects, material variability, modeling assumptions, etc. can cause a substantial disconnect between simulation and reality. Packaging these complex computational models within an UQ framework, however, can be a significant challenge due to the need to repeatedly evaluate the model when even a single evaluation is time-consuming.
This talk discusses efforts at NASA Langley Research Center (LaRC) to enable rapid UQ for problems with expensive computational models. Under the High Performance Computing Incubator (HPCI) program at LaRC, several open-source software libraries are being developed and released to provide access to general-purpose, state-of-the-art UQ algorithms. The common denominator of these methods is that they all expose parallelism among the model evaluations needed for UQ and, as such, are implemented to leverage HPC resources when available to achieve tremendous computational speedup. While the methods and software presented are broadly applicable, they will be demonstrated in the context of applications that have particular interest at NASA, including structural health management and trajectory simulation.
|
James Warner |
Breakout |
| 2019 |
| When Validation Fails: Analysis of Data from an Imperfect Test Chamber
Abstract
For chemical/biological testing, test chambers are sometimes designed with a vapor or aerosol homogeneity requirement. For example, a test community may require that the difference in concentration between any two test locations in a chamber be no greater than 20 percent. To validate the chamber, testers must demonstrate that such a requirement is met with a specified amount of certainty, such as 80 percent. With a validated chamber, multiple systems can be simultaneously tested at different test locations with the assurance that each system is exposed to nearly the same concentration.
In some cases, however, homogeneity requirements are difficult to achieve. This presentation demonstrates a valid Bayesian method for testing probability of detection as a function of concentration in a chamber that fails to meet a homogeneity requirement. The demonstrated method of analysis is based on recent experience with an actual test chamber. Multiple systems are tested simultaneously at different locations in the chamber. Because systems tested in the chamber are exposed to different concentrations depending on these locations, the differences must be quantified to the greatest extent possible. To this end, data from the failed validation efforts are used to specify informative prior distributions for probability-of-detection modeling. Because these priors quantify and incorporate uncertainty in model parameters, they ensure that the final probability-of-detection model constitutes a valid comparison of the performance of the different systems.
|
Kendal Ferguson |
Breakout |
| 2019 |
| Technical Leadership Panel-Tuesday Afternoon | Dr. Catherine Warner Science Advsior DOT&E |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | Paul Roberts Chief Engineer Engineering and Safety Center |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | Frank Peri Deputy Director Langley Engineering Directorate |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | CAPT Peter Matisoo Technical Director COTF |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | Jeff Olinger Technical Director AFOTEC |
Breakout | 2016 | |
| Combining information for Realiability Assesment-Tuesday Morning | Dr. Alyson Wilson North Carolina State University |
Breakout |
| 2016 |
| Reliability Growth in T&E – Summary of National Research Council’s Committee on National Statistics Report Finding-Tuesday Morning | Dr. Art Fries Research Staff Member IDA |
Breakout |
| 2016 |
| Design and Analysis of Margin Testing in Support of Product Qualification for High Reliability Systems | Dr. Justin Newcomer Sandia National Lab |
Breakout | 2016 | |
| Three Case Studies Comparing Traditional versus Modern Test Designs
Abstract
There are many testing situations that historically involve a large number of runs. The use of experimental design methods can reduce the number of runs required to obtain the information desired. Example applications include wind tunnel test campaigns, computational experiments and live fire tests. In this work we present three case studies conducted under the auspices of the Science of Test Research Consortium comparing the information obtained via a historical experimental approach with the information obtained via an experimental design approach. The first case study involves a large scale wind tunnel experimental campaign. The second involves a computational fluid dynamics model of a missile through various speeds and angles of attack. The third case involves ongoing live fire testing involve hot surface testing. In each case, results suggest a tremendous opportunity to reduce experimental test efforts without losing test information.
|
Dr. Ray Hill Air Force instite of Technology |
Breakout |
| 2016 |
| Supersaturated Designs: Construction and Analysis
Abstract
An important property of any experimental design is its ability to detect active factors. For supersaturated designs, in which model parameters outnumber experimental runs, power is even more critical. In this talk, we review several popular supersaturated design construction criteria and analysis methods. We then demonstrate how simulation studies can be useful for practitioners in selecting a supersaturated design with regards to power to detect active factors. One of our findings based on an extensive simulation study is that although differences clearly exist among analysis methods, most supersaturated design construction methods are indistinguishable in terms of power. This conclusion can be reassuring for practitioners as supersaturated designs can then be sensibly chosen based upon convenience. For instance, the Bayesian D-optimal supersaturated designs can be easily constructed in JMP and SAS for any run size and number of factors. On the other hand, software for constructing E(s2)-optimal supersaturated designs is not as accessible.
|
Dr. David Edwards Virginia Commonwealth University |
Breakout | 2016 | |
| Statistically Defensible Experiment Design for Wind Tunnel Characterization of Subscale Parachutes for Mission to Mars
Abstract
https://s3.amazonaws.com/workshop-archives-2016/IDA+Workshop+2016/testsciencemeeting.ida.org/pdfs/1b-ExperimentalDesignMethodsandApplications.pdf
|
Dr. Drew Landman Old Dominion Univerity |
Breakout |
| 2016 |
| Managing Uncertainty in the Context of Risk Acceptance Decision Making at NASA: Thinking Beyond the Model
Abstract
NASA has instituted requirements for establishing Agency-level safety thresholds and goals that define “long-term targeted and maximum tolerable levels of risk to the crew as guidance to developers in evaluating ‘how safe is safe enough’ for a given type of mission.” With the adoption of this policy for human space flight and with ongoing Agency efforts to increase formality in the development and review of the basis for risk acceptance, the decision-support demands placed on risk models are becoming more stringent at NASA. While these models play vital roles in informing risk acceptance decisions, they are vulnerable to incompleteness of risk identification, as well as to incomplete understanding of probabilities of occurrence, potentially leaving a substantial portion of the actual risk unaccounted for, especially for new systems. This presentation argues that management of uncertainty about the “actual” safety performance of a system must take into account the contribution of unknown and/or underappreciated (UU) risk. Correspondingly, responsible risk-acceptance decision-making requires the decision-maker to think beyond the model and address factors (e.g., organizational and management factors) that live outside traditional engineering risk models. This presentation advocates the use of a safety-case approach to risk acceptance decision-making.
|
Dr. Homayoon Dezfuli NASA |
Breakout |
| 2016 |
| Acceptability of Radiation Detection Systems
Abstract
The American National Standards Institute (ANSI) maintains a set of test standards that provide methods to characterize and determine the acceptability of radiation detection systems for use in Homeland security. With a focus on the environmental, electromagnetic, and mechanical functionality tests, we describe the test formulation and discuss challenges faced in administering the standard to include the assurance of comparable evaluations across multiple test facilities and the handling of systems that provide a non-standard, unit-less response. We present proposed solutions to these difficulties that are currently being considered in updated versions of the ANSI standards. We briefly describe a decision analytic approach that could allow for the removal of minimum performance requirements from the standards and enable the end user to determine system suitability based on operation-specific requirements.
|
Dr. Dennis Leber NIST |
Breakout |
| 2016 |
| Risk Analysis for Orbital Debris Wire Harness Failure Assessment for the Joint Polar Satellite System
Abstract
This paper presents the results of two hypervelocity impact failure probabilistic risk assessments for critical wire bundles exposed aboard the Joint Polar Satellite System (JPSS-1) to an increased orbital debris environment at its 824 km, 98.8 deg inclination orbit. The first “generic” approach predicted the number of wires broken by orbital debris ejecta emerging from normal impact with multi-layer insulation (MLI) covering 36-, 18-, and 6-strand wire bundles at a 5 cm standoff using the Smooth Particle Hydrodynamic (SPH) code. This approach also included a mathematical approach for computing the probability that redundant wires were impacted then severed within the bundle. Based in part on the high computed risk of a critical wire bundle failure from the generic approach, an enhanced orbital debris protection design was examined, consisting of betacloth-reinforced MLI suspended at a 5 cm standoff over a seven layer betacloth and Kevlar blanket, draped over the exposed wire bundles. A second SPH-based risk assessment was conducted that also included the beneficial effects from the high (75 degree) obliquity of orbital debris impact and shadowing by other spacecraft components, and resulted in a considerably reduced likelihood of critical wire bundle failure compared to the original baseline design.
|
Dr. Joel Williamsen IDA |
Breakout |
| 2016 |
| Application of Statistical Engineering to Mixture Problems With Process Variables
Abstract
Statistical engineering has been defined as: “The study of how to best utilize statistical concepts, methods, and tool, and integrate them with IT and other relevant sciences to generate improved results.” A key principle is that significant untapped benefits are often achievable by integrating multiple methods in novel ways to address a problem, without having to invent new statistical techniques. In this presentation, we discuss the application of statistical engineering to the problem of design and analysis of mixture experiments when process variables are also involved. In such cases, models incorporating interaction between the mixture and process variables have been developed, but tend to require large designs and models. By considering models nonlinear in the parameters, also well known in the literature, we demonstrate how experimenters can utilize an alternative, iterative strategy in attacking such problems. We show that this strategy potentially saves considerable experimental time and effort, while producing models that are nearly as accurate as much larger linear models. These results are illustrated using two published data sets and one new data set, all involving interactive mixture and process variable problems.
|
Dr. Roger Hoerl Union College |
Breakout |
| 2016 |
| The Sixth Sense: Clarity through Statistical Engineering
Abstract
Two responses to an expensive, time consuming test on a final product will be referred to as “adhesion” and “strength”. A screening test was performed on compounds that comprise the final product. These screening tests are multivariate profile measurements. Previous models to predict the expensive, time consuming test lacked accuracy and precision. Data visualization was used to guide a statistical engineering model that makes use of multiple statistical techniques. The modeling approach raised some interesting statistical questions for partial least square models regarding over-fitting and cross validation. Ultimately, the model interpretation and the visualization both make engineering sense and led to interesting insights regarding the product development program and screening compounds.
|
Dr. Jennifer Van-Mullekom DuPont |
Breakout | 2016 | |
| Case Studies for Statistical Engineering Applied to Powered Rotorcraft Wind-Tunnel Tests
Abstract
Co-Authors: Sean A. Commo, Ph.D., P.E. and Peter A. Parker, Ph.D., P.E. NASA Langley Research Center, Hampton, Virginia, USA Austin D. Overmeyer, Philip E. Tanner, and Preston B. Martin, Ph.D. U.S. Army Research, Development, and Engineering Command, Hampton, Virginia, USA. The application of statistical engineering to helicopter wind-tunnel testing was explored during two powered rotor entries. The U.S. Army Aviation Development Directorate Joint Research Program Office and the NASA Revolutionary Vertical Lift Project performed these tests jointly at the NASA Langley Research Center. Both entries were conducted in the 14- by 22-Foot Subsonic Tunnel with a small segment of the overall tests devoted to developing case studies of a statistical engineering approach. Data collected during each entry were used to estimate response surface models characterizing vehicle performance, a novel contribution of statistical engineering applied to powered rotor-wing testing. Additionally, a 16- to 47-times reduction in the number of data points required was estimated when comparing a statistically-engineered approach to a conventional one-factor-at-a-time approach.
|
Dr. Sean Commo NASA |
Breakout | 2016 | |
| Bayesian Adaptive Design for Conformance Testing with Bernoulli Trials
Abstract
Co-authors: Adam L. Pintar, Blaza Toman, and Dennis Leber. A task of the Domestic Nuclear Detection Office (DNDO) is the evaluation of radiation and nuclear (rad/nuc) detection systems used to detect and identify illicit rad/nuc materials. To obtain estimated system performance measures, such as probability of detection, and to determine system acceptability, the DNDO sometimes conduct large scale field tests of these systems at great cost. Typically, non adaptive designs are employed where each rad/nuc test source is presented to each system under test a predetermined and fixed number of times. This approach can lead to unnecessary cost if the system is clearly acceptable or unacceptable. In this presentation, an adaptive design with Bayesian decision theoretic foundations is discussed as an alternative to, and contrasted with, the more common single stage design. Although the basis of the method is Bayesian decision theory, designs may be tuned to have desirable type I and II error rates. While the focus of the presentation is a specific DNDO example, the method is applicable widely. Further, since constructing the designs is somewhat compute intensive, software in the form of an R package will be shown and is available upon request.
|
Dr. Adamn Pintar NIST |
Breakout |
| 2016 |
| Statistical Models for Combining Information: Styrker Reliability Case Study | Dr. Rebecca Dickinson IDA |
Breakout |
| 2016 |
| Bayesian Estimation of Reliability Growth | Dr. Jim Brownlow U.S. Air Force 812TSS/ENT |
Breakout |
| 2016 |
| Recent use of Statistical Methods in NASA Aeroscience Testing Research and Development Activities
Abstract
Over the past 10 years, a number of activities have incorporated statistical methods for the purpose of database development and associated uncertainty modeling. This presentation will highlight approaches taken in aerodynamic database development for space vehicle projects, specifically the Orion spacecraft and abort vehicle, and the Space Launch System (SLS) launch vehicle. Additionally, statistical methods have been incorporated into the Commercial Supersonic Transport Project for test technique development and optimization as well as a certification prediction methodology, which is planned to be verified with the Low-Boom Flight Demonstrator data. Discussion will conclude with the use of statistical methods for quality control and assurance in the NASA Langley Research Center ground testing facilities related to our Check Standard Project and characterization and calibration testing.
|
Dr. Eric Walker NASA |
Breakout | 2016 | |
| “High Velocity Analytics for NASA JPL Mars Rover Experimental Design”
Abstract
Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used.
|
Dr. Jim Wisnowski Co-founder/Principle Adsurgo Bio
James Wisnowski provides training and consulting services in Design of Experiments, Predictive Analytics, Reliability Engineering, Quality Engineering, Text Mining, Data Visualization, and Forecasting to government and industry. Previously, he spent a career in analytics for the government. He retired from the Air Force having had leadership positions at the Pentagon, Air Force Academy, Air Force Operational Test and Evaluation Center, and units across the Air Force. He has published numerous papers in technical journals and presented several invited conference presentations. He was co-author of the Design and Analysis of Experiments by Douglas Montgomery: A Supplement for using JMP.
|
Breakout |
| 2016 |
| “High Velocity Analytics for NASA JPL Mars Rover Experimental Design”
Abstract
Rigorous characterization of system capabilities is essential for defensible decisions in test and evaluation (T&E). Analysis of designed experiments is not usually associated “big” data analytics as there are typically a modest number of runs, factors, and responses. The Mars Rover program has recently conducted several disciplined DOEs on prototype coring drill performance with approximately 10 factors along with scores of responses and hundreds of recorded covariates. The goal is to characterize the ‘atthis-time’ capability to confirm what the scientists and engineers already know about the system, answer specific performance and quality questions across multiple environments, and inform future tests to optimize performance. A ‘rigorous’ characterization required that not just one analytical path should be taken, but a combination of interactive data visualization, classic DOE analysis screening methods, and newer methods from predictive analytics such as decision trees. With hundreds of response surface models across many test series and qualitative factors, these methods used had to efficiently find the signals hidden in the noise. Participants will be guided through an end-to-end analysis workflow with actual data from many tests (often Definitive Screening Designs) of the Rover prototype coring drill. We will show data assembly, data cleaning (e.g. missing values and outliers), data exploration with interactive graphical designs, variable screening, response partitioning, data tabulation, model building with stepwise and other methods, and model diagnostics. Software packages such as R and JMP will be used.
|
Dr. Heath Rushing Co-founder/Principle Adsurgo Bio
Heath Rushing is the cofounder of Adsurgo and author of the book Design and Analysis of Experiments by Douglas Montgomery: A Supplement for using JMP. Previously, he was the JMP Training Manager at SAS, a quality engineer at Amgen, an assistant professor at the Air Force Academy, and a scientific analyst for OT&E in the Air Force. In addition, over the last six years, he has taught Science of Tests (SOT) courses to T&E organizations throughout the DoD.
|
Breakout |
| 2016 |
| An Engineer and a Statistician Walk into a Bar: A Statistical Engineering Negotiation
Abstract
A question about whether how post-processing affects the flammability of a given metal an oxygen enriched environment could be answered with the sensitivity of a standard test that is typically used to measure differences in flammability between different metals. The principal investigator was familiar with design of experiments (DOE) and wanted to optimize the value of the information to be gained. This talk will focus on the interchange between the engineer and a statistician. Their negotiations will illustrate the process of clarifying the problem, planning the test including choosing factors and responses, running the experiment and analyzing and reporting on the data. It will focus on the choices made to help squeeze extra knowledge from each test run and leverage statistics to result in increased engineering insight.
|
Mr. Ken Johnson NASA |
Breakout |
| 2016 |
| Making Statistically Defensible Testing ‘The way we do things around here’
Abstract
For the past 7 years, USAF DT&E has been exploring ways to adapt the principles of experimental design to rapidly evolving developmental test articles and test facilities – often with great success. This paper discusses three case studies that span the range of USAF DT&E activities from EW to Ground Test to Flight Test and shows the truly revolutionary impact Fisher’s DOE can have on development. The Advanced Strategic and Tactical Expendable (ASTE) testing began in 1990 to develop, enhance, and test new IR flares, flare patterns, and dispense tactics. More than 60 aircraft & flare types have been tested. The typical output is a “Pancake Plot” of 200+ “cases” of flare, aspect angle, range, elevation, and flare effectiveness using a stop-light chart (red-yellow-green) approach. In usual testing – ~3000 flare engagements costing $1M to participate. The response, troublingly enough, is binary – 15-30 binomial response trials measuring P(Decoy). Binary responses are information-poor. Legacy testing does not assess present statistical power in reporting P(decoy) results. Analysts investigated replacing P(Decoy) w/ continuous metrics – e.g. time to decoy. This research is ongoing. We found we could spread the replicates out to examine 3x to 5x more test conditions without affecting power materially. Analysis with the Generalized Linear Model (GLZ) replaced legacy “cases” analysis with 75% improvement to confidence intervals with same data. We are seeking to build a Monte-Carlo simulation to estimate how many runs are required in a logistics regression model to achieve adequate power. We hope to reduce customer expenditures for flare information by as much as 50%. Co-authors J. Higdon, B, Knight. AEDC completed a major upgrade with new nozzle hardware to vary Mach. The Arnold Transonic Wind Tunnel 4T spans the range of flows from subsonic to approximately M9. The new wind tunnel was to be computer controlled. A number of key instrumentation improvements were made at the same time. The desire was to calibrate the resulting modified tunnel. The last calibration of 4T was 25 years ago in 1990. The calibration ranged across the full range of Mach and pressure capabilities, spanning a four-D space: pressure, Mach, wall angle, and wall porosity. Both the traditional OFAT effort – vary one factor at a time – and a parallel DOE effort were run to compare design, execution, modeling, and prediction capabilities against cost and time to run. The robust embedded face-centered CCD DOE design (J. Simpson and D. Landman ’05) employed 75 vs. 176 OFAT runs. The RSM design achieved 57% run savings. Due to an admirable discipline in randomization during the DOE trials, the smaller design required longer to run. As a result of using the DOE approach, engineers found it easier predict offcondition tunnel operating characteristics using RSM models, optimize facility flow quality for any given test condition. In addition, the RSM regression models support future “spot check” calibration in future by comparing predictions to measured values. If the measurement falls within the prediction interval, the existing calibration still appropriate. AEDC is using split-plot style designs a for current wind tunnel probe calibration. Co-author: Dr Dough Garrard.
|
Mr. Greg Hutto Air Force 96th Test Wing |
Breakout | 2016 | |
| Using Sequential Testing to Address Complex Full-Scale Live Fire Test and Evaluation
Abstract
Co-authors: Dr. Darryl Ahner, Director, STAT COE Dr. Lenny Truett, STAT COE Mr. Scott Wacker, 96 TG/OL-ACS. This presentation will present the benefits of sequential testing and demonstrate how sequential testing can be used to address complex test conditions by developing well controlled early experiments to explore basic questions before proceeding to full-scale testing. This approach can result in increased knowledge and decreased cost. As of FY13 the Air Force had spent an estimated $47M on dry bay fire testing making fire the largest cost contributor for Live Fire Test and Evaluation (LFT&E) programs. There is currently an estimated 60% uncertainty in total platform vulnerable area (Av) driven by probability of kill (PK) due to ballistically ignited fires. A large part of this uncertain comes from the fact that current spurt modeling does not predict fuel spurt delay with reasonable accuracy despite a large amount of test data. A low-cost sequential approach was developed to improve spurts models. Initial testing used a spherical projectile to test 10 different factors in a definitive screening design. Once the list of factors was refined, a second phase of testing determined if a suitable methodology could be developed to scaled results using water as a surrogate for JP-8 fuel. Finally testing was performed with cubical projectiles to evaluate the effect of fragment orientation. The entire cost for this effort was less than one or two typical full-scale live fire tests.
|
Dr. Darryl Ahner AFIT, STAT COE |
Breakout | 2016 | |
| Aerospace Measurement and Experimental System Development Characterization
Abstract
Co-Authors: Sean A. Commo, Ph.D., P.E. and Peter A. Parker, Ph.D., P.E. NASA Langley Research Center, Hampton, Virginia, USA Austin D. Overmeyer, Philip E. Tanner, and Preston B. Martin, Ph.D. U.S. Army Research, Development, and Engineering Command, Hampton, Virginia, USA. The application of statistical engineering to helicopter wind-tunnel testing was explored during two powered rotor entries. The U.S. Army Aviation Development Directorate Joint Research Program Office and the NASA Revolutionary Vertical Lift Project performed these tests jointly at the NASA Langley Research Center. Both entries were conducted in the 14- by 22-Foot Subsonic Tunnel with a small segment of the overall tests devoted to developing case studies of a statistical engineering approach. Data collected during each entry were used to estimate response surface models characterizing vehicle performance, a novel contribution of statistical engineering applied to powered rotor-wing testing. Additionally, a 16- to 47-times reduction in the number of data points required was estimated when comparing a statistically-engineered approach to a conventional one-factor-at-a-time approach.
|
Mr. Ray Rhew NASA |
Breakout |
| 2016 |
| Leveraging Design for Variation to Improve both Testing and Design – A Case Study on Probabilistic Design of Bearings
Abstract
In this case study we demonstrate an application of Pratt & Whitney’s “Design for Variation” discipline applied to the task of a roller bearing design. The ultimate goal, in this application, was to utilize test data from the “real world” to calibrate a computer model used for design and ensure that roller bearing designs obtained from this model were optimized for maximum robustness to major sources of variation in bearing manufacture and operation. The “Design for Variation” process provides engineers with many useful analysis results even before real world data is applied: high fidelity sensitivity analysis, uncertainty analysis (quantifying a baseline risk of failing to meet design intent) and model verification. The combining of real world data and Bayesian statistical methods that Design for Variation employs to calibrate models, however, goes a step further, validating the accuracy of the model’s outputs and quantifying any bias between the model and the real world. As a result of this application, the designers were able to identify the sources of the bias and correct the model’s physics-based aspects to more accurately model reality. The improved model is now integrated into all successive bearing design activities. The benefits of this method, which only required a small amount of high quality test data, are now available to all present and future roller bearing designs.
|
Dr. Jaime O’connell Pratt and Whitney |
Breakout |
| 2016 |
| Importance of Modeling and Simulation in Testing and the Need for Rigorous Validation
Abstract
Modeling and simulation (M&S) is often an important element of operational evaluations of effectiveness, suitability, survivability, and lethality. For example, the testing of new systems designed to operate against advanced foreign threats, as well as the testing of systems of systems, will involve the use of M&S to examine scenarios that cannot be created using live testing. In order to have an adequate understanding of, and confidence in, the results obtained from M&S, statistically rigorous techniques should be applied to the validation process wherever possible. Design of experiments methodologies should be employed to determine what live and simulation data are needed to support rigorous validation, and formal statistical tests should be used to compare live and simulated data. This talk will discuss the importance of M&S in operational testing through a few examples, and outline several statistically rigorous techniques for validation.
|
Dr. Kelly McGinnity IDA |
Breakout |
| 2016 |
| Operational Cybersecurity Testing
Abstract
The key to acquiring a cybersecure system is the ability to drive considerations about security from the operational level into the tactics and procedures. For T&E to support development and acquisition decisions, we must also adopt the perspective that a cyberattack is an attack on the mission using technology. A well-defined process model linking tools, tasks, and operators to mission performance supports this perspective. We will discuss an approach based on best practices learned from various DHS programs.
|
Mr. Alex Hoover DHS |
Breakout |
| 2016 |
| Dr. Tye Botting Research Staff Member IDA |
Breakout | 2016 | ||
| Recent Advances in Measuring Display Clutter
Abstract
Display clutter has been defined as an unintended effect of display imagery obscuring or confusing other information or that may not be relevant to the task at hand. Negative effects of clutter on user performance have been documented; however, some work suggests differential effects with workload variations and measurement method. Existing measures of clutter either focus on physical display characteristics or user perceptions and they generally exhibit weak correlations with task performance, limiting utility for application in safety-critical domains. These observations have led to a new integrated measure of clutter accounting for display data, user knowledge and patterns of visual attention. Due to limited research on clutter effects in domains other than aviation, empirical studies have been conducted to evaluate the new measure in automobile driving. Data-driven measures and subjective perceptions of clutter were collected along with patterns of visual attention allocation when drivers searched ‘high’ and ‘low’ clutter navigation displays. The experimental paradigm was manipulated to include both presentation-based trials with static display images or use of a dynamic, driving simulator. The new integrated measure was more strongly correlated with driver performance than other, previously-developed measures of clutter. Results also revealed clutter to significantly alter attention and degrade performance with static displays but to have little to no effects in driving simulation. Findings corroborate trends in the literature that clutter has its greatest effects on behavior in domains requiring extended attention to displays, such as map-search, compared to use of displays to support secondary tasks, such as nav aids in driving. Integrating display data and user knowledge factors with patterns of attention shows promise for clutter measurement
|
Dr. David Kaber NCSU |
Breakout |
| 2017 |
| Recent Advances in Measuring Display Clutter
Abstract
Display clutter has been defined as an unintended effect of display imagery obscuring or confusing other information or that may not be relevant to the task at hand. Negative effects of clutter on user performance have been documented; however, some work suggests differential effects with workload variations and measurement method. Existing measures of clutter either focus on physical display characteristics or user perceptions and they generally exhibit weak correlations with task performance, limiting utility for application in safety-critical domains. These observations have led to a new integrated measure of clutter accounting for display data, user knowledge and patterns of visual attention. Due to limited research on clutter effects in domains other than aviation, empirical studies have been conducted to evaluate the new measure in automobile driving. Data-driven measures and subjective perceptions of clutter were collected along with patterns of visual attention allocation when drivers searched ‘high’ and ‘low’ clutter navigation displays. The experimental paradigm was manipulated to include both presentation-based trials with static display images or use of a dynamic, driving simulator. The new integrated measure was more strongly correlated with driver performance than other, previously-developed measures of clutter. Results also revealed clutter to significantly alter attention and degrade performance with static displays but to have little to no effects in driving simulation. Findings corroborate trends in the literature that clutter has its greatest effects on behavior in domains requiring extended attention to displays, such as map-search, compared to use of displays to support secondary tasks, such as nav aids in driving. Integrating display data and user knowledge factors with patterns of attention shows promise for clutter measurement
|
Dr. Carl Pankok |
Breakout |
| 2017 |
| Trust in Automation
Abstract
This brief talk will focus on the process of human-machine trust in context of automated intelligence tools. The trust process is multifaceted and this talk will define concepts such as trust, trustworthiness, trust behavior, and will examine how these constructs might be operationalized in user studies. The talk will walk through various aspects of what might make an automated intelligence tool more or less trustworthy. Further, the construct of transparency will be discussed as a mechanism to foster shared awareness and shared intent between humans and machines.
|
Dr. Joseph Lyons Technical Advisor Air Force Research Laboratory |
Breakout |
| 2017 |
| Dose-Response Model of Recent Sonic Boom Community Annoyance Data
Abstract
To enable quiet supersonic passenger flight overland, NASA is providing national and international noise regulators with a low-noise sonic boom database. The database will consist of dose-response curves, which quantify the relationship between low-noise sonic boom exposure and community annoyance. The recently-updated international standard for environmental noise assessment, ISO 1996-1:2016, references multiple fitting methods for dose-response analysis. One of these fitting methods, Fidell’s community tolerance level method, is based on theoretical assumptions that fix the slope of the curve, allowing only the intercept to vary. This fitting method is applied to an existing pilot sonic boom community annoyance data set from 2011 with a small sample size. The purpose of this exercise is to develop data collection and analysis recommendations for future sonic boom community annoyance surveys.
|
Dr. Jonathan Rathsam NASA |
Breakout | 2017 | |
| Overview of Statistical Validation Tools
Abstract
When Modeling and Simulation (M&S) is used as part of operational evaluations of effectiveness, suitability, survivability, or lethality, the M&S capability should first be rigorously validated to ensure it is representing the real world accurately enough for the intended use. Specifically, we need to understand and characterize the usefulness and limitations of the M&S, especially in terms of uncertainty! Many statistical techniques are available to compare M&S output with live test data. This presentation will describe and present results from a simulation study conducted to determine which techniques provide the highest statistical power to detect differences in mean and variance between live and sim for a variety of data types and sizes.
|
Dr. Kelly McGinnity IDA |
Breakout |
| 2017 |
| Improving the Rigor of Navy M&S VV&A through the application of Design of Experiments Methodologies and related Statistical Techniques | Dr. Stargel Doane COTF |
Breakout |
| 2017 |
| Validation of AIM-9X Modeling and Simulation
Abstract
One use for Modeling and Simulation (M&S) in Test and Evaluation (T&E) is to produce weapon miss distances to evaluate the effectiveness of a weapons. This is true for the Air Intercept Missile-9X (AIM-9X) T&E community. Since flight testing is expensive, the test program uses relatively few flight tests at critical conditions, and supplements those data with large numbers of miss distances from simulated tests across the weapons operational space. However, before the model and simulation is used to predict performance it must first be validated. Validation is an especially daunting task when working with a limited number of live test data. In this presentation we shown that even with a limited number of live test points (e.g. 16 missile fires), we can still perform a statistical analysis for the validation. Specifically, we introduce a validation technique known as Fisher’s Combined Probability Test and we show how to apply Fisher’s test to validate the AIM-9X model and simulation.
|
Dr. Rebecca Dickinson Research Staff Member IDA |
Breakout |
| 2017 |
| Software Reliability Modeling
Abstract
Many software reliability models characterize the number of faults detected during the testing process as a function of testing time, which is performed over multiple stages. Typically, the later stages are progressively more expensive because of the increased number of personnel and equipment required to support testing as the system nears completion. Such transitions from one stage of testing to the next change in the operational environment. One statistical approach to combine software reliability growth models in a manner capable of characterizing multi-stage testing is the concept of a change-point process, where the intensity of a process experiences a distinct change at one or more discrete times during testing. Thus, change-point processes can be used to model change in the failure rate of software due to changes in the testing strategy and environment, integration testing, and resource allocation as it proceeds through multiple stages of testing.
This presentation generalizes change-point models to the heterogeneous case, where fault detection before and after a change-point can be characterized by distinct nonhomogeneous Poisson processes (NHPP). Experimental results suggest that heterogeneous change-point models better characterize some failure data sets, which can improve the applicability of software reliability models to large-scale software systems that are tested over multiple stages.
|
Dr. Lance Fiondella UMASS |
Breakout |
| 2017 |
| Reliability Growth Modeling
Abstract
Several optimization models are described for allocating resources to different testing activities in a system’s reliability growth program. These models assume availability of an underlying reliability growth model for the system, and capture the tradeoffs associated with focusing testing resources at various levels (e.g., system, subsystem, component) and/or how to divide resources within a given level. In order to demonstrate insights generated by solving the model, we apply the optimization models to an example series-parallel system in which reliability growth is assumed to follow the Crow/AMSAA reliability growth model. We then demonstrate how the optimization models can be extended to incorporate uncertainty in Crow/AMSAA parameters.
|
Dr. Kellly Sullivan University of Arkansas |
Breakout |
| 2017 |
| Updating R and Reliability Training with Bill Meeker
Abstract
Since its publication, Statistical Methods for Reliability Data by W. Q. Meeker and L. A. Escobar has been recognized as a foundational resource in analyzing failure time to and survival data. Along with the text, the authors provided an S-Plus software package, called SPLIDA, to help readers utilize the methods presented in the text. Today, R is the most popular statistical computing language in the world, largely supplanting S-Plus. The SMRD package is the result of a multi-year effort to completely rebuild SPLIDA, to take advantage of the improved graphics and workflow capabilities available in R. This presentation introduces the SMRD package, outlines the improvements and shows how the package works seamlessly with the rmarkdown and shiny packages to dramatically speed up your workflow. The presentation concludes with a discussion on what improvements still need to be made prior to publishing the package on the CRAN.
|
Dr. Jason Freels AFIT |
Breakout |
| 2017 |
| Introduction to Bayesian Statistics
Abstract
One of the most powerful features of Bayesian analyses is the ability to combine multiple sources of information in a principled way to perform inference. For example, this feature can be particularly valuable in assessing the reliability of systems where testing is limited for some reason (e.g., expense, treaty). At their most basic, Bayesian methods for reliability develop informative prior distributions using expert judgment or similar systems. Appropriate models allow the incorporation of many other sources of information, including historical data, information from similar systems, and computer models. I will introduce the approach and then consider examples from defense acquisition and lifecycle extension, focusing on the strengths and weaknesses of the Bayesian analyses.
|
Dr. Alyson Wilson North Carolina State University |
Breakout |
| 2017 |
| How do the Framework and Design of Experiments Fundamentally Help?
Abstract
The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals.
The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface.
When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help.
This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.
|
Dr. Luis A. Cortes |
Breakout | 2017 | |
| How do the Framework and Design of Experiments Fundamentally Help?
Abstract
The Military Global Positioning System (GPS) User Equipment (MGUE) program is the user segment of the GPS Enterprise—a program on the Deputy Assistant Secretary of Defense for Developmental Test and Evaluation (DASD(DT&E)) Space and Missile Defense Systems portfolio. The MGUE program develops and test GPS cards capable of using Military-Code (M Code) and legacy signals.
The program’s DT&E strategy is challenging. The GPS cards provide new, untested capabilities. Milestone A was approved on 2012 with sole source contracts released to three vendors for Increment 1. An Acquisition Decision Memorandum directs the program to support a Congressional Mandate to provide GPS M Code-capable equipment for use after FY17. Increment 1 provides GPS receiver form factors for the ground domain interface as well as for the aviation and maritime domain interface.
When reviewing DASD(DT&E) Milestone B (MS B) Assessment Report, Mr. Kendall expressed curiosity about how the Developmental Evaluation Framework (DEF) and Design of Experiments (DOE) help.
This presentation describes how the DEF and DOE methods help producing more informative and more economical developmental tests than what was originally under consideration by the test community—decision-quality information with a 60% reduction in test cycle time. It provides insight into how the integration of the DEF and DOE improved the overall effectiveness of the DT&E strategy, illustrates the role of modeling and simulation (M&S) in the test design process, provides examples of experiment designs for different functional and performance areas, and illustrates the logic involved in balancing risks and test resources. The DEF and DOE methods enables the DT&E strategy to fully exploit early discovery, to maximize verification and validation opportunities, and to characterize system behavior across the technical requirements space.
|
Mr. Mike Sheeha MITRE |
Breakout | 2017 | |
| Allocating Information Gathering Efforts for Selection Decisions
Abstract
Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.
|
Dr. Dennis Leber NIST |
Breakout |
| 2017 |
| Allocating Information Gathering Efforts for Selection Decisions
Abstract
Selection decisions, such as procurement decisions, are often based on multiple performance attributes whose values are estimated using data (samples) collected through experimentation. Because the sampling (measurement) process has uncertainty, more samples provide better information. With a limited test budget to collect information to support such a selection decision, determining the number of samples to observe from each alternative and attribute is a critical information gathering decision. In this talk we present a sequential allocation scheme that uses Bayesian updating and maximizes the probability of selecting the true best alternative when the attribute value samples contain Gaussian measurement error. In this sequential approach, the test-designer uses the current knowledge of the attribute values to identify which attribute and alternative to sample next; after that sample, the test-designer chooses another attribute and alternative to sample, and this continues until no more samples can be made. We present the results of a simulation study that illustrates the performance advantage of the proposed sequential allocation scheme over simpler and more common fixed allocation approaches.
|
Dr. Jeffrey Herrmann University of Maryland |
Breakout |
| 2017 |
| Statistical Methods for Programmatic Assessment and Anomaly Detection | Mr. Douglas Brown BAH |
Breakout |
| 2017 |
| Statistical Methods for Programmatic Assessment and Anomaly Detection | Dr. Ray McCollum BAH |
Breakout |
| 2017 |
| Flight Test and Evaluation of Airborne Spacing Application
Abstract
NASA’s Airspace Technology Demonstration (ATD) project was developed to facilitate the transition of mature air traffic management technologies from the laboratory to operational use. The first ATD focused on an integrated set of advanced NASA technologies to enable efficient arrival operations in high-density terminal airspace. This integrated arrival solution was validated and verified in laboratories and transitioned to a field prototype for an operational demonstration. Within NASA, this was a collaborative effort between Ames and Langley Research Centers involving a multi-year iterative experimentation process consisting of a series of sequential batch computer simulations and human-in-the-loop experiments, culminating in a flight test. Designing and analyzing the flight test involved a number of statistical challenges. There were several variables which are known to impact the performance of the system, but which could not be controlled in an operational environment. Changes in the schedule due to weather and the dynamic positioning of the aircraft on the arrival routes resulted in the need for a design that could be modified in real-time. This presentation describes a case study from a recent NASA flight test, highlights statistical challenges, and discusses lessons learned.
|
Dr. Sara Wilson NASA |
Breakout |
| 2017 |
| Testing and Estimation in Sequential High-Dimension Data
Abstract
Many modern processes generate complex data records not readily analyzed by traditional techniques. For example, a single observation from a process might be a radar signal consisting of n pairs of bivariate data described via some functional relation between reflection and direction. Methods are examined here for detecting changes in such sequences from some known or estimated nominal state. Additionally, estimates of the degree of change (scale, location, frequency, etc.) are desirable and discussed. The proposed methods are designed to take advantage of all available data in a sequence. This can become unwieldy for long sequences of large-sized observations, so dimension reduction techniques are needed. In order for these methods to be as widely applicable as possible, we make limited distributional assumptions and so we propose new nonparametric and Bayesian tools to implement these estimators.
|
Dr. Eric Chicken Florida State University |
Breakout |
| 2017 |
| Blast Noise Event Classification from a Spectrogram
Abstract
Spectrograms (i.e., squared magnitude of short-time Fourier transform) are commonly used as features to classify audio signals in the same way that social media companies (e.g., Google, Facebook, Yahoo) use images to classify or automatically tag people in photos. However, a serious problem arises when using spectrograms to classify acoustic signals, in that the user must choose the input parameters (hyperparameters), and such choices can have a drastic effect on the accuracy of the resulting classifier. Further, considering all possible combinations of the hyperparameters is a computationally intractable problem. In this study, we simplify the problem making it computationally tractable, explore the utility of response surface methods for sampling the hyperparameter space, and find that response surface methods are a computationally efficient means of identifying the hyperparameter combinations that are likely to give the best classification results.
|
Dr. Edward Nykaza Army Engineering Research and Development Center, Construction Engineering Research Laboratory |
Breakout |
| 2017 |
| Automated Software Testing Best Practices and Framework: A STAT COE Project
Abstract
The process for testing military systems which are largely software intensive involves techniques and procedures often different from those for hardware-based systems. Much of the testing can be performed in laboratories at many of the acquisition stages, up to operational testing. Testing software systems is not different from testing hardware-based systems in that testing earlier and more intensively benefits the acquisition program in the long run. Automated testing of software systems enables more frequent and more extensive testing, allowing for earlier discovery of errors and faults in the code. Automated testing is beneficial for unit, integrated, functional and performance testing, but there are costs associated with automation tool license fees, specialized manpower, and the time to prepare and maintain the automation scripts. This presentation discusses some of the features unique to automated software testing and offers a framework organizations can implement to make the business case for, to organize for, and to execute and benefit from automating the right aspects of their testing needs. Automation has many benefits in saving time and money, but is most valuable in freeing test resources to perform higher value tasks.
|
Dr. Jim Simpson JK Analytics |
Breakout |
| 2017 |
| Automated Software Testing Best Practices and Framework: A STAT COE Project
Abstract
The process for testing military systems which are largely software intensive involves techniques and procedures often different from those for hardware-based systems. Much of the testing can be performed in laboratories at many of the acquisition stages, up to operational testing. Testing software systems is not different from testing hardware-based systems in that testing earlier and more intensively benefits the acquisition program in the long run. Automated testing of software systems enables more frequent and more extensive testing, allowing for earlier discovery of errors and faults in the code. Automated testing is beneficial for unit, integrated, functional and performance testing, but there are costs associated with automation tool license fees, specialized manpower, and the time to prepare and maintain the automation scripts. This presentation discusses some of the features unique to automated software testing and offers a framework organizations can implement to make the business case for, to organize for, and to execute and benefit from automating the right aspects of their testing needs. Automation has many benefits in saving time and money, but is most valuable in freeing test resources to perform higher value tasks.
|
Dr. Jim Wisnowski Adsurgo |
Breakout |
| 2017 |
| Software Test Techniques
Abstract
In recent years, software testing techniques based on formal methods have made their way into industrial practice as a supplement to system and unit testing. I will discuss three core techniques that have proven particularly amenable to transition: 1) Concolic execution, which enables the automatic generation of high-coverage test suites; 2) Property-based randomized testing, which automatically checks sequences of API calls to ensure that expected high-level behavior occurs; and 3) Bounded model checking, which enables systematic exploration of both concrete systems and high-level models to check temporal properties, including ordering of events and timing requirements.
|
Dr. Jose Calderon Galois |
Breakout |
| 2017 |
| Combinational Testing
Abstract
Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced
|
Dr. Raghu Kacker NIST |
Breakout |
| 2017 |
| Combinational Testing
Abstract
Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost. Combinatorial testing takes advantage of the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. The talk explains background, method, and tools available for combinatorial testing. New results on using combinatorial methods for oracle-free testing of certain types of applications will also be introduced
|
Dr. Rick Kuhn NIST |
Breakout |
| 2017 |
| Introduction to Human Measurement | Dr. Cynthia Null NASA |
Breakout | 2017 | |
| The (Empirical) Case for Analyzing Likert-Type Data with Parametric Tests
Abstract
Surveys are commonly used to evaluate the quality of human-system interactions during the operational testing of military systems. Testers use Likert-type response options to measure the intensity of operators’ subjective experiences (e.g., usability, workload) while operating the system. Recently, appropriate methods for analyzing Likert data have become a point of contention within the operational test community. Some argue that Likert data can be analyzed with parametric techniques whereas others argue that only non-parametric techniques should be used. However, the reasons stated for holding a particular view are rarely tied to findings in the empirical literature. This presentation sheds light on this debate by reviewing existing research on how parametric statistics affect the conclusions drawn from Likert data and debunk common myths and misunderstandings about the nature of Likert data within the operational test community and academia.
|
Dr. Heather Wojton Research Staff Member IDA |
Breakout |
| 2017 |
| The System Usability Scale: A measurement Instrument Should Suit the Measurement Needs
Abstract
The System Usability Scale (SUS) was developed by John Brooke in 1986 “to take a quick measurement of how people perceived the usability of (office) computer systems on which they were working.” The SUS is a 10-item, generic usability scale that is assumed to be system agnostic, and it results in a numerical score that ranges from 0-100. It has been widely employed and researched with non-military systems. More recently, it has been strongly recommended for use with military systems in operational test and evaluation, in part because of its widespread commercial use, but largely because it produces a numerical score that makes it amendable to statistical operations.
Recent lessons learned with SUS in operational test and evaluation strongly question its use with military systems, most of which differ radically from non-military systems. More specifically, (1) usability measurement attributes need to be tailored to the specific system under test and meet the information needs of system users, and (2) a SUS numerical cutoff score of 70—a common benchmark with non-military systems—does not accurately reflect “system usability” from an operator or test team perspective. These findings will be discussed in a psychological and human factors measurement context, and an example of system-specific usability attributes will be provided as a viable way forward. In the event that the SUS is used in operational test and evaluation, some recommendations for interpreting the outcomes will be provided.
|
Mr. Keith Kidder AFOTEC |
Breakout |
| 2017 |
| Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation
Abstract
Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.
|
Mr. Scott West Aerospace Corporation |
Breakout |
| 2017 |
| Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation
Abstract
Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.
|
Mr. Martin Annett Aerospace Corporation |
Breakout |
| 2017 |
| Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation
Abstract
Frangible Joints are linear pyrotechnic devices used to separate launch vehicle and spacecraft stages and fairings. Advantages of these systems include low mass, low dynamic shock, and low debris. However the primary disadvantage for human space flight applications is the design’s use of a single explosive cord to effect function, rendering the device zero fault tolerant. Commercial company proposals to utilize frangible joints in human space flight applications spurred a NASA Engineering and Safety Center (NESC) assessment of the reliability of frangible joints. Empirical test and LS-DYNA based finite element analysis was used to understand and assess the design and function, and a deterministic system Design of Experiments (dsDOE) study was conducted to assess the sensitivity of function to frangible joint design variables and predict the device’s design reliability. The collaboration between statistical engineering experts and LS-DYNA analysis experts enabled a comprehensive understanding of these devices.
|
Dr. James Womach Aerospace Corporation |
Breakout |
| 2017 |
| Machine Learning: Overview and Applications to Test
Abstract
“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning.
Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”
|
Lt. Takayuki Iguchi AFOTEC |
Breakout |
| 2017 |
| Machine Learning: Overview and Applications to Test
Abstract
“Machine learning is quickly gaining importance in being able to infer meaning from large, high-dimensional datasets. It has even demonstrated performance meeting or exceeding human capabilities in conducting a particular set of tasks such as speech recognition and image recognition. Employing these machine learning capabilities can lead to increased efficiency in data collection, processing, and analysis. Presenters will provide an overview of common examples of supervised and unsupervised learning tasks and algorithms as an introduction to those without experience in machine learning.
Presenters will also provide motivation for machine learning tasks and algorithms in a variety of test and evaluation settings. For example, in both developmental and operational test, restrictions on instrumentation, number of sorties, and the amount of time allocated to analyze collected data make data analysis challenging. When instrumentation is unavailable or fails, a common back-up data source is an over-the-shoulder video recording or recordings of aircraft intercom and radio transmissions, which traditionally are tedious to analyze. Machine learning based image and speech recognition algorithms can assist in extracting information quickly from hours of video and audio recordings. Additionally, unsupervised learning techniques may be used to aid in the identification of influences of logged or uncontrollable factors in many test and evaluation settings. Presenters will provide a potential example for the application of unsupervised learning techniques to test and evaluation.”
|
Lt. Megan Lewis AFOTEC |
Breakout |
| 2017 |
| Range Adversarial Planning Tool for Autonomy Test and Evaluation | Dr. Chad Hawthorne JHU/APL |
Breakout | 2017 | |
| Search for Extended Test Design Methods for Complex Systems of Systems | Dr. Alex Alaniz AFOTEC |
Breakout |
| 2017 |
| The Future of Engineering at NASA Langley
Abstract
In May 2016, the NASA Langley Research Center’s Engineering Director stood up a group consisting of employees within the directorate to assess the current state of engineering being done by the organization. The group was chartered to develop ideas, through investigation and benchmarking of other organizations within and outside of NASA, for how engineering should look in the future. This effort would include brainstorming, development of recommendations, and some detailed implementation plans which could be acted upon by the directorate leadership as part of an enduring activity. The group made slow and sporadic progress in several specific, self-selected areas including: training and development; incorporation of non-traditional engineering disciplines; capturing and leveraging historical data and knowledge; revolutionizing project documentation; and more effective use of design reviews.
The design review investigations have made significant progress by leveraging lessons learned and techniques gained by collaboration with operations research analysts within the local Lockheed Martin Center for Innovation (the “Lighthouse”) and pairing those techniques with advanced data analysis tools available through the IBM Watson Content Analytics environment. Trials with these new techniques are underway but show promising results for the future of providing objective, quantifiable data from the design review environment – an environment which to this point has remained essentially unchanged for the past 50 years.
|
Mr. Joe Gasbarre NASA |
Breakout |
| 2017 |
| Combinatorial Testing for Link-16 Developmental Test and Evaluation
Abstract
Due to small Tactical Data Link testing windows, only commonly used messages are tested resulting in the evaluation of only a small subset of all possible Link 16 messages. To increase the confidence that software design and implementation issues are discovered in the earliest phases of government acceptance testing, Marine Corps Tactical Systems Support Activity (MCTSSA) Instrumentation and Data Management Section (IDMS) successfully implemented an extension of the traditional form of Design of Experiments (DOE), called Combinatorial Testing (CT). CT was utilized to reduce the human bias and inconsistencies involved in Link 16 testing and replace them with a thorough test that can validate a system’s ability to properly consume all of the possible valid combinations of Link 16 message field values. MCTSSA’s unique team of subject matter experts was able to bring together the tenants of virtualization, automation, C4I Air systems testing, tactical data link testing, and Design of Experiments methodology, to invent a testing paradigm that will exhaustively evaluate tactical Air systems. This presentation will give an overview of how CT was implemented for the test.
|
Mr. Tim Mclean MCTSSA |
Breakout |
| 2017 |
| Censored Data Analysis for Performance Data
Abstract
Binomial metrics like probability-to-detect or probability-to-hit typically provide operationally meaningful and easy to interpret test outcomes. However, they are information poor metrics and extremely expensive to test. The standard power calculations to size a test employ hypothesis tests, which typically result in many tens to hundreds of runs. In addition to being expensive, the test is most likely inadequate for characterizing performance over a variety of conditions due to the inherently large statistical uncertainties associated with binomial metrics. A solution is to convert to a continuous variable, such as miss distance or time-to-detect. The common objection to switching to a continuous variable is that the hit/miss or detect/non-detect binomial information is lost, when the fraction of misses/no-detects is often the most important aspect of characterizing system performance. Furthermore, the new continuous metric appears to no longer be connected to the requirements document, which was stated in terms of a probability. These difficulties can be overcome with the use of censored data analysis. This presentation will illustrate the concepts and benefits of this approach, and will illustrate a simple analysis with data, including power calculations to show the cost savings for employing the methodology.
|
Dr. Bram Lillard IDA |
Breakout |
| 2017 |
| Estimating the Distribution of an Extremum using a Peaks-Over-Threshold Model and Monte Carlo Simulation
Abstract
Estimating the probability distribution of an extremum (maximum or minimum), for some fixed amount of time, using a single time series typically recorded for a shorter amount of time, is important in many application areas, e.g., structural design, reliability, quality, and insurance. When designing structural members, engineers are concerned with maximum wind effects, which are functions of wind speed. With respect to reliability and quality, extremes experienced during storage or transport, e.g., extreme temperatures, may substantially impact product quality, lifetime, or both. Insurance companies are of course concerned about very large claims.
In this presentation, a method to estimate the distribution of an extremum using a well-known peaks-over-threshold (POT) model and Monte Carlo simulation is presented. Since extreme values have long been a subject of study, some brief history is first discussed. The POT model that underlies the approach is then laid out. A description of the algorithm follows. It leverages pressure data collected on scale models of buildings in a wind tunnel for context. Essentially, the POT model is fitted to the observed data and then used to simulate many times series of the desired length. The empirical distribution of the extrema is obtained from the simulated series. Uncertainty in the estimated distribution is quantified by a bootstrap algorithm. Finally, an R package implementing the computations is discussed.
|
Dr. Adam Pintar NIST |
Breakout |
| 2017 |
| Design & Analysis of a Computer Experiment for an Aerospace Conformance Simulation Study
Abstract
Within NASA’s Air Traffic Management Technology Demonstration # 1 (ATD-1), Interval Management (IM) is a flight deck tool that enables pilots to achieve or maintain a precise in-trail spacing behind a target aircraft. Previous research has shown that violations of aircraft spacing requirements can occur between an IM aircraft and its surrounding non-IM aircraft when it is following a target on a separate route. This talk focuses on the experimental design and analysis of a computer experiment which models the airspace configuration of interest in order to determine airspace/aircraft conditions leading to spacing violations during IM operation. We refer to multi-layered nested continuous factors as those that are continuous and ordered in their selection; they can only be selected sequentially with a level selected for one factor affecting the range of possible values for each subsequently nested factor. While each factor is nested within another factor, the exact nesting relationships have no closed form solution. In this talk, we describe our process of engineering an appropriate space-filling design for this situation. Using this space-filling design and Gaussian process modeling, we found that aircraft delay assignments and wind profiles significantly impact the likelihood of spacing violations and the interruption of IM operations.
|
Dr. David Edwards Virginia Commonwealth University |
Breakout |
| 2017 |
| Optimal Multi-Response Designs
Abstract
The problem of constructing a design for an experiment when multiple responses are of interest does not have a clear answer, particularly when the response variables are of different types. Planning an experiment for an air-to-air missile simulation, for example, might have the following responses simultaneously: hit or miss the target (a binary response) and the time to acquire the target (a continuous response). With limited time and resources and only one experiment possible, the question of selecting an appropriate design to model both responses is important. In this presentation, we discuss a method for creating designs when two responses, each with a different distribution (normal, binomial, or Poisson), are of interest. We demonstrate the proposed method using various weighting schemes for the two models to show how the designs change as the weighting scheme changes. In addition, we explore the effect of the specified priors for the nonlinear models on these designs.
|
Dr. Sarah Burke STAT COE |
Breakout | 2017 | |
| Augmenting Definitive Screening Designs
Abstract
Jones and Nachtsheim (2011) introduced a class of three-level screening designs called definitive screening designs (DSDs). The structure of these designs results in the statistical independence of main effects and two-factor interactions; the absence of complete confounding among two-factor interactions; and the ability to estimate all quadratic effects. Because quadratic effects can be estimated, DSDs can allow for the screening and optimization of a system to be performed in one step, but only when the number of terms found to be active during the screening phase of analysis is less than about half the number or runs required by the DSD (Errore, et al., 2016). Otherwise, estimation of second-order models requires augmentation of the DSD. In this paper we explore the construction of series of augmented designs, moving from the starting DSD to designs capable of estimating the full second-order model. We use power calculations, model-robustness criteria, and model-discrimination criteria to determine the number of runs by which to augment in order to identify the active second-order effects with high probability.
|
Ms. Abby Nachtsheim ASU |
Breakout |
| 2017 |
| Integrated Uncertainty Quantification for Risk and Resource Management: Building Confidence in Design | Dr. Eric Walker NASA |
Breakout |
| 2017 |
| Background of NASA’s Juncture Flow Validation Test | Mr. Joseph Morrison NASA |
Breakout | 2017 | |
| A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime | Dr. Eric Walker NASA |
Breakout |
| 2017 |
| A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime | Mr. Joseph Morrison NASA |
Breakout |
| 2017 |
| Communicating Complex Statistical Methodologies to Leadership
Abstract
More often than not, the data we analyze for the military is plagued with statistical issues. Multicollinearity, small sample sizes, quasi-experimental designs, and convenience samples are some examples of what we commonly see in military data. Many of these complications can be resolved either in the design or analysis stage with appropriate statistical procedures. But, to keep our work useful, usable, and transparent to the military leadership who sponsors it, we must strike the elusive balance between explaining and justifying our design and analysis techniques and not inundating our audience with unnecessary details. It can be even more difficult to get military leadership to understand the statistical problems and solutions so well that they are enthused and supportive of our approaches. Using literature written on the subject as well as a variety of experiences, we will showcase several examples, as well as present ideas for keeping our clients actively engaged in statistical methodology discussions.
|
Dr. Jane Pinelis Johns Hopkins University Applied Physics Lab or JHU |
Breakout |
| 2017 |
| Communicating Complex Statistical Methodologies to Leadership
Abstract
More often than not, the data we analyze for the military is plagued with statistical issues. Multicollinearity, small sample sizes, quasi-experimental designs, and convenience samples are some examples of what we commonly see in military data. Many of these complications can be resolved either in the design or analysis stage with appropriate statistical procedures. But, to keep our work useful, usable, and transparent to the military leadership who sponsors it, we must strike the elusive balance between explaining and justifying our design and analysis techniques and not inundating our audience with unnecessary details. It can be even more difficult to get military leadership to understand the statistical problems and solutions so well that they are enthused and supportive of our approaches. Using literature written on the subject as well as a variety of experiences, we will showcase several examples, as well as present ideas for keeping our clients actively engaged in statistical methodology discussions.
|
Mr. Paul Johnson MCOTEA |
Breakout |
| 2017 |
| DOE Case Studies in Aerospace Research and Development
Abstract
This presentation will provide a high level view of recent DOE applications to aerospace research. Two broad categories are defined, aerodynamic force measurement system calibrations and aircraft model wind tunnel aerodynamic characterization. Each case study will outline the application of DOE principles including design choices, accommodations for deviations from classical DOE approaches, discoveries, and practical lessons learned.
Case Studies
Aerodynamic Force Measurement System Calibrations
Large External Wind Tunnel Balance Calibration
– Fractional factorial
– Working with non-ideal factor settings
– Customer driven uncertainty assessment
Internal Balance Calibration Including Temperature
– Restrictions to randomization – split plot design requirements
– Constraints to basic force model
– Crossed design approach
Aircraft Model Wind Tunnel Aerodynamic Characterization
The NASA/Boeing X-48B Blended Wing Body Low-Speed Wind Tunnel Test
– Overcoming a culture of OFAT
– General approach to investigating a new aircraft configuration
– Use of automated wind tunnel models and randomization
– Power of residual analysis in detecting problems
NASA GL–10 UAV Aerodynamic Characterization
– Use of the Nested-Face Centered Design for aerodynamic characterization
– Issues working with over 20 factors
– Discoveries
|
Dr. Drew Landman Old Dominion University |
Breakout |
| 2017 |
| Sample Size and Considerations for Statistical Power
Abstract
Sample size drives the resources and supports the conclusions of operational test. Power analysis is a common statistical methodology used in planning efforts to justify the number of samples. Power analysis is sensitive to extreme performance (e.g. 0.1% correct responses or 99.999% correct responses) relative to a threshold value, extremes in response variable variability, numbers of factors and levels, system complexity, and a myriad of other design- and system-specific criteria. This discussion will describe considerations (correlation/aliasing, operational significance, thresholds, etc.) and relationships (design, difference to detect, noise, etc.) associated with power. The contribution of power to design selection or adequacy must often be tempered when significant uncertainty or test resources constraints exist. In these situations, other measures of merit and alternative analytical approaches become at least as important as power in the development of designs that achieve the desired technical adequacy. In conclusion, one must understand what power is, what factors influence the calculation, and when to leverage alternative measures of merit.
|
Mr. Vance Oas AFOTEC |
Breakout |
| 2017 |
| Sample Size and Considerations for Statistical Power
Abstract
Sample size drives the resources and supports the conclusions of operational test. Power analysis is a common statistical methodology used in planning efforts to justify the number of samples. Power analysis is sensitive to extreme performance (e.g. 0.1% correct responses or 99.999% correct responses) relative to a threshold value, extremes in response variable variability, numbers of factors and levels, system complexity, and a myriad of other design- and system-specific criteria. This discussion will describe considerations (correlation/aliasing, operational significance, thresholds, etc.) and relationships (design, difference to detect, noise, etc.) associated with power. The contribution of power to design selection or adequacy must often be tempered when significant uncertainty or test resources constraints exist. In these situations, other measures of merit and alternative analytical approaches become at least as important as power in the development of designs that achieve the desired technical adequacy. In conclusion, one must understand what power is, what factors influence the calculation, and when to leverage alternative measures of merit.
|
Mr. Nick Garcia AFOTEC |
Breakout |
| 2017 |
| Experimental Design for Composite Pressure Vessel Life Prediction
Abstract
One of the major pillars of experimental design is sequential learning. The experimental design should not be viewed as a “one-shot” effort, but rather as a series of experiments where each stage builds upon information learned from the previous study. It is within this realm of sequential learning that experimentation soundly supports the application of the scientific method.
This presentation illustrates the value of sequential experimentation and also the connection between the scientific method and experimentation through a discussion of a multi-stage project supported by NASA’s Engineering Safety Center (NESC) where the objective was to assess the safety of composite overwrapped pressure vessels (COPVs). The analytical team was tasked with devising a test plan to model stress rupture failure risk in carbon fiber strands that encase the COPVs with the goal of understanding the reliability of the strands at use conditions for the expected mission life. This presentation highlights the recommended experimental design for the strand tests and then discusses the benefits that resulted from the suggested sequential testing protocol.
|
Dr. Anne Drsicoll Virginia Tech |
Breakout |
| 2017 |
| Improving Sensitivity Experiments
Abstract
This presentation will provide a brief overview of sensitivity testing, and emphasize applications to several products and system of importance to the Defense as well as private industry, including Insensitive Energetics, Ballistic testing of protective armor, testing of munition fuzes and Microelectromechanical Systems (MEMS) components, and safety testing of high-pressure test ammunition, and packaging for high-value materials.
|
Mr. Douglas Ray US Army RDECOM ARDEC |
Breakout |
| 2017 |
| Improving Sensitivity Experiments
Abstract
This presentation will provide a brief overview of sensitivity testing, and emphasize applications to several products and system of importance to the Defense as well as private industry, including Insensitive Energetics, Ballistic testing of protective armor, testing of munition fuzes and Microelectromechanical Systems (MEMS) components, and safety testing of high-pressure test ammunition, and packaging for high-value materials.
|
Mr. Kevin Singer US Army |
Breakout |
| 2017 |
| Sequential Experimentation for a Binary Response – The Break Separation Method
Abstract
Binary response experiments are common in epidemiology, biostatistics as well as in military applications. The Up and Down method, Langlie’s Method, Neyer’s method, K in a Row method and 3 Phase Optimal Design are methods used for sequential experimental design when there is a single continuous variable and a binary response. During this talk, we will discuss a new sequential experimental design approach called the Break Separation Method (BSM). BSM provides an algorithm for determining sequential experimental trials that will be used to find a median quantile and fit a logistic regression model using Maximum Likelihood estimation. BSM results in a small sample size and is designed to efficiently compute the median quantile.
|
Mr. Darsh Thakkar RIT-S |
Breakout |
| 2017 |
| Sequential Experimentation for a Binary Response – The Break Separation Method
Abstract
Binary response experiments are common in epidemiology, biostatistics as well as in military applications. The Up and Down method, Langlie’s Method, Neyer’s method, K in a Row method and 3 Phase Optimal Design are methods used for sequential experimental design when there is a single continuous variable and a binary response. During this talk, we will discuss a new sequential experimental design approach called the Break Separation Method (BSM). BSM provides an algorithm for determining sequential experimental trials that will be used to find a median quantile and fit a logistic regression model using Maximum Likelihood estimation. BSM results in a small sample size and is designed to efficiently compute the median quantile.
|
Dr. Rachel Silvestrini RIT-S |
Breakout |
| 2017 |
| Carrier Reliability Model Validation
Abstract
Model Validation for Simulations of CVN-78 Sortie Generation
As part of the test planning process, IDA is examining flight operations on the Navy’s newest carrier, CVN-78. The analysis uses a model, the IDA Virtual Carrier Model (IVCM), to examine sortie generation rates and whether aircraft can complete missions on time. Before using IVCM, it must be validated. However, CVN-78 has not been delivered to the Navy, and data from actual operations are to validate the model. Consequently, we will validate IVCM by comparing it to another model. This is a reasonable approach when a model is used in general analyses such as test planning, but is not acceptable when a model is used in the assessment of system effectiveness and suitability. The presentation examines the use of various statistical tools – Wilcoxon Rank Sum Test, Kolmogorov-Smirnov Test, and lognormal regression – to examine whether the results from two models provide similar results and to quantify the magnitude of any differences. From the analysis, IDA concluded that locations and distribution shapes are consistent, and that the differences between the models are less than 15 percent, which is acceptable for test planning.
|
Dr. Dean Thomas IDA |
Breakout | 2017 | |
| Model Based Systems Engineering Panel Discussion
Abstract
This panel will share status, experiences and expectations within DoD and NASA for transitioning Systems Engineering to a more integrated digital engineering domain. A wide range of perspectives will be provided, covering the implementation waterfront of practitioner, management, research and strategy. Panelist will also be prepared to discuss more focused areas of digital systems engineering, such as test and evaluation, and engineering statistics.
|
Mr. John Holladay NASA |
Breakout |
| 2017 |
| Resampling Methods
Abstract
Resampling Methods: This tutorial presents widely used resampling methods to include bootstrapping, cross-validation, and permutation tests. Underlying theories will be presented briefly, but the primary focus will be on applications. A new graph-theoretic approach to change detection will be discussed as a specific application of permutation testing. Examples will be demonstrated in R; participants are encouraged to bring their own portable computers to follow along using datasets provided by the instructor.
|
Dr. David Ruth United States Naval Academy |
Breakout |
| 2017 |
| Structured Decision Making
Abstract
Difficult choices are often required in a decision-making process where resources and budgets are increasingly constrained. This talk demonstrates a structured decision-making approach using layered Pareto fronts to prioritize the allocation of funds between munitions stockpiles based on their estimated reliability, the urgency of needing available units, and the consequences if adequate numbers of units are . This case study illustrates the process of first identifying appropriate metrics that summarize important dimensions of the decision, and then eliminating non-contenders from further consideration in an objective stage. The final subjective stage incorporates subject matter expert priorities to select the four stockpiles to receive additional maintenance and surveillance funds based on understanding the trade-offs and robustness to various user priorities.
|
Dr. Christine Anderson Cook LANL |
Breakout |
| 2017 |
| Do Asymmetries in Nuclear Arsenals Matter?
Abstract
The importance of the nuclear balance vis-a-vis our principal adversary has been the subject of intense but unresolved debate in the international security community for almost seven decades. Perspectives on this question underlie national security policies regarding potential unilateral reductions in strategic nuclear forces, the imbalance of nonstrategic nuclear weapons in Europe, nuclear crisis management, nuclear proliferation, and nuclear doctrine.
The overwhelming majority of past studies of the role of the nuclear balance in nuclear crisis evolution and outcome have been qualitative and focused on the relative importance of the nuclear balance and national resolve. Some recent analyses have invoked statistical methods, however, these quantitative studies have generated intense controversy because of concerns with analytic rigor. We apply a multi-disciplinary approach that combines historical case study, international relations theory, and appropriate statistical analysis. This approach results in defensible findings on causal mechanisms that regulate nuclear crisis resolution. Such findings should inform national security policy choices facing the Trump administration.
|
Dr. Jane Pinelis Johns Hopkins University Applied Physics Lab or JHU |
Breakout |
| 2017 |
| Communication in Statistics & the Five Hardest Concepts | Dr. Jennifer Van-Mellekom Virginia Tech |
Breakout | 2017 | |
| Uncertainty Quantification: What is it and Why it is Important to Test, Evaluation, and Modeling and Simulation in Defense and Aerospace
Abstract
Uncertainty appears in many aspects of systems design including stochastic design parameters, simulation inputs, and forcing functions. Uncertainty Quantification (UQ) has emerged as the science of quantitative characterization and reduction of uncertainties in both simulation and test results. UQ is a multidisciplinary field with a broad base of methods including sensitivity analysis, statistical calibration, uncertainty propagation, and inverse analysis. Because of their ability to bring greater degrees of confidence to decisions, uncertainty quantification methods are playing a greater role in test, evaluation, and modeling and simulation in defense and aerospace. The value of UQ comes with better understanding of risk from assessing the uncertainty in test and modeling and simulation results.
The presentation will provide an overview of UQ and then discuss the use of some advanced statistical methods, including DOEs and emulation for multiple simulation solvers and statistical calibration, for efficiently quantifying uncertainties. These statistical methods effectively link test, evaluation and modeling and simulation by coordinating the valuation of uncertainties, simplifying verification and validation activities.
|
Dr. Peter Qian University of Wisconsin and SmartUQ |
Breakout |
| 2017 |
| Model Uncertainty and its Inclusion in Testing Results
Abstract
Answers to real world questions are often based on the use of judiciously chosen mathematical/statistical/physical models. In particular, assessment of failure probabilities of physical systems rely heavily on such models. Since no model describes the real world exactly, sensitivity analyses are conducted to examine influences of (small) perturbations of an assumed model. In this talk we present a structured approach, using an “Assumptions Lattice” and corresponding “Uncertainty Pyramid”, for transparently conveying the influence of various assumptions on analysis conclusions. We illustrate this process in the context of a simple multicomponent system.
|
Dr. Stever Lund NIST |
Breakout |
| 2017 |
| VV&UQ – Uncertainty Quantification for Model-Based Engineering of DoD Systems
Abstract
The US Army ARDEC has recently established an initiative to integrate statistical and probabilistic techniques into engineering modeling and simulation (M&S) analytics typically used early in the design lifecycle to guide technology development. DOE-driven Uncertainty Quantification techniques, including statistically rigorous model verification and validation (V&V) approaches, enable engineering teams to identify, quantify, and account for sources of variation and uncertainties in design parameters, and identify opportunities to make technologies more robust, reliable, and resilient earlier in the product’s lifecycle. Several recent armament engineering case studies – each with unique considerations and challenges – will be discussed.
|
Mr. Douglas Ray US Army RDECOM ARDEC |
Breakout |
| 2017 |
| VV&UQ – Uncertainty Quantification for Model-Based Engineering of DoD Systems
Abstract
The US Army ARDEC has recently established an initiative to integrate statistical and probabilistic techniques into engineering modeling and simulation (M&S) analytics typically used early in the design lifecycle to guide technology development. DOE-driven Uncertainty Quantification techniques, including statistically rigorous model verification and validation (V&V) approaches, enable engineering teams to identify, quantify, and account for sources of variation and uncertainties in design parameters, and identify opportunities to make technologies more robust, reliable, and resilient earlier in the product’s lifecycle. Several recent armament engineering case studies – each with unique considerations and challenges – will be discussed.
|
Ms. Melissa Jablonski US Army |
Breakout |
| 2017 |
| Data Visualization
Abstract
Teams of people with many different talents and skills work together at NASA to
improve our understanding of our planet Earth, our Sun and solar system, and the
Universe. The Earth System is made up of complex interactions and dependencies
of the solar, oceanic, terrestrial, atmospheric, and living components. Solar storms
have been recognized as a cause of technological problems on Earth since the invention
of the telegraph in the 19th century. Solar flares, coronal holes, and coronal mass
ejections (CME’s) can emit large bursts of radiation, high speed electrons and
protons, and other highly energetic particles that are released from the sun, and are
sometimes directed at Earth. These particles and radiation can damage satellites in space,
shutdown power grids on earth, cause GPS outages, and have serious health concerns to humans
flying at high altitudes on earth, as well as astronauts in space. NASA builds and operates a fleet
of satellites to study the sun and a fleet of satellites and aircraft to observe the Earth system.
NASA’s Computer Models combine the observations with numerical models, to understand
how these systems work. Using satellite observations alongside computer models we can combine
many pieces of information to form a coherent view of Earth and the Sun. NASA research helps us
understand how processes combine to affect life on Earth: this includes severe weather, health,
changes in climate, and space weather. The Scientific Visualization Studio wants you to learn
about NASA programs through visualization. The SVS works closely with scientists
in the creation of data visualizations, animations, and images in order to promote
a greater understanding of Earth and Space Science research activities at NASA
and within the academic research community supported by NASA.
|
Ms. Lori Perklins NASA |
Breakout | 2017 | |
| Big Data, Big Think
Abstract
The NASA Big Data, Big Think team jump-starts coordination, strategy, and progress for NASA applications of Big Data Analytics techniques, fosters collaboration and teamwork among centers and improves agency-wide understanding of Big Data research techniques & technologies and their application to NASA mission domains. The effort brings the Agency’s Big Data community together and helps define near term projects and leverages expertise throughout the agency. This presentation will share examples of Big Data activities from the Agency and discuss knowledge areas and experiences, including data management, data analytics and visualization.
|
Mr. Robert Beil NASA |
Breakout |
| 2017 |
| Project Data Flow Is an Engineered System
Abstract
Data within a project, investigation or test series are often seen as a bunch of numbers that were produced. While this is part of the story, it forgets the most important part: the data’s users. This more powerful process begins with early focus on planning, executing and managing data flow within a test or project as a system, treating each handoff between internal and external stakeholders a system interface. This presentation will persuade you why data production should be replaced by the idea of a data supply chain focused on goals and customers. The presenter will outline how this could be achieved in your team. The talk is aimed at not only project and data managers, but also team members who produce or use data. Retooling team thinking and processes along these lines will help communication, facilitate availability, display and understanding of data by any stakeholder, make data verification, validation and analysis easier, and help keep team members focused on what is necessary and important: solving the problem at hand.
|
Mr. Ken Johnson NASA |
Breakout |
| 2017 |
| Comparing Experimental Designs
Abstract
This tutorial will show how to compare and choose experimental designs based on multiple criteria. Answers to questions like “Which Design of Experiments (DOE) is better/best?” will be answered by looking at both data and graphics that show the relative performance of the designs based on multiple criteria, including; power of the designs for different model terms, how well the designs minimize predictive variance across the design space, to what level are model terms confounded or correlated, what are the relative efficiencies that measure how well coefficients are estimated or how well predictive variance is minimized. Many different case studies of screening, response surface, and screening augmented to response surface designs will be compared. Designs with both continuous and categorical factors, and with constraints on the experimental region will also be compared.
|
Dr. Tom Donnelly JMP |
Breakout |
| 2017 |
| Testing Autonomous Systems
Abstract
Autonomous robotic systems (hereafter referred to simply as autonomous systems) have attracted interest in recent years as capabilities improve to operate in unstructured, dynamic environments without continuous human guidance. Acquisition of autonomous systems potentially decrease personnel costs and provide a capability to operate in dirty, dull, or dangerous mission segments or achieve greater operational performance. Autonomy enables a particular action of a system to be automatic or, within programmed boundaries, self-governing. For our purposes, autonomy is defined as the system having a set of intelligence-based capabilities (i.e., learned behaviors) that allows it to respond to situations that were not pre-programmed or anticipated (i.e. learning-based responses) prior to system deployment. Autonomous systems have a degree of self-governance and self-directed behavior, possibly with a human’s proxy for decisions. Because of these intelligence-based capabilities, autonomous systems pose new challenges in conducting test and evaluation that assures adequate performance, safety, and cybersecurity outcomes. We propose an autonomous systems architecture concept and map the elements of a decision theoretic view of a generic decision problem to the components of this architecture. These models offer a foundation for developing a decision-based, common framework for autonomous systems. We also identify some of the various challenges faced by the Department of Defense (DoD) test and evaluation community in assuring the behavior of autonomous systems as well as test and evaluation requirements, processes, and methods needed to address these challenges.
|
Dr. Darryl Ahner Director AFIT |
Breakout |
| 2018 |
| Screening Experiments with Partial Replication
Abstract
Small screening designs are frequently used in the initial stages of experimentation with the goal of identifying important main effects as well as to gain insight on potentially important two-factor interactions. Commonly utilized experimental designs for screening (e.g., resolution III or IV two-level fractional factorials, Plackett-Burman designs, etc.) are unreplicated and as such, provide no unbiased estimate of experimental error. However, if statistical inference is considered an integral part of the experimental analysis, one view is that inferential procedures should be performed using the unbiased pure error estimate. As full replication of an experiment may be quite costly, partial replication offers an alternative for obtaining a model independent error estimate. Gilmour and Trinca (2012, Applied Statistics) introduce criteria for the design of optimal experiments for statistical inference (providing for the optimal selection of replicated design points). We begin with an extension of their work by proposing a Bayesian criterion for the construction of partially replicated screening designs with less dependence on an assumed model. We then consider the use of the proposed criterion within the context of multi-criteria design selection where estimation and protection against model misspecification are considered. Insights for analysis and model selection in light of partial replication will be provided.
|
Mr. David Edwards Virginia Commonwealth University |
Breakout |
| 2018 |
| What is Bayesian Experimental Design?
Abstract
In an experiment with a single factor with three levels, treatments A, B, and C, a single treatment is to be applied to each of several experimental units selected from some set of units. The response variable is continuous, and differences in its value show the relative effectiveness of the treatments. An experimental design will dictate which treatment is applied to what units. Since differences in the response variable are used to judge differences between treatments, the most important goal of the design is to prevent the treatment effect being masked by some unrelated property of the experimental units. Second important function of the design is to ensure power, that is, that if the treatments are not equally effective, the differences in the response variable are likely to be larger than background noise. Classical experimental design theory uses three principles: replication, randomization, and blocking, to produce an experimental design. Replication refers to how many units are used, blocking is a possible grouping of the units to reduce between unit heterogeneity, and randomization governs the assignment of units to treatment. Classical experimental designs are balanced as much as possible, that is, the three treatments are applied the same number of times, in each potential block of units. Bayesian experimental design aims to make use of additional related information, often called prior information, to produce a design. The information may be in the form of related experimental results, for example, treatments A and B may have been previously studied. It could be additional information about the experimental units, or about the response variable. This additional information could be used to change the usual blocking, to reduce the number of units assigned to treatments A and B compared to C, and/or reduce the total number of units needed to ensure power. This talk aims to explain Bayesian design concepts and illustrate them on realistic examples.
|
Blaza Toman Statistical Engineering Division, NIST |
Breakout |
| 2018 |
| DOE and Test Automation for System of Systems TE
Abstract
Rigorous, efficient and effective test science techniques are individually taking hold in many software centric DoD acquisition programs, both in developmental and operational test regimes. These techniques include agile software development, cybersecurity test and evaluation (T&E), design and analysis of experiments and automated software testing. Many software centric programs must also be tested together with other systems to demonstrate they can be successfully integrated into a more complex systems of systems. This presentation focuses on the two test science disciplines of designed experiments (DOE) and automated software testing (AST) and describes how they can be used effectively and leverage one another in planning for and executing a system of systems test strategy. We use the Navy’s Distributed Common Ground System as an example.
|
Jim Simpson JK Analytics |
Breakout |
| 2018 |
| Building A Universal Helicopter Noise Model Using Machine Learning
Abstract
Helicopters serve a number of useful roles within the community; however, community acceptance of helicopter operations is often limited by the resulting noise. Because the noise characteristics of helicopters depend strongly on the operating condition of the vehicle, effective noise abatement procedures can be developed for a particular helicopter type, but only when the noisy regions of the operating envelope are identified. NASA Langley Research Center—often in collaboration with other US Government agencies, industry, and academia—has conducted noise measurements for a wide variety of helicopter types, from light commercial helicopters to heavy military utility helicopters. While this database is expansive, it covers only a fraction of helicopter types in current commercial and military service and was measured under a limited set of ambient conditions and vehicle configurations. This talk will describe a new “universal” helicopter noise model suitable for planning helicopter noise abatement procedures. Modern machine learning techniques will be combined with the principle of nondimensionalization and applied to NASA’s helicopter noise data in order to develop a model capable of estimating the noisy operating states of any conventional helicopter under any specific ambient conditions and vehicle configurations.
|
Eric Greenwood Aeroacoustics Branch |
Breakout |
| 2018 |
| Machine Learning to Assess Piolts’ Cognitive State
Abstract
The goal of the Crew State Monitoring (CSM) project is to use machine learning models trained with physiological data to predict unsafe cognitive states in pilots such as Channelized Attention (CA) and Startle/Surprise (SS). These models will be used in a real-time system that predicts a pilot’s mental state every second, a tool that can be used to help pilots recognize and recover from these mental states. Pilots wore different sensors that collected physiological data such as a 20-channel electroencephalography (EEG), respiration, and galvanic skin response (GSR). Pilots performed non-flight benchmark tasks designed to induce these states, and a flight simulation with “surprising” or “channelizing” events. The team created a pipeline to generate pilot-dependent models that trains on benchmark data, tune on a portion of a flight task, and be deployed onto the remaining flight task. The model is a series of anomaly-detection based ensembles, where each ensemble focuses on predicting a single state. Ensembles were comprised of several anomaly detectors such as One Class SVMs, each focusing on a different subset of sensor data. We will discuss the performance of these models, as well as the ongoing research generalizing models across pilots and improving accuracy.
|
Tina Heinich Computer Engineer, OCIO Data Science Team AST, Data Systems |
Breakout |
| 2018 |
| CYBER Penetration Testing and Statistical Analysis in DT&E
Abstract
Reconnaissance, footprinting, and enumeration are critical steps in the CYBER penetration testing process because if these steps are not fully and extensively executed, the information available for finding a system’s vulnerabilities may be limited. During the CYBER testing process, penetration testers often find themselves doing the same initial enumeration scans over and over for each system under test. Because of this, automated scripts have been developed that take these mundane and repetitive manual steps and perform them automatically with little user input. Once automation is present in the penetration testing process, Scientific Test and Analysis Techniques (STAT) can be incorporated. By combining automation and STAT in the CYBER penetration testing process, Mr. Tim McLean at Marine Corps Tactical Systems Support Activity (MCTSSA) coined a new term called CYBERSTAT. CYBERSTAT is applying scientific test and analysis techniques to offensive CYBER penetration testing tools with an important realization that CYBERSTAT assumes the system under test is the offensive penetration test tool itself. By applying combinatorial testing techniques to the CYBER tool, the CYBER tool’s scope is expanded beyond “one at a time” uses as the combinations of the CYBER tool’s capabilities and options are explored and executed as test cases against the target system. In CYBERSTAT, the additional test cases produced by STAT can be run automatically using scripts. This talk will show how MCTSSA is preparing to use CYBERSTAT in the Developmental Test and Evaluation process of USMC Command and Control systems.
|
Timothy McLean |
Breakout |
| 2018 |
| Method for Evaluating the Quality of Cybersecurity Defenses
Abstract
This presentation discusses a methodology to use knowledge of cyber attacks and defender responses from operational assessments to gain insights into the defensive posture and to inform a strategy for improvement. The concept is to use the attack thread as the instrument to probe and measure the detection capability of the cyber defenses. The data enable a logistic regression approach to provide a quantitative basis for the analysis and recommendations.
|
Dr, Shawn Whetstone Research Staff Member IDA |
Breakout |
| 2018 |
| Metrics to Characterize Temporal patterns in Lifespans of Artifacts
Abstract
Over the past decade, uncertainty quantification has become an integral part of engineering design and analysis. Both NASA and the DoD are making significant investments to advance the science of uncertainty quantification, increase the knowledge base, and strategically expanding its use. This increased use of uncertainty based results improves investment strategies and decision making. However, in complex systems, many challenges still exist when dealing with uncertainty in cases that have sparse, unreliable, poorly understood, and/or conflicting data. Often times, assumptions are made regarding the statistical nature of data that may not be well grounded and the impact of those assumptions is not well understood, which can lead to ill-informed decision making. This talk will focus on the quantification of uncertainty when both well characterized, aleatory, and not well known, epistemic, uncertainty sources exist. Particular focus is given to the treatment and management of epistemic uncertainty. A summary of non-probabilistic methods will be presented along with the propagation of mixed uncertainty and optimization under uncertainty. A discussion of decision making under uncertainty is also included to illustrate the use of uncertainty quantification.
|
Soumyo MoitraNA Software Engineering Institute Carnegie Mellon Univeristy |
Breakout |
| 2018 |
| Uncertainty Quantification with Mixed Uncertainty Sources
Abstract
Over the past decade, uncertainty quantification has become an integral part of engineering design and analysis. Both NASA and the DoD are making significant investments to advance the science of uncertainty quantification, increase the knowledge base, and strategically expanding its use. This increased use of uncertainty based results improves investment strategies and decision making. However, in complex systems, many challenges still exist when dealing with uncertainty in cases that have sparse, unreliable, poorly understood, and/or conflicting data. Often times, assumptions are made regarding the statistical nature of data that may not be well grounded and the impact of those assumptions is not well understood, which can lead to ill-informed decision making. This talk will focus on the quantification of uncertainty when both well characterized, aleatory, and not well known, epistemic, uncertainty sources exist. Particular focus is given to the treatment and management of epistemic uncertainty. A summary of non-probabilistic methods will be presented along with the propagation of mixed uncertainty and optimization under uncertainty. A discussion of decision making under uncertainty is also included to illustrate the use of uncertainty quantification.
|
Tom West |
Breakout | 2018 | |
| System Level Uncertainty Quanification for Low-Boom Supersonic Flight Vehicles
Abstract
Under current FAA regulations, civilian aircraft may not operate at supersonic speeds over land. However, over the past few decades, there have been renewed efforts to invest in technologies to mitigate sonic boom from supersonic aircraft through advances in both vehicle design and sonic boom prediction. NASA has heavily invested in tools and technologies to enable commercial supersonic flight and currently has several technical challenges related to sonic boom reduction. One specific technical challenge relates to the development of tools and methods to predict, under uncertainty, the noise on the ground generated by an aircraft flying at supersonic speeds. In attempting to predict ground noise, many factors from multiple disciplines must be considered. Further, classification and treatment of uncertainties in coupled systems, mutlifidelity simulations, experimental data, and community responses are all concerns in system level analysis of sonic boom prediction. This presentation will introduce the various methodologies and techniques utilized for uncertainty quantification with a focus on the build up to system level analysis. An overview of recent research activities and case studies investigating the impact of various disciplines and factors on variance in ground noise will be discussed.
|
Ben Phillips |
Breakout | 2018 | |
| Uncertainty Quantification and Analysis at The Boeing Company
Abstract
The Boeing Company is assessing uncertainty quantification methodologies across many phases of aircraft design in order to establish confidence in computational fluid dynamics-based simulations of aircraft performance. This presentation provides an overview of several of these efforts. First, the uncertainty in aerodynamic performance metrics of a commercial aircraft at transonic cruise due to turbulence model and flight condition variability is assessed using 3D CFD with non-intrusive polynomial chaos and second order probability. Second, a sample computation of uncertainty in increments is performed for an engineering trade study, leading to the development of a new method for propagating input-uncontrolled uncertainties as well as input-controlled uncertainties. This type of consideration is necessary to account for variability associated with grid convergence on different configurations, for example. Finally, progress toward applying the computed uncertainties in forces and moments into an aerodynamic database used for flight simulation will be discussed. This approach uses a combination of Gaussian processes and multiple-fidelity Kriging meta-modeling to synthesize the required data.
|
John Schaefer Sandia National Labortories |
Breakout |
| 2018 |
| Interface design for analysts in a data and analysis-rich environment
Abstract
Increasingly humans will rely on the outputs of our computational partners to make sense of the complex systems in our world. To be employed, the statistical and algorithmic analysis tools that are deployed to analysts’ toolboxes must afford their proper use and interpretation. Interface design for these tool users should provide decision support appropriate for the current stage of sensemaking. Understanding how users build, test, and elaborate their mental models of complex systems can guide the development of robust interfaces.
|
Karin Butler Sandia National Labortories |
Breakout |
| 2018 |
| Asparagus is the most articulate vegetable ever
Abstract
During the summer of 2001, Microsoft launched Windows XP, which was lauded by many users as the most reliable and usable operating system at the time. Miami Herald columnist, Dave Berry, responded to this praise by stating that “this is like saying asparagus is the most articulate vegetable ever.” Whether you agree or disagree with Dave Berry, these users’ reactions are relative (to other operating systems and to other past and future versions). This is due to an array of technological factors that have facilitated human-system improvements. Automation is often cited as improving human-system performance across many domains. It is true that when the human and automation are aligned, performance improves. But, what about the times that this is not the case? This presentation will describe the myths and facts about human-system performance and increasing levels of automation through examples of human-system R&D conducted on a satellite ground system. Factors that affect human-system performance and a method to characterize mission performance as it relates to increasing levels of automation will also be discussed.
|
Kerstan Cole |
Breakout | 2018 | |
| Mitigating Pilot Disorientation with Synthetic Vision Displays
Abstract
Loss of control in flight has been a leading cause of accidents and incidents in commercial aviation worldwide. The Commercial Aviation Safety Team (CAST) requested studies on virtual day-visual meteorological conditions displays, such as synthetic vision, in order to combat loss of control. Over the last four years NASA has conducted a series of experiments evaluating the efficacy of synthetic vision displays for increased spatial awareness. Commercial pilots with various levels of experience from both domestic and international airlines were used as subjects. This presentation describes the synthetic vision research and how pilot subjects affected experiment design and statistical analyses.
|
Kathryn Ballard NASA |
Breakout |
| 2018 |
| Sage III Seu Statistical Analysis Model
Abstract
The Stratospheric Aerosol and Gas Experiment (SAGE III) aboard the International Space Station (ISS) was experiencing a series of anomalies called Single Event Upsets (SEUs). Booz Allen Hamilton was tasked with conducting a statistical analysis to model the incidence of SEUs in the SAGE III equipment aboard the ISS. The team identified factors correlated with SEU incidences, set up a model to track degradation of Sage III, and showed current and past probabilities as a function of the space environment. The space environment of SAGE III was studied to identify possible causes of SEUs. The analysis revealed variables most correlated with the anomalies, including solar wind strength, solar and geomagnetic field behavior, and location/orientation of the ISS, sun, and moon. The data was gathered from a variety of sources including US government agencies, foreign and domestic academic centers, and state-of-the-art simulation algorithms and programs. Logistic regression was used to analyze SEUs and gain preliminary results. The data was divided into small time intervals to approximate independence and allow logistic regression. Due to the rarity of events the initial model results were based on few SEUs. The team set up a Graphical User Interface (GUI) program to automatically analyze new data as it became available to the SAGE III team. A GUI was built to allow the addition of more data over the life of the SAGE III mission. As more SEU incidents occur and are entered into the model, its predictive power will grow significantly. The GUI enables the user to easily rerun the regression analysis and visualize its results to inform operational decision making.
|
Ray McCollum Booz Allen Hamilton |
Breakout |
| 2018 |
| Initial Validation of the Trust of Automated System Test
Abstract
Automated systems are technologies that actively select data, transform information, make decisions, and control processes. The U.S. military uses automated systems to perform search and rescue and reconnaissance missions, and to assume control of aircraft to avoid ground collision. Facilitating appropriate trust in automated systems is essential to improving the safety and performance of human-system interactions. In two studies, we developed and validated an instrument to measure trust in automated systems. In study 1, we demonstrated that the scale has a 2-factor structure and demonstrates concurrent validity. We replicated these results using an independent sample in study 2.
|
Dr. Heather Wojton Research Staff Member IDA |
Breakout |
| 2018 |
| Operational Evaluation of a Flight-deck Software Application
Abstract
Traffic Aware Strategic Aircrew Requests (TASAR) is a NASA-developed operational concept for flight efficiency and route optimization for the near-term airline flight deck. TASAR provides the aircrew with a cockpit automation tool that leverages a growing number of information sources on the flight deck to make fuel- and time-saving route optimization recommendations while in route. In partnership with a commercial airline, a research prototype software that implements TASAR has been installed on three aircraft to enable the evaluation of this software in operational use. During the flight trials, data are being collected to quantify operational performance, which will enable NASA to improve algorithms and enhance functionality in the software based on real-world user experience. This presentation highlights statistical challenges and discusses lessons learned during the initial stages of the operational evaluation.
|
Sara Wilson NASA |
Breakout |
| 2018 |
| Comparing M&S Output to Live Test Data: A Missile System Case Study
Abstract
In the operational testing of DoD weapons systems, modeling and simulation (M&S) is often used to supplement live test data in order to support a more complete and rigorous evaluation. Before the output of the M&S is included in reports to decision makers, it must first be thoroughly verified and validated to show that it adequately represents the real world for the purposes of the intended use. Part of the validation process should include a statistical comparison of live data to M&S output. This presentation includes an example of one such validation analysis for a tactical missile system. In this case, the goal is to validate a lethality model that predicts the likelihood of destroying a particular enemy target. Using design of experiments, along with basic analysis techniques such as the Kolmogorov-Smirnov test and Poisson regression, we can explore differences between the M&S and live data across multiple operational conditions and quantify the associated uncertainties.
|
Dr. Kelly Avery Reasearch Staff member IDA |
Breakout |
| 2018 |
| NASA’s Human Exploration Research Analog (HERA): An analog mission for isolation, confinement, and remote conditions in space exploration scenarios
Abstract
Shelley Cazares served as a crewmember of the 14th mission of NASA’s Human Exploration Research Analog (HERA). In August 2017, Dr. Cazares and her three crewmates were enclosed in an approximately 600-sq. ft. simulated spacecraft for an anticipated 45 days of confined isolation at Johnson Space Center, Houston, TX. In preparation for long-duration missions to Mars in the 2030s and beyond, NASA seeks to understand what types of diets, habitats, and activities can keep astronauts healthy and happy on deep space voyages. To collect this information, NASA is conducting several analog missions simulating the conditions astronauts face in space. HERA is a set of experiments to investigate the effects of isolation, confinement, and remote conditions in space exploration scenarios. Dr. Cazares will discuss the application procedure, the pre-mission training process, the life and times inside the habitat during the mission, and her crew’s emergency evacuation from the habitat due to the risk of rising floodwaters in Hurricane Harvey.
|
Shelley Cazares NASA |
Breakout | 2018 | |
| Reliability Fundamentals and Analysus Lessons Learned
Abstract
Although reliability analysis is a part of Operational Test and Evaluation, it is uncommon for analysts to have a background in reliability theory or experience applying it. This presentation highlights some lessons learned from reliability analysis conducted on several AFOTEC test programs. Topics include issues related to censored data, limitations and alternatives to using the exponential distribution, and failure rate analysis using test data.
|
Dan Telford AFOTEC |
Breakout |
| 2018 |
| Design and Analysis of Nonlinear Models for the Mars 2020 Rover
Abstract
The Mars Rover 2020 team commonly faces nonlinear behavior across the test program that is often closely related to the underlying physics. Classical and newer response surface designs do well with quadratic approximations while space filling designs have proven useful for modeling & simulation of complex surfaces. This talk specifically covers fitting nonlinear equations based on engineering functional forms as well as sigmoid and exponential decay curves. We demonstrate best practices on how to design and augment nonlinear designs using the Bayesian D-Optimal Criteria. Several examples, to include drill bit degradation, illustrate the relative ease of implementation with popular software and the utility of these methods.
|
Jim Wisnowski |
Breakout |
| 2018 |
| Cases of Second-Order Split-Plot Designs
Abstract
The fundamental principles of experiment design are factorization, replication, randomization, and local control of error. In many industries, however, departure from these principles is commonplace. Often in our experiments complete randomization is not feasible because the factor level settings are hard, impractical, or inconvenient to change or the resources available to execute under homogeneous conditions are limited. These restrictions in randomization lead to split-plot experiments. We are also often interested in fitting second-order models leading to second-order split-plot experiments. Although response surface methodology has grown tremendously since 1951, the lack of alternatives for second-order split-plots remains largely unexplored. The literature and textbooks offer limited examples and provide guidelines that often are too general. This deficit of information leaves practitioners ill prepared to face the many roadblocks associated with these types of designs. This presentation provides practical strategies to help practitioners in dealing with second-order split-plot and by extension, split-split-plot experiments, including an innovative approach for the construction of a response surface design referred to as second-order sub-array Cartesian product split-plot design. This new type of design, which is an alternative to other classes of split-plot designs that are currently in use in defense and industrial applications, is economical, has a low prediction variance of the regression coefficients, and low aliasing between model terms. Based on an assessment using well accepted key design evaluation criterion, second-order sub-array Cartesian product split-plot designs perform as well as historical designs that have been considered standards up to this point.
|
Dr. Luis Cortes MITRE |
Breakout |
| 2018 |
| The Development and Execution of Split-Plot Designs In Navy Operational Test and Evaluation: A Practitioner’s Perspective
Abstract
Randomization is one of the basic principles of experimental design and the associated statistical methods. In Navy operational testing, complete randomization is often not possible due to scheduling or execution constraints. Given these constraints, operational test designers often utilize split-plot designs to accommodate the hard-to-change nature of various factors of interest. Several case studies will be presented to provide insight into the challenges associated with Navy operational test design and execution.
|
Stargel Doane |
Breakout | 2018 | |
| B-52 Radar Modernization Test Design Considerations
Abstract
Inherent system processes, restrictions on collection, or cost may impact the practical execution of an operational test. This study presents the use of blocking and split-plot designs when complete randomization is not feasible in operational test. Specifically, the USAF B-52 Radar Modernization Program test design is used to present tradeoffs of different design choices and the impacts of those choices on cost, operational relevance, and analytical rigor.
|
Stuart Corbett AFOTEC |
Breakout |
| 2018 |
| B-52 Radar Modernization Test Design Considerations
Abstract
Inherent system processes, restrictions on collection, or cost may impact the practical execution of an operational test. This study presents the use of blocking and split-plot designs when complete randomization is not feasible in operational test. Specifically, the USAF B-52 Radar Modernization Program test design is used to present tradeoffs of different design choices and the impacts of those choices on cost, operational relevance, and analytical rigor.
|
Joseph Maloney AFOTEC |
Breakout |
| 2018 |
| Experimental Design of a Unique Force Measurement System Calibration
Abstract
Aerodynamic databases for space flight vehicles rely on wind-tunnel tests utilizing precision force measurement systems (FMS). Recently, NASA’s Space Launch System (SLS) program has conducted numerous wind-tunnel testing. This presentation will focus on the calibration of a unique booster FMS through the use of design of experiments (DoE) and regression modeling. Utilization of DoE resulted in a sparse, time-efficient, design with results exceeding researcher’s expectations.
|
Ken Toro |
Breakout | 2018 | |
| Application of Design of Experiments to a Calibration of the National Transonic Facility
Abstract
Recent work at the National Transonic Facility (NTF) at the NASA Langley Research Center has shown that a substantial reduction in freestream pressure fluctuations can be achieved by positioning the moveable model support walls and plenum re-entry flaps to choke the flow just downstream of the test section. This choked condition reduces the upstream propagation of disturbances from the diffuser into the test section, resulting in improved Mach number control and reduced freestream variability. The choked conditions also affect the Mach number gradient and distribution in the test section, so a calibration experiment was undertaken to quantify the effects of the model support wall and re-entry flap movements on the facility freestream flow using a centerline static pipe. A design of experiments (DOE) approach was used to develop restricted-randomization experiments to determine the effects of total pressure, reference Mach number, model support wall angle, re-entry flap gap height, and test section longitudinal location on the centerline static pressure and local Mach number distributions for a reference Mach number range from 0.7 to 0.9. Tests were conducted using air as the test medium at a total temperature of 120 °F as well as for gaseous nitrogen at cryogenic total temperatures of -50, -150, and -250 °F. The resulting data were used to construct quadratic polynomial regression models for these factors using a Restricted Maximum Likelihood (REML) estimator approach. Independent validation data were acquired at off-design conditions to check the accuracy of the regression models. Additional experiments were designed and executed over the full Mach number range of the facility (0.2 £ Mref £ 1.1) at each of the four total temperature conditions, but with the model support walls and re-entry flaps set to their nominal positions, in order to provide calibration regression models for operational experiments where a choked condition downstream of the test section is either not feasible or not required. This presentation focuses on the design, execution, analysis, and results for the two experiments performed using air at a total temperature of 120 °F. Comparisons are made between the regression model output and validation data, as well as the legacy NTF calibration results, and future work is discussed.
|
Matt Rhode NASA |
Breakout |
| 2018 |
| Application of Design of Experiments to a Calibration of the National Transonic Facility
Abstract
Recent work at the National Transonic Facility (NTF) at the NASA Langley Research Center has shown that a substantial reduction in freestream pressure fluctuations can be achieved by positioning the moveable model support walls and plenum re-entry flaps to choke the flow just downstream of the test section. This choked condition reduces the upstream propagation of disturbances from the diffuser into the test section, resulting in improved Mach number control and reduced freestream variability. The choked conditions also affect the Mach number gradient and distribution in the test section, so a calibration experiment was undertaken to quantify the effects of the model support wall and re-entry flap movements on the facility freestream flow using a centerline static pipe. A design of experiments (DOE) approach was used to develop restricted-randomization experiments to determine the effects of total pressure, reference Mach number, model support wall angle, re-entry flap gap height, and test section longitudinal location on the centerline static pressure and local Mach number distributions for a reference Mach number range from 0.7 to 0.9. Tests were conducted using air as the test medium at a total temperature of 120 °F as well as for gaseous nitrogen at cryogenic total temperatures of -50, -150, and -250 °F. The resulting data were used to construct quadratic polynomial regression models for these factors using a Restricted Maximum Likelihood (REML) estimator approach. Independent validation data were acquired at off-design conditions to check the accuracy of the regression models. Additional experiments were designed and executed over the full Mach number range of the facility (0.2 £ Mref £ 1.1) at each of the four total temperature conditions, but with the model support walls and re-entry flaps set to their nominal positions, in order to provide calibration regression models for operational experiments where a choked condition downstream of the test section is either not feasible or not required. This presentation focuses on the design, execution, analysis, and results for the two experiments performed using air at a total temperature of 120 °F. Comparisons are made between the regression model output and validation data, as well as the legacy NTF calibration results, and future work is discussed.
|
Matt Bailey Jacobs Technology Inc |
Breakout |
| 2018 |
| The Use of DOE vs OFAT in the Calibration of AEDC Wind Tunnels
Abstract
The use of statistically rigorous methods to support testing at Arnold Engineering Development Complex (AEDC) has been an area of focus in recent years. As part of this effort, the use of Design of Experiments (DOE) has been introduced for calibration of AEDC wind tunnels. Historical calibration efforts used One- Factor-at-a-Time (OFAT) test matrices, with a concentration on conditions of interest to test customers. With the introduction of DOE, the number of test points collected during the calibration decreased, and were not necessary located at historical calibration points. To validate the use of DOE for calibration purposes, the 4-ft Aerodynamic Wind Tunnel 4T was calibrated using both DOE and OFAT methods. The results from the OFAT calibration were compared to model developed from the DOE data points and it was determined that the DOE model sufficiently captured the tunnel behavior within the desired levels of uncertainty. DOE analysis also showed that within Tunnel 4T, systematic errors are insignificant as indicated by agreement noted between the two methods. Based on the results of this calibration, a decision was made to apply DOE methods to future tunnel calibrations, as appropriate. The development of the DOE matrix in Tunnel 4T required the consideration of operational limitations, measurement uncertainties, and differing tunnel behavior over the performance map. Traditional OFAT methods allowed tunnel operators to set conditions efficiently while minimizing time consuming plant configuration changes. DOE methods, however, require the use of randomization which had the potential to add significant operation time to the calibration. Additionally, certain tunnel parameters, such as variable porosity, are only of interest in a specific region of the performance map. In addition to operational concerns, measurement uncertainty was an important consideration for the DOE matrix. At low tunnel total pressures, the uncertainty in the Mach number measurements increase significantly. Aside from introducing non-constant variance into the calibration model, the large uncertainties at low pressures can increase overall uncertainty in the calibration in high pressure regions where the uncertainty would otherwise be lower. At high pressures and transonic Mach numbers, low Mach number uncertainties are required to meet drag count uncertainty requirements. To satisfy both the operational and calibration requirements, the DOE matrix was divided into multiple independent models over the tunnel performance map. Following the Tunnel 4T calibration, AEDC calibrated the Propulsion Wind Tunnel 16T, Hypersonic Wind Tunnels B and C, and the National Full-Scale Aerodynamics Complex (NFAC). DOE techniques were successfully applied to the calibration of Tunnel B and NFAC, while a combination of DOE and OFAT test methods were used in Tunnel 16T because of operational and uncertainty requirements over a portion of the performance map. Tunnel C was calibrated using OFAT because of operational constraints. The cost of calibrating these tunnels has not been significantly reduced through the use of DOE, but the characterization of test condition uncertainties is firmly based in statistical methods.
|
Rebecca Rought AEDC/TSTA |
Breakout |
| 2018 |
| Initial Investigation into the Psychoacoustic Properties of Small Unmanned Aerial System Noise
Abstract
For the past several years, researchers at NASA Langley have been engaged in a series of projects to study the degree to which existing facilities and capabilities, originally created for work on full-scale aircraft, are extensible to smaller scales – those of the small unmanned aerial systems (sUAS, also UAVs and, colloquially, `drones’) that have been showing up in the nation’s airspace. This paper follows an effort that has led to an initial human-subject psychoacoustic test regarding the annoyance generated by sUAS noise. This effort spans three phases: 1. the collection of the sounds through field recordings, 2. the formulation and execution of a psychoacoustic test using those recordings, 3. the analysis of the data from that test. The data suggests a lack of parity between the noise of the recorded sUAS and that of a set of road vehicles that were also recorded and included in the test, as measured by a set of contemporary noise metrics.
|
Andrew Chrisrian Structural Acoustics Branch |
Breakout |
| 2018 |
| Combining Human Factors Data and Models of Human Performance
Abstract
As systems and missions become increasingly complex, the roles of humans throughout the mission life cycle is evolving. In areas, such as maintenance and repair, hands-on tasks still dominate, however, new technologies have changed many tasks. For example, some critical human tasks have moved from manual control to supervisory control, often of systems at great distances (e.g., remotely piloting a vehicle, or science data collection on Mars). While achieving mission success remains the key human goal, almost all human performance metrics focus on failures rather than successes. This talk will examine the role of humans in creating mission success as well as new approaches for system validation testing needed to keep up with evolving systems and human roles.
|
Cynthia Null Technicial Fellow for Human Factors |
Breakout |
| 2018 |
| A Multi-method, Triangulation Approach to Operational Testing
Abstract
Humans are not produced in quality-controlled assembly lines, and we typically are much more variable than the mechanical systems we employ. This mismatch means that when characterizing the effectiveness of a system, the system must be considered in the context of its users. Accurate measurement is critical to this endeavor, yet while human variability is large, effort to reduce measurement error of those humans is relatively small. The following talk discusses the importance of using multiple measurement methods—triangulation—to reduce error and increase confidence when characterizing the quality of HSI. A case study from an operational test of an attack helicopter demonstrates how triangulation enables more actionable recommendations.
|
Dr. Daniel Porter Research Staff Member IDA |
Breakout |
| 2018 |
| Illustrating the Importance of Uncertainty Quantification (UQ) in Munitions Modeling
Abstract
The importance of the incorporation of Uncertainty Quantification (UQ) techniques into the design and analysis of Army systems is discussed. Relevant examples are presented where UQ would have been extremely useful. The intent of the presentation is to show the broad relevance of UQ and how, in the future, it will greatly improve the time to fielding and quality of developmental items.
|
Donald Calucci |
Breakout | 2018 | |
| Space-Filling Designs for Robustness Experiments
Abstract
To identify the robust settings of the control factors, it is very important to understand how they interact with the noise factors. In this article, we propose space-filling designs for computer experiments that are more capable of accurately estimating the control-by-noise interactions. Moreover, the existing space-filling designs focus on uniformly distributing the points in the design space, which are not suitable for noise factors because they usually follow non-uniform distributions such as normal distribution. This would suggest placing more points in the regions with high probability mass. However, noise factors also tend to have a smooth relationship with the response and therefore, placing more points towards the tails of the distribution is also useful for accurately estimating the relationship. These two opposing effects make the experimental design methodology a challenging problem. We propose optimal and computationally efficient solutions to this problem and demonstrate their advantages using simulated examples and a real industry example involving a manufacturing packing line.
|
Roshan Vengazhiyil |
Breakout | 2018 | |
| Introduction of Uncertainty Quantification and Industry Challenges
Abstract
Uncertainty is an inescapable reality that can be found in nearly all types of engineering analyses. It arises from sources like measurement inaccuracies, material properties, boundary and initial conditions, and modeling approximations. For example, the increasing use of numerical simulation models throughout industry promises improved design and insight at significantly lower costs and shorter timeframes than purely physical testing. However, the addition of numerical modeling has also introduced complexity and uncertainty to the process of generating actionable results. It has become not only possible, but vital to include Uncertainty Quantification (UQ) in engineering analysis. The competitive benefits of UQ include reduced development time and cost, improved designs, better understanding of risk, and quantifiable confidence in analysis results and engineering decisions. Unfortunately, there are significant cultural and technical challenges which prevent organizations from utilizing UQ methods and techniques in their engineering practice. This presentation will introduce UQ methodology and discuss the past and present strategies for addressing these challenges, making it possible to use UQ to enhance engineering processes with fewer resources and in more situations. Looking to the future, anticipated challenges will be discussed along with an outline of the path towards making UQ a common practice in engineering.
|
Peter Chien |
Breakout | 2018 | |
| The Effect of Extremes in Small Sample Size on Simple Mixed Models: A Comparison of Level-1 and Level-2 Size
Abstract
Mixed models are ideally suited to analyzing nested data from within-persons designs, designs that are advantageous in applied research. Mixed models have the advantage of enabling modeling of random effects, facilitating an accounting of the intra-person variation captured by multiple observations of the same participants and suggesting further lines of control to the researcher. However, the sampling requirements for mixed models are prohibitive to other areas which could greatly benefit from them. This simulation study examines the impact of small sample sizes in both levels of the model on the fixed effect bias, type I error, and power of a simple mixed model analysis. Despite the need for adjustments to control for type I error inflation, findings indicate that smaller samples than previously recognized can be used for mixed models under certain conditions prevalent in applied research. Examination of the marginal benefit of increases in sample subject and observation size provides applied researchers with guidance for developing mixed-model repeated measures designs that maximize power.
|
Dr. Kristina Carter Research Staff Member IDA |
Breakout |
| 2018 |
| Evaluating Deterministic Models of Time Series by Comparison to Observations
Abstract
A standard paradigm for assessing the quality of model simulations is to compare what these models produce to experimental or observational samples of what the models seek to predict. Often these comparisons are based on simple summary statistics, even when the objects of interest are time series. Here, we propose a method of evaluation through probabilities derived from tests of hypotheses that model-simulated and observed time sequences share common signals. The probabilities are based on the behavior of summary statistics of model output and observational data, over ensembles of pseudo-realizations. These are obtained by partitioning the original time sequences into signal and noise components, and using a parametric bootstrap to create pseudo-realizations of the noise. We demonstrate with an example from climate model evaluation for which this methodology was developed.
|
Amy Braverman Jet Propulsion Laboratory, California Institute of Technology |
Breakout |
| 2018 |
| Challenger Challenge: Pass-Fail Thinking Increases Risk Measurably
Abstract
Binomial (pass-fail) response metrics are more far more commonly used in test, requirements, quality and engineering than they need to be. In fact, there is even an engineering school of thought that they’re superior to continuous-variable metrics. This is a serious, even dangerous problem in aerospace and other industries: think the Space Shuttle Challenger accident. There are better ways. This talk will cover some examples of methods available to engineers and statisticians in common statistical software. It will not dig far into the mathematics of the methods, but will walk through where each method might be most useful and some of the pitfalls inherent in their use – including potential sources of misinterpretation and suspicion by your teammates and customers. The talk is geared toward engineers, managers and professionals in the –ilities who run into frustrations dealing with pass-fail data and thinking.
|
Ken Johnson Applied Statistician NASA Engineering and Safety Center |
Breakout |
| 2018 |
| Doppler Assisted Sensor Fusion for Tracking and Exploitation
Abstract
We have developed a new sensor fusion approach called Doppler Assisted Sensor Fusion (DASF), which pairs a range rate profile from one moving sensor with location accuracy with another range rate profile from another sensor with high location accuracy. This paring provides accurate identification, location, and tracking of moving emitters, with low association latency. The approach we use for data fusion is distinct from previous approaches. In the conventional approach, post detection data from the each sensor is overlaid with data from another sensor in an attempt to associate the data outputs. For the DASF approach the fusion is at the sensor level, the first sensor collects data and provides the standard identification in addition a unique emitter range rate profile. This profile is used to associate the emitter signature to a range-rate signature obtained by the geolocation sensor. The geolocation sensor then provides the desired location accuracy. We will provide results using real tracking data scenarios.
|
J. Derek Tucker Sandia National Laboratories |
Breakout |
| 2018 |
| XPCA: A Copula-based Generalization of PCA for Ordinal Data
Abstract
Principal Component Analysis is a standard tool in an analyst’s toolbox. The standard practice of rescaling each column can be reframed as a copula-based decomposition in which the marginal distributions are fit with a univariate Gaussian distribution and the joint distribution is modeled with a Gaussian copula. In this light, we present an alternative to traditional PCA we call XPCA by relaxing the marginal Gaussian assumption and instead fit each marginal distribution with the empirical distribution function. Interval-censoring methods are used to account for the discrete nature of the empirical distribution function when fitting the Gaussian copula model. In this talk, we derive the XPCA estimator and inspect the differences in fits on both simulated and real data applications.
|
Cliff Anderson-Bergman Sandia National Laboratories |
Breakout |
| 2018 |
| Insights, Predictions, and Actions: Descriptive Definitions of Data Science, Machine Learning, and Artificial Intelligence
Abstract
The terms “Data Science”, “Machine Learning”, and “Artificial Intelligence” have become increasingly common in popular media, professional publications, and even in the language used by DoD leadership. But these terms are often not well understood, and may be used incorrectly and interchangeably. Even a textbook definition of these fields is unlikely to help with the distinction, as many definitions tend to lump everything under the umbrella of computer science or introduce unnecessary buzzwords. Leveraging a framework first proposed by David Robinson, Chief Data Scientist at DataCamp, we forgo the textbook definitions and instead focus on practical distinctions between the work of practitioners in each field, using examples relevant to the test and evaluation community where applicable.
|
Dr. Andrew Flack Research Staff Member IDA |
Breakout |
| 2018 |
| Bayesian Calibration and Uncertainty Analysis: A Case Study Using a 2-D CFD Turbulence Model
Abstract
The growing use of simulations in the engineering design process promises to reduce the need for extensive physical testing, decreasing both development time and cost. However, as mathematician and statistician George E. P. Box said, “Essentially, all models are wrong, but some are useful.” There are many factors that determine simulation or, more broadly, model accuracy. These factors can be condensed into noise, bias, parameter uncertainty, and model form uncertainty. To counter these effects and ensure that models faithfully match reality to the extent required, simulation models must be calibrated to physical measurements. Further, the models must be validated, and their accuracy must be quantified before they can be relied on in lieu of physical testing. Bayesian calibration provides a solution for both requirements: it optimizes tuning of model parameters to improve simulation accuracy, and estimates any remaining discrepancy which is useful for model diagnosis and validation. Also, because model discrepancy is assumed to exist in this framework, it enables robust calibration even for inaccurate models. In this paper, we present a case study to investigate the potential benefits of using Bayesian calibration, sensitivity analyses, and Monte Carlo analyses for model improvement and validation. We will calibrate a 7-parameter k-𝜎 CFD turbulence model simulated in COMSOL Multiphysics®. The model predicts coefficient of lift and drag for an airfoil defined using a 6049-series airfoil parameterization from the National Advisory Committee for Aeronautics (NACA). We will calibrate model predictions using publicly available wind tunnel data from the University of Illinois Urbana-Champaign’s (UIUC) database. Bayesian model calibration requires intensive sampling of the simulation model to determine the most likely distribution of calibration parameters, which can be a large computational burden. We greatly reduce this burden by following a surrogate modeling approach, using Gaussian process emulators to mimic the CFD simulation. We train the emulator by sampling the simulation space using a Latin Hypercube (LHD) Design of Experiment (DOE), and assess the accuracy of the emulator using leave-oneout Cross Validation (CV) error. The Bayesian calibration framework involves calculating the discrepancy between simulation results and physical test results. We also use Gaussian process emulators to model this discrepancy. The discrepancy emulator will be used as a tool for model validation; characteristic trends in residual errors after calibration can indicate underlying model form errors which were not addressed via tuning the model calibration parameters. In this way, we will separate and quantify model form uncertainty and parameter uncertainty. The results of a Bayesian calibration include a posterior distribution of calibration parameter values. These distributions will be sampled using Monte Carlo methods to generate model predictions, whereby new predictions have a distribution of values which reflects the uncertainty in the tuned calibrated parameter. The resulting output distributions will be compared against physical data and the uncalibrated model to assess the effects of the calibration and discrepancy model. We will also perform global, variance based sensitivity analysis on the uncalibrated model and the calibrated models, and investigate any changes in the sensitivity indices from uncalibrated to calibrated.
|
Peter Chien |
Contributed | 2018 | |
| Assessing Human Visual Inspection for Acceptance Testing: An Attribute Agreement Analysis Case Study
Abstract
In today’s manufacturing, inspection, and testing world, understanding the capability of the measurement system being used via the use of Measurement Systems Analyses (MSA) is a crucial activity that provides the foundation for the use of Design of Experiments (DOE) and Statistical Process Control (SPC). Although undesirable, there are times when human observation is the only measurement system available. In these types of situations, traditional MSA tools are often ineffectual due to the nature of the data collected. When there are no other alternatives, we need some method for assessing the adequacy and effectiveness of the human observations. When multiple observers are involved, Attribute Agreement Analyses are a powerful tool for quantifying the Agreement and Effectiveness of a visual inspection system. This talk will outline best practices and rules of thumb for Attribute Agreement Analyses, and will highlight a recent Army case study to further demonstrate the tool’s use and potential.
|
Christopher Drake Lead Statistician QE&SA Statistical Methods & Analysis Group |
Contributed |
| 2018 |
| Leveraging Anomaly Detection for Aircraft System Health Data Stability Reporting
Abstract
Detecting and diagnosing aircraft system health poses a unique challenge as system complexity increases and software is further integrated. Anomaly detection algorithms systematically highlight unusual patterns in large datasets and are a promising methodology for detecting aircraft system health. The F-35A fighter aircraft is driven by complex, integrated subsystems with both software and hardware components. The F-35A operational flight program is the software that manages each subsystem within the aircraft and the flow of required information and support between subsystems. This information and support are critical to the successful operation of many subsystems. For example, the radar system supplies information to the fusion engine, without which the fusion engine would fail. ACC operational testing can be thought of as equivalent to beta testing for operational flight programs. As in other software, many faults result in minimal loss of functionality and are often unnoticed by the user. However, there are times when a software fault might result in catastrophic functionality loss (i.e., subsystem shutdown). It is critical to catch software problems that will result in catastrophic functionality loss before the flight software is fielded to the combat air forces. Subsystem failures and degradations can be categorized and quantified using simple system health data codes (e.g., degrade, fail, healthy). However, because the integrated nature of the F-35A, a subsystem degradation may be caused by another subsystem. The 59th Test and Evaluation Squadron collects autonomous system data, pilot questionnaires, and health report codes for F-35A subsystems. Originally, this information was analyzed using spreadsheet tools (i.e., Microsoft Excel). Using this method, analysts were unable to examine all subsystems or attribute cause for subsystem faults. The 59 TES is developing a new process that leverages anomaly detection algorithms to isolate flights with unusual patterns of subsystem failures and within those flights, highlight what subsystem faults are correlated with increased subsystem failures. This presentation will compare the performance of several anomaly detection algorithms (e.g., K-means, K-nearest neighbors, support vector machines) using simulated F-35A data.
|
Kyle Gartrell |
Contributed |
| 2018 |
| Application of Adaptive Sampling to Advance the Metamodeling and Uncertainty Quantification Process
Abstract
Over the years the aerospace industry has continued to implement design of experiments and metamodeling (e.g., response surface methodology) in order to shift the knowledge curve forward in the systems design process. While the adoption of these methods is still incomplete across aerospace sub-disciplines, they comprise the state-of-the-art during systems design and for design evaluation using modeling and simulation or ground testing. In the context of modeling and simulation, while national infrastructure in high performance computing becomes higher performance, so do the demands placed on those resources in terms of simulation fidelity and number of researchers. Furthermore, with recent emphasis placed on the uncertainty quantification of aerospace system design performance, the number of simulation cases needed to properly characterize a system’s uncertainty across the entire design space increases by orders of magnitude, further stressing available resources. This leads to advanced development groups either sticking to ad hoc estimates of uncertainty (e.g., subject matter expert estimates based on experience) or neglecting uncertainty quantification all together. Advancing the state-of-the-art of aerospace systems design and evaluation requires a practical adaptive sampling scheme that responds to the characteristics of the underlying design or uncertainty space. For example, when refining a system metamodel gradually, points should be chosen for design variable combinations that are located in high curvature regions or where metamodel uncertainty is the greatest. The latter method can be implemented by defining a functional form of the metamodel variance and using it to define the next best point to sample. For schemes that require n points to be sampled simultaneously, considerations can be made to ensure proper sample dispersion. The implementation of adaptive sampling schemes to the design and evaluation process will enable similar fidelity with fewer samples of the design space compared to fixed or ad hoc sampling methods (i.e., shorter time or human resources required). Alternatively, the uncertainty of the design space can be reduced to a greater extent for the same number of samples or with fewer samples using higher fidelity simulations. The purpose of this presentation will be to examine the benefits of adaptive sampling as applied to challenging design problems. Emphasis will be placed on methods that are accessible to engineering
|
Erik Axdahl Hypersonic Airbreathing Propulsion Branch NASA Langley Research Center |
Contributed |
| 2018 |
| Application of Adaptive Sampling to Advance the Metamodeling and Uncertainty Quantification Process
Abstract
Over the years the aerospace industry has continued to implement design of experiments and metamodeling (e.g., response surface methodology) in order to shift the knowledge curve forward in the systems design process. While the adoption of these methods is still incomplete across aerospace sub-disciplines, they comprise the state-of-the-art during systems design and for design evaluation using modeling and simulation or ground testing. In the context of modeling and simulation, while national infrastructure in high performance computing becomes higher performance, so do the demands placed on those resources in terms of simulation fidelity and number of researchers. Furthermore, with recent emphasis placed on the uncertainty quantification of aerospace system design performance, the number of simulation cases needed to properly characterize a system’s uncertainty across the entire design space increases by orders of magnitude, further stressing available resources. This leads to advanced development groups either sticking to ad hoc estimates of uncertainty (e.g., subject matter expert estimates based on experience) or neglecting uncertainty quantification all together. Advancing the state-of-the-art of aerospace systems design and evaluation requires a practical adaptive sampling scheme that responds to the characteristics of the underlying design or uncertainty space. For example, when refining a system metamodel gradually, points should be chosen for design variable combinations that are located in high curvature regions or where metamodel uncertainty is the greatest. The latter method can be implemented by defining a functional form of the metamodel variance and using it to define the next best point to sample. For schemes that require n points to be sampled simultaneously, considerations can be made to ensure proper sample dispersion. The implementation of adaptive sampling schemes to the design and evaluation process will enable similar fidelity with fewer samples of the design space compared to fixed or ad hoc sampling methods (i.e., shorter time or human resources required). Alternatively, the uncertainty of the design space can be reduced to a greater extent for the same number of samples or with fewer samples using higher fidelity simulations. The purpose of this presentation will be to examine the benefits of adaptive sampling as applied to challenging design problems. Emphasis will be placed on methods that are accessible to engineering
|
Robert Baurle Hypersonic Airbreathing Propulsion Branch NASA Langley Research Center |
Contributed |
| 2018 |
| Application of Statistical Methods and Designed Experiments to Development of Technical Requirements
Abstract
The Army relies heavily on the voice of the customer to develop and refine technical requirements for developmental systems, but too often the approach is reactive. The ARDEC (Armament Research, Development & Engineering Center) Statistics Group at Picatinny Arsenal, NJ, working closely with subject matter experts, has been implementing market research and web development techniques and Design of Experiments (DOE) best practices to design and analyze surveys that provide insight into the customer’s perception of utility for various developmental commodities. Quality organizations tend to focus on ensuring products meet technical requirements, with far less of an emphasis placed on whether or not the specification actually captures customer needs. The employment of techniques and best practices spanning the fields of Market Research, Design of Experiments, and Web Development (choice design, conjoint analysis, contingency analysis, psychometric response scales, stratified random sampling) converge towards a more proactive and risk-mitigating approach to the development of technical and training requirements, and encourages strategic decision-making when faced with the inarticulate nature of human preference. Establishing a hierarchy of customer preference for objective and threshold values of key performance parameters enriches the development process of emerging systems by making the process simultaneously more effective and more efficient.
|
Eli Golden U.S. ARMY ARMAMENT RESEARCH, DEVELOPMENT & ENGINEERING CENTER |
Contributed |
| 2018 |
| Infrastructure Lifetimes
Abstract
Infrastructure refers to the structures, utilities, and interconnected roadways that support the work carried out at a given facility. In the case of the Lawrence Livermore National Laboratory infrastructure is considered exclusive of scientific apparatus, safety and security systems. LLNL inherited it’s infrastructure management policy from the University of California which managed the site during LLNL’s first 5 decades. This policy is quite different from that used in commercial property management. Commercial practice weighs reliability over cost by replacing infrastructure at industry standard lifetimes. LLNL practice weighs overall lifecycle cost seeking to mitigate reliability issues through inspection. To formalize this risk management policy a careful statistical study was undertaken using 20 years of infrastructure replacement data. In this study care was taken to adjust for left truncation as-well-as right censoring. 57 distinct infrastructure class data sets were fitted using MLE to the Generalized Gamma distribution. This distribution is useful because it produces a weighted blending of discrete failure (Weibull model) and complex system failure (Lognormal model). These parametric fittings then yielded median lifetimes and conditional probabilities of failure. From conditional probabilities bounds on budget costs could be computed as expected values. This has provided a scientific basis for rational budget management as-well-as aided operations by prioritizing inspection, repair and replacement activities.
|
Erika Taketa Lawrence Livermore National Laboratory |
Contributed |
| 2018 |
| Infrastructure Lifetimes
Abstract
Infrastructure refers to the structures, utilities, and interconnected roadways that support the work carried out at a given facility. In the case of the Lawrence Livermore National Laboratory infrastructure is considered exclusive of scientific apparatus, safety and security systems. LLNL inherited it’s infrastructure management policy from the University of California which managed the site during LLNL’s first 5 decades. This policy is quite different from that used in commercial property management. Commercial practice weighs reliability over cost by replacing infrastructure at industry standard lifetimes. LLNL practice weighs overall lifecycle cost seeking to mitigate reliability issues through inspection. To formalize this risk management policy a careful statistical study was undertaken using 20 years of infrastructure replacement data. In this study care was taken to adjust for left truncation as-well-as right censoring. 57 distinct infrastructure class data sets were fitted using MLE to the Generalized Gamma distribution. This distribution is useful because it produces a weighted blending of discrete failure (Weibull model) and complex system failure (Lognormal model). These parametric fittings then yielded median lifetimes and conditional probabilities of failure. From conditional probabilities bounds on budget costs could be computed as expected values. This has provided a scientific basis for rational budget management as-well-as aided operations by prioritizing inspection, repair and replacement activities.
|
William Romine Lawrence Livermore National Laboratory |
Contributed |
| 2018 |
| Workforce Analytics
Abstract
Several statistical methods have been used effectively to model workforce behavior, specifically attrition due to retirement and voluntary separation[1]. Additionally various authors have introduced career development[2] as a meaningful aspect of workforce planning. While both general and more specific attrition modeling techniques yield useful results only limited success has followed attempts to quantify career stage transition probabilities. A complete workforce model would include quantifiable flows both vertically and horizontally in the network described pictorially here at a single time point in Figure 1. The horizontal labels in Figure 1 convey one possible meaning assignable to career stage transition – in this case, competency. More formal examples might include rank within a hierarchy such as in a military organization or grade in a civil service workforce. In the case of the Nuclear Weapons labs knowing that the specialized, classified knowledge needed to deal with Stockpile Stewardship is being preserved as evidenced by the production of Masters, individuals capable of independent technical work, is also of interest to governmental oversight. In this paper we examine the allocation of labor involved in a specific Life Extension program at LLNL. This growing workforce is described by discipline and career stage to determine how well the Norden-Rayleigh development cost model[3] fits the data. Since this model underlies much budget estimation within both DOD and NNSA the results should be of general interest. Data is also examined as a possible basis for quantifying horizontal flows in Figure 1.
|
William Romine Lawrence Livermore National Laboratory |
Contributed | 2018 | |
| Optimizing for Mission Success in Highly Uncertain Scenarios
Abstract
Optimization under uncertainty increases the complexity of a problem as well as the computing resources required to solve it. As the amount of uncertainty is increased, these difficulties are exacerbated. However, when optimizing for mission-level objectives, rather than component- or system-level objectives, an increase in uncertainty is inevitable. Previous research has found methods to perform optimization under uncertainty, such as robust design optimization or reliability-based design optimization. These are generally executed at a product component quality level, to minimize variability and stay within design tolerances but are not tailored to capture the high amount of variability in a mission-level problem. . In this presentation, an approach for formulating and solving highly stochastic mission-level optimization problems is described. A case study is shown using an unmanned aerial system (UAS) on a search mission while an “enemy” UAS attempts to interfere. This simulation, modeled in the Unity Game Engine, has highly stochastic outputs, where the time to mission success varies by multiple orders of magnitude, but the ultimate goal is a binary output representing mission success or failure. The results demonstrate the capabilities and challenges of optimization in these types of mission scenarios.
|
Brian Chell |
Contributed | 2018 | |
| Test Planning for Observational Studies using Poisson Process Modeling
Abstract
Operational Test (OT) is occasionally conducted after a system is already fielded. Unlike a traditional test based on Design of Experiments (DOE) principles, it is often not possible to vary the levels of the factors of interest. Instead the test is of an observational nature. Test planning for observational studies involves choosing where, when, and how long to evaluate a system in order to observe the possible combinations of factor levels that define the battlespace. This presentation discusses a test-planning method that uses Poisson process modeling as a way to estimate the length of time required to observe factor level combinations in the operational environment.
|
Brian Stone AFOTEC |
Contributed |
| 2018 |
| Model credibility in statistical reliability analysis with limited data
Abstract
Due to financial and production constraints, it has become increasingly common for analysts and test planners in defense applications to find themselves working with smaller amounts of data than seen in industry. These same analysts are also being asked to make strong statistical statements based on this limited data. For example, a common goal is ‘demonstrating’ a high reliability requirement with sparse data. In such situations, strong modeling assumptions are often used to achieve the desired precision. Such model-driven actions contain levels of risk that customers may not be aware of and may be too high to be considered acceptable. There is a need to articulate and mitigate risk associated with model form error in statistical reliability analysis. In this work, we review different views on model credibility from the statistical literature and discuss how these notions of credibility apply in data-limited settings. Specifically, we consider two different perspectives on model credibility: (1) data-driven credibility metrics for model fit, (2) credibility assessments based on consistency of analysis results with prior beliefs. We explain how these notions of credibility can be used to drive test planning and recommend an approach to presenting analysis results in data-limited settings. We apply this approach to two case studies from reliability analysis: Weibull analysis and Neyer-D optimal test plans.
|
Caleb King Sandia National Laboratories |
Contributed |
| 2018 |
| Sound level recommendations for quiet sonic boom dose-response community surveys
Abstract
The current ban on commercial overland supersonic flight may be replaced by a noise limit on sonic boom sound level. NASA is establishing a quiet sonic boom database to guide the new regulation. The database will consist of multiple community surveys used to model the dose-response relationship between sonic boom sound levels and human annoyance. There are multiple candidate dose-response modeling techniques, such as classical logistic regression and multilevel modeling. To plan for these community surveys, recommendations for data collection will be developed from pilot community test data. Two important aspects are selecting sample size and sound level range. Selection of sample size must be strategic as large sample sizes are costly whereas small sample sizes may result in more uncertainty in the estimates. Likewise, there are trade-offs associated with selection of the sound level range. If the sound level range includes excessively high sound levels, the public may misunderstand the potential impact of quiet sonic booms, resulting in a negative backlash against a promising technological advancement. Conversely, a narrow range that includes only low sound levels might exclude the eventual noise limit. This presentation will focus on recommendations for sound level range given the expected shape of the dose-response curve.
|
Jasme Lee North Carlina State Univeristy |
Contributed |
| 2018 |
| Development of a Locking Setback Mass for Cluster Munition Applications: A UQ Case Study
Abstract
The Army is currently developing a cluster munition that is required to meet functional reliability requirements of 99%. This effort focuses on the design process for a setback lock within the safe and arm (S&A) device in the submunition fuze. This lock holds the arming rotor in place, thus preventing the fuze from beginning its arming sequence until the setback lock detracts during a launch event. Therefore, the setback lock is required to not arm (remain in place) during a drop event (safety) and to arm during a launch event (reliability). In order to meet these requirements, uncertainty quantification techniques were used to evaluate setback lock designs. We designed a simulation experiment, simulated the setback lock behavior in a drop event and in a launch event, fit a model to the results, and optimized the design for safety and reliability. Currently, 8 candidate designs that meet the requirements are being manufactured, and adaptive sensitivity testing is planned to inform the surrogate models and improve their predictive capability. A final optimized design will be chosen based on the improved models, and realistic drop safety and arm reliability predictions will be obtained using Monte-Carlo simulations of the surrogate models.
|
Melissa Jablonski U.S. ARMY ARMAMENT RESEARCH, DEVELOPMENT & ENGINEERING CENTER |
Contributed |
| 2018 |
| Wednesday Keynote Speaker I | Dr. Peter Parker Team Lead, Advanced Measurement Systems NASA Bio
Dr. Parker is Team Lead for Advanced Measurement Systems at the National Aeronautics and Space Administration’s Langley Research Center in Hampton, Virginia. He serves an Agency-wide statistical expert across all of NASA’s mission directorates of Exploration, Aeronautics, and Science to infuse statistical thinking, engineering, and methods including statistical design of experiments, response surface methodology, and measurement system characterization. His expertise is in collaboratively integrating research objectives, measurement sciences, test design, and statistical methods to produce actionable knowledge for aerospace research and development.
He holds a B.S. in Mechanical Engineering, a M.S. in Applied Physics and Computer Science and a M.S. and Ph.D. in Statistics from Virginia Tech. Dr. Parker is a senior member of the American Institute for Aeronautics and Astronautics, American Society for Quality, and the American Statistical Association. Dr. Parker currently Chairs the American Society for Quality’s Publication Management Board and previously served as Editor-in-Chief of the journal Quality Engineering.
|
Keynote |
| 2019 |
| Tuesday Keynote | Dr. David Chu President Institute for Defense Analyses Bio
David Chu serves as President of the Institute for Defense Analyses. IDA is a non-profit corporation operating in the public interest. Its three federally funded research and development centers provide objective analyses of national security issues and related national challenges, particularly those requiring extraordinary scientific and technical expertise.
As president, Dr. Chu directs the activities of more than 1,000 scientists and technologists. Together, they conduct and support research requested by federal agencies involved in advancing national security and advising on science and technology issues.
Dr. Chu served in the Department of Defense as Under Secretary of Defense for Personnel and Readiness from 2001-2009, and earlier as Assistant Secretary of Defense and Director for Program Analysis and Evaluation from 1981-1993.
From 1978-1981 he was the Assistant Director of the Congressional Budget Office for National Security and International Affairs.
Dr. Chu served in the U. S. Army from 1968-1970. He was an economist with the RAND Corporation from 1970-1978, director of RAND’s Washington Office from 1994-1998, and vice president for its Army Research Division from 1998-2001.
He earned a bachelor of arts in economics and mathematics, and his doctorate in economics, from Yale University.
Dr. Chu is a member of the Defense Science Board and a Fellow of the National Academy of Public Administration. He is a recipient of the Department of Defense Medal for Distinguished Public Service with Gold Palm, the Department of Veterans Affairs Meritorious Service Award, the Department of the Army Distinguished Civilian Service Award, the Department of the Navy Distinguished Public Service Award, and the National Academy of Public Administration’s National Public Service Award.
|
Keynote |
2019 | |
| Thursday Keynote Speaker II | Mr. Michael Little Program Manager, Advanced Information Systems Technology Earth Science Technology Office, NASA Headquarters Bio
Over the past 45 years, Mike’s primary focus has been on the management of research and development, focusing on making the results more useful in meeting the needs of the user community. Since 1984, he has specialized in communications, data and processing systems, including projects in NASA, the US Air Force, the FAA and the Census Bureau. Before that, he worked on Major System Acquisition Programs, in the Department of Defense including Marine Corps combat vehicles and US Navy submarines.
Currently, Mike manages a comprehensive program to provide NASA’s Earth Science research efforts with the information technologies it will need in the 2020-2035 time-frame to characterize, model and understand the Earth. This Program addresses the full range of data lifecycle from generating data using instruments and models, through the management of the data and including the ways in which information technology can help to exploit the data. Of particular interest today are the ways in which NASA can measure and understand transient and transitional phenomena and the impact of climate change. The AIST Program focuses the application of applied math and statistics, artificial intelligence, case-based reasoning, machine learning and automation to improve our ability to use observational data and model output in understanding Earth’s physical processes and natural phenomena.
Training and odd skills:
Application of cloud computing
US Government Computer Security
US Navy Nuclear Propulsion operations and maintenance on two submarines
|
Keynote |
| 2019 |
| Thursday Keynote Speaker I | Dr. Wendy Martinez Director, Mathematical Statistics Research Center, Bureau of Labor Statistics ASA President-Elect (2020) Bio
Wendy Martinez has been serving as the Director of the Mathematical Statistics Research Center at the Bureau of Labor Statistics (BLS) for six years. Prior to this, she served in several research positions throughout the Department of Defense. She held the position of Science and Technology Program Officer at the Office of Naval Research, where she established a research portfolio comprised of academia and industry performers developing data science products for the future Navy and Marine Corps. Her areas of interest include computational statistics, exploratory data analysis, and text data mining. She is the lead author of three books on MATLAB and statistics. Dr. Martinez was elected as a Fellow of the American Statistical Association (ASA) in 2006 and is an elected member of the International Statistical Institute. She was honored by the American Statistical Association when she received the ASA Founders Award at the JSM 2017 conference. Wendy is also proud and grateful to have been elected as the 2020 ASA President.
|
Keynote |
| 2019 |
| Wednesday Keynote Speaker II | Dr. Laura Freeman Associate Director, ISL Hume Center for National Security and Technology, Virginia Tech Bio
Dr. Laura Freeman is an Assistant Director of the Operational Evaluation Division at the Institute for Defense Analyses. In that position, she established and developed an interdisciplinary analytical team of statisticians, psychologists, and engineers to advance scientific approaches to DoD test and evaluation. Her focus areas include test design, statistical data analysis, modeling and simulation validation, human-system interactions, reliability analysis, software testing, and cybersecurity testing. Dr. Freeman currently leads a research task for the Chief Management Officer (CMO) aiming to reform DoD testing. She guides an interdisciplinary team in recommending changes and developing best practices. Reform initiatives include incorporating mission context early in the acquisition lifecycle, integrating all test activities, and improving data management processes.
During 2018, Dr. Freeman served as that acting Senior Technical Advisor for Director Operational Test and Evaluation (DOT&E). As the Senior Technical Advisor, Dr. Freeman provided leadership, advice, and counsel to all personnel on technical aspects of testing military systems. She served as a liaison with Service technical advisors, General Officers, and members of the Senior Executive Service on key technical issues. She reviewed test strategies, plans, and reports from all systems on DOT&E oversight.
During her tenure at IDA, Dr. Freeman has designed tests and conducted statistical analyses for programs of national importance including weapon systems, missile defense, undersea warfare systems, command and control systems, and most recently the F-35. She prioritizes supporting the analytical community in the DoD workforce. She developed and taught numerous courses on advanced test design and statistical analysis, including two new Defense Acquisition University (DAU) Courses on statistical methods. She is a founding organizer of DATAWorks (Defense and Aerospace Test and Analysis Workshop), a workshop designed to share new methods, provide training, and share best practices between NASA, the DoD, and National Labs.
Dr. Freeman is the recipient of the 2017 IDA Goodpaster Award for Excellence in Research and the 2013 International Test and Evaluation Association (ITEA) Junior Achiever Award. She is a member of the American Statistical Association, the American Society for Quality, the International Statistical Engineering Association, and ITEA. She serves on the editorial boards for Quality Engineering, Quality Reliability Engineering International, and the ITEA Journal. Her areas of statistical expertise include designed experiments, reliability analysis, and industrial statistics.
Prior to joining IDA in 2010, Dr. Freeman worked at SAIC providing statistical guidance to the Director, Operational Test and Evaluation. She also consulted with NASA on various projects. In 2008, Dr. Freeman established the Laboratory for Interdisciplinary Statistical Analyses at Virginia Tech and Served as its inaugural Director.
Dr. Freeman has a B.S. in Aerospace Engineering, a M.S. in Statistics and a Ph.D. in Statistics, all from Virginia Tech. Her Ph.D. research was on design and analysis of experiments for reliability data.
|
Keynote |
| 2019 |
| Thursday Lunchtime Keynote Speaker | Dr. T. Charles Clancy Bradley Professor of Electrical and Computer Engineering Virginia Tech Bio
Charles Clancy is the Bradley Professor of Electrical and Computer Engineering at Virginia Tech where he serves as the Executive Director of the Hume Center for National Security and Technology. Clancy leads a range of strategic programs at Virginia Tech related to security, including the Commonwealth Cyber Initiative. Prior to joining VT in 2010, Clancy was an engineering leader in the National Security Agency, leading research programs in digital communications and signal processing. He received his PhD from the University of Maryland, MS from University of Illinois, and BS from the Rose-Hulman Institute of Technology. He is co-author to over 200 peer-reviewed academic publications, six books, over twenty patents, and co-founder to five venture-backed startup companies.
|
Keynote |
| 2019 |
| Wednesday Keynote Speaker III | Mr. Timothy Dare Deputy Director, Developmental Test, Evaluation, and Prototyping SES OUSD(R&E) Bio
Mr. Timothy S. Dare is the Deputy Director for Developmental Test, Evaluation and Prototyping (DD(DTEP)). As the DD(DTEP), he serves as the principal advisor on developmental test and evaluation (DT&E) to the Secretary of Defense, Under Secretary of Defense for Research and Engineering, and Director of Defense Research and Engineering for Advanced Capabilities. Mr. Dare is responsible for DT&E policy and guidance in support of the acquisition of major Department of Defense (DoD) systems, and providing advocacy, oversight, and guidance to the DT&E acquisition workforce. He informs policy and advances leading edge technologies through the development of advanced technology concepts, and developmental and operational prototypes. By working closely with interagency partners, academia, industry and governmental labs, he identifies, develops and demonstrates multi-domain technologies and concepts that address high-priority DoD, multi-Service, and Combatant Command warfighting needs.
Prior to his appointment in December 2018, Mr. Dare was a Senior Program Manager for program management and capture at Lockheed Martin (LM) Space. In this role he was responsible for the capture and execution phases of multiple Intercontinental Ballistic Missile programs for Minuteman III, including a new airborne Nuclear Command and Control (NC2) development program. His major responsibilities included establishing program working environments at multiple locations, policies, processes, staffing, budget and technical baselines.
Mr. Dare has extensive T&E and prototyping experience. As the Engineering Program Manager for the $1.8B Integrated Space C2 programs for NORAD/NORTHCOM systems at Cheyenne Mountain, Mr. Dare was the Integration and Test lead focusing on planning, executing, and evaluating the integration and test phases (developmental and operational T&E) for Missile Warning and Space Situational Awareness (SSA) systems. Mr. Dare has also been the Engineering Lead/Integration and Test lead on other systems such as the Hubble Space Telescope; international border control systems; artificial intelligence (AI) development systems (knowledge-based reasoning); Service-based networking systems for the UK Ministry of Defence; Army C2 systems; Space Fence C2; and foreign intelligence, surveillance, and reconnaissance systems.
As part of the Department’s strategic defense portfolio, Mr. Dare led the development of advanced prototypes in SSA C2 (Space Fence), Information Assurance (Single Sign-on), AI systems, and was the sponsoring program manager for NC2 capability development.
Mr. Dare is a graduate of Purdue University and is a member of both the Association for Computing Machinery and Program Management Institute. He has been recognized by the U.S. Air Force for his contributions supporting NORAD/NORTHCOM’s strategic defense missions, and the National Aeronautics and Space Administration for his contributions to the original Hubble Space Telescope program. Mr. Dare holds a U.S. Patent for Single Sign-on architectures.
|
Keynote |
| 2019 |
| Wednesday Lunchtime Keynote Speaker | Dr. Jared Freeman Chief Scientist of Aptima and Chair of the Human Systems Division National Defense Industry Association Bio
Jared Freeman, Ph.D., is Chief Scientist of Aptima and Chair of the Human Systems Division of the National Defense Industry Association. His research and publications address measurement, assessment, and enhancement of human learning, cognition, and performance in technologically complex military environments.
|
Keynote |
| 2019 |
| Welcoming & Opening Keynote-Tuesday AM | Dr. Mike Gilmore Director DOT&E Bio
Link to Bio unavail
|
Keynote |
| 2016 |
| Lunch with Keynote Leadership Perspective | Dr. Jon Hollday NASA Bio
Link to Bio unavail
|
Keynote |
2016 | |
| STAT Engineering Keynote-Wednesday AM | Dr. Christine Anderson Cook Statistics Los Alamos National Lab Bio
s in the areas of complex system reliability, non-proliferation, malware detection and statistical process control. Before joining LANL, she was a faculty member in the Department of Statistics at Virginia Tech for 8 years. Her research areas include response surface methodology, design of experiments, reliability, multiple criterion optimization and graphical methods. She has authored more than 130 articles in statistics and quality peerreviewed journals, and has been a long time contributor to the Quality Progress Statistics Roundtable column. In 2012, she edited a special issue in Quality Engineering on Statistical Engineering with Lu Lu. She is an elected fellow of the American Statistical Association and the American Society for Quality. In 2012 she was honored with the ASQ Statistics Division William G. Hunter Award. In 2011 she received the 26th Annual Governor’s Award for Outsta
|
Keynote |
2016 | |
| Lunch With Leadership Perspective-Wednesday PM | Dr. David Brown Deputy Assistant Secretary of Defense, Developmental Test & Evaluation Bio
Link to Bio unavail
|
Keynote |
2016 | |
| Test and Evaluation Matter for Warfighter and We Need your Help to Make it Better | Dr. Katherine Warner Science Advisor within Secretary of Defense DOT&E Bio
Dr. Catherine Warner serves as the Science Advisor for the Director, Operational Test and Evaluation within the Office of the Secretary of Defense. Dr. Warner has been involved with operational test and evaluation since 1991, when she became a research staff member at the Institute for Defense Analyses (IDA). In that capacity, Dr. Warner performed and directed analysis of operational tests for Army, Navy, and Air Force systems in support of DOT&E. From 2005 – 2010, Dr. Warner was an Assistant Director at IDA and also served as the lead for the Air Warfare group. Her analysis portfolio included major aircraft systems such as the F-22, F/A-18E/F, V-22, and H-1. Prior to that, Dr. Warner was the lead analyst for Unmanned Aerial Vehicle (UAV) systems including Predator, Shadow, Hunter, and Global Hawk. In 2013 at the request of the Defense Information Systems Agency (DISA), Dr. Warner deployed to Kabul, Afghanistan for 16 months in support of NATO’s International Security Assistance Force (ISAF) and the US Operation Enduring Freedom (OEF). She lead a team of Information Technology specialists in advising Afghanistan’s Ministry of Communications and Information Technology on enhancing national communication capabilities for the security and economic growth of the country. The primary focus of this team included supporting the completion of Afghanistan’s National Fiber Optic Ring, Spectrum Management, and Cyber Security.Dr. Warner previously worked at the Lawrence Livermore National Laboratory. She grew up in Albuquerque, New Mexico, attended the University of New Mexico and San Jose State University where she earned both B.S. and M.S. degrees in Chemistry. She also earned both M.A. and Ph.D. degrees in Chemistry from Princeton University.
|
Keynote |
| 2017 |
| Retooling Design and Development | Mr. Chris Singer NASA Deputy Chief Engineer NASA Bio
Christopher (Chris) E. Singer is the NASA Deputy, Chief Engineer responsible for integrating engineering across the Agencies 10 field centers. Prior to this appointment in April 2016, he served as the Engineering Director at NASA’s Marshall Space Flight Center in Huntsville, Alabama. Appointed in 2011, Mr. Singer led an organization of 1,400 civil service and 1,200 support contractor employees responsible for the design, testing, evaluation, and operation of hardware and software associated with space transportation, spacecraft systems, science instruments and payloads under development at the Marshall Center. The Engineering Directorate also manages NASA’s Payload Operations Center at Marshall, which is the command post for scientific research activities on-board the International Space Station.
Mr. Singer began his NASA career in 1983 as a rocket engine specialist. In 1992, he served a one-year assignment at NASA Headquarters in Washington, DC, as senior manager for the space shuttle main engine and external tank in the Space Shuttle Support Office. In 1994, Mr. Singer supervised the development and implementation of safety improvements and upgrades to shuttle propulsion components. In 2000, he was appointed chief engineer in the Space Transportation Directorate then was selected as deputy director of Marshall’s Engineering Directorate from 2004 to 2011.
Mr. Singer is an AIAA Associate Fellow. In 2006, he received the Presidential Rank Award for Meritorious Executives — the highest honor for career federal employees. He was awarded the NASA Outstanding Leadership Medal in 2001 and 2008 for his leadership. In 1989, he received the prestigious Silver Snoopy Award from the Astronaut Corps for his contributions to the success of human spaceflight missions.
A native of Nashville, Tennessee, Mr. Singer earned a bachelor’s degree in mechanical engineering in 1983 from Christian Brothers University in Memphis, Tennessee.
Chris enjoys woodworking, fishing and Hang gliding. Chris is married to the former Jody Adams of Hartselle, Alabama. They have three children and live in Huntsville, Alabama.
|
Keynote |
| 2017 |
| Reflections on Statistical Engineering and it’s Application | Dr. Geoff Vining Professor Virginia Tech Bio
Geoff Vining is a Professor of Statistics at Virginia Tech. From 1999 – 2006, he also was the department head. He currently is the ASQ Treasurer for 2016 and Past-Chair of the ASQ Technical Communities Council. He is a Fellow of the ASQ, a Fellow of the American Statistical Association (ASA), and an Elected Member of the International Statistical Institute.
Dr. Vining served as Editor of the Journal of Quality Technology from 1998 – 2000 and as Editor-in-Chief of Quality Engineering from 2008-2009. He also has served as Chair of the ASQ Publications Management Board, as Chair of the ASQ Statistics Division, and as Chair of the ASA Quality and Productivity Section.
Dr. Vining won the 2010 Shewhart Medal, the ASQ career award given annually to the person not previously so honored who has demonstrated the most outstanding technical leadership in the field of modern quality control, especially through the development to its theory, principles, and techniques. He also received the 2015 Box Medal from the European Network for Business and Industrial Statistics (ENBIS). This medal recognizes each year an extraordinary statistician who has remarkably contributed with his/her work to the development and the application of statistical methods in European business and industry. In 2013, he received an Engineering Excellence Award from the NASA Engineering and Safety Center. He received the 2011 William G. Hunter Award from the ASQ Statistics Division for excellence in statistics as a communicator, a consultant, an educator, an innovator, an integrator of statistics with other disciplines and an implementer who obtains meaningful results. He won the 1990 ASQ Brumbaugh Award for the paper published in an ASQ journal that made the greatest contribution to the development of industrial applications of quality control and the 2005 Lloyd Nelson Award from the Statistics Division for the paper published in the Journal of Quality Technology that had the greatest immediate impact to practitioners.
Dr. Vining is the author of three textbooks. He is an internationally recognized expert in the use of experimental design for quality, productivity, and reliability improvement and in the application of statistical process control. He has extensive consulting experience, most recently with NASA and the U.S. Department of Defense.
|
Keynote |
| 2017 |
| Developmental Test and Evaluation | Dr. Brian Hall Principal Deputy Director DoD Bio
Dr. Hall was appointed to the Senior Executive Service in November 2014 as the Principal Deputy Director for Developmental Test and Evaluation, and is currently serving as the Principle Deputy Director of the Test Resource Management Center (TRMC). In this position, he oversees matters concerning the Nation’s critical test range infrastructure, science and technology efforts, development of the biennial Strategic Plan for DoD Test and Evaluation (T&E) resources, as well as certification of the Services’ T&E budgets.
Prior to this position, Dr. Hall was the Technical Advisor for Operational Test and Evaluation (OT&E) of all Land and Expeditionary Warfare systems in OSD-DOT&E. In this position, advised the highest authorities in DoD for OT&E, observed operational testing, and coauthored Operational Assessments and Beyond Low-Rate Initial Production reports submitted to the 4 Congressional Defense Committees.
Prior to serving on the OSD staff, Dr. Hall was the Division Chief for Aviation, Missiles, and C4ISR Systems in the Army Test and Evaluation Command (ATEC), where he was responsible for supervising the development of test plans and evaluations that directly supported milestone decision reviews and materiel fielding/production decisions of more than 300 Army programs. While with the Army, Dr. Hall was one of the leading reliability experts that helped establish the Center for Reliability Growth at Aberdeen Proving Ground, as well as develop and administer the ATEC/AMSAA 3-day reliability course to improve defense acquisition learning.
Over his career, Dr. Hall has led studies, developed methodologies, presented research, published papers, crafted policy, and authored policy implementation guides. He has also developed staff, advised numerous defense programs, and served as an executive member of, or invited contributor to: tri-service DoD Blue Ribbon Panels; National Academy of Science studies; and DoD working-groups to improve system reliability.
Dr. Hall is a Senior Service College graduate and has earned advanced degrees in Applied Mathematics, Reliability Engineering, and Strategic Studies. He is a domain expert in Reliability Engineering, as well as in Reliability Growth Management and Methodology. He has developed statistical methods and reliability growth models that have been: published in international journals, incorporated into Military Handbooks, adopted by Operational Test Agency policy, and utilized to shape growth plans and assess reliability maturity of numerous DoD systems.
|
Keynote |
| 2017 |
| Mr. Dave Duma Acting Director, Operationak Test and Evaluation, Office of the Secretary of Defense DOT&E Bio
Mr. Duma is the Acting Director, Operational Test and Evaluation as of January 20, 2017. Mr. Duma was appointed as the Principal Deputy Director, Operational Test and Evaluation in January 2002. In this capacity he is responsible for all functional areas assigned to the office. He participates in the formulation, development, advocacy, and oversight of policies of the Secretary of Defense and in the development and implementation of test and test resource programs. He oversees the planning, conduct, analysis, evaluation, and reporting of operational and live fire testing. He serves as the Appropriation Director and Comptroller for the Operational Test and Evaluation, Defense Appropriation and coordinates all Planning, Programming, and Budgeting Execution matters. He previously served as Acting Director, Operational Test and Evaluation from February 2005 to July 2007 and again from May 2009 to September 2009.
Mr. Duma also served as the Acting Deputy Director, Operational Test and Evaluation from January 1992 to June 1994. In this capacity he was responsible for oversight of the planning, conduct, analysis, and reporting of operational test and evaluation for all major conventional weapons systems in the Department of Defense. He supervised the development of evaluation plans and test program strategies, observed the conduct of operational test events, evaluated operational field tests of all armed services and submitted final reports for Congress.
Mr. Duma returned to government service from the commercial sector. In private industry he worked a variety of projects involving test and evaluation; requirements generation; command, control, communications, intelligence, surveillance and reconnaissance; modeling and simulation; and software development.
Mr. Duma has 30 years of naval experience during which he was designated as a Joint Service Officer. He served as the Director, Test and Evaluation Warfare Systems for the Chief of Naval Operations, the Deputy Commander, Submarine Squadron TEN, and he commanded the nuclear powered submarine USS SCAMP (SSN 588).
Mr. Duma holds Masters of Science degrees in National Security and Strategic Studies and in Management. He holds a Bachelor of Science degree in Nuclear Engineering. He received the U.S. Presidential Executive Rank Award on two occasions; in 2008, the Meritorious Executive Award and in 2015, the Distinguished Executive Rank Award. He is a member of the International Test and Evaluation Association.
|
Keynote |
2017 | ||
| Opening Keynote | Dr. David Chu President IDA Bio
David Chu serves as President of the Institute for Defense Analyses. IDA is a non-profit corporation operating in the public interest. Its three federally funded research and development centers provide objective analyses of national security issues and related national challenges, particularly those requiring extraordinary scientific and technical expertise. As president, Dr. Chu directs the activities of more than 1,000 scientists and technologists. Together, they conduct and support research requested by federal agencies involved in advancing national security and advising on science and technology issues. Dr. Chu served in the Department of Defense as Under Secretary of Defense for Personnel and Readiness from 2001-2009, and earlier as Assistant Secretary of Defense and Director for Program Analysis and Evaluation from 1981-1993. From 1978-1981 he was the Assistant Director of the Congressional Budget Office for National Security and International Affairs. Dr. Chu served in the U. S. Army from 1968-1970. He was an economist with the RAND Corporation from 1970-1978, director of RAND’s Washington Office from 1994-1998, and vice president for its Army Research Division from 1998-2001. He earned a bachelor of arts in economics and mathematics, and his doctorate in economics, from Yale University. Dr. Chu is a member of the Defense Science Board and a Fellow of the National Academy of Public Administration. He is a recipient of the Department of Defense Medal for Distinguished Public Service with Gold Palm, the Department of Veterans Affairs Meritorious Service Award, the Department of the Army Distinguished Civilian Service Award, the Department of the Navy Distinguished Public Service Award, and the National Academy of Public Administration’s National Public Service Award.
|
Keynote |
| 2018 |
| Closing Remarks | Mr. Robert Behler Director DOT&E Bio
Robert F. Behler was sworn in as Director of Operational Test and Evaluation on December 11, 2017. A Presidential appointee confirmed by the United States Senate, he serves as the senior advisor to the Secretary of Defense on operational and live fire test and evaluation of Department of Defense weapon systems. Prior to his appointment, he was the Chief Operating Officer and Deputy Director of the Carnegie Mellon University Software Engineering Institute (SEI), a Federally Funded Research and Development Center. SEI is a global leader in advancing software development and cybersecurity to solve the nation’s toughest problems through focused research, development, and transition to the broader software engineering community. Before joining the SEI, Mr. Behler was the President and CEO of SRC, Inc. (formerly the Syracuse Research Corporation). SRC is a not-for-profit research and development corporation with a forprofit manufacturing subsidiary that focuses on radar, electronic warfare and cybersecurity technologies. Prior to working at SRC, Mr. Behler was the General Manager and Senior Vice President of the MITRE Corp where he provided leadership to more than 2,500 technical staff in 65 worldwide locations. He joined MITRE from the Johns Hopkins University Applied Physics Laboratory where he was a General Manager for more than 350 scientists and engineers as they made significant contributions to critical Department of Defense (DOD) precision engagement challenges. General Behler served 31 years in the United States Air Force, retiring as a Major General in 2003. During his military career, he was the Principal Adviser for Command and Control, Intelligence, Surveillance and Reconnaissance (C21SR) to the Secretary and Chief of Staff of the U.S. Air Force (USAF). International assignments as a general officer included the Deputy Commander for NATO’s Joint Headquarters North in Stavanger, Norway. He was the Director of the Senate Liaison Office for the USAF during the 104th congress. Mr. Behler also served as the assistant for strategic systems to the Director of Operational Test and Evaluation. As an experimental test pilot, he flew more than 65 aircraft types. Operationally he flew worldwide reconnaissance missions in the fastest aircraft in the world, the SR-71 Blackbird. Mr. Behler is a Fellow of the Society of Experimental Test Pilots and an Associate Fellow of the American Institute of Aeronautics and Astronautics. He is a graduate of the University of Oklahoma where he received a B.S. and M.S. in aerospace engineering, has a MBA from Marymount University and was a National Security Fellow at the JFK School of Government at Harvard University. Mr. Behler has recently been on several National Research Council studies for the National Academy of Sciences including: “Critical Code,” “Software Producibility, Achieving Effective Acquisition of Information Technology in the Department of Defense” and “Development Planning: A Strategic Approach to Future Air Force Capabilities.”
|
Keynote |
2018 | |
| Testing and Analytical Challenges on the Path to Hypersonic Flight | Dr. Mark Lewis Director IDA-STPI Bio
Dr. Mark J. Lewis is the Director of IDA’s Science and Technology Policy Institute, a federally funded research and development center. He leads an organization that provides analysis of national and international science and technology issues to the Office of Science and Technology Policy in the White House, as well as other Federal agencies including the National Institutes of Health, the National Science Foundation, NASA, the Department of Energy, Homeland Security, and the Federal Aviation Administration, among others.Prior to taking charge of STPI, Dr. Lewis served as the Willis Young, Jr. Professor and Chair of the Department of Aerospace Engineering at the University of Maryland. A faculty member at Maryland for 24 years, Dr. Lewis taught and conducted basic and applied research. From 2004 to 2008, Dr. Lewis was the Chief Scientist of the U.S. Air Force. From 2010 to 2011, he was President of the American Institute of Aeronautics and Astronautics (AIAA).Dr. Lewis attended the Massachusetts Institute of Technology, where he received a Bachelor of Science degree in aeronautics and astronautics, Bachelor of Science degree in earth and planetary science (1984), Master of Science (1985), and Doctor of Science (1988) in aeronautics and astronautics.Dr. Lewis is the author of more than 300 technical publications and has been an adviser to more than 70 graduate students. Dr. Lewis has also served on various advisory boards for NASA, the Air Force, and DoD, including two terms on the Air Force Scientific Advisory Board, the NASA Advisory Council, and the Aeronautics and Space Engineering Board of the National Academies.Dr. Lewis’s awards include the Meritorious Civilian Service Award and Exceptional Civilian Service Award; he was also recognized as the 1994 AIAA National Capital Young Scientist/Engineer of the Year, IECEC/ AIAA Lifetime Achievement Award, and is an Aviation Week and Space Technology Laureate (2007).
|
Keynote |
| 2018 |
| Journey to a Data Centric Approcach for National Security | Dr. Marcey Hoover Quality Assurance Director Sandia National Labortories Bio
As Quality Assurance Director, Dr. Marcey Hoover is responsible for designing and sustaining the Laboratories’ quality assurance system and the associated technical capabilities needed for flawless execution of safe, secure, and efficient work to deliver exceptional products and services to its customers.Marcey previously served as the Senior Manager responsible for developing the science and engineering underpinning efforts to predict and influence the behavior of complex, highly interacting systems critical to our nation’s security posture. In her role as Senior Manager and Chief of Operations for Sandia’s Energy and Climate program, Marcey was responsible for strategic planning, financial management, business development, and communications. In prior positions, she managed organizations responsible for (1) quality engineering on new product development programs, (2) research and development of advanced computational techniques in the engineering sciences, and (3) development and execution of nuclear weapon testing and evaluation programs. Marcey has also led several executive office functions, including corporate- level strategic planning.Active in both the American Statistical Association and the American Society for Quality (ASQ), Marcey served two terms as the elected ASQ Statistics Division Treasurer. She was recognized as the Outstanding Alumni of the Purdue University Statistics Department in 2009 and nominated in 2011 for the YWCA Middle Rio Grande Women on the Move award. She currently serves on both the Purdue Strategic Research Advisory Council and the Statistics Alumni Advisory Board, and as a mentor for Big Brothers Big Sisters of New Mexico.Marcey received her bachelor of science degree in mathematics from Michigan State University, and her master of science and doctor of philosophy degrees in mathematical statistics from Purdue University.
|
Keynote |
| 2018 |
| NASA AERONAUTICS | Mr. Bob Pearce Deputy Associate Administrator Strategy, Aeronautics Research Mission DirectorateNASA Bio
Mr. Pearce is responsible for leading aeronautics research mission strategic planning to guide the conduct of the agency’s aeronautics research and technology programs, as well as leading ARMD portfolio planning and assessments, mission directorate budget development and approval processes, and review and evaluation of all of NASA’s aeronautics research mission programs for strategic progress and relevance. Pearce is also currently acting director for ARMD’s Airspace Operations and Safety Program, and responsible for the overall planning, management and evaluation of foundational air traffic management and operational safety research. Previously he was director for strategy, architecture and analysis for ARMD, responsible for establishing a strategic systems analysis capability focused on understanding the system-level impacts of NASA’s programs, the potential for integrated solutions, and the development of high-leverage options for new investment and partnership. From 2003 until July 2010, Pearce was the deputy director of the FAA-led Next Generation Air Transportation System (NextGen) Joint Planning and Development Office (JPDO). The JPDO was an interagency office tasked with developing and facilitating the implementation of a national plan to transform the air transportation system to meet the long-term transportation needs of the nation. Prior to the JPDO, Pearce held various strategic and program management positions within NASA. In the mid-1990s he led the development of key national policy documents including the National Science and Technology Council’s “Goals for a National Partnership in Aeronautics Research and Technology” and the “Transportation Science and Technology Strategy.” These two documents provided a substantial basis for NASA’s expanded investment in aviation safety and airspace systems. He began his career as a design engineer at the Grumman Corporation, working on such projects as the Navy’s F-14 Tomcat fighter and DARPA’s X-29 Forward Swept Wing Demonstrator. Pearce also has experience from the Department of Transportation’s Volpe National Transportation Systems Center where he made contributions in the area of advanced concepts for intercity transportation systems. Pearce has received NASA’s Exceptional Service Medal for sustained excellence in planning and advocating innovative aeronautics programs in conjunction with the White House and other federal agencies. He received NASA’s Exceptional Achievement Medal for outstanding leadership of the JPDO in support of the transformation of the nation’s air transportation system. Pearce has also received NASA’s Cooperative External Achievement Award and several Exceptional Performance and Group Achievement Awards.He earned a bachelor’s of science degree in mechanical and aerospace engineering from Syracuse University, and a master’s of science degree in technology and policy from the Massachusetts Institute of Technology.
|
Keynote |
| 2018 |
| High-Effective Statistical Collaboration. The Art and the Science | Dr. Peter Parker Team Lead Advance Measurement Systems NASA Bio
Dr. Parker is Team Lead for Advanced Measurement Systemsat the National Aeronautics and Space Administration’s Langley Research Center in Hampton, Virginia. He serves an Agency-wide statistical expert across all of NASA’s mission directorates of Exploration, Aeronautics, and Science to infuse statistical thinking, engineering, and methods including statistical design of experiments, response surface methodology, and measurement system characterization. His expertise is in collaboratively integrating research objectives, measurement sciences, test design, and statistical methods to produce actionable knowledge for aerospace research and development. He holds a B.S. in Mechanical Engineering, a M.S. in Applied Physics and Computer Science and a M.S. and Ph.D. in Statistics from Virginia Tech. Dr. Parker is a senior member of the American Institute for Aeronautics and Astronautics, American Society for Quality, and the American Statistical Association. Dr. Parker currently Chairs the American Society for Quality’s Publication Management Board and previously served as Editor-in-Chief of the journal Quality Engineering.
|
Keynote |
| 2018 |
| Consensus Building | Dr. Antonio Possolo NIST Fellow, Chief Statistician National Institute of Standards and Technology. Bio
Antonio Possolo holds a Ph.D. in statistics from Yale University, and has been practicing the statistical arts for more than 35 years, in industry (General Electric, Boeing), academia (Princeton University, University of Washington in Seattle, Classical University of Lisboa), and government. He is committed to the development and application of probabilistic and statistical methods that contribute to advances in science and technology, and in particular to measurement science.
|
Keynote |
| 2018 |
| Opening Keynote | Mr. Dave Duma Assistant Director Operational Test and Evaluation Bio
Mr. Duma is the Acting Director, Operational Test and Evaluation as of January 20, 2017. Mr. Duma was appointed as the Principal Deputy Director, Operational Test and Evaluation in January 2002. In this capacity he is responsible for all functional areas assigned to the office. He participates in the formulation, development, advocacy, and oversight of policies of the Secretary of Defense and in the development and implementation of test and test resource programs. He oversees the planning, conduct, analysis, evaluation, and reporting of operational and live fire testing. He serves as the Appropriation Director and Comptroller for the Operational Test and Evaluation, Defense Appropriation and coordinates all Planning, Programming, and Budgeting Execution matters. He previously served as Acting Director, Operational Test and Evaluation from February 2005 to July 2007 and again from May 2009 to September 2009.Mr. Duma also served as the Acting Deputy Director, Operational Test and Evaluation from January 1992 to June 1994. In this capacity he was responsible for oversight of the planning, conduct, analysis, and reporting of operational test and evaluation for all major conventional weapons systems in the Department of Defense. He supervised the development of evaluation plans and test program strategies, observed the conduct of operational test events, evaluated operational field tests of all armed services and submitted final reports for Congress.Mr. Duma returned to government service from the commercial sector. In private industry he worked a variety of projects involving test and evaluation; requirements generation; command, control, communications, intelligence, surveillance and reconnaissance; modeling and simulation; and software development.Mr. Duma has 30 years of naval experience during which he was designated as a Joint Service Officer. He served as the Director, Test and Evaluation Warfare Systems for the Chief of Naval Operations, the Deputy Commander, Submarine Squadron TEN, and he commanded the nuclear powered submarine USS SCAMP (SSN 588).Mr. Duma holds Masters of Science degrees in National Security and Strategic Studies and in Management. He holds a Bachelor of Science degree in Nuclear Engineering. He received the U.S. Presidential Executive Rank Award on two occasions; in 2008, the Meritorious Executive Award and in 2015, the Distinguished Executive Rank Award. He is a member of the International Test and Evaluation Association.
|
Keynote |
| 2018 |
| Bayesian Analysis
Abstract
This course will cover the basics of the Bayesian approach to practical and coherent statistical inference. Particular attention will be paid to computational aspects, including MCMC. Examples/practical hands-on exercises will the run gamut from toy illustration to real-world data analysis from all areas of science, with R implementations/coaching provided. The course closely follows P.D. Hoff’s “A First Course in Bayesian Statistical Methods”—Springer 2009. Some examples are borrowed from two other texts which are nice references to have. J. Albert’s’ “Bayesian Computation with R”— Springer 2nd ed. 2009; and “A. Gelman, J.B. Carlin, H.S. Stern, D. Dunson, A. Vehtari and D.B. Rubin’ s “Bayesian Data Analysis”—3rd ed. 2013.
|
Dr. Robert Gramacy Virginia Tech |
Short Course |
| 2019 |
| Categorical Data Analysis
Abstract
Categorical data is abundant in the 21st century, and its analysis is vital to advance research across many domains. Thus, data-analytic techniques that are tailored for categorical data are an essential part of the practitioner’s toolset. The purpose of this short course is to help attendees develop and sharpen their abilities with these tools. Topics covered in this short course will include logistic regression, ordinal regression, and classification, and methods to assess predictive accuracy of these approaches will be discussed. Data will be analyzed using the R software package, and course content loosely follow Alan Agresti’s excellent textbook An Introduction to Categorical Data Analysis, Third Edition.
|
Dr. Christopher Franck Virginia Tech |
Short Course |
| 2019 |
| Multivariate Data Analysis
Abstract
In this one-day workshop, we will explore five techniques that are commonly used to model human behavior: principal component analysis, factor analysis, cluster analysis, mixture modeling, and multidimensional scaling. Brief discussions of the theory of each method will be provided, along with some examples showing how the techniques work and how the results are interpreted in practice. Accompanying R-code will be provided so attendees are able to implement these methods on their own.
|
Dr. Doug Steinley University of Missouri |
Short Course |
| 2019 |
| Statistical Methods for Modeling and Simulation Verification and Validation
Abstract
Statistical Methods for Modeling and Simulation Verification and Validation is a 1-day tutorial in applied statistical methods for the planning, designing and analysis of simulation experiments and live test events for the purposes of verifying and validating models and simulations. The course covers the fundamentals of verification and validation of models and simulations, as it is currently practiced and as is suggested for future applications. The first session is largely an introduction to modeling and simulation concepts, verification and validation policies, along with a basic introduction to data visualization and statistical methods. Session 2 covers the essentials of experiment design for simulations and live test events. The final session focuses on analysis techniques appropriate for designed experiments and tests, as well as observational data, for the express purpose of simulation validation. We look forward to your participation in this course.
|
Dr. Jim Simpson, Dr. Jim Wisnowski, and Dr. Stargel Doane JK Analytics LLC, Adsurgo LLC, Analytical Arts LLC |
Short Course |
| 2019 |
| Uncertainty Quantification
Abstract
We increasingly rely on mathematical and statistical models to predict
phenomena ranging from nuclear power plant design to profits made in financial markets. When assessing the feasibility of these predictions, it is critical to quantify uncertainties associated with the models, inputs to the models, and data used to calibrate the models. The synthesis of statistical and mathematical techniques, which can be used to quantify input and response uncertainties for simulation codes that can take hours to days to run, comprises the evolving field of uncertainty quantification. The use of data, to improve the predictive accuracy of models, is central to uncertainty quantification so we will begin by providing an overview of how Bayesian techniques can be used to construct distributions for model inputs. We will subsequently describe the computational issues associated with propagating these distributions through complex models to construct prediction intervals for statistical quantities of interest such as expected
profits or maximal reactor temperatures. Finally, we will describe the use of
sensitivity analysis to isolate critical model inputs and surrogate model
construction for simulation codes that are too complex for direct statistical
analysis. All topics will be motivated by examples arising in engineering, biology, and economics.
|
Dr. Ralph Smith North Carolina State University |
Short Course |
| 2019 |
| Design of Experiments
Abstract
Overview/Course Outcomes-
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.
The course outcomes are:
• Ability to plan and execute experiments
• Ability to collect data and analyze and interpret these data to provide the knowledge required for business success
• Knowledge of a wide range of modern experimental tools that enable practitioners to customize their experiment to meet practical resource constraints
The topics covered during the course are:
• Fundamentals of DOX – randomization, replication, and blocking.
• Planning for a designed experiment – type and size of design, factor selection, levels and ranges, response measurement, sample sizes.
• Graphical and statistical approaches to DOX analysis.
• Blocking to eliminate the impact of nuisance factors on experimental results.
• Factorial experiments and interactions.
• Fractional factorials – efficient and effective use of experimental resources.
• Optimal designs
• Response surface methods
• A demonstration illustrating and comparing the effectiveness of different experimental design strategies.
This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course
Who Should Attend-
The course is suitable for participants from an engineering or technical background. Participants will need some previous experience and background in statistical methods.
Reference Materials-
The course is based on the textbook Design and Analysis of Experiments, 9th Edition, by Douglas C. Montgomery. JMP Software will be discussed and illustrated.
|
Dr. Doug Montgomery and Dr. Caleb King Arizona State University, JMP |
Short Course | 2019 | |
| Overview of Design of Experiments
Abstract
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance. This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course
|
Dr. Doug Montgomery Professor Arizon State University Bio
Douglas C. Montgomery, Ph.D., is Regent’s Professor of Industrial Engineering and Statistics and ASU Foundation Professor of Engineering at Arizona State University. He was the John M. Fluke Distinguished Professor of Engineering, Director of Industrial Engineering and Professor of Mechanical Engineering at the University of Washington in Seattle. He was a Professor of Industrial and Systems Engineering at the Georgia Institute of Technology. He holds BSIE, MS and Ph.D. degrees from Virginia Tech. Dr. Montgomery’s industrial experience includes engineering assignments with Union Carbide Corporation and Eli Lilly and Company. He also has extensive consulting experience. Dr. Montgomery’s professional interests focus on industrial statistics, including design of experiments, quality and reliability engineering, applications of linear models, and time series analysis and forecasting. He also has interests in operations research and statistical methods applied to modeling and analyzing manufacturing systems. He was a Visiting Professor of Engineering at the Monterey Institute of Technology in Monterey, Mexico, and a University Distinguished Visitor at the University of Manitoba. Dr. Montgomery has conducted basic research in empirical stochastic modeling, process control, and design of experiments. The Department of Defense, the Office of Naval Research, the National Science Foundation, the United States Army, and private industry have sponsored his research. He has supervised 66 doctoral dissertations and over 40 MS theses and MS Statistics Projects.
|
Short Course |
| 2016 |
| Experiences in Reliability Analysis
Abstract
Reliability assurance processes in manufacturing industries require data-driven information for making product-design decisions. Life tests, accelerated life tests, and accelerated degradation tests are commonly used to collect reliability data. Data from products in the field provide another important source of useful reliability information. Due to complications like censoring, multiple failure modes, and the need for extrapolation, these reliability studies typically yield data that require special statistical methods. This presentation will describe the analyses of a collection of different life data analysis applications in the area of product reliability. Methods used in the analyses include Weibull and lognormal analysis, analysis of data with multiple failure modes, accelerated test analysis, analysis of both repeated measures and destructive degradation data and the analysis of recurrence data from repairable systems.
|
Dr. Bill Meeker Professor Iowa State University Bio
William Meeker is Professor of Statistics and Distinguished Professor of Liberal Arts and Sciences at Iowa State University. He is a Fellow of the American Statistical Association, the American Society for Quality, and the American Association for the Advancement of Science, and a past Editor of Technometrics. He is co-author of the books Statistical Methods for Reliability Data with Luis Escobar (1998), and Statistical Intervals with Gerald Hahn (1991), and of numerous publications in the engineering and statistical literature. He has won numerous awards for his research and service, including the Brumbaugh, Hunter, Sacks, Shewhart, Youden, and Wilcoxon awards. He has done research and consulted extensively on problems in reliability data analysis, warranty analysis, accelerated testing, nondestructive evaluation, and statistical computing.
|
Short Course |
| 2016 |
| Advanced Regression
Abstract
This course assumes that the students have previous exposure to simple and multiple linear regression topics, at least from their undergraduate education; however, the course does not assume that the students are very current with this material. The course goal is to provide more insights and details into some of the more important topics. The presentation emphasizes the use of software for performing the analysis.
|
Dr. Geoff Vining Professor Virginia Tech University Bio
Geoff Vining is a Professor of Statistics at Virginia Tech. From 1999 – 2006, he also was the department head. He currently is a member of the American Society for Quality (ASQ) Board of Directors and Past-Chair of the ASQ Technical Communities Council. In 2016, he serves as the ASQ Treasurer. He is a Fellow of the ASQ, a Fellow of the American Statistical Association (ASA), and an Elected Member of the International Statistical Institute. Dr. Vining served as Editor of the Journal of Quality Technology from 1998 – 2000 and as Editor-in-Chief of Quality Engineering from 2008-2009. Dr. Vining has authored or co-authored three textbooks, all in multiple editions. He was several of the most important awards in industrial statistics/quality engineering including the ASQ’s Shewhart Medal, Brumbaugh Award, and Hunter Award along with the ENBIS Box Medal. He is an internationally recognized expert in the use of experimental design for quality and productivity improvement and in the application of statistical process control. He has extensive consulting experience, most recently with NASA and the U.S. Department of Defense.
|
Short Course |
| 2016 |
| Introduction to Bayesian
Abstract
This course will cover the basics of the Bayesian approach to practical and coherent statistical inference. Particular attention will be paid to computational aspects, including MCMC. Examples will the run gamut from toy illustration to real-world data analysis from all areas of science, with R implementations provided.
|
Dr. Robert Gramacy Associate Professor University of Chicago Bio
Professor Gramacy is an Associate Professor of Econometrics and Statistics in the Booth School of business, and a fellow of the Computation Institute at The University of Chicago. His research interests include Bayesian modeling methodology, statistical computing, Monte Carlo inference, nonparametric regression, sequential design, and optimizaton under uncertainty. He specializes in areas of real-data analysis where the ideal modeling apparatus is impractical, or where the current solutions are inefficient and thus skimp on fidelity.
|
Short Course | 2016 | |
| Introduction To Design of Experiments
Abstract
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.
|
Dr. Doug Montgomery Professor Arizona State University |
Short Course | 2017 | |
| Introduction To Design of Experiments
Abstract
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.
|
Dr. Brad Jones Professor Arizona State University |
Short Course | 2017 | |
| Data Farming
Abstract
This tutorial is designed for newcomers to simulation-based experiments. Data farming is the process of using computational experiments to “grow” data, which can then be analyzed using statistical and visualization techniques to obtain insight into complex systems. The focus of the tutorial will be on gaining practical experience with setting up and running simulation experiments, leveraging recent advances in large-scale simulation experimentation pioneered by the Simulation Experiments & Efficient Designs (SEED) Center for Data Farming at the Naval Postgraduate School (http://harvest.nps.edu). Participants will be introduced to fundamental concepts, and jointly explore simulation models in an interactive setting. Demonstrations and written materials will supplement guided, hands-on activities through the setup, design, data collection, and analysis phases of an experiment-driven simulation study.
|
Dr. Susan Sanchez Naval Postgraduate School |
Short Course |
| 2017 |
| Split-Plot and Restricted Randomization Designs
Abstract
Have you ever built what you considered to be the ideal designed experiment, then passed it along to be run and learn later that your recommended run order was ignored? Or perhaps you were part of a test execution team and learned too late that one or more of your experimental factors are difficult or time-consuming to change. We all recognize that the best possible guard against lurking background noise is complete randomization, but often we find that a randomized run order is extremely impractical or even infeasible. Split-plot design and analysis methods have been around for over 80 years, but only in the last several years have the methods fully matured and been made available in commercial software. This class will introduce you to the world of practical split-plot design and analysis methods. We’ll provide you the skills to effectively build designs appropriate to your specific needs and demonstrate proper analysis techniques using general linear models, available in the statistical software. Topics include split-plots for 2-level and mixed-level factor sets, for first and second order models, as well as split-split-plot designs.
|
Dr. Jim Simpson JK Analytics |
Short Course |
| 2017 |
| Statistical Models for Reliability Data
Abstract
Engineers in manufacturing industries require data-driven reliability information for making business, product-design, and engineering decisions. The owners and operators of fleets of systems also need reliability information to make good decisions. The course will focus on concepts, examples, models, data analysis, and interpretation of reliability data analyses. Examples and exercises will include product field (maintenance or warranty) data and accelerated life test data. After completing this course, participants will be able to recognize and properly deal with different kinds of reliability data and properly interpret important reliability metrics. Topics will include the use probability plots to identify appropriate distributional models (e.g. Weibull and lognormal distributions), estimating important quantities like distribution quantiles and failure probabilities, the analysis of data with multiple failure modes, and the analysis of recurrence data from a fleet of systems or a reliability growth program.
|
Dr. William Meeker Professor Iowa State University |
Short Course |
| 2017 |
| Data Visualization
Abstract
Data visualization allows us to quickly explore and discover relationships graphically and interactively. We will provide the foundations for creating better graphical information to accelerate the insight discovery process and enhance the understandability of reported results. First principles and the “human as part of the system” aspects of information visualization from multiple leading sources such as Harvard Business Review, Edward Tufte, and Stephen Few will be explored using representative example data sets. We will discuss best practices for graphical excellence to most effectively, clearly, and efficiently communicate your story. We will explore visualizations applicable across the conference themes (computational modeling, DOE, statistical engineering, modeling & simulation, and reliability) for univariate, multivariate, time-dependent, and geographical data.
|
Dr. Jim Wisnowski Adsurgo |
Short Course |
| 2017 |
| Modern Response Surface Methods & Computer Experiments
Abstract
This course details statistical techniques at the interface between mathematical modeling via computer simulation, computer model meta-modeling (i.e., emulation/surrogate modeling), calibration of computer models to data from field experiments, and model-based sequential design and optimization under uncertainty (a.k.a. Bayesian Optimization). The treatment will include some of the historical methodology in the literature, and canonical examples, but will primarily concentrate on modern statistical methods, computation and implementation, as well as modern application/data type and size. The course will return at several junctures to real-word experiments coming from the physical and engineering sciences, such as studying the aeronautical dynamics of a rocket booster re-entering the atmosphere; modeling the drag on satellites in orbit; designing a hydrological remediation scheme for water sources threatened by underground contaminants; studying the formation of super-nova via radiative shock hydrodynamics. The course material will emphasize deriving and implementing methods over proving theoretical properties.
|
Dr. Robert Gramacy Virginia Polytechnic Institute and State University |
Short Course |
| 2018 |
| Overview of Design of Experiments
Abstract
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.The course outcomes are: • Ability to plan and execute experiments. • Ability to collect data and analyze and interpret these data to provide the knowledge required for business success. • Knowledge of a wide range of modern experimental tools that enable practitioners to customize their experiment to meet practical resource constraintsThe topics covered during the course are: • Fundamentals of DOX – randomization, replication, and blocking. • Planning for a designed experiment – type and size of design, factor selection, levels and ranges, response measurement, sample sizes. • Graphical and statistical approaches to DOX analysis. • Blocking to eliminate the impact of nuisance factors on experimental results. • Factorial experiments and interactions. • Fractional factorials – efficient and effective use of experimental resources. • Optimal designs. • Response surface methods. • A demonstration illustrating and comparing the effectiveness of different experimental design strategies. This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course.
|
Dr. Bradley Jones Distinguished Research Fellow JMP Division/SAS |
Short Course |
| 2018 |
| Overview of Design of Experiments
Abstract
Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.The course outcomes are: • Ability to plan and execute experiments. • Ability to collect data and analyze and interpret these data to provide the knowledge required for business success. • Knowledge of a wide range of modern experimental tools that enable practitioners to customize their experiment to meet practical resource constraintsThe topics covered during the course are: • Fundamentals of DOX – randomization, replication, and blocking. • Planning for a designed experiment – type and size of design, factor selection, levels and ranges, response measurement, sample sizes. • Graphical and statistical approaches to DOX analysis. • Blocking to eliminate the impact of nuisance factors on experimental results. • Factorial experiments and interactions. • Fractional factorials – efficient and effective use of experimental resources. • Optimal designs. • Response surface methods. • A demonstration illustrating and comparing the effectiveness of different experimental design strategies. This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course.
|
Dr. Doug Motgomery Regents’ Professor of Industrial Engineering and Statistics ASU Foundation Professor of Engineering Arizona State University |
Short Course |
| 2018 |
| Introduction to R
Abstract
This course is designed to introduce participants to the R programming language and the R studio editor. R is a free and open-source software for summarizing data, creating visuals of data, and conducting statistical analyses. R can offer many advantages over programs such as Excel including faster computation, customized analyses, access to the latest statistical techniques, automation of tasks, and the ability to easily reproduce research. After completing this course, a new user should be able to: • Import/export data from/to external files.
• Create and manipulate new variables.
• Conduct basic statistical analyses (such as t-tests and linear regression).
• Create basic graphs.
• Install and use R packages
Participants should bring a laptop for the interactive components of the course.
|
Dr. Justin Post North Carlina State Univeristy |
Short Course |
| 2018 |
| Survey Construction and Analysis
Abstract
In this course, we introduce the main concepts of the survey methodology process – from survey sampling design to analyzing the data obtained from complex survey designs. The course topics include: 1. Introduction to the Survey Process
2. R Tools
3. Sampling Designs – Simple Random Sampling, Cluster Sampling, Stratified Sampling, and more
4. Weighting and Variance Estimation
5. Exploratory Data Analysis
6. Complex Survey Analysis
We use a combination of lectures and hands-on exercises using R. Students are expected to have R and associated packages installed on their computers. We will send a list of required packages before the course. We also use data from Department of Defense surveys, where appropriate.
|
Dr. Wendy Matrinez Bureau of Labor and Statistics |
Short Course |
| 2018 |
| Survey Construction and Analysis
Abstract
In this course, we introduce the main concepts of the survey methodology process – from survey sampling design to analyzing the data obtained from complex survey designs. The course topics include: 1. Introduction to the Survey Process
2. R Tools
3. Sampling Designs – Simple Random Sampling, Cluster Sampling, Stratified Sampling, and more
4. Weighting and Variance Estimation
5. Exploratory Data Analysis
6. Complex Survey Analysis
We use a combination of lectures and hands-on exercises using R. Students are expected to have R and associated packages installed on their computers. We will send a list of required packages before the course. We also use data from Department of Defense surveys, where appropriate.
|
Dr. MoonJung Cho Bureau of Labor and Statistics |
Short Course |
| 2018 |
| Uncertainty Quantification
Abstract
We increasingly rely on mathematical and statistical models to predict phenomena ranging from nuclear power plant design to profits made in financial markets. When assessing the feasibility of these predictions, it is critical to quantify uncertainties associated with the models, inputs to the models, and data used to calibrate the models. The synthesis of statistical and mathematical techniques, which can be used to quantify input and response uncertainties for simulation codes that can take hours to days to run, comprises the evolving field of uncertainty quantification. The use of data, to improve the predictive accuracy of models, is central to uncertainty quantification so we will begin by providing an overview of how Bayesian techniques can be used to construct distributions for model inputs. We will subsequently describe the computational issues associated with propagating these distributions through complex models to construct prediction intervals for statistical quantities of interest such as expected profits or maximal reactor temperatures. Finally, we will describe the use of sensitivity analysis to isolate critical model inputs and surrogate model construction for simulation codes that are too complex for direct statistical analysis. All topics will be motivated by examples arising in engineering, biology, and economics.
|
Dr. Ralph Smith North Carlina State Univeristy |
Short Course |
| 2018 |
| Tutorial: Combinatorial Methods for Testing and Analysis of Critical Software and Security Systems
Abstract
Combinatorial methods have attracted attention as a means of providing strong assurance at reduced cost, but when are these methods practical and cost-effective? This tutorial includes two sections on the basis and application of combinatorial test methods: The first section explains the background, process, and tools available for combinatorial testing, with illustrations from industry experience with the method. The focus is on practical applications, including an industrial example of testing to meet FAA-required standards for life-critical software for commercial aviation. Other example applications include modeling and simulation, mobile devices, network configuration, and testing for a NASA spacecraft. The discussion will also include examples of measured resource and cost reduction in case studies from a variety of application domains.
The second part explains combinatorial testing-based techniques for effective security testing of software components and large-scale software systems. It will develop quality assurance and effective re-verification for security testing of web applications and testing of operating systems. It will further address how combinatorial testing can be applied to ensure proper error-handling of network security protocols and provide the theoretical guarantees for detecting Trojans injected in cryptographic hardware. Procedures and techniques, as well as workarounds will be presented and captured as guidelines for a broader audience.
|
Rick Kuhn, Dimitris Simos, and Raghu Kacker National Institute of Standards & Technology |
Tutorial |
| 2019 |
| Tutorial: Cyber Attack Resilient Weapon Systems
Abstract
This tutorial is an abbreviated version of a 36-hour short course recently provided by UVA to a class composed of engineers working at the Defense Intelligence Agency. The tutorial provides a definition for cyber attack resilience that is an extension of earlier definitions of system resilience that were not focused on cyber attacks. Based upon research results derived by the University of Virginia over an eight year period through DoD/Army/AF/Industry funding , the tutorial will illuminate the following topics: 1) A Resilence Design Requirements methodology and the need for supporting analysis tools, 2) a System Architecture approach for achieving resilience, 3) Example resilience design patterns and example prototype implementations, 4) Experimental results regarding resilience-related roles and readiness of system operators, and 5) Test and Evaluation Issues. The tutorial will be presented by UVA Munster Professor Barry Horowitz.
|
Barry Horowitz Professor, Systems Engineering University of Virginia |
Tutorial |
| 2019 |
| Tutorial: Learning Python and Julia
Abstract
In recent years, the programming language Python with its supporting ecosystem has established itself as a significant capability to support the activities of the typical data scientist. Recently, version 1.0 of the programming language Julia has been released; from a software engineering perspective, it can be viewed as a modern alternative. This tutorial presents both Python and Julia from both a user and developer point of view. From a user’s point of view, the basic syntax of each, along with fundamental prerequisite knowledge presented. From a developers point of view the underlying infrastructure of the programming language / interpreter / compiler is discussed.
|
Douglas Hodson Associate Professor Air Force Institute of Technology |
Tutorial | 2019 | |
| Tutorial: Statistics Boot Camp
Abstract
In the test community, we frequently use statistics to extract meaning from data. These inferences may be drawn with respect to topics ranging from system performance to human factors. In this mini-tutorial, we will begin by discussing the use of descriptive and inferential statistics, before exploring the basics of interval estimation and hypothesis testing. We will introduce common statistical techniques and when to apply them, and conclude with a brief discussion of how to present your statistical findings graphically for maximum impact.
|
Kelly Avery IDA |
Tutorial |
| 2019 |
| Tutorial: Reproducible Research
Abstract
Analyses are “reproducible” if the same methods applied to the same data produce identical results when run again by another researcher (or you in the future). Reproducible analyses are transparent and easy for reviewers to verify, as results and figures can be traced directly to the data and methods that produced them. There are also direct benefits to the researcher. Real-world analysis workflows inevitably require changes to incorporate new or additional data, or to address feedback from collaborators, reviewers, or sponsors. These changes are easier to make when reproducible research best practices have been considered from the start.
Poor reproducibility habits result in analyses that are difficult or impossible to review, are prone to compounded mistakes, and are inefficient to re-run in the future. They can lead to duplication of effort or even loss of accumulated knowledge when a researcher leaves your organization. With larger and more complex datasets, along with more complex analysis techniques, reproducibility is more important than ever.
Although reproducibility is critical, it is often not prioritized either due to a lack of time or an incomplete understanding of end-to-end opportunities to improve reproducibility.
This tutorial will discuss the benefits of reproducible research and will demonstrate ways that analysts can introduce reproducible research practices during each phase of the analysis workflow: preparing for an analysis, performing the analysis, and presenting results. A motivating example will be carried throughout to demonstrate specific techniques, useful tools, and other tips and tricks where appropriate. The discussion of specific techniques and tools is non-exhaustive; we focus on things that are accessible and immediately useful for someone new to reproducible research. The methods will focus mainly on work performed using R, but the general concepts underlying reproducible research techniques can be implemented in other analysis environments, such as JMP and Excel, and are briefly discussed.
By implementing the approaches and concepts discussed during this tutorial, analysts in defense and aerospace will be equipped to produce more credible and defensible analyses of T&E data.
|
Andrew Flack, Kevin Kirshenbaum, and John Haman IDA |
Tutorial |
| 2019 |
| Tutorial: Developing Valid and Reliable Scales
Abstract
The DoD uses psychological measurement to aid in decision-making about a variety of issues including the mental health of military personnel before and after combat, and the quality of human-systems interactions. To develop quality survey instruments (scales) and interpret the data obtained from these instruments appropriately, analysts and decision-makers must understand the factors that affect the reliability and validity of psychological measurement. This tutorial covers the basics of scale development and validation and discusses current efforts by IDA, DOT&E, ATEC, and JITC to develop validated scales for use in operational test and evaluation.
|
Heather Wojton & Shane Hall IDA / USARMY ATEC |
Tutorial |
| 2019 |
| The Bootstrap World
Abstract
Bootstrapping is a powerful tool for statistical estimation and inference. In this tutorial, we will use operational test scenarios to provide context when exploring examples ranging from the simple (estimating a sample mean) to the complex (estimating a confidence interval for system availability). Areas of focus will include point estimates, confidence intervals, parametric bootstrapping and hypothesis testing with the bootstrap. The strengths and weaknesses of bootstrapping will also be discussed.
|
Dr. Matt Avery Research Staff Member IDA |
Tutorial |
| 2016 |
|
Abstract
This tutorial will provide attendees with a live demo of an open source software reliability tool to automatically apply models to data. Functionality to be illustrated includes how to: Select and view data in time between failures, cumulative failures, and failure intensity formats. Apply trend tests to determine if a data set exhibits reliability growth, which is a prerequisite to apply software reliability growth models. Apply models to a data set . Apply measures of model goodness of fit to obtain quantitative guidance to select one or more models based on the needs of the user .Query model results to determine the additional testing time required to achieve a desired reliability. Following this live demonstration an overview of the underlying mathematical theory will be presented, including: Representation of failure data formats. Laplace trend test and running arithmetic average. Maximum likelihood estimation. Failure rate and failure counting software reliability models. Akaike information criterion and predictive sum of squares error.
|
Dr. Lance Fiondella Univeristy of Massachusetts, Dartmouth |
Tutorial |
| 2016 |
| A Statistical Tool for Efficient and information-Rich Testing
Abstract
Binomial metrics like probability-to-detect or probability-to-hit typically provide operationally meaningful and easy to interpret test outcomes. However, they are informationpoor metrics and extremely expensive to test. The standard power calculations to size a test employ hypothesis tests, which typically result in many tens to hundreds of runs. In addition to being expensive, the test is most likely inadequate for characterizing performance over a variety of conditions due to the inherently large statistical uncertainties associated with binomial metrics. A solution is to convert to a continuous variable, such as miss distance or time-todetect. The common objection to switching to a continuous variable is that the hit/miss or detect/non-detect binomial information is lost, when the fraction of misses/no-detects is often the most important aspect of characterizing system performance. Furthermore, the new continuous metric appears to no longer be connected to the requirements document, which was stated in terms of a probability. These difficulties can be overcome with the use of censored data analysis. This presentation will illustrate the concepts and benefits of this approach, and will illustrate a simple analysis with data, including power calculations to show the cost savings for employing the methodology.
|
Dr. Bram Lillard Research Staff Member IDA |
Tutorial |
| 2016 |
| Power Anyalysis Concepts | Dr. Jim Simpson JK Analytics |
Tutorial |
| 2016 |
| Introduction to Survey Design
Abstract
Surveys are a common tool for assessing user experiences with systems in various stages of development. This mini-tutorial introduces the social and cognitive processes involved in survey measurement and addresses best practices in survey design. Clarity of question wording, appropriate scale use, and methods for reducing survey-fatigue are emphasized. Attendees will learn practical tips to maximize the information gained from user surveys and should bring paper and pencils to practice writing and evaluating questions.
|
Dr. Heather Wojton Research Staff Member IDA |
Tutorial |
| 2016 |
| Introduction to Survey Design
Abstract
Surveys are a common tool for assessing user experiences with systems in various stages of development. This mini-tutorial introduces the social and cognitive processes involved in survey measurement and addresses best practices in survey design. Clarity of question wording, appropriate scale use, and methods for reducing survey-fatigue are emphasized. Attendees will learn practical tips to maximize the information gained from user surveys and should bring paper and pencils to practice writing and evaluating questions.
|
Dr. Justin Mary Research Staff Member IDA |
Tutorial |
| 2016 |
| Introduction to Survey Design
Abstract
Surveys are a common tool for assessing user experiences with systems in various stages of development. This mini-tutorial introduces the social and cognitive processes involved in survey measurement and addresses best practices in survey design. Clarity of question wording, appropriate scale use, and methods for reducing survey-fatigue are emphasized. Attendees will learn practical tips to maximize the information gained from user surveys and should bring paper and pencils to practice writing and evaluating questions.
|
Mr. Jonathan Snavely IDA |
Tutorial |
| 2016 |
| Bayesian Data Analysis in R/STAN
Abstract
In an era of reduced budgets and limited testing, verifying that requirements have been met in a single test period can be challenging, particularly using traditional analysis methods that ignore all available information. The Bayesian paradigm is tailor made for these situations, allowing for the combination of multiple sources of data and resulting in more robust inference and uncertainty quantification. Consequently, Bayesian analyses are becoming increasingly popular in T&E. This tutorial briefly introduces the basic concepts of Bayesian Statistics, with implementation details illustrated in R through two case studies: reliability for the Core Mission functional area of the Littoral Combat Ship (LCS) and performance curves for a chemical detector in the Common Analytical Laboratory System (CALS) with different agents and matrices. Examples are also presented using RStan, a high-performance open-source software for Bayesian inference on multi-level models.
|
Dr. Kassandra Fronczyk IDA |
Tutorial |
| 2016 |
| Bayesian Data Analysis in R/STAN
Abstract
In an era of reduced budgets and limited testing, verifying that requirements have been met in a single test period can be challenging, particularly using traditional analysis methods that ignore all available information. The Bayesian paradigm is tailor made for these situations, allowing for the combination of multiple sources of data and resulting in more robust inference and uncertainty quantification. Consequently, Bayesian analyses are becoming increasingly popular in T&E. This tutorial briefly introduces the basic concepts of Bayesian Statistics, with implementation details illustrated in R through two case studies: reliability for the Core Mission functional area of the Littoral Combat Ship (LCS) and performance curves for a chemical detector in the Common Analytical Laboratory System (CALS) with different agents and matrices. Examples are also presented using RStan, a high-performance open-source software for Bayesian inference on multi-level models.
|
Dr. James Brownlow U.S. Air Force 812TSS/ENT |
Tutorial |
| 2016 |
| Presenting Complex Statistical Methodologies to Military Leadership
Abstract
More often than not, the data we analyze for the military is plagued with statistical issues. Multicollinearity, small sample sizes, quasi-experimental designs, and convenience samples are some examples of what we commonly see in military data. Many of these complications can be resolved either in the design or analysis stage with appropriate statistical procedures. But, to keep our work useful, usable, and transparent to the military leadership who sponsors it, we must strike the elusive balance between explaining and justifying our design and analysis techniques and not inundating our audience with unnecessary details. It can be even more difficult to get military leadership to understand the statistical problems and solutions so well that they are enthused and supportive of our approaches. Using literature written on the subject as well as a variety of experiences, we will showcase several examples, as well as present ideas for keeping our clients actively engaged in statistical methodology discussions.
|
Dr. Jane Pinelis John Hopkins University, Applied Physics Lab |
Tutorial |
| 2016 |
| Sensitivity Experiments
Abstract
A sensitivity experiment is a special type of experimental design that is used when the response variable is binary and the covariate is continuous. Armor protection and projectile lethality tests often use sensitivity experiments to characterize a projectile’s probability of penetrating the armor. In this minitutorial we illustrate the challenge of modeling a binary response with a limited sample size, and show how sensitivity experiments can mitigate this problem. We review eight different single covariate sensitivity experiments and present a comparison of these designs using simulation. Additionally, we cover sensitivity experiments for cases that include more than one covariate, and highlight recent research in this area. The mini-tutorial concludes with a case study by Greg Hutto on Army grenade fuze testing, titled “Preventing Premature ZAP: EMPATHY Capacitive Design With 3 Phase Optimal Design (3pod).”
|
Dr. Thomas Johnson Research Staff Member IDA |
Tutorial |
| 2016 |
| Sensitivity Experiments
Abstract
A sensitivity experiment is a special type of experimental design that is used when the response variable is binary and the covariate is continuous. Armor protection and projectile lethality tests often use sensitivity experiments to characterize a projectile’s probability of penetrating the armor. In this minitutorial we illustrate the challenge of modeling a binary response with a limited sample size, and show how sensitivity experiments can mitigate this problem. We review eight different single covariate sensitivity experiments and present a comparison of these designs using simulation. Additionally, we cover sensitivity experiments for cases that include more than one covariate, and highlight recent research in this area. The mini-tutorial concludes with a case study by Greg Hutto on Army grenade fuze testing, titled “Preventing Premature ZAP: EMPATHY Capacitive Design With 3 Phase Optimal Design (3pod).”
|
Mr. Greg Hutton U.S. Air Force , 96 Test Wing |
Tutorial |
| 2016 |
| Creating Shiny Apps in R for Sharing Automated Statistical Products
Abstract
Interactive web apps can be built straight from R with the R package, Shiny. hiny apps are becoming more prevalent as a way to automate statistical products and share them with others who do not know R. This tutorial will cover Shiny app syntax and how to create basic Shiny apps. Participants will create basic apps by working through several examples and explore how to change and improve these apps. Participants will leave the session with the tools to create their own complicated applications. Participants will need a computer with R, R Studio, and the shiny R package installed.
|
Mr. Randy Griffiths U.S. Army Evaluation Center |
Tutorial |
| 2018 |
| Determination of Power for Complex Experimental Designs
Abstract
Power tells us the probability of rejecting the null hypothesis for an effect of a given size, and helps us select an appropriate design prior to running the experiment. The key to computing power for an effect is determining the size of the effect. We describe a general approach for sizing effects that covers a wide variety of designs including two-level factorials, multilevel factorials with categorical levels, split-plot and response surface designs. The application of power calculations to DoE is illustrated by way of several case studies. These case studies include both continuous and binomial responses. In the case of response surface designs, the fitted model is usually used for drawing contour maps, 3D surfaces, making predictions, or performing optimization. For these purposes, it is important that the model adequately represent the response behavior over the region of interest. Therefore, power to detect individual model parameters is not a good measure of what we are designing for. A discussion and pertinent examples will show attendees how the precision of the fitted surface (i.e. the precision of the predicted response) relative to the noise is a critical criterion in design selection. In this presentation, we introduce a process to determine if the design has adequate precision for DoE needs.
|
Mr. Pat Whitcomb Stat-Ease, Inc |
Tutorial |
| 2018 |
| Operational Testing of Cyber Systems
Abstract
Previous operational tests that included cybersecurity focused in on vulnerabilities discovered at the component level and ad hoc system level exploitation attacks during adversarial assessments. The subsequent evaluation of vulnerabilities and attacks as it relates to the overall resilience of the system were largely qualitative in nature and chalk full of human centered biases making them unreliable estimators of system resilience in a cyber contested environment. To mitigate these shortcomings this tutorial will present an approach for more structured operational tests based on common search algorithms; and, more rigorous quantitative measurements and analysis based on actuarial methods for estimating resilience.
|
Mr. Paul Johnson MCOTEA |
Tutorial |
| 2018 |
| Demystifying Data Science
Abstract
Data science is the new buzz word – it is being touted as the solution for everything from curing cancer to self-driving cars. How is data science related to traditional statistics methods? Is data science just another name for “big data”? In this mini-tutorial, we will begin by discussing what data science is (and is not). We will then discuss some of the key principles of data science practice and conclude by examining the classes of problems and methods that are included in data science.
|
Dr. Alyson Wilson Laboratory for Analytic Sciences North Carolina State Univeristy |
Tutorial |
| 2018 |
| Statistics Boot Camp
Abstract
In the test community, we frequently use statistics to extract meaning from data. These inferences may be drawn with respect to topics ranging from system performance to human factors. In this mini-tutorial, we will begin by discussing the use of descriptive and inferential statistics. We will continue by discussing commonly used parametric and nonparametric statistics within the defense community, ranging from comparisons of distributions to comparisons of means. We will conclude with a brief discussion of how to present your statistical findings graphically for maximum impact.
|
Dr. Stephanie Lane Research Staff Member IDA |
Tutorial |
| 2018 |
| Robust Parameter Design
Abstract
The Japanese industrial engineer, Taguchi, introduced the concept of robust parameter design in the 1950s. Since then, it has seen widespread, successful application in automotive and aerospace applications. Engineers have applied this methodology both to physical and computer experimentation. This tutorial provides a basic introduction to these concepts, with an emphasis on how robust parameter design provides a proper basis for the evaluation and confirmation of system performance. The goal is to show how to modify basic robust parameter designs to meet the specific needs of the weapons testing community.This tutorial targets systems engineers, analysts, and program managers who must evaluate and confirm complex system performance. The tutorial illustrates new ideas that are useful for the evaluation and the confirmation of the performance for such systems.What students will learn: • The basic concepts underlying robust parameter design
• The importance of the statistical concept of interaction to robust parameter design
• How statistical interaction is the key concept underlying much of the evaluation and confirmation of system performance, particularly of weapon systems
|
Dr. Geoff Vining |
Tutorial |
| 2018 |
| Exploratory Data Analysis
Abstract
After decades of seminal methodological research on the subject—accompanied by a myriad of applications—John Tukey formally created the statistical discipline known as EDA with the publication of his book “Exploratory Data Analysis” in 1977. The breadth and depth of this book was staggering, and its impact pervasive, running the gamut from today’s routine teaching of box plots in elementary schools, to the existent core philosophy of data exploration “in-and-for-itself” embedded in modern day statistics and AI/ML. As important as EDA was at its inception, it is even more essential now, with data sets increasing in both complexity and size. Given a science & engineering problem/question, and given an existing data set, we argue that the most important deliverable in the problem-solving process is data-driven insight; EDA visualization techniques lie at the core of extracting that insight. This talk has 3 parts: 1. Data Diamond: In light of the focus of DATAWorks to share essential methodologies for operational testing/evaluation, we first present a problem-solving framework (simple in form but rich in content) constructed and fine-tuned over 4 decades of scientific/engineering problem-solving: the data diamond. This data-centric structure has proved essential for systematically approaching a variety of research and operational problems, for determining if the data on hand has the capacity to answer the question at hand, and for identifying weaknesses in the total experimental effort that might compromise the rigor/correctness of derived solutions. 2. EDA Methods & Block Plot: We discuss those EDA graphical tools that have proved most important/insightful (for the presenter) in attacking the wide variety of physical/chemical/ biological/engineering/infotech problems existent in the NIST environment. Aside from some more commonly-known EDA tools in use, we discuss the virtues/applications of the block plot, which is a tool specifically designed for the “comparative” problem type–ascertaining as to whether the (yes/no) conclusion about the statistical significance of a single factor under study, is in fact robustly true over the variety of other factors (material/machine/method/operator/ environment, etc.) that co-exist in most systems. The testing of army bullet-proof vests is used as an example. 3. 10-Step DEX Sensitivity Analysis: Since the rigor/robustness of testing & evaluation conclusions are dictated not only by the choice of (post-data) analysis methodologies, but more importantly by the choice of (pre-data) experiment design methodologies, we demonstrate a recommended procedure for the important “sensitivity analysis” problem–determining what factors most affect the output of a multi-factor system. The deliverable is a ranked list (ordered by magnitude) of main effects (and interactions). Design-wise, we demonstrate the power and efficiency of orthogonal fractionated 2-level designs for this problem; analysis-wise, we present a structured 10-step graphical analysis which provides detailed data-driven insight into what “drives” the system, what optimal settings exist for the system, what prediction model exists for the system, and what direction future experiments should be to further optimize the system. The World Trade Center collapse analysis is used as an example.
|
Dr. Jim Filliben |
Tutorial | 2018 | |
| Quality Control and Statistical Process Control
Abstract
formance. On the other hand, the need to draw causal inference about factors not under the researchers’ control, calls for a specialized set of techniques developed for observational studies. The persuasiveness and adequacy of such an analysis depends in part on the ability to recover metrics from the data that would approximate those of an experiment. This tutorial will provide a brief overview of the common problems encountered with lack of randomization, as well as suggested approaches for rigorous analysis of observational studies.
|
Dr. Jane Pinelis Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools
Abstract
In this session, researchers from the Institute for Defense Analyses (IDA) present a collection of statistical tools designed to meet ongoing and emerging needs for planning, designing, and evaluating operational tests. We first present a suite of interactive applications hosted on testscience.org that are designed to address common analytic needs in the operational test community. These freely available resources include tools for constructing confidence intervals, computing statistical power, comparing distributions, and computing Bayesian reliability. Next, we discuss four dedicated software tools: JEDIS – a JMP Add-In for automating power calculations for designed experiments
skpr – an R package for generating optimal experimental designs and easily evaluating power for normal and non-normal response variables
ciTools – an R package for quickly and simply generating confidence intervals and quantifying uncertainty for simple and complex linear models
nautilus – an R package for visualizing and analyzing aspects of sensor performance, such as detection range and track completeness
|
Dr. Matthew Avery Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools
Abstract
In this session, researchers from the Institute for Defense Analyses (IDA) present a collection of statistical tools designed to meet ongoing and emerging needs for planning, designing, and evaluating operational tests. We first present a suite of interactive applications hosted on testscience.org that are designed to address common analytic needs in the operational test community. These freely available resources include tools for constructing confidence intervals, computing statistical power, comparing distributions, and computing Bayesian reliability. Next, we discuss four dedicated software tools: JEDIS – a JMP Add-In for automating power calculations for designed experiments
skpr – an R package for generating optimal experimental designs and easily evaluating power for normal and non-normal response variables
ciTools – an R package for quickly and simply generating confidence intervals and quantifying uncertainty for simple and complex linear models
nautilus – an R package for visualizing and analyzing aspects of sensor performance, such as detection range and track completeness
|
Dr. Tyler Morgan-Wall Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools
Abstract
In this session, researchers from the Institute for Defense Analyses (IDA) present a collection of statistical tools designed to meet ongoing and emerging needs for planning, designing, and evaluating operational tests. We first present a suite of interactive applications hosted on testscience.org that are designed to address common analytic needs in the operational test community. These freely available resources include tools for constructing confidence intervals, computing statistical power, comparing distributions, and computing Bayesian reliability. Next, we discuss four dedicated software tools: JEDIS – a JMP Add-In for automating power calculations for designed experiments
skpr – an R package for generating optimal experimental designs and easily evaluating power for normal and non-normal response variables
ciTools – an R package for quickly and simply generating confidence intervals and quantifying uncertainty for simple and complex linear models
nautilus – an R package for visualizing and analyzing aspects of sensor performance, such as detection range and track completeness
|
Dr. Benjamin Ashwell Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools
Abstract
In this session, researchers from the Institute for Defense Analyses (IDA) present a collection of statistical tools designed to meet ongoing and emerging needs for planning, designing, and evaluating operational tests. We first present a suite of interactive applications hosted on testscience.org that are designed to address common analytic needs in the operational test community. These freely available resources include tools for constructing confidence intervals, computing statistical power, comparing distributions, and computing Bayesian reliability. Next, we discuss four dedicated software tools: JEDIS – a JMP Add-In for automating power calculations for designed experiments
skpr – an R package for generating optimal experimental designs and easily evaluating power for normal and non-normal response variables
ciTools – an R package for quickly and simply generating confidence intervals and quantifying uncertainty for simple and complex linear models
nautilus – an R package for visualizing and analyzing aspects of sensor performance, such as detection range and track completeness
|
Dr. Kevin Kirshenbaum Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools
Abstract
In this session, researchers from the Institute for Defense Analyses (IDA) present a collection of statistical tools designed to meet ongoing and emerging needs for planning, designing, and evaluating operational tests. We first present a suite of interactive applications hosted on testscience.org that are designed to address common analytic needs in the operational test community. These freely available resources include tools for constructing confidence intervals, computing statistical power, comparing distributions, and computing Bayesian reliability. Next, we discuss four dedicated software tools: JEDIS – a JMP Add-In for automating power calculations for designed experiments
skpr – an R package for generating optimal experimental designs and easily evaluating power for normal and non-normal response variables
ciTools – an R package for quickly and simply generating confidence intervals and quantifying uncertainty for simple and complex linear models
nautilus – an R package for visualizing and analyzing aspects of sensor performance, such as detection range and track completeness
|
Dr. Stephanie Lane Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools
Abstract
In this session, researchers from the Institute for Defense Analyses (IDA) present a collection of statistical tools designed to meet ongoing and emerging needs for planning, designing, and evaluating operational tests. We first present a suite of interactive applications hosted on testscience.org that are designed to address common analytic needs in the operational test community. These freely available resources include tools for constructing confidence intervals, computing statistical power, comparing distributions, and computing Bayesian reliability. Next, we discuss four dedicated software tools: JEDIS – a JMP Add-In for automating power calculations for designed experiments
skpr – an R package for generating optimal experimental designs and easily evaluating power for normal and non-normal response variables
ciTools – an R package for quickly and simply generating confidence intervals and quantifying uncertainty for simple and complex linear models
nautilus – an R package for visualizing and analyzing aspects of sensor performance, such as detection range and track completeness
|
Dr. Jason Sheldon Research Staff Member IDA |
Tutorial |
| 2018 |
DATAWorks Sessions Archive
| Session Title | Speaker | Type | Materials | Yr |
|---|---|---|---|---|
| Open Architecture Tradeoffs (OAT): A simple, computational game engine for rapidly exploring hypotheses in Battle Management Command and Control (BMC2) | Shelley Cazares |
Breakout |
| 2019 |
| Comparison of Methods for Testing Uniformity to Support the Validation of Simulation Models used for Live-Fire Testing | Shannon Shelburne |
Breakout |
| 2019 |
| Valuing Human Systems Integration: A Test and Data Perspective | Jeffrey Thomas |
Breakout |
| 2019 |
| Air Force Human Systems Integration Program | Anthony Thomas |
Breakout |
| 2019 |
| Toward Real-Time Decision Making in Experimental Settings | Devin Francom |
Breakout | 2019 | |
| Area Validation for Applications with Mixed Uncertainty | Laura White |
Breakout |
| 2019 |
| A 2nd-Order Uncertainty Quantification Framework Applied to a Turbulence Model Validation Effort | Robert Baurle |
Breakout |
| 2019 |
| Sources of Error and Bias in Experiments with Human Subjects | Poornima Madhavan |
Breakout | 2019 | |
| Deep Reinforcement Learning | Benjamin Bell |
Breakout |
| 2019 |
| Decentralized Signal Processing and Distributed Control for Collaborative Autonomous Sensor Networks | Ryan Goldhahn |
Breakout | 2019 | |
| An Overview of Uncertainty-Tolerant Decision Support Modeling for Cybersecurity | Samrat Chatterjee |
Breakout | 2019 | |
| Software Reliability and Security Assessment: Automation and Frameworks | Lance Fiondella |
Breakout |
| 2019 |
| AI & ML in Complex Environment | Tien Pham |
Breakout | 2019 | |
| Waste Not, Want Not: A Methodological Illustration of Quantitative Text Analysis | Laura Castro-Schilo |
Breakout |
| 2019 |
| Machine Learning Prediction With Streamed Sensor Data: Fitting Neural Networks using Functional Principal Components | Chris Gotwalt |
Breakout |
| 2019 |
| Screening Designs for Resource Constrained Deterministic M&S Experiments: A Munitions Case Study | Christopher Drake |
Breakout |
| 2019 |
| The Isle of Misfit Designs: A Guided Tour of Optimal Designs That Break the Mold | Caleb King |
Breakout |
| 2019 |
| Adapting Operational Test to Rapid-Acquisition Programs | Panel Discussion |
Breakout |
| 2019 |
| Sample Size Calculations for Quiet Sonic Boom Community Surveys | Jasme Lee |
Breakout |
| 2019 |
| Improved Surface Gunnery Analysis with Continuous Data | Benjamin Ashwell & V. Bram Lillard |
Breakout |
| 2019 |
| Human in the Loop Experiment Series Evaluating Synthetic Vision Displays for Enhanced Airplane State Awareness | Kathryn Ballard |
Breakout |
| 2019 |
| Statistical Engineering and M&S in the Design and Development of DoD Systems | Doug Ray & Melissa Jablonski |
Breakout | 2019 | |
| Uncertainty Quantification: Combining Large Scale Computational Models with Physical Data for Inference | Dave Higdon |
Breakout |
| 2019 |
| A User-Centered Design Approach to Military Software Development | Pam Savage-Knepshield |
Breakout |
| 2019 |
| Engineering first, Statistics second: Deploying Statistical Test Optimization (STO) for Cyber | Kedar Phadke |
Breakout | 2019 | |
| Validation and Uncertainty Quantification of Complex Models | Kassie Fronczyk |
Breakout |
| 2019 |
| Communicating Statistical Concepts and Results: Lessons Learned from the US Service Academies | Panel Discussion |
Breakout | 2019 | |
| Multivariate Density Estimation and Data-enclosing Sets Using Sliced-Normal Distributions | Luis Crespo |
Breakout |
| 2019 |
| A Statistical Approach for Uncertainty Quantification with Missing Data | Mark Andrews |
Breakout | 2019 | |
| Exploring Problems in Shipboard Air Defense with Modeling | Ralph Donnelly & Benjamin Ashwell |
Breakout |
| 2019 |
| Sequential Testing for Fast Jet Life Support Systems | Darryl Ahner |
Breakout |
| 2019 |
| Bayesian Component Reliability Estimation: F-35 Case Study | V. Bram Lillard & Rebecca Medlin |
Breakout |
| 2019 |
| Applying Functional Data Analysis throughout Aerospace Testing | David Harrison |
Breakout |
| 2019 |
| Adopting Optimized Software Test Design Methods at Scale | Justin Hunter |
Breakout |
| 2019 |
| A Quantitative Assessment of the Science Robustness of the Europa Clipper Mission | Kelli McCoy |
Breakout |
| 2019 |
| Identifying and Contextualizing Maximum Instrument Fault Rates and Minimum Instrument Recovery Times for Europa Clipper Science through Applied Statistics and Strategic Visualizations | Thomas Youmans |
Breakout |
| 2019 |
| Design and Analysis of Experiments for Europa Clipper’s Science Sensitivity Model | Amy Braverman |
Breakout |
| 2019 |
| SLS Structural Dynamics Sensor Optimization Study | Ken Toro & Jon Stallrich |
Breakout | 2019 | |
| The 80/20 rule, can and should we break it using efficient data management tools? | Ghaleb Abdulla |
Breakout |
| 2019 |
| Time Machine Learning: Getting Navy Maintenance Duration Right | Tim Kao |
Breakout |
| 2019 |
| Behavioral Analytics: Paradigms and Performance Tools of Engagement in System Cybersecurity | Robert Gough |
Breakout |
| 2019 |
| 3D Mapping, Plotting, and Printing in R with Rayshader | Tyler Morgan-Wall |
Breakout | 2019 | |
| Functional Data Analysis for Design of Experiments | Tom Donnelly |
Breakout |
| 2019 |
| Test and Evaluation of Emerging Technologies | Dr. Greg Zacharias, Chief Scientist Operational Test and Evaluation |
Breakout |
| 2019 |
| Challenges in Test and Evaluation of AI: DoD’s Project Maven | Jane Pinelis |
Breakout | 2019 | |
| Demystifying the Black Box: A Test Strategy for Autonomy | Dan Porter |
Breakout |
| 2019 |
| Satellite Affordability in LEO (SAL) | Matthew Avery |
Breakout |
| 2019 |
| Statistical Process Control and Capability Study on the Water Content Measurements in NASA Glenn’s Icing Research Tunnel | Emily Timko |
Breakout | 2019 | |
| Reasoning about Uncertainty with the Stan Modeling Language | John Haman |
Breakout |
| 2019 |
| Target Location Error Estimation Using Parametric Models | James Brownlow |
Breakout |
| 2019 |
| Anatomy of a Cyberattack: Standardizing Data Collection for Adversarial and Defensive Analyses | Jason Schlup |
Breakout |
| 2019 |
| A Survey of Statistical Methods in Aeronautical Ground Testing | Drew Landman |
Breakout |
| 2019 |
| Your Mean May Not Mean What You Mean It to Mean | Ken Johnson |
Breakout |
| 2019 |
| A Causal Perspective on Reliability Assessment | Lauren Hund |
Breakout |
| 2019 |
| Hypergames for Control System Security | Arnab Bhattacharya |
Breakout |
| 2019 |
| Probabilistic Data Synthesis to Provide a Defensible Risk Assessment for Army Munition | Kevin Singer |
Breakout |
| 2019 |
| Using Bayesian Neural Networks for Uncertainty Quantification of Hyperspectral Image Target Detection | Daniel Ries |
Breakout |
| 2019 |
| Constructing Designs for Fault Location | Erin Lanus |
Breakout |
| 2019 |
| Accelerating Uncertainty Quantification for Complex Computational Models | James Warner |
Breakout |
| 2019 |
| When Validation Fails: Analysis of Data from an Imperfect Test Chamber | Kendal Ferguson |
Breakout |
| 2019 |
| Technical Leadership Panel-Tuesday Afternoon | Dr. Catherine Warner Science Advsior DOT&E |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | Paul Roberts Chief Engineer Engineering and Safety Center |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | Frank Peri Deputy Director Langley Engineering Directorate |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | CAPT Peter Matisoo Technical Director COTF |
Breakout | 2016 | |
| Technical Leadership Panel-Tuesday Afternoon | Jeff Olinger Technical Director AFOTEC |
Breakout | 2016 | |
| Combining information for Realiability Assesment-Tuesday Morning | Dr. Alyson Wilson North Carolina State University |
Breakout |
| 2016 |
| Reliability Growth in T&E – Summary of National Research Council’s Committee on National Statistics Report Finding-Tuesday Morning | Dr. Art Fries Research Staff Member IDA |
Breakout |
| 2016 |
| Design and Analysis of Margin Testing in Support of Product Qualification for High Reliability Systems | Dr. Justin Newcomer Sandia National Lab |
Breakout | 2016 | |
| Three Case Studies Comparing Traditional versus Modern Test Designs | Dr. Ray Hill Air Force instite of Technology |
Breakout |
| 2016 |
| Supersaturated Designs: Construction and Analysis | Dr. David Edwards Virginia Commonwealth University |
Breakout | 2016 | |
| Statistically Defensible Experiment Design for Wind Tunnel Characterization of Subscale Parachutes for Mission to Mars | Dr. Drew Landman Old Dominion Univerity |
Breakout |
| 2016 |
| Managing Uncertainty in the Context of Risk Acceptance Decision Making at NASA: Thinking Beyond the Model | Dr. Homayoon Dezfuli NASA |
Breakout |
| 2016 |
| Acceptability of Radiation Detection Systems | Dr. Dennis Leber NIST |
Breakout |
| 2016 |
| Risk Analysis for Orbital Debris Wire Harness Failure Assessment for the Joint Polar Satellite System | Dr. Joel Williamsen IDA |
Breakout |
| 2016 |
| Application of Statistical Engineering to Mixture Problems With Process Variables | Dr. Roger Hoerl Union College |
Breakout |
| 2016 |
| The Sixth Sense: Clarity through Statistical Engineering | Dr. Jennifer Van-Mullekom DuPont |
Breakout | 2016 | |
| Case Studies for Statistical Engineering Applied to Powered Rotorcraft Wind-Tunnel Tests | Dr. Sean Commo NASA |
Breakout | 2016 | |
| Bayesian Adaptive Design for Conformance Testing with Bernoulli Trials | Dr. Adamn Pintar NIST |
Breakout |
| 2016 |
| Statistical Models for Combining Information: Styrker Reliability Case Study | Dr. Rebecca Dickinson IDA |
Breakout |
| 2016 |
| Bayesian Estimation of Reliability Growth | Dr. Jim Brownlow U.S. Air Force 812TSS/ENT |
Breakout |
| 2016 |
| Recent use of Statistical Methods in NASA Aeroscience Testing Research and Development Activities | Dr. Eric Walker NASA |
Breakout | 2016 | |
| “High Velocity Analytics for NASA JPL Mars Rover Experimental Design” | Dr. Jim Wisnowski Co-founder/Principle Adsurgo |
Breakout |
| 2016 |
| “High Velocity Analytics for NASA JPL Mars Rover Experimental Design” | Dr. Heath Rushing Co-founder/Principle Adsurgo |
Breakout |
| 2016 |
| An Engineer and a Statistician Walk into a Bar: A Statistical Engineering Negotiation | Mr. Ken Johnson NASA |
Breakout |
| 2016 |
| Making Statistically Defensible Testing ‘The way we do things around here’ | Mr. Greg Hutto Air Force 96th Test Wing |
Breakout | 2016 | |
| Using Sequential Testing to Address Complex Full-Scale Live Fire Test and Evaluation | Dr. Darryl Ahner AFIT, STAT COE |
Breakout | 2016 | |
| Aerospace Measurement and Experimental System Development Characterization | Mr. Ray Rhew NASA |
Breakout |
| 2016 |
| Leveraging Design for Variation to Improve both Testing and Design – A Case Study on Probabilistic Design of Bearings | Dr. Jaime O’connell Pratt and Whitney |
Breakout |
| 2016 |
| Importance of Modeling and Simulation in Testing and the Need for Rigorous Validation | Dr. Kelly McGinnity IDA |
Breakout |
| 2016 |
| Operational Cybersecurity Testing | Mr. Alex Hoover DHS |
Breakout |
| 2016 |
| Dr. Tye Botting Research Staff Member IDA |
Breakout | 2016 | ||
| Recent Advances in Measuring Display Clutter | Dr. David Kaber NCSU |
Breakout |
| 2017 |
| Recent Advances in Measuring Display Clutter | Dr. Carl Pankok |
Breakout |
| 2017 |
| Trust in Automation | Dr. Joseph Lyons Technical Advisor Air Force Research Laboratory |
Breakout |
| 2017 |
| Dose-Response Model of Recent Sonic Boom Community Annoyance Data | Dr. Jonathan Rathsam NASA |
Breakout | 2017 | |
| Overview of Statistical Validation Tools | Dr. Kelly McGinnity IDA |
Breakout |
| 2017 |
| Improving the Rigor of Navy M&S VV&A through the application of Design of Experiments Methodologies and related Statistical Techniques | Dr. Stargel Doane COTF |
Breakout |
| 2017 |
| Validation of AIM-9X Modeling and Simulation | Dr. Rebecca Dickinson Research Staff Member IDA |
Breakout |
| 2017 |
| Software Reliability Modeling | Dr. Lance Fiondella UMASS |
Breakout |
| 2017 |
| Reliability Growth Modeling | Dr. Kellly Sullivan University of Arkansas |
Breakout |
| 2017 |
| Updating R and Reliability Training with Bill Meeker | Dr. Jason Freels AFIT |
Breakout |
| 2017 |
| Introduction to Bayesian Statistics | Dr. Alyson Wilson North Carolina State University |
Breakout |
| 2017 |
| How do the Framework and Design of Experiments Fundamentally Help? | Dr. Luis A. Cortes |
Breakout | 2017 | |
| How do the Framework and Design of Experiments Fundamentally Help? | Mr. Mike Sheeha MITRE |
Breakout | 2017 | |
| Allocating Information Gathering Efforts for Selection Decisions | Dr. Dennis Leber NIST |
Breakout |
| 2017 |
| Allocating Information Gathering Efforts for Selection Decisions | Dr. Jeffrey Herrmann University of Maryland |
Breakout |
| 2017 |
| Statistical Methods for Programmatic Assessment and Anomaly Detection | Mr. Douglas Brown BAH |
Breakout |
| 2017 |
| Statistical Methods for Programmatic Assessment and Anomaly Detection | Dr. Ray McCollum BAH |
Breakout |
| 2017 |
| Flight Test and Evaluation of Airborne Spacing Application | Dr. Sara Wilson NASA |
Breakout |
| 2017 |
| Testing and Estimation in Sequential High-Dimension Data | Dr. Eric Chicken Florida State University |
Breakout |
| 2017 |
| Blast Noise Event Classification from a Spectrogram | Dr. Edward Nykaza Army Engineering Research and Development Center, Construction Engineering Research Laboratory |
Breakout |
| 2017 |
| Automated Software Testing Best Practices and Framework: A STAT COE Project | Dr. Jim Simpson JK Analytics |
Breakout |
| 2017 |
| Automated Software Testing Best Practices and Framework: A STAT COE Project | Dr. Jim Wisnowski Adsurgo |
Breakout |
| 2017 |
| Software Test Techniques | Dr. Jose Calderon Galois |
Breakout |
| 2017 |
| Combinational Testing | Dr. Raghu Kacker NIST |
Breakout |
| 2017 |
| Combinational Testing | Dr. Rick Kuhn NIST |
Breakout |
| 2017 |
| Introduction to Human Measurement | Dr. Cynthia Null NASA |
Breakout | 2017 | |
| The (Empirical) Case for Analyzing Likert-Type Data with Parametric Tests | Dr. Heather Wojton Research Staff Member IDA |
Breakout |
| 2017 |
| The System Usability Scale: A measurement Instrument Should Suit the Measurement Needs | Mr. Keith Kidder AFOTEC |
Breakout |
| 2017 |
| Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation | Mr. Scott West Aerospace Corporation |
Breakout |
| 2017 |
| Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation | Mr. Martin Annett Aerospace Corporation |
Breakout |
| 2017 |
| Deterministic System Design of Experiments Based Frangible Joint Design Reliability Estimation | Dr. James Womach Aerospace Corporation |
Breakout |
| 2017 |
| Machine Learning: Overview and Applications to Test | Lt. Takayuki Iguchi AFOTEC |
Breakout |
| 2017 |
| Machine Learning: Overview and Applications to Test | Lt. Megan Lewis AFOTEC |
Breakout |
| 2017 |
| Range Adversarial Planning Tool for Autonomy Test and Evaluation | Dr. Chad Hawthorne JHU/APL |
Breakout | 2017 | |
| Search for Extended Test Design Methods for Complex Systems of Systems | Dr. Alex Alaniz AFOTEC |
Breakout |
| 2017 |
| The Future of Engineering at NASA Langley | Mr. Joe Gasbarre NASA |
Breakout |
| 2017 |
| Combinatorial Testing for Link-16 Developmental Test and Evaluation | Mr. Tim Mclean MCTSSA |
Breakout |
| 2017 |
| Censored Data Analysis for Performance Data | Dr. Bram Lillard IDA |
Breakout |
| 2017 |
| Estimating the Distribution of an Extremum using a Peaks-Over-Threshold Model and Monte Carlo Simulation | Dr. Adam Pintar NIST |
Breakout |
| 2017 |
| Design & Analysis of a Computer Experiment for an Aerospace Conformance Simulation Study | Dr. David Edwards Virginia Commonwealth University |
Breakout |
| 2017 |
| Optimal Multi-Response Designs | Dr. Sarah Burke STAT COE |
Breakout | 2017 | |
| Augmenting Definitive Screening Designs | Ms. Abby Nachtsheim ASU |
Breakout |
| 2017 |
| Integrated Uncertainty Quantification for Risk and Resource Management: Building Confidence in Design | Dr. Eric Walker NASA |
Breakout |
| 2017 |
| Background of NASA’s Juncture Flow Validation Test | Mr. Joseph Morrison NASA |
Breakout | 2017 | |
| A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime | Dr. Eric Walker NASA |
Breakout |
| 2017 |
| A Study to Investigate the Use of CFD as a Surrogate for Wind Tunnel Testing in the High Supersonic Speed Regime | Mr. Joseph Morrison NASA |
Breakout |
| 2017 |
| Communicating Complex Statistical Methodologies to Leadership | Dr. Jane Pinelis Johns Hopkins University Applied Physics Lab or JHU |
Breakout |
| 2017 |
| Communicating Complex Statistical Methodologies to Leadership | Mr. Paul Johnson MCOTEA |
Breakout |
| 2017 |
| DOE Case Studies in Aerospace Research and Development | Dr. Drew Landman Old Dominion University |
Breakout |
| 2017 |
| Sample Size and Considerations for Statistical Power | Mr. Vance Oas AFOTEC |
Breakout |
| 2017 |
| Sample Size and Considerations for Statistical Power | Mr. Nick Garcia AFOTEC |
Breakout |
| 2017 |
| Experimental Design for Composite Pressure Vessel Life Prediction | Dr. Anne Drsicoll Virginia Tech |
Breakout |
| 2017 |
| Improving Sensitivity Experiments | Mr. Douglas Ray US Army RDECOM ARDEC |
Breakout |
| 2017 |
| Improving Sensitivity Experiments | Mr. Kevin Singer US Army |
Breakout |
| 2017 |
| Sequential Experimentation for a Binary Response – The Break Separation Method | Mr. Darsh Thakkar RIT-S |
Breakout |
| 2017 |
| Sequential Experimentation for a Binary Response – The Break Separation Method | Dr. Rachel Silvestrini RIT-S |
Breakout |
| 2017 |
| Carrier Reliability Model Validation | Dr. Dean Thomas IDA |
Breakout | 2017 | |
| Model Based Systems Engineering Panel Discussion | Mr. John Holladay NASA |
Breakout |
| 2017 |
| Resampling Methods | Dr. David Ruth United States Naval Academy |
Breakout |
| 2017 |
| Structured Decision Making | Dr. Christine Anderson Cook LANL |
Breakout |
| 2017 |
| Do Asymmetries in Nuclear Arsenals Matter? | Dr. Jane Pinelis Johns Hopkins University Applied Physics Lab or JHU |
Breakout |
| 2017 |
| Communication in Statistics & the Five Hardest Concepts | Dr. Jennifer Van-Mellekom Virginia Tech |
Breakout | 2017 | |
| Uncertainty Quantification: What is it and Why it is Important to Test, Evaluation, and Modeling and Simulation in Defense and Aerospace | Dr. Peter Qian University of Wisconsin and SmartUQ |
Breakout |
| 2017 |
| Model Uncertainty and its Inclusion in Testing Results | Dr. Stever Lund NIST |
Breakout |
| 2017 |
| VV&UQ – Uncertainty Quantification for Model-Based Engineering of DoD Systems | Mr. Douglas Ray US Army RDECOM ARDEC |
Breakout |
| 2017 |
| VV&UQ – Uncertainty Quantification for Model-Based Engineering of DoD Systems | Ms. Melissa Jablonski US Army |
Breakout |
| 2017 |
| Data Visualization | Ms. Lori Perklins NASA |
Breakout | 2017 | |
| Big Data, Big Think | Mr. Robert Beil NASA |
Breakout |
| 2017 |
| Project Data Flow Is an Engineered System | Mr. Ken Johnson NASA |
Breakout |
| 2017 |
| Comparing Experimental Designs | Dr. Tom Donnelly JMP |
Breakout |
| 2017 |
| Testing Autonomous Systems | Dr. Darryl Ahner Director AFIT |
Breakout |
| 2018 |
| Screening Experiments with Partial Replication | Mr. David Edwards Virginia Commonwealth University |
Breakout |
| 2018 |
| What is Bayesian Experimental Design? | Blaza Toman Statistical Engineering Division, NIST |
Breakout |
| 2018 |
| DOE and Test Automation for System of Systems TE | Jim Simpson JK Analytics |
Breakout |
| 2018 |
| Building A Universal Helicopter Noise Model Using Machine Learning | Eric Greenwood Aeroacoustics Branch |
Breakout |
| 2018 |
| Machine Learning to Assess Piolts’ Cognitive State | Tina Heinich Computer Engineer, OCIO Data Science Team AST, Data Systems |
Breakout |
| 2018 |
| CYBER Penetration Testing and Statistical Analysis in DT&E | Timothy McLean |
Breakout |
| 2018 |
| Method for Evaluating the Quality of Cybersecurity Defenses | Dr, Shawn Whetstone Research Staff Member IDA |
Breakout |
| 2018 |
| Metrics to Characterize Temporal patterns in Lifespans of Artifacts | Soumyo MoitraNA Software Engineering Institute Carnegie Mellon Univeristy |
Breakout |
| 2018 |
| Uncertainty Quantification with Mixed Uncertainty Sources | Tom West |
Breakout | 2018 | |
| System Level Uncertainty Quanification for Low-Boom Supersonic Flight Vehicles | Ben Phillips |
Breakout | 2018 | |
| Uncertainty Quantification and Analysis at The Boeing Company | John Schaefer Sandia National Labortories |
Breakout |
| 2018 |
| Interface design for analysts in a data and analysis-rich environment | Karin Butler Sandia National Labortories |
Breakout |
| 2018 |
| Asparagus is the most articulate vegetable ever | Kerstan Cole |
Breakout | 2018 | |
| Mitigating Pilot Disorientation with Synthetic Vision Displays | Kathryn Ballard NASA |
Breakout |
| 2018 |
| Sage III Seu Statistical Analysis Model | Ray McCollum Booz Allen Hamilton |
Breakout |
| 2018 |
| Initial Validation of the Trust of Automated System Test | Dr. Heather Wojton Research Staff Member IDA |
Breakout |
| 2018 |
| Operational Evaluation of a Flight-deck Software Application | Sara Wilson NASA |
Breakout |
| 2018 |
| Comparing M&S Output to Live Test Data: A Missile System Case Study | Dr. Kelly Avery Reasearch Staff member IDA |
Breakout |
| 2018 |
| NASA’s Human Exploration Research Analog (HERA): An analog mission for isolation, confinement, and remote conditions in space exploration scenarios | Shelley Cazares NASA |
Breakout | 2018 | |
| Reliability Fundamentals and Analysus Lessons Learned | Dan Telford AFOTEC |
Breakout |
| 2018 |
| Design and Analysis of Nonlinear Models for the Mars 2020 Rover | Jim Wisnowski |
Breakout |
| 2018 |
| Cases of Second-Order Split-Plot Designs | Dr. Luis Cortes MITRE |
Breakout |
| 2018 |
| The Development and Execution of Split-Plot Designs In Navy Operational Test and Evaluation: A Practitioner’s Perspective | Stargel Doane |
Breakout | 2018 | |
| B-52 Radar Modernization Test Design Considerations | Stuart Corbett AFOTEC |
Breakout |
| 2018 |
| B-52 Radar Modernization Test Design Considerations | Joseph Maloney AFOTEC |
Breakout |
| 2018 |
| Experimental Design of a Unique Force Measurement System Calibration | Ken Toro |
Breakout | 2018 | |
| Application of Design of Experiments to a Calibration of the National Transonic Facility | Matt Rhode NASA |
Breakout |
| 2018 |
| Application of Design of Experiments to a Calibration of the National Transonic Facility | Matt Bailey Jacobs Technology Inc |
Breakout |
| 2018 |
| The Use of DOE vs OFAT in the Calibration of AEDC Wind Tunnels | Rebecca Rought AEDC/TSTA |
Breakout |
| 2018 |
| Initial Investigation into the Psychoacoustic Properties of Small Unmanned Aerial System Noise | Andrew Chrisrian Structural Acoustics Branch |
Breakout |
| 2018 |
| Combining Human Factors Data and Models of Human Performance | Cynthia Null Technicial Fellow for Human Factors |
Breakout |
| 2018 |
| A Multi-method, Triangulation Approach to Operational Testing | Dr. Daniel Porter Research Staff Member IDA |
Breakout |
| 2018 |
| Illustrating the Importance of Uncertainty Quantification (UQ) in Munitions Modeling | Donald Calucci |
Breakout | 2018 | |
| Space-Filling Designs for Robustness Experiments | Roshan Vengazhiyil |
Breakout | 2018 | |
| Introduction of Uncertainty Quantification and Industry Challenges | Peter Chien |
Breakout | 2018 | |
| The Effect of Extremes in Small Sample Size on Simple Mixed Models: A Comparison of Level-1 and Level-2 Size | Dr. Kristina Carter Research Staff Member IDA |
Breakout |
| 2018 |
| Evaluating Deterministic Models of Time Series by Comparison to Observations | Amy Braverman Jet Propulsion Laboratory, California Institute of Technology |
Breakout |
| 2018 |
| Challenger Challenge: Pass-Fail Thinking Increases Risk Measurably | Ken Johnson Applied Statistician NASA Engineering and Safety Center |
Breakout |
| 2018 |
| Doppler Assisted Sensor Fusion for Tracking and Exploitation | J. Derek Tucker Sandia National Laboratories |
Breakout |
| 2018 |
| XPCA: A Copula-based Generalization of PCA for Ordinal Data | Cliff Anderson-Bergman Sandia National Laboratories |
Breakout |
| 2018 |
| Insights, Predictions, and Actions: Descriptive Definitions of Data Science, Machine Learning, and Artificial Intelligence | Dr. Andrew Flack Research Staff Member IDA |
Breakout |
| 2018 |
| Bayesian Calibration and Uncertainty Analysis: A Case Study Using a 2-D CFD Turbulence Model | Peter Chien |
Contributed | 2018 | |
| Assessing Human Visual Inspection for Acceptance Testing: An Attribute Agreement Analysis Case Study | Christopher Drake Lead Statistician QE&SA Statistical Methods & Analysis Group |
Contributed |
| 2018 |
| Leveraging Anomaly Detection for Aircraft System Health Data Stability Reporting | Kyle Gartrell |
Contributed |
| 2018 |
| Application of Adaptive Sampling to Advance the Metamodeling and Uncertainty Quantification Process | Erik Axdahl Hypersonic Airbreathing Propulsion Branch NASA Langley Research Center |
Contributed |
| 2018 |
| Application of Adaptive Sampling to Advance the Metamodeling and Uncertainty Quantification Process | Robert Baurle Hypersonic Airbreathing Propulsion Branch NASA Langley Research Center |
Contributed |
| 2018 |
| Application of Statistical Methods and Designed Experiments to Development of Technical Requirements | Eli Golden U.S. ARMY ARMAMENT RESEARCH, DEVELOPMENT & ENGINEERING CENTER |
Contributed |
| 2018 |
| Infrastructure Lifetimes | Erika Taketa Lawrence Livermore National Laboratory |
Contributed |
| 2018 |
| Infrastructure Lifetimes | William Romine Lawrence Livermore National Laboratory |
Contributed |
| 2018 |
| Workforce Analytics | William Romine Lawrence Livermore National Laboratory |
Contributed | 2018 | |
| Optimizing for Mission Success in Highly Uncertain Scenarios | Brian Chell |
Contributed | 2018 | |
| Test Planning for Observational Studies using Poisson Process Modeling | Brian Stone AFOTEC |
Contributed |
| 2018 |
| Model credibility in statistical reliability analysis with limited data | Caleb King Sandia National Laboratories |
Contributed |
| 2018 |
| Sound level recommendations for quiet sonic boom dose-response community surveys | Jasme Lee North Carlina State Univeristy |
Contributed |
| 2018 |
| Development of a Locking Setback Mass for Cluster Munition Applications: A UQ Case Study | Melissa Jablonski U.S. ARMY ARMAMENT RESEARCH, DEVELOPMENT & ENGINEERING CENTER |
Contributed |
| 2018 |
| Wednesday Keynote Speaker I | Dr. Peter Parker Team Lead, Advanced Measurement Systems NASA |
Keynote |
| 2019 |
| Tuesday Keynote | Dr. David Chu President Institute for Defense Analyses |
Keynote |
2019 | |
| Thursday Keynote Speaker II | Mr. Michael Little Program Manager, Advanced Information Systems Technology Earth Science Technology Office, NASA Headquarters |
Keynote |
| 2019 |
| Thursday Keynote Speaker I | Dr. Wendy Martinez Director, Mathematical Statistics Research Center, Bureau of Labor Statistics ASA President-Elect (2020) |
Keynote |
| 2019 |
| Wednesday Keynote Speaker II | Dr. Laura Freeman Associate Director, ISL Hume Center for National Security and Technology, Virginia Tech |
Keynote |
| 2019 |
| Thursday Lunchtime Keynote Speaker | Dr. T. Charles Clancy Bradley Professor of Electrical and Computer Engineering Virginia Tech |
Keynote |
| 2019 |
| Wednesday Keynote Speaker III | Mr. Timothy Dare Deputy Director, Developmental Test, Evaluation, and Prototyping SES OUSD(R&E) |
Keynote |
| 2019 |
| Wednesday Lunchtime Keynote Speaker | Dr. Jared Freeman Chief Scientist of Aptima and Chair of the Human Systems Division National Defense Industry Association |
Keynote |
| 2019 |
| Welcoming & Opening Keynote-Tuesday AM | Dr. Mike Gilmore Director DOT&E |
Keynote |
| 2016 |
| Lunch with Keynote Leadership Perspective | Dr. Jon Hollday NASA |
Keynote |
2016 | |
| STAT Engineering Keynote-Wednesday AM | Dr. Christine Anderson Cook Statistics Los Alamos National Lab |
Keynote |
2016 | |
| Lunch With Leadership Perspective-Wednesday PM | Dr. David Brown Deputy Assistant Secretary of Defense, Developmental Test & Evaluation |
Keynote |
2016 | |
| Test and Evaluation Matter for Warfighter and We Need your Help to Make it Better | Dr. Katherine Warner Science Advisor within Secretary of Defense DOT&E |
Keynote |
| 2017 |
| Retooling Design and Development | Mr. Chris Singer NASA Deputy Chief Engineer NASA |
Keynote |
| 2017 |
| Reflections on Statistical Engineering and it’s Application | Dr. Geoff Vining Professor Virginia Tech |
Keynote |
| 2017 |
| Developmental Test and Evaluation | Dr. Brian Hall Principal Deputy Director DoD |
Keynote |
| 2017 |
| Mr. Dave Duma Acting Director, Operationak Test and Evaluation, Office of the Secretary of Defense DOT&E |
Keynote |
2017 | ||
| Opening Keynote | Dr. David Chu President IDA |
Keynote |
| 2018 |
| Closing Remarks | Mr. Robert Behler Director DOT&E |
Keynote |
2018 | |
| Testing and Analytical Challenges on the Path to Hypersonic Flight | Dr. Mark Lewis Director IDA-STPI |
Keynote |
| 2018 |
| Journey to a Data Centric Approcach for National Security | Dr. Marcey Hoover Quality Assurance Director Sandia National Labortories |
Keynote |
| 2018 |
| NASA AERONAUTICS | Mr. Bob Pearce Deputy Associate Administrator Strategy, Aeronautics Research Mission DirectorateNASA |
Keynote |
| 2018 |
| High-Effective Statistical Collaboration. The Art and the Science | Dr. Peter Parker Team Lead Advance Measurement Systems NASA |
Keynote |
| 2018 |
| Consensus Building | Dr. Antonio Possolo NIST Fellow, Chief Statistician National Institute of Standards and Technology. |
Keynote |
| 2018 |
| Opening Keynote | Mr. Dave Duma Assistant Director Operational Test and Evaluation |
Keynote |
| 2018 |
| Bayesian Analysis | Dr. Robert Gramacy Virginia Tech |
Short Course |
| 2019 |
| Categorical Data Analysis | Dr. Christopher Franck Virginia Tech |
Short Course |
| 2019 |
| Multivariate Data Analysis | Dr. Doug Steinley University of Missouri |
Short Course |
| 2019 |
| Statistical Methods for Modeling and Simulation Verification and Validation | Dr. Jim Simpson, Dr. Jim Wisnowski, and Dr. Stargel Doane JK Analytics LLC, Adsurgo LLC, Analytical Arts LLC |
Short Course |
| 2019 |
| Uncertainty Quantification | Dr. Ralph Smith North Carolina State University |
Short Course |
| 2019 |
| Design of Experiments | Dr. Doug Montgomery and Dr. Caleb King Arizona State University, JMP |
Short Course | 2019 | |
| Overview of Design of Experiments | Dr. Doug Montgomery Professor Arizon State University |
Short Course |
| 2016 |
| Experiences in Reliability Analysis | Dr. Bill Meeker Professor Iowa State University |
Short Course |
| 2016 |
| Advanced Regression | Dr. Geoff Vining Professor Virginia Tech University |
Short Course |
| 2016 |
| Introduction to Bayesian | Dr. Robert Gramacy Associate Professor University of Chicago |
Short Course | 2016 | |
| Introduction To Design of Experiments | Dr. Doug Montgomery Professor Arizona State University |
Short Course | 2017 | |
| Introduction To Design of Experiments | Dr. Brad Jones Professor Arizona State University |
Short Course | 2017 | |
| Data Farming | Dr. Susan Sanchez Naval Postgraduate School |
Short Course |
| 2017 |
| Split-Plot and Restricted Randomization Designs | Dr. Jim Simpson JK Analytics |
Short Course |
| 2017 |
| Statistical Models for Reliability Data | Dr. William Meeker Professor Iowa State University |
Short Course |
| 2017 |
| Data Visualization | Dr. Jim Wisnowski Adsurgo |
Short Course |
| 2017 |
| Modern Response Surface Methods & Computer Experiments | Dr. Robert Gramacy Virginia Polytechnic Institute and State University |
Short Course |
| 2018 |
| Overview of Design of Experiments | Dr. Bradley Jones Distinguished Research Fellow JMP Division/SAS |
Short Course |
| 2018 |
| Overview of Design of Experiments | Dr. Doug Motgomery Regents’ Professor of Industrial Engineering and Statistics ASU Foundation Professor of Engineering Arizona State University |
Short Course |
| 2018 |
| Introduction to R | Dr. Justin Post North Carlina State Univeristy |
Short Course |
| 2018 |
| Survey Construction and Analysis | Dr. Wendy Matrinez Bureau of Labor and Statistics |
Short Course |
| 2018 |
| Survey Construction and Analysis | Dr. MoonJung Cho Bureau of Labor and Statistics |
Short Course |
| 2018 |
| Uncertainty Quantification | Dr. Ralph Smith North Carlina State Univeristy |
Short Course |
| 2018 |
| Tutorial: Combinatorial Methods for Testing and Analysis of Critical Software and Security Systems | Rick Kuhn, Dimitris Simos, and Raghu Kacker National Institute of Standards & Technology |
Tutorial |
| 2019 |
| Tutorial: Cyber Attack Resilient Weapon Systems | Barry Horowitz Professor, Systems Engineering University of Virginia |
Tutorial |
| 2019 |
| Tutorial: Learning Python and Julia | Douglas Hodson Associate Professor Air Force Institute of Technology |
Tutorial | 2019 | |
| Tutorial: Statistics Boot Camp | Kelly Avery IDA |
Tutorial |
| 2019 |
| Tutorial: Reproducible Research | Andrew Flack, Kevin Kirshenbaum, and John Haman IDA |
Tutorial |
| 2019 |
| Tutorial: Developing Valid and Reliable Scales | Heather Wojton & Shane Hall IDA / USARMY ATEC |
Tutorial |
| 2019 |
| The Bootstrap World | Dr. Matt Avery Research Staff Member IDA |
Tutorial |
| 2016 |
| Dr. Lance Fiondella Univeristy of Massachusetts, Dartmouth |
Tutorial |
| 2016 | |
| A Statistical Tool for Efficient and information-Rich Testing | Dr. Bram Lillard Research Staff Member IDA |
Tutorial |
| 2016 |
| Power Anyalysis Concepts | Dr. Jim Simpson JK Analytics |
Tutorial |
| 2016 |
| Introduction to Survey Design | Dr. Heather Wojton Research Staff Member IDA |
Tutorial |
| 2016 |
| Introduction to Survey Design | Dr. Justin Mary Research Staff Member IDA |
Tutorial |
| 2016 |
| Introduction to Survey Design | Mr. Jonathan Snavely IDA |
Tutorial |
| 2016 |
| Bayesian Data Analysis in R/STAN | Dr. Kassandra Fronczyk IDA |
Tutorial |
| 2016 |
| Bayesian Data Analysis in R/STAN | Dr. James Brownlow U.S. Air Force 812TSS/ENT |
Tutorial |
| 2016 |
| Presenting Complex Statistical Methodologies to Military Leadership | Dr. Jane Pinelis John Hopkins University, Applied Physics Lab |
Tutorial |
| 2016 |
| Sensitivity Experiments | Dr. Thomas Johnson Research Staff Member IDA |
Tutorial |
| 2016 |
| Sensitivity Experiments | Mr. Greg Hutton U.S. Air Force , 96 Test Wing |
Tutorial |
| 2016 |
| Creating Shiny Apps in R for Sharing Automated Statistical Products | Mr. Randy Griffiths U.S. Army Evaluation Center |
Tutorial |
| 2018 |
| Determination of Power for Complex Experimental Designs | Mr. Pat Whitcomb Stat-Ease, Inc |
Tutorial |
| 2018 |
| Operational Testing of Cyber Systems | Mr. Paul Johnson MCOTEA |
Tutorial |
| 2018 |
| Demystifying Data Science | Dr. Alyson Wilson Laboratory for Analytic Sciences North Carolina State Univeristy |
Tutorial |
| 2018 |
| Statistics Boot Camp | Dr. Stephanie Lane Research Staff Member IDA |
Tutorial |
| 2018 |
| Robust Parameter Design | Dr. Geoff Vining |
Tutorial |
| 2018 |
| Exploratory Data Analysis | Dr. Jim Filliben |
Tutorial | 2018 | |
| Quality Control and Statistical Process Control | Dr. Jane Pinelis Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools | Dr. Matthew Avery Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools | Dr. Tyler Morgan-Wall Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools | Dr. Benjamin Ashwell Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools | Dr. Kevin Kirshenbaum Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools | Dr. Stephanie Lane Research Staff Member IDA |
Tutorial |
| 2018 |
| Evolving Statistical Tools | Dr. Jason Sheldon Research Staff Member IDA |
Tutorial |
| 2018 |