DATAWorks Sessions Archive

Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.

Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.

This tutorial is designed for newcomers to simulation-based experiments. Data farming is the process of using computational experiments to “grow” data, which can then be analyzed using statistical and visualization techniques to obtain insight into complex systems. The focus of the tutorial will be on gaining practical experience with setting up and running simulation experiments, leveraging recent advances in large-scale simulation experimentation pioneered by the Simulation Experiments & Efficient Designs (SEED) Center for Data Farming at the Naval Postgraduate School (http://harvest.nps.edu). Participants will be introduced to fundamental concepts, and jointly explore simulation models in an interactive setting. Demonstrations and written materials will supplement guided, hands-on activities through the setup, design, data collection, and analysis phases of an experiment-driven simulation study.

Have you ever built what you considered to be the ideal designed experiment, then passed it along to be run and learn later that your recommended run order was ignored? Or perhaps you were part of a test execution team and learned too late that one or more of your experimental factors are difficult or time-consuming to change. We all recognize that the best possible guard against lurking background noise is complete randomization, but often we find that a randomized run order is extremely impractical or even infeasible. Split-plot design and analysis methods have been around for over 80 years, but only in the last several years have the methods fully matured and been made available in commercial software. This class will introduce you to the world of practical split-plot design and analysis methods. We’ll provide you the skills to effectively build designs appropriate to your specific needs and demonstrate proper analysis techniques using general linear models, available in the statistical software. Topics include split-plots for 2-level and mixed-level factor sets, for first and second order models, as well as split-split-plot designs.

Engineers in manufacturing industries require data-driven reliability information for making business, product-design, and engineering decisions. The owners and operators of fleets of systems also need reliability information to make good decisions. The course will focus on concepts, examples, models, data analysis, and interpretation of reliability data analyses. Examples and exercises will include product field (maintenance or warranty) data and accelerated life test data. After completing this course, participants will be able to recognize and properly deal with different kinds of reliability data and properly interpret important reliability metrics. Topics will include the use probability plots to identify appropriate distributional models (e.g. Weibull and lognormal distributions), estimating important quantities like distribution quantiles and failure probabilities, the analysis of data with multiple failure modes, and the analysis of recurrence data from a fleet of systems or a reliability growth program.

Data visualization allows us to quickly explore and discover relationships graphically and interactively. We will provide the foundations for creating better graphical information to accelerate the insight discovery process and enhance the understandability of reported results. First principles and the “human as part of the system” aspects of information visualization from multiple leading sources such as Harvard Business Review, Edward Tufte, and Stephen Few will be explored using representative example data sets. We will discuss best practices for graphical excellence to most effectively, clearly, and efficiently communicate your story. We will explore visualizations applicable across the conference themes (computational modeling, DOE, statistical engineering, modeling & simulation, and reliability) for univariate, multivariate, time-dependent, and geographical data.

This course details statistical techniques at the interface between mathematical modeling via computer simulation, computer model meta-modeling (i.e., emulation/surrogate modeling), calibration of computer models to data from field experiments, and model-based sequential design and optimization under uncertainty (a.k.a. Bayesian Optimization). The treatment will include some of the historical methodology in the literature, and canonical examples, but will primarily concentrate on modern statistical methods, computation and implementation, as well as modern application/data type and size. The course will return at several junctures to real-word experiments coming from the physical and engineering sciences, such as studying the aeronautical dynamics of a rocket booster re-entering the atmosphere; modeling the drag on satellites in orbit; designing a hydrological remediation scheme for water sources threatened by underground contaminants; studying the formation of super-nova via radiative shock hydrodynamics. The course material will emphasize deriving and implementing methods over proving theoretical properties.

Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.The course outcomes are: • Ability to plan and execute experiments. • Ability to collect data and analyze and interpret these data to provide the knowledge required for business success. • Knowledge of a wide range of modern experimental tools that enable practitioners to customize their experiment to meet practical resource constraintsThe topics covered during the course are: • Fundamentals of DOX – randomization, replication, and blocking. • Planning for a designed experiment – type and size of design, factor selection, levels and ranges, response measurement, sample sizes. • Graphical and statistical approaches to DOX analysis. • Blocking to eliminate the impact of nuisance factors on experimental results. • Factorial experiments and interactions. • Fractional factorials – efficient and effective use of experimental resources. • Optimal designs. • Response surface methods. • A demonstration illustrating and comparing the effectiveness of different experimental design strategies. This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course.

Well-designed experiments are a powerful tool for developing and validating cause and effect relationships when evaluating and improving product and process performance and for operational testing of complex systems. Designed experiments are the only efficient way to verify the impact of changes in product or process factors on actual performance.The course outcomes are: • Ability to plan and execute experiments. • Ability to collect data and analyze and interpret these data to provide the knowledge required for business success. • Knowledge of a wide range of modern experimental tools that enable practitioners to customize their experiment to meet practical resource constraintsThe topics covered during the course are: • Fundamentals of DOX – randomization, replication, and blocking. • Planning for a designed experiment – type and size of design, factor selection, levels and ranges, response measurement, sample sizes. • Graphical and statistical approaches to DOX analysis. • Blocking to eliminate the impact of nuisance factors on experimental results. • Factorial experiments and interactions. • Fractional factorials – efficient and effective use of experimental resources. • Optimal designs. • Response surface methods. • A demonstration illustrating and comparing the effectiveness of different experimental design strategies. This course is focused on helping you and your organization make the most effective utilization of DOX. Software usage is fully integrated into the course.

This course is designed to introduce participants to the R programming language and the R studio editor. R is a free and open-source software for summarizing data, creating visuals of data, and conducting statistical analyses. R can offer many advantages over programs such as Excel including faster computation, customized analyses, access to the latest statistical techniques, automation of tasks, and the ability to easily reproduce research. After completing this course, a new user should be able to: • Import/export data from/to external files. • Create and manipulate new variables. • Conduct basic statistical analyses (such as t-tests and linear regression). • Create basic graphs. • Install and use R packages Participants should bring a laptop for the interactive components of the course.

In this course, we introduce the main concepts of the survey methodology process – from survey sampling design to analyzing the data obtained from complex survey designs. The course topics include: 1. Introduction to the Survey Process 2. R Tools 3. Sampling Designs – Simple Random Sampling, Cluster Sampling, Stratified Sampling, and more 4. Weighting and Variance Estimation 5. Exploratory Data Analysis 6. Complex Survey Analysis We use a combination of lectures and hands-on exercises using R. Students are expected to have R and associated packages installed on their computers. We will send a list of required packages before the course. We also use data from Department of Defense surveys, where appropriate.

In this course, we introduce the main concepts of the survey methodology process – from survey sampling design to analyzing the data obtained from complex survey designs. The course topics include: 1. Introduction to the Survey Process 2. R Tools 3. Sampling Designs – Simple Random Sampling, Cluster Sampling, Stratified Sampling, and more 4. Weighting and Variance Estimation 5. Exploratory Data Analysis 6. Complex Survey Analysis We use a combination of lectures and hands-on exercises using R. Students are expected to have R and associated packages installed on their computers. We will send a list of required packages before the course. We also use data from Department of Defense surveys, where appropriate.

We increasingly rely on mathematical and statistical models to predict phenomena ranging from nuclear power plant design to profits made in financial markets. When assessing the feasibility of these predictions, it is critical to quantify uncertainties associated with the models, inputs to the models, and data used to calibrate the models. The synthesis of statistical and mathematical techniques, which can be used to quantify input and response uncertainties for simulation codes that can take hours to days to run, comprises the evolving field of uncertainty quantification. The use of data, to improve the predictive accuracy of models, is central to uncertainty quantification so we will begin by providing an overview of how Bayesian techniques can be used to construct distributions for model inputs. We will subsequently describe the computational issues associated with propagating these distributions through complex models to construct prediction intervals for statistical quantities of interest such as expected profits or maximal reactor temperatures. Finally, we will describe the use of sensitivity analysis to isolate critical model inputs and surrogate model construction for simulation codes that are too complex for direct statistical analysis. All topics will be motivated by examples arising in engineering, biology, and economics.

This mini-tutorial provides an introduction to combinatorial interaction testing (CIT). The main idea behind CIT is to pseudo-exhaustively test software and hardware systems by covering combinations of components in order to detect faults. In 90 minutes, we provide an overview of this domain that includes the following topics: the role of CIT in software and hardware testing, how it complements and differs from design of experiments, considerations such as variable strength and constraints, the typical combinatorial arrays used for constructing test suites, and existing tools for test suite construction. Last, defense systems are increasingly relying on software with embedded machine learning (ML), yet ML poses unique challenges to applying conventional software testing due to characteristics such as the large input space, effort required for white box testing, and emergent behaviors apparent only at integration or system levels. As a well-studied black box approach to testing integrated systems with a pseudo-exhaustive strategy for handling large input spaces, CIT provides a good foundation for testing ML. In closing, we present recent research adapting concepts of combinatorial coverage to test design for ML.

This mini-tutorial session discusses the practical application of neural networks from a lay person’s perspective and will walk through a hands-on case study in which we build, train, and analyze a few neural network models using TensorFlow. The course will review the basics of neural networks and touch on more complex neural network architecture variants for deep learning applications. Deep learning techniques are becoming more prevalent throughout the development of autonomous and AI-enabled systems, and this session will provide students with the foundational intuition needed to understand these systems.

Categorical data is abundant in the 21st century, and its analysis is vital to advance research across many domains. Thus, data-analytic techniques that are tailored for categorical data are an essential part of the practitioner’s toolset. The purpose of this short course is to help attendees develop and sharpen their abilities with these tools. Topics covered in this short course will include binary and multi-category logistic regression, ordinal regression, and classification, and methods to assess predictive accuracy of these approaches will be discussed. Data will be analyzed using the R software package, and course content loosely follow Alan Agresti’s excellent textbook “An Introduction to Categorical Data Analysis, Third Edition.”

Having completed a first course in DOE and begun to apply these concepts, engineers and scientists quickly learn that test and evaluation often demands knowledge beyond the use of classical designs.  This one-day short course, taught by an engineer from a practitioner’s perspective, targets this problem.  Three broad areas are covered:

• Optimal designs address questions such as how to accommodate constraints in the design space, specify unique models, and fractionate general factorial designs. Choices for methods revolve around objectives. For example, is the priority to best estimate model parameters or to reduce overall response prediction error?
• Split plot designs allow restrictions to randomization for hard-to-change factors. These designs, including optimal adaptations, are now supported by most commercial experimental design software and within the reach of the practitioner.
• Sequential design approaches strive to minimize resource requirements at the onset of a test. A phased approach includes a plan to add runs to achieve specific objectives following an initial test analysis.  The approach is especially effective in high-dimensional spaces where the influence of all factors is in question and/or model order is unknown.  One example is the use of screening designs where all factors are included in the first test with the expectation that many will be found insignificant.

The course format is to introduce relevant background material, discuss case studies, and provide software demonstrations.   Case studies and demonstrations are derived from a variety of sources, including aerospace testing and DOD T&E. Learn design approaches, design comparison metrics, best practices, and lessons learned from the instructor’s experience.   A first course in Design of Experiments is a prerequisite.

R is one of the major platforms for doing statistical analysis and research.  This course introduces the powerful and popular R software through the use of the RStudio IDE.  This course covers the use of the tidyverse suite of packages to import raw data (readr), do common data manipulations (dplyr and tidyr), and summarize data numerically (dplyr) and graphically (ggplot2).  In order to promote reproducibility of analyses, we will discuss how to code using R Markdown – a method of R coding that allows one to easily create PDF and HTML documents that interweave narrative, R code, and results. List of packages to install: tidyverse, GGally, Lahman, tinytex

Cyber resilience is the ability to anticipate, withstand, recover from, and adapt to adverse conditions, stresses, attacks, or compromises on systems that use or are enabled by cyber resources. As a property defined in terms of system behavior, cyber resilience presents special challenges from a test and evaluation perspective. Typically, system requirements are specified in terms of technology function and can be tested through manipulation of the systems operational environment, controls, or inputs. Resilience, however, is a high-level property relating to the capacity of the system to recover from unwanted loss of function. There are no commonly accepted definitions of how to measure this system property. Moreover, by design, resilience behaviors are exhibited only when the system has lost critical functions. The implication is that the test and evaluation of requirements for operational resilience will involve creating, emulating, or reasoning about the internal systems states that might result from successful attacks.

This tutorial will introduce the Framework for Operational Resilience in Engineering and System Test (FOREST), a framework that supports the derivation of measures and metrics for developmental and operational test plans and activities for cyber resilience in cyber-physical systems. FOREST aims to provide insights to support the development of testable requirements for cyber resilience and the design of systems with immunity to new vulnerabilities and threat tactics. FOREST’s elements range from attack sensing to the existence and characterization of resilience modes of operation to operator decisions and forensic evaluation. The framework is meant to be a reusable, repeatable, and practical framework that calls for system designers to describe a system’s operational resilience design in a designated, partitioned manner that aligns with resilience requirements and directly relates to the development of associated test concepts and performance metrics.

The tutorial introduces model-based systems engineering (MBSE) tools and associated engineering methods that complement FOREST and support the architecting, design, or engineering aspects of cyber resilience. Specifically, it features Mission Aware, a MBSE meta-model and associated requirements and architecture analysis process targeted to decomposition of loss scenarios into testable resilience features in a system design. FOREST, Mission Aware, and associated methodologies and digital engineering tools will be applied to two case studies for cyber resilience: (1) Silverfish, a hypothetical networked munition system and (2) an oil distribution pipeline. The case studies will lead to derivations of requirements for cyber resilience and survivability, along with associated measures and metrics.