Session Title | Speaker | Type | Materials | Year |
---|---|---|---|---|
Keynote Wednesday Lunchtime Keynote Speaker |
Jared Freeman Chief Scientist of Aptima and Chair of the Human Systems Division National Defense Industry Association ![]() (bio)
Jared Freeman, Ph.D., is Chief Scientist of Aptima and Chair of the Human Systems Division of the National Defense Industry Association. His research and publications address measurement, assessment, and enhancement of human learning, cognition, and performance in technologically complex military environments. |
Keynote |
![]() | 2019 |
Keynote Welcoming & Opening Keynote-Tuesday AM |
Mike Gilmore Director DOT&E ![]() (bio)
Link to Bio unavail |
Keynote | Materials | 2016 |
Breakout What is Bayesian Experimental Design? (Abstract)
In an experiment with a single factor with three levels, treatments A, B, and C, a single treatment is to be applied to each of several experimental units selected from some set of units. The response variable is continuous, and differences in its value show the relative effectiveness of the treatments. An experimental design will dictate which treatment is applied to what units. Since differences in the response variable are used to judge differences between treatments, the most important goal of the design is to prevent the treatment effect being masked by some unrelated property of the experimental units. Second important function of the design is to ensure power, that is, that if the treatments are not equally effective, the differences in the response variable are likely to be larger than background noise. Classical experimental design theory uses three principles: replication, randomization, and blocking, to produce an experimental design. Replication refers to how many units are used, blocking is a possible grouping of the units to reduce between unit heterogeneity, and randomization governs the assignment of units to treatment. Classical experimental designs are balanced as much as possible, that is, the three treatments are applied the same number of times, in each potential block of units. Bayesian experimental design aims to make use of additional related information, often called prior information, to produce a design. The information may be in the form of related experimental results, for example, treatments A and B may have been previously studied. It could be additional information about the experimental units, or about the response variable. This additional information could be used to change the usual blocking, to reduce the number of units assigned to treatments A and B compared to C, and/or reduce the total number of units needed to ensure power. This talk aims to explain Bayesian design concepts and illustrate them on realistic examples. |
Blaza Toman Statistical Engineering Division, NIST |
Breakout | Materials | 2018 |
Breakout When Validation Fails: Analysis of Data from an Imperfect Test Chamber (Abstract)
For chemical/biological testing, test chambers are sometimes designed with a vapor or aerosol homogeneity requirement. For example, a test community may require that the difference in concentration between any two test locations in a chamber be no greater than 20 percent. To validate the chamber, testers must demonstrate that such a requirement is met with a specified amount of certainty, such as 80 percent. With a validated chamber, multiple systems can be simultaneously tested at different test locations with the assurance that each system is exposed to nearly the same concentration. In some cases, however, homogeneity requirements are difficult to achieve. This presentation demonstrates a valid Bayesian method for testing probability of detection as a function of concentration in a chamber that fails to meet a homogeneity requirement. The demonstrated method of analysis is based on recent experience with an actual test chamber. Multiple systems are tested simultaneously at different locations in the chamber. Because systems tested in the chamber are exposed to different concentrations depending on these locations, the differences must be quantified to the greatest extent possible. To this end, data from the failed validation efforts are used to specify informative prior distributions for probability-of-detection modeling. Because these priors quantify and incorporate uncertainty in model parameters, they ensure that the final probability-of-detection model constitutes a valid comparison of the performance of the different systems. |
Kendal Ferguson | Breakout |
![]() | 2019 |
Contributed Workforce Analytics (Abstract)
Several statistical methods have been used effectively to model workforce behavior, specifically attrition due to retirement and voluntary separation[1]. Additionally various authors have introduced career development[2] as a meaningful aspect of workforce planning. While both general and more specific attrition modeling techniques yield useful results only limited success has followed attempts to quantify career stage transition probabilities. A complete workforce model would include quantifiable flows both vertically and horizontally in the network described pictorially here at a single time point in Figure 1. The horizontal labels in Figure 1 convey one possible meaning assignable to career stage transition – in this case, competency. More formal examples might include rank within a hierarchy such as in a military organization or grade in a civil service workforce. In the case of the Nuclear Weapons labs knowing that the specialized, classified knowledge needed to deal with Stockpile Stewardship is being preserved as evidenced by the production of Masters, individuals capable of independent technical work, is also of interest to governmental oversight. In this paper we examine the allocation of labor involved in a specific Life Extension program at LLNL. This growing workforce is described by discipline and career stage to determine how well the Norden-Rayleigh development cost model[3] fits the data. Since this model underlies much budget estimation within both DOD and NNSA the results should be of general interest. Data is also examined as a possible basis for quantifying horizontal flows in Figure 1. |
William Romine Lawrence Livermore National Laboratory |
Contributed | 2018 |
|
Breakout XPCA: A Copula-based Generalization of PCA for Ordinal Data (Abstract)
Principal Component Analysis is a standard tool in an analyst’s toolbox. The standard practice of rescaling each column can be reframed as a copula-based decomposition in which the marginal distributions are fit with a univariate Gaussian distribution and the joint distribution is modeled with a Gaussian copula. In this light, we present an alternative to traditional PCA we call XPCA by relaxing the marginal Gaussian assumption and instead fit each marginal distribution with the empirical distribution function. Interval-censoring methods are used to account for the discrete nature of the empirical distribution function when fitting the Gaussian copula model. In this talk, we derive the XPCA estimator and inspect the differences in fits on both simulated and real data applications. |
Cliff Anderson-Bergman Sandia National Laboratories |
Breakout | Materials | 2018 |
Breakout Your Mean May Not Mean What You Mean It to Mean (Abstract)
The average and standard deviation of, say, strength or dimensional test data are basic engineering math, simple to calculate. What those resulting values actually mean, however, may not be simple, and can be surprisingly different from what a researcher wants to calculate and communicate. Mistakes can lead to overlarge estimates of spread, structures that are over- or under-designed and other challenges to understanding or communicating what your data is really telling you. This talk will discuss some common errors and missed opportunities seen in engineering and scientific analyses along with mitigations that can be applied through smart and efficient test planning and analysis. It will cover when – and when not – to report a simple mean of a dataset based on the way the data was taken; why ignoring this often either hides or overstates risk; and a standard method for planning tests and analyses to avoid this problem. And it will cover what investigators can correctly (or incorrectly) say about means and standard deviations of data, including how and why to describe uncertainty and assumptions depending on what a value will be used for. The presentation is geared toward the engineer, scientist or project manager charged with test planning, data analysis or understanding findings from tests and other analyses. Attenders’ basic understanding of quantitative data analysis is recommended; more-experienced participants will grasp correspondingly more nuance from the pitch. Some knowledge of statistics is helpful, but not required. Participants will be challenged to think about an average as not just “the average”, but a valuable number that can and must relate to the engineering problem to be solved, and must be firmly based in the data. Attenders will leave the talk with a more sophisticated understanding of this basic, ubiquitous but surprisingly nuanced statistic and greater appreciation of its power as an engineering tool. |
Ken Johnson | Breakout |
![]() | 2019 |
Breakout
|
Tye Botting Research Staff Member IDA |
Breakout | 2016 |
|
Tutorial
(Abstract)
This tutorial will provide attendees with a live demo of an open source software reliability tool to automatically apply models to data. Functionality to be illustrated includes how to: Select and view data in time between failures, cumulative failures, and failure intensity formats. Apply trend tests to determine if a data set exhibits reliability growth, which is a prerequisite to apply software reliability growth models. Apply models to a data set . Apply measures of model goodness of fit to obtain quantitative guidance to select one or more models based on the needs of the user .Query model results to determine the additional testing time required to achieve a desired reliability. Following this live demonstration an overview of the underlying mathematical theory will be presented, including: Representation of failure data formats. Laplace trend test and running arithmetic average. Maximum likelihood estimation. Failure rate and failure counting software reliability models. Akaike information criterion and predictive sum of squares error. |
Lance Fiondella Univeristy of Massachusetts, Dartmouth |
Tutorial | Materials | 2016 |
Keynote
|
Dave Duma Acting Director, Operationak Test and Evaluation, Office of the Secretary of Defense DOT&E ![]() (bio)
Mr. Duma is the Acting Director, Operational Test and Evaluation as of January 20, 2017. Mr. Duma was appointed as the Principal Deputy Director, Operational Test and Evaluation in January 2002. In this capacity he is responsible for all functional areas assigned to the office. He participates in the formulation, development, advocacy, and oversight of policies of the Secretary of Defense and in the development and implementation of test and test resource programs. He oversees the planning, conduct, analysis, evaluation, and reporting of operational and live fire testing. He serves as the Appropriation Director and Comptroller for the Operational Test and Evaluation, Defense Appropriation and coordinates all Planning, Programming, and Budgeting Execution matters. He previously served as Acting Director, Operational Test and Evaluation from February 2005 to July 2007 and again from May 2009 to September 2009. Mr. Duma also served as the Acting Deputy Director, Operational Test and Evaluation from January 1992 to June 1994. In this capacity he was responsible for oversight of the planning, conduct, analysis, and reporting of operational test and evaluation for all major conventional weapons systems in the Department of Defense. He supervised the development of evaluation plans and test program strategies, observed the conduct of operational test events, evaluated operational field tests of all armed services and submitted final reports for Congress. Mr. Duma returned to government service from the commercial sector. In private industry he worked a variety of projects involving test and evaluation; requirements generation; command, control, communications, intelligence, surveillance and reconnaissance; modeling and simulation; and software development. Mr. Duma has 30 years of naval experience during which he was designated as a Joint Service Officer. He served as the Director, Test and Evaluation Warfare Systems for the Chief of Naval Operations, the Deputy Commander, Submarine Squadron TEN, and he commanded the nuclear powered submarine USS SCAMP (SSN 588). Mr. Duma holds Masters of Science degrees in National Security and Strategic Studies and in Management. He holds a Bachelor of Science degree in Nuclear Engineering. He received the U.S. Presidential Executive Rank Award on two occasions; in 2008, the Meritorious Executive Award and in 2015, the Distinguished Executive Rank Award. He is a member of the International Test and Evaluation Association. |
Keynote | 2017 |
Session Title | Speaker | Type | Materials | Year |
---|---|---|---|---|
Keynote Wednesday Lunchtime Keynote Speaker |
Jared Freeman Chief Scientist of Aptima and Chair of the Human Systems Division National Defense Industry Association ![]() |
Keynote |
![]() | 2019 |
Keynote Welcoming & Opening Keynote-Tuesday AM |
Mike Gilmore Director DOT&E ![]() |
Keynote | Materials | 2016 |
Breakout What is Bayesian Experimental Design? |
Blaza Toman Statistical Engineering Division, NIST |
Breakout | Materials | 2018 |
Breakout When Validation Fails: Analysis of Data from an Imperfect Test Chamber |
Kendal Ferguson | Breakout |
![]() | 2019 |
Contributed Workforce Analytics |
William Romine Lawrence Livermore National Laboratory |
Contributed | 2018 |
|
Breakout XPCA: A Copula-based Generalization of PCA for Ordinal Data |
Cliff Anderson-Bergman Sandia National Laboratories |
Breakout | Materials | 2018 |
Breakout Your Mean May Not Mean What You Mean It to Mean |
Ken Johnson | Breakout |
![]() | 2019 |
Breakout
|
Tye Botting Research Staff Member IDA |
Breakout | 2016 |
|
Tutorial
|
Lance Fiondella Univeristy of Massachusetts, Dartmouth |
Tutorial | Materials | 2016 |
Keynote
|
Dave Duma Acting Director, Operationak Test and Evaluation, Office of the Secretary of Defense DOT&E ![]() |
Keynote | 2017 |
2019-06-24