DATAWorks Sessions Archive

Graph link prediction is an important task in cyber-security: relationships between entities within a computer network, such as users interacting with computers, or clients connecting to servers, can provide key insights into adversary behaviour. Poisson matrix factorization (PMF) is a popular model for link prediction in large networks, particularly useful for its scalability. An extension to PMF to include scenarios that are commonly encountered in cyber-security applications are presented. Specifically, an extension is proposed to include known covariates associated with the graph nodes as well as a seasonal variation to handle dynamic networks.

The mission of the Human Behavior and Cybersecurity Capability at MITRE is to leverage human behavior to reduce cybersecurity risk using behavioral sciences to understand and strengthen the human firewall. The Capability consists of a team of experienced behavioral scientists who bring applied subject-matter-expertise in human behavior to cybersecurity challenges, with demonstrated successes and thought leadership in insider threat and usable security. This presentation will introduce the four focus areas of the Capability: Insider Threat; Usable Security and Technology Adoption; Cybersecurity Assessment and Exercise Support; and Cybersecurity Risk Perceptions and Awareness. We will then discuss the methods and metrics used to study human behavior within each of these focal areas. Insider Threat Assessment: Our insider threat behavioral risk assessments use qualitative research practices to elicit discussion and disclosure of risks from the perspective of potential insiders. MITRE’s Insider Threat Framework, developed to address the challenges experienced by the National Critical Infrastructure in classifying and assessing insider threats, is a data-driven framework that includes psycho-social and cyber-physical characteristics that could be common, observable indicators for insider attacks. The continuous approach to developing this framework includes consistently structuring, hand-coding, collating and analyzing a large dataset (5,000-10,000) of raw insider threat investigation case files shared directly from multiple organizations. The aggregated framework, but not the sensitive raw data, is shared with insider threat programs to operationalize and facilitate the identification, prevention, detection and mitigation of insider threats. Usable Security and Technology Adoption: Our goal of studying the human aspects of usable security technologies is to improve decision making and enhance technology adoption by users. We will present examples of how we have applied psychological principles and behavioral methods to design valid and reliable approaches to evaluate the feasibility of new security programs, products, and resources such as AI-enabled security technology. Cybersecurity Assessment and Exercise Support: Complex collaborations among cybersecurity teams are becoming increasingly important but can trigger new challenges that impact mission success. We assess, evaluate and train individuals and teams to improve knowledge among cyber professionals during cybersecurity exercises. Examples of our work include cognitive adaptability and exercise assessments, development of team collaboration and performance metrics, team sense-making, and cyber team resilience. Cybersecurity Risk Perceptions and Awareness: Perceptions of cybersecurity risks and threats, and resulting decisions about how to approach or mitigate them have the potential to impact the effectiveness of cybersecurity programs. We evaluate how people, processes and technology impact tactical and strategic risk-based decisions and apply behavioral concepts to inform the cybersecurity risk framework and awareness programs.

As defense systems become more complex, multi-domain, and interdependent, the problem arises: What is the best way to determine what we need to test, and how much testing is adequate? A methodology, based on systems engineering, was developed specifically for use in Live Fire Test and Evaluation (LFT&E); however, the process can be applied to operational or developmental testing as well. The use of Systems Engineering principles involves understanding the prioritized warfighter (user) needs, and applies a definition process that identifies critical test issues. The main goals of this methodology are to clearly define the system performance priorities, based on the critical mission risks if performance is insufficient, and then to generate various solutions that will mitigate those associated risks. The systems engineering process also helps develop a common language and framework so all parties can discuss tradeoffs between cost, schedule and performance. It also produces specific products for each step of the process that ensure that each step has been adequately addressed. Most importantly, the methodology should enable better communication among and within the program, test, and oversight teams by clarifying mission and test priorities, clarifying test objectives, evaluating risks from proposed testing and demonstrated performance, and reporting decision-quality information. The result of implementing the methodology is two-fold: It produces a test strategy that prioritizes testing based on the criticality and uncertainty of the system’s performance; and it guides the development of a system evaluation that clearly links test outcomes to overall mission risks.

The Europa Clipper mission must comply with the NASA Planetary Protection requirement in NASA Procedural Requirement (NPR) 8020.12D, which states: “The probability of inadvertent contamination of an ocean or other liquid water body must be less than 1×10-4 per mission”. Mathematical approaches designed to assess compliance with this requirement have been offered in the past, but no accepted methodology was in-place to trace the end-to-end probability of contamination: from a terrestrial microorganism surviving in a non-terrestrial ocean, to the impact scenario that put them there, to potential flight system failures that led to impact, back to the initial bioburden launched with the spacecraft. As a result, hardware could presumably be either over- or under-cleaned. Over-specified microbial reduction protocols can greatly add to the cost (and schedule) of a project. On the other hand, if microbes on hardware are not sufficiently eliminated, there is increased risk of potentially contaminating another body with terrestrial organisms – adversely affecting scientific exploration and possibly conflicting with international treaty. The anticipated Mars Sample Return Campaign would be subject to a similar challenge regarding returning Martian material to Earth. A proposed requirement is “The MSR campaign shall have a probability of releasing unsterilized Martian particles, with diameters ≥ 50 nanometers (TBC), into Earth’s biosphere ≤ 1×10-6”. A similar question arises: what is required in terms of ensuring sterilization or containing Martian particles in order to meet the requirement? The mathematical framework and other interesting sensitivities that the analysis has revealed for each mission are discussed.

Structural Equation Modeling (SEM) is an analytical framework that offers unique opportunities for investigating human-system interactions. SEM is used heavily in the social and behavioral sciences, where emphasis is placed on (1) explanation rather than prediction, and (2) measuring variables that are not observed directly. The framework facilitates modeling of survey data through confirmatory factor analysis and latent (i.e., unobserved) variable regression models. We provide a general introduction to SEM by describing what it is, the unique features it offers to analysts and researchers, and how it is easily implemented in JMP Pro 15.1. The introduction relies on a fun example everyone can relate to. Then, we shed light on a few published studies that have used SEM to unveil insights on human performance factors and the mechanisms by which performance is affected. The key goal of this presentation is to provide general exposure to a modeling tool that is likely new to most in the fields of defense and aerospace.

The DoD’s challenge to provide capability at the “Speed of Relevance” has generated many new strategies to adapt to rapid development and acquisition. As a result, Operational Test Agencies (OTA) have had to adjust their test processes to accommodate rapid, but incremental delivery of capability to the warfighter. The Air Force Operational Test and Evaluation Center (AFOTEC) developed the Adaptive Relevant Testing (ART) concept to answer the challenge. In this session, AFOTEC Test Analysts will brief examples and lessons learned from implementing the ART principles on the KC-46A acquisition program to identify problems early and promote the delivery of individual capabilities as they are available to test. The AFOTEC goal is to accomplish these incremental tests while maintaining a rigorous statistical evaluation in a relevant and timely manner. This discussion will explain in detail how the KC-46A Initial Operational Test and Evaluation (IOT&E) was accomplished in a unique way that allowed the test team to discover, report on, and correct major system deficiencies much earlier than traditional methods.

Introduction: Information Operations (IO) are a key component of our adversaries’ strategy to undermine U.S. military power without escalating to more traditional (and more easily identifiable) military strikes. Social media activity is one method of IO. In 2017 and 2018, Twitter suspended thousands of accounts likely belonging to the Kremlin-backed Internet Research Agency (IRA). Clemson University archived a large subset of these tweets (2.9M tweets posted by over 2800 IRA accounts), tagged each tweet with metadata (date, time, language, supposed geographical region, number of followers, etc.), and published this dataset on the polling aggregation website FiveThirtyEight.

Requirements specification flaws are still the biggest contributing factor to most accidents related to software. Most NASA projects have safety-critical requirements that, if implemented incorrectly, could lead to serious safety implications and/or mission-ending scenarios. There are normally thousands of system-/subsystem-/component-level requirements that need to be analyzed for safety criticality early in the project development life cycle. Manually processing such requirements is typically time-consuming and prone to error. To address this, we implemented and tested text classification models to identify requirements that are safety-critical within project documentation. We found that a naïve Bayes classifier was able to identify all safety-critical requirements with an average false positive rate of 41.35%. Future models trained on larger project requirement datasets may achieve even better performance, reducing the burden of processing requirements on safety and mission assurance personnel and improving the safety of NASA projects.

Having multiple objectives is common is experimental design. However, a design that is optimal for a normal response can be very different from a design that is optimized for a nonnormal response. This application uses a weighted optimality criterion to identify an optimal design with continuous factors for three different response distributions. Both linear and nonlinear models are incorporated with normal, binomial and Poisson response variables. A JMP script employs a coordinate exchange algorithm that seeks to identify a design that is useful for all three of these responses. The impact from varying the prior distributions on the nonlinear parameters as well as changing the weights on the responses in the criterion is considered.

The Air Force maintains a space catalog of orbital information on tens-of-thousands of space objects, including active satellites and satellite debris. They also computationally project where they expect objects to be in the future. Each day, the Air Force issues warnings to satellite owner-operators about potential conjunctions (space objects passing near each other), which often results in one or both of the satellites maneuvering (if possible) for safety of flight. This problem grows worse as mega-constellations, such as SpaceX’s Starlink, are launched.

Recently, a Department of Energy (DOE) report was released on the concept of scientic machine learning (SML), which is broadly dened as “a computational tech- nology that can be trained, with scientic data, to augment or automate human skills.” [1] As the demand for machine learning (ML) in science and engineering rapidly in- creases, it is important to have condence that the output of the ML algorithm is representative of the phenomena, processes, or physics being modeled. This is espe- cially important in high-stakes elds such as defense and aerospace. In the DOE report, three research themes were highlighted with the aim of providing condence in ML im- plementations. In particular, ML algorithms should be domain-aware, interpretable, and robust. Deep learning has become a ubiquitous term over the past decade due to its ability to model high-dimensional complex processes, but domain awareness, interpretability, and robustness in these large neural networks (NNs) are often hard to achieve. Recent advances in physics-informed neural networks (PINNs) are promising in that they can provide both domain awareness and a degree of interpretability [2, 3, 4]. These al- gorithms take advantage of the breadth of scientic knowledge built over centuries by fusing governing partial dierential equations into the NN training process. In this way, PINNs output physically admissible solutions. However, PINNs are generally deter- ministic, meaning interpretability and robustness suffer as it is unclear how uncertainty affects the model. Another noteworthy deep learning algorithm is the generative adversarial network (GAN). GANs are capable of modeling probability distributions in both forward and inverse problems, and thus have received a ood of inerest with over 15,000 citations of the seminal paper [5] in six years. A natural next step is to combine both PINNs and GANs to address all three themes laid out in [1]. The resultant physics-informed GAN (PI-GAN) is capable of both modeling physical processes and simultaneously quantifying uncertainty. A limited number of works have already demonstrated the success of PI-GANs [6, 7]. This talk will present an introduction to PI-GANs as well as an example of current NASA research implementing these networks.

The ability of spaceborne remote sensing data to address important Earth and climate science problems rests crucially on how well the underlying geophysical quantities can be inferred from these observations. Remote sensing instruments measure parts of the electromagnetic spectrum and use computational algorithms to infer the unobserved true physical states. However, the accompanying uncertainties, if they are provided at all, are usually incomplete. There are many reasons why including but not limited to unknown physics, computational artifacts and compromises, unknown uncertainties in the inputs, and more.

In this talk I will describe a practical methodology for uncertainty quantification of physical state estimates derived from remote sensing observing systems. The method we propose combines Monte Carlo simulation experiments with statistical modeling to approximate conditional distributions of unknown true states given point estimates produced by imperfect operational algorithms. Our procedure is carried out post-hoc; that is, after the operational processing step because it is not feasible to redesign and rerun operational code. I demonstrate the procedure using four months of data from NASA’s Orbiting Carbon Observatory-2 mission, and compare our results to those obtained by validation against data from the Total Carbon Column Observing Network where it exists.

Testing today’s complex systems often requires advanced statistical methods to properly characterize and optimize measures of performance as a function of input factors. One promising area to more precisely model system behavior when the response is a curve or function over several measured time units is Functional Data Analysis (FDA). Some input factors such as sensor data could also be functions rather than held constant as is often assumed over the duration of the test run. Recent enhancements in common statistical software programs used across DoD and NASA now make FDA much more accessible to the analytical test community. This presentation will address the fundamental principles, workflow, and interpretation of results from FDA using a designed experiment for an autonomous system as an example. Additionally, we will address how to use FDA to establish a level of equivalence for modeling & simulation verification & validation efforts.