Replacing Binary with Continuous Response Variables: JCAD

Continuous data provide the same information as binary data while augmenting it with rich details. Moreover, they do so with greater efficiency, as fewer test resources are required to provide the same level of information as their binary counterparts. The following example pertains to testing the Joint Chemical Agent Detector (JCAD; see Figure 1). It compares the required sample size as well as the results for the two different detection measurements: continuous (time to detection) and binary (detect/non-detect).

Scenario & Test goal

JCAD is a handheld device designed to detect chemical warfare agents under varying environmental conditions of temperature and humidity. After initial developmental testing, Design of Experiments (DOE) was used to characterize the detector’s performance across this operational envelope by choosing specific temperature and humidity combinations to test detection performance. These combinations are the specific test points (and replications of test points) that were chosen using DOE techniques because it would be impossible to test every possible condition. After constructing the test design, but before implementing it, the evaluators asked, “What test resources are needed to detect an improvement in the probability that JCAD will detect a chemical agent within 1 minute?”

Method

The evaluators made several decisions in order to conduct a series of power analyses. They decided that a 10% change in probability of detection between conditions (i.e., effect size) was the smallest detectable difference of interest. They would like an 80% chance of correctly identifying a real change of this size (power = 0.80). Also, they would accept a 10% chance of concluding that there was a change when, in fact, there was none (α = 0.10; confidence level = 90%). By recording both binary (detect/non-detect) and continuous (time to detect) response variables, the evaluators were able to compare the test resources required for each response variable type to show a 10% increase in the probability of detection between conditions.

Results

The number of replicates required when using a binary response variable is almost three times the number of replicates required for the continuous response variable (Table 1). Power analyses showed that using the continuous response variable would require 65% fewer test resources (total test points) to arrive at the same conclusion.

After executing the design, the evaluators found that the continuous measure substantially reduced uncertainty in the conclusions that could be drawn from the test, as confidence intervals around the predicted probability of detection were 300% smaller for the continuous response data than the binary response data (modeled for mean agent concentration). Moreover, it is clear that the requirement of 85% probability of detection is met when using the continuous metric, whereas this is not so clear with the binary data results (Table 2).


These results were possible because of the additional information continuous responses provide over binary response variables (See Figure 2). For example, there is a clear linear trend between concentration level and detection time. Moreover, Figure 2 shows that the detections will likely occur before the 60 seconds requirement at the mean concentration.

This case study shows the benefits of using continuous data over dichotomous (pass/fail) data. It is important to remember that data with a higher scale always can be converted to lower scale data, but not vice versa. Therefore, whenever possible, continuous data should be collected and the requirements should be recast in terms of mean or median with an associated variance.