Probability of Detection: Q53


Binary and Continuous Data: Q53 Counterfire Radar Analysis Case Study

It’s important to consider all the factors that may influence system performance. For some systems, there may be many factors to consider, so a systematic approach that can account for multiple factors simultaneously is desirable. Linear models are a broad, flexible class of tools that excel in these situations. The following example shows how two types of linear models were used to help evaluate the AN/TPQ-53 Counterfire Radar. A logistic regression is used to estimate probability of detection across a complex factor space, and a lognormal regression is used to estimate target location error.


Scenario & Test Goal

Soldiers emplacing the Q-53 radar
Figure 1. Soldiers emplacing the Q-53 radar during the Initial Operational Test & Evaluation.

Mortar, rocket, and artillery fire posed a significant threat to U.S. forces in Afghanistan and Iraq and will continue to pose a significant threat to ground troops in future conflicts. The AN/TPQ-53 Counterfire Radar (Figure 1) is a ground-based radar designed to detect incoming mortar, artillery, and rocket projectiles; predict impact locations; and locate the threat geographically. The Q-53 has a variety of operating modes designed to help optimize its search. The 360-degree mode searches for projectiles in all directions around the radar, while 90-degree search modes can be used to search for threats at longer ranges in a specific sector. In addition to the various operating modes, the Q-53 radar’s detection performance can vary depending on characteristics of incoming projectile trajectory relative to the radar’s position (Figure 2). Determining how much the radar’s performance varies across all these factors is essential to inform users of the capabilities and limitations of this system as well as to identify deficiencies in need of correction.

Figure 2

Method

The Q-53 Initial Operational Test and Evaluation (IOT&E) replicated typical Q-53 combat missions as accurately as possible given test constraints. Many missions were observed by two radars, so a single threat fire mission could be detected by two radars. Testers fired 2,873 projectiles, which resulted in 323 usable fire missions. The data are heavily imbalanced due to the free-play nature of the test.

Figure 3 shows the raw probability of detection data. Each point represents a fire mission, with the size of the point determined by the number of shots taken in the fire mission – ranging from a single shot to as many as 20 projectiles. The percentage of those shots detected by the Q-53 counterfire radar is shown on the y-axis. The colors of the points show the munition type, while different operating modes and fire rates are separated across the x-axis.

Figure 3

There is clearly a substantial variability in probability of detection across different combinations of operating mode, munition, and rate of fire. There are geometric differences between operating modes, complicating the definition of a shot’s geometry. A logistic regression can be used to identify which factors drive performance and to estimate probability of detection across the operational space. Logistic regression is a common analysis method for binary response variables, enabling analysts to estimate the effect of each factor while accounting for the others. The general logistic regression equation is

\(log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 x_1 + … + \beta_px_p.\)

In our case, is the probability of detection and βi and x  represent the coefficients and factors, respectively. This approach relates the log of the odds ratio of probability of detection to the various factors that affect the probability of detection. Unlike a more naïve approach that looks at factors one at a time, this method allows us to attribute changes in probability of detection to specific factors. This is especially important when the data are unbalanced and not generated from an experimental design. In cases such as these, one-factor-at-a-time analyses are especially error-prone and can incorrectly identify irrelevant factors as important while failing to identify the true drivers of system performance.

In addition to detecting incoming projectiles, the Q-53 counterfire radar also estimates the location from which the detected projectiles were fired. The radar tracks the projectile through most of its flight and then use that trajectory to estimate the location of the threat that fired to projectile. (This is the point of origin of the trajectory.) The distance between this point of origin and the location estimated by the Q-53 is referred to as target location error (TLE). TLEs present a different challenge, because while these measurements are continuous, they are not normally distributed, so standard approaches will produce biased results.

Figure 4 shows quantile plots of TLEs for the 360-degree operating mode, broken down by munition type. These quantile plots are arranged so data originating from a normal distribution will fall along the straight lines shown in the plot. The further away the data points fall from the straight line, the more the actual data distribution differs from a normal distribution. The chart on the left plots the raw data and reveals that they fall far from the straight lines. The plot on the right shows the same data on a log scale; the data fall much closer to the straight lines, which indicates that a lognormal distribution better represents the actual distribution of the data. As a result, the TLE data was analyzed using a lognormal regression. This approach allows us to take the skewness of the data into account so that the fit has the same characteristics as the data.

Figure 4

Results

The logistic regression model, once determined from the data, showed that – in addition to projectile type, operating mode, and rate of fire – radar weapon range, quadrant elevation (QE), aspect angle, and shot range had an impact on the probability of detection. Figure 5 shows how the probability of detection changes as the distance between the weapon and the Q-53 counterfire radar increases when the system is in the 360-degree operating mode observing single-fire artillery engagements. The data also revealed that radar-weapon range and quadrant elevation had large effects on Q-53’s ability to detect incoming projectiles.

Figure 5

These factors are linked to the time the projectile travels through the radar search sector. High arcing shots (larger values for quadrant elevation) are easier to see than shots with more shallow trajectories that stay closer to the ground (low quadrant elevation) and are more likely to be masked by terrain. The distance between the radar and the threat’s origin is very important. Other physical factors, such as shot range (not shown on the picture above) and aspect angle, may be statistically significant but have minimal practical effect on Q-53 probability of detection.

The logistic regression approach we employed also allows us to analyze the effects of these factors simultaneously and observe how they interact. In the Figure 5, as the radar-weapon range increases, the probability of detection drops sharply around 12,000 meters for shots with shallow trajectories (QE=30 degrees, shown with the blue lines). For the shots with more arc (QE=60 degrees, shown with black lines), the Q-53 is still able to detect with high probability at much farther ranges. While these factors have large effects, other factors such as aspect angle have relatively minor effects on probability to detect. Comparing the left and right panels, we can see that a 30-degree change in aspect angle results in a change in probability of detection no larger than 7 percent. This analysis illustrates the relative size of these factors on probability to detect.

The results for the continuous outcome variable are shown in Figure 6. The right side of the figure shows the TLE for mortars and the left side shows the TLE for artillery and rockets. The green lines show the system’s requirements, and the black lines show the estimated median TLE along with 80 percent confidence intervals. While TLE for mortars showed substantial variability, the large number of mortar fire missions allows us to make precise estimates of median TLE. The analysis revealed that the estimated median TLE tends to increase (get worse) as radar-weapon range increases. While the Q-53 is more accurate at estimating a mortar’s location than the location of artillery and rocket weapons, the requirements for artillery and rockets were less stringent. This regression approach accounts for the variety of factors impacting system performance, resulting in rigorous system evaluation.

Figure 6

 

 

 


 

Leave a Reply