Squared Prediction Variance Graphs

Prediction Variance & Scaled Prediction Variance

Prediction Variance
In addition to coefficient estimation, one key reason we conduct experiments is to predict future performance of a system.  Prediction variance describes the error involved with making a prediction using a regression model.  Consider an operational test that consists of runs and factors.  The corresponding first order regression model is y=X\beta+\epsilon  .  The response variable y is size [N x 1], X is the [N x k] design matrix, \beta is a [k x 1] vector of coefficients, and \epsilon is a [N x 1] vector of random errors that has E(\epsilon) = 0  and var(\epsilon) = \sigma^2I_N.  The predicted value at any point in the design space is \hat{y} = x'_{0}\hat\beta, where \hat\beta=(X'X)^{-1}X'y is the maximum likelihood estimator of \beta and x_0 = [ \begin{array}{ccccc} 1&x_1&x_2&...&x_k \end{array}].  The prediction variance at any point in the operational envelope, x_0, is defined as V=\sig^2x'_0(X'X)^{-1}x_0.  Thus, prediction variance is a function of the designed experiment (X), the location in the design space where the prediction is made (X_0), and the overall variance in the response (\sigma^2).  Since \sigma^2 is unknown, prediction variance can be difficult to use for evaluating the merit of an experimental design.
Scaled Prediction Variance
Scaled Prediction Variance (SPV), on the other hand, normalizes the prediction variance by \sigma^2 so that SPV is a function of N, X , and x_0, that is SPV = \frac{NV}{\sig^2} = Nx'_0(X'X)^{-1}x_o. The benefit of SPV is that it can be used to evaluate a designed experiment prior to running the test and collecting data.  Multiple designed experiments can be postulated for a single test event and compared using SPV and the best design can be selected.  When assessing a design in this way, it is important to consider the full range of values each factor can take.  For categorical factors, this is just a matter of considering prediction at each level of the relevant factors.  For continuous variables, graphical methods such as contour plots are available. To get a better understanding of SPV, consider a notional penetration test example.  In this example only 10 shots are available to characterize penetration depth of a small arms munition impacting an armor plate on a light combat vehicle as a function of muzzle velocity and range to target.  Additionally, the test team plans to fit a second order regression model and has two candidate experimental designs shown in Table 1.  The levels of muzzle velocity and range are expressed in normalized units between minus one and one.  The SPV contour plots for each experiment are shown in Figure 1. Table 1 SPV designs table Figure 1 SPV A vs BCareful inspection of Figure 1 shows that while Design B has a larger region with the minimal SPV (less than 3.5), a greater portion of the design space for Design A has an SPV less than 4.0. Meanwhile, the SPV near the extremes is much greater for Design B.  Based on these observations, Design A is the preferred experimental design for the limited 10 shots. In this example, it is reasonable to characterize SPV because it is a two-dimensional problem. In cases where there are more than two factors, when the characterization is not straight forward, there are different types of summarizing graphs that can be used to compare experimental designs, such as the Fraction of Design Space plot.  

Leave a Reply