DEVELOPING OPTIMAL PREDICTION EQUATIONS IN MULTIVARIATE REGRESSION ANALYSIS

Defining the ‘Full’ Model

• select either a linear or quadratic model

For Ŷ = f(X₁, X₂, X₃)

where (1) the number of terms = p

(2) each term can incorporate one or more transformed or untransformed
variables

Ŷ = a + b₁X₁ + b₂X₂ + b₃X₃

or

Ŷ = a + b₁X₁ + b₂X₂ + b₃X₃ + b₄X₁² + b₅X₂² + b₆X₃² + b₇X₁X₂ + b₈X₁X₃ + b₉X₂X₃

• n ≥ p + 1 must be true

• if k = the number of predictor variables then the number of terms in the full quadratic model (p) is calculated as follows: p = [(k + 1)([k + 2]/2)] - 1

• need to find the “most parsimonious” model (i.e., the one containing only terms that make a statistically significant contribution to the regression of Y – this is the “optimized” model)

Ordinary Regression for Y vs X₁, X₂, …, X_p

• do F tests of terms using the Type III Sums of Squares, where each term is treated as if it were the last one added to the model

• use a Residuals Plot to assess the ‘goodness-of-fit’ of the model to the data

(Y-Ŷ) vertical axis

vs

Ŷ horizontal axis

• keep the terms that pass the F test then repeat the regression analysis with just the retained terms – this is the final regression prediction equation

Stepwise Regression for Y vs X₁, X₂, …, X_p

1. Forward Selection where the model initially contains no terms

Step 1: find the first term to add to the model using ‘total’ correlations

i. calculate R_L for all possible one-term models

ii. add the term with the highest R_L to the model

iii. do an F-test (Type III) on the added term – if significant at, say, 95% it stays

in the model; otherwise the regression analysis is finished

Step 2: find the second term to add to the model using ‘partial’ correlations and/or F tests

Approach One

i. calculate linear partial correlations between all remaining (outside) terms and

Y (i.e., with the effects of the one term already in the model removed)

ii. add the term with the highest partial correlation to the model

iii. do an F-test (Type III) on the added term – if significant it stays in the model;

otherwise the regression analysis is finished

F_last = [SSR_diff / 1] / [Σ(Y-Ŷ)² / (n-p-1)] where p = no. of terms currently in model

Approach Two

i. do an F-test (Type III) on all outside terms, treating each as if it were the last

term entered

ii. add the term with the largest significant F_last

Note 1: default value for “F-to-enter” (i.e., F_crit) is usually set at 4.0

(= approx. value of F for n-p-1 > 30 at 95% significance level)

Note 2: SAS reports ‘p’ rather than ‘F_crit’ where (1-p)•100 is the percent significance level

Step 3 and beyond: Step 2 is repeated recursively for all terms outside the model until no more terms can be added to the model

Note: some programs also report an “adjusted R²” (R²_adj), which can be used instead of F tests

R²_adj = R² – ([p • (1-R²)] / [n-p-1])

whereas the ordinary R² continues to increase with each added term, the R²_adj will start to decrease when non-contributing terms are added

2. Backward Selection or Elimination where the model initially contains all the terms

Step 1: find the first term to remove from the model

i. calculate F_last statistics for all terms, treating each as if it were the last one

added to the model (Type III)

ii. remove the term with the smallest non-significant F_last

Step 2 and beyond: Step 1 is repeated recursively for all the remaining terms inside the model until no more terms can be removed from the model

Note 1: default value for “F-to-remove” (i.e., F_crit) is usually set at 4.0

Note 2: SAS reports ‘p’ rather than ‘F_crit’ where (1-p)•100 is the percent significance level

Note 3: Backward Selection is usually preferred over Forward Selection because it recognizes that two or more terms already in the model can together make a greater contribution than the sum of the individual contributions of the same terms taken one at a time as in Forward Selection

3. ‘Stepwise’ Procedure which is a combination of Forward and Backward Selections

The model initially contains no terms.

Step 1: the term with the largest significant F_last (Type III) is entered into the model

Step 2: the term with the next largest significant F_last is entered

Step 3: for the terms already in the model, F_last is calculated for each one and the one with the smallest insignificant F_last is removed (i.e., returned to the pool of other terms outside the model)

Step 4 and beyond: Steps 2 and 3 are repeated recursively until no more terms can be added or removed

Note 1: as the F-to-remove approaches zero, the procedure approaches pure Foreward Selection

Note 2: the F-to-remove must be smaller than the F-to-enter, otherwise the same term will be continuously added and removed

Note 3: the optimal values for F-to-remove (~3) and F-to-enter (~4) are very problematical and for this reason the Stepwise Procedure is unpopular with some data analysts

4. All Possible Subsets or R² Procedure (graphical without significance testing)

Objective: identify the model with the fewest terms but highest ‘meaningful’ R²

Step 1: calculate R² for all possible subsets of the ‘p’ terms; there will be a total of

2^p-1 subsets

Step 2: plot a scree (i.e., slope) diagram of R² vs the best model for a given number of terms; the first model where the curve begins to flatten out (i.e., at the slope break) is the optimal model

Option A: the terms in the best earlier models are always incorporated into all subsequent models

Option B: the model with the highest R² is always selected regardless of which terms are included

Note: R²_adj is usually preferred to R² because the curve will stop rising and start to drop as non-contributing terms are added in the larger models