Chapter 12 — Front Matter
Purpose and Overview
A model with one predictor is a starting point. A model that accounts for multiple sources of variation is how inference connects to reality.
Chapter 11 introduced linear regression as a tool for modeling the relationship between one predictor and one response. That framework is powerful, but the world rarely cooperates with single-predictor simplicity. The weight of a patient depends not only on height but also on age, sex, activity level, and diet. The salary of an employee depends not only on years of experience but also on field, education level, geographic region, and job title. The yield of a crop depends not only on rainfall but also on soil composition, temperature, and fertilizer type.
Multiple regression extends the regression framework to accommodate several predictors simultaneously. This extension changes what coefficients mean, how assumptions work, how models are evaluated, and how conclusions must be interpreted. Multiple regression is one of the most widely used statistical tools in science, economics, medicine, social research, engineering, and data science.
This chapter also introduces analysis of variance, known as ANOVA. Where regression asks how a numerical response changes with one or more predictors, one-way ANOVA asks whether the means of a numerical response differ across several groups. Both methods belong to a unified framework called the general linear model, and the F-test appears prominently in both.
14Sections
12Objectives
28Key Terms
10+Worked Examples
Learning Objectives
- 1Explain why multiple predictors may be needed and describe how omitted variable bias arises.
- 2State the multiple regression model and derive the least squares estimator in matrix form.
- 3Interpret partial regression coefficients correctly, including the phrase "holding all other predictors fixed."
- 4Construct indicator variables for categorical predictors and interpret their coefficients relative to a reference category.
- 5Interpret an interaction term and explain the hierarchy principle.
- 6Calculate and interpret R² and explain why it increases mechanically with every predictor added.
- 7Calculate and use adjusted R² and describe AIC and BIC as model selection criteria.
- 8Explain why automated stepwise selection should be used with caution and describe best practices for model selection.
- 9Partition total variation into between-group and within-group components and construct a one-way ANOVA table.
- 10Compute the F-statistic, identify its distribution under H₀, and state a conclusion.
- 11Apply the overall and partial F-tests in multiple regression.
- 12Identify and avoid the most common multiple regression and ANOVA mistakes.
Sections 12.1 – 12.3
Multiple Regression: Model, Estimation, and Interpretation
Motivation · Model and Matrix Form · Partial Coefficient Interpretation
Section 12.1
Why One Predictor May Not Be Enough
The Limits of Simple Regression
Recall from Chapter 11 that simple linear regression models the conditional mean of a response variable Y given a single predictor X:
E[Y | X = x] = β₀ + β₁x
This model is elegant and interpretable. Each one-unit increase in X is associated with a β₁-unit change in the mean of Y. But consider a concrete example. Suppose we study resting heart rate as a function of age. The estimated slope might be positive and significant. Now suppose we also record whether each person exercises regularly. If regular exercisers tend to be younger in our sample, the regression of heart rate on age partly reflects exercise status rather than age itself. When we add exercise status as a second predictor, the coefficient on age adjusts — it now estimates the association between age and heart rate among people with the same exercise status. The two models answer genuinely different questions.
This is the fundamental motivation for multiple regression: to estimate the association between one predictor and the response while accounting for other predictors that might confound or modify that association.
Confounding and Omitted Variable Bias
A confounder is a variable associated with both the predictor of interest and the response. When a confounder is omitted, the estimated coefficient on the predictor of interest absorbs its effect, creating omitted variable bias.
A classic illustration: the strong positive correlation between ice cream sales and drowning deaths. A simple regression might yield a significant positive slope. The confounder is temperature — warm weather causes both more ice cream purchases and more swimming, and more swimming leads to more drowning incidents. Including temperature in the model would reduce or eliminate the apparent association.
No regression can adjust for confounders that were never measured. Multiple regression controls for measured confounders only.
When Multiple Regression Is Appropriate
Use multiple regression when several predictors are plausibly related to the response, when confounding must be addressed, when precision can be improved by accounting for additional variation, or when the scientific question itself involves multiple variables simultaneously.
Section 12.2
Multiple Regression Models
The General Model
A multiple regression model with p predictors has the form:
Y = β₀ + β₁X₁ + β₂X₂ + ··· + βₚXₚ + ε
Assumptions of Multiple Regression
1. Linearity — The conditional mean of Y given X₁, …, Xₚ is linear in those predictors.
2. Independence — The errors εᵢ are independent across observations.
3. Equal variance — Var(εᵢ) = σ² for all i (homoscedasticity).
4. Normality — εᵢ ~ N(0, σ²).
5. No perfect multicollinearity — No predictor is an exact linear combination of the others. (New in multiple regression.)
Least Squares Estimation in Matrix Form
Stack all n observations into a design matrix X (n × (p+1), first column all ones) and a response vector y (n × 1). The least squares estimator minimizes:
RSS(β) = ‖y − Xβ‖² = Σᵢ(yᵢ − β₀ − β₁xᵢ₁ − ··· − βₚxᵢₚ)²
Setting the gradient to zero yields the normal equations:
XᵀX β̂ = Xᵀy
When XᵀX is invertible (no perfect multicollinearity), the unique solution is:
β̂ = (XᵀX)⁻¹ Xᵀy
Generalizing Chapter 11
The formula β̂ = (XᵀX)⁻¹Xᵀy generalizes the simple regression slope formula. With p = 1 predictor, it reduces exactly to the Chapter 11 formulas for β̂₀ and β̂₁.
The residual variance is estimated by:
σ̂² = RSS / (n − p − 1)
where n − p − 1 is the degrees of freedom for error: n observations minus p + 1 estimated parameters. Under the regression assumptions, the Gauss-Markov theorem extends: β̂ is the best linear unbiased estimator (BLUE). Under normality:
β̂ ~ N(β, σ²(XᵀX)⁻¹)
Section 12.3
Interpreting Regression Coefficients
The Central Interpretive Principle
In multiple regression, β̂ⱼ is the estimated change in the mean of Y per one-unit increase in Xⱼ, holding all other predictors fixed. This phrase — holding all other predictors fixed, or equivalently adjusting for all other predictors — is critical and cannot be dropped.
Example 12.3.1
Interpreting partial slopes
- ModelAnnual salary (thousands) = 38.4 + 2.1·Experience + 11.7·GradDegree, where GradDegree = 1 if yes, 0 if no.
- β̂₁ = 2.1Among employees with the same education level, each additional year of experience is associated with an estimated $2,100 higher annual salary.
- β̂₂ = 11.7Among employees with the same years of experience, having a graduate degree is associated with an estimated $11,700 higher salary.
- Key pointNeither coefficient can be interpreted in isolation. Adding experience to the model changes what the education coefficient means, because the two predictors are correlated.
Sign Changes and Suppression
A coefficient can change sign when additional predictors are included. This is not an error — it is a consequence of adjusting for correlated variables. A predictor that appears positively associated with the response in a simple regression can be negatively associated after controlling for a confounder. Never interpret simple regression coefficients as measuring direct effects when multiple predictors are plausibly relevant.
Inference for Individual Coefficients
Under normality, the test statistic for H₀: βⱼ = 0 is:
t = β̂ⱼ / SE(β̂ⱼ) ~ t(n − p − 1) under H₀
A (1 − α) confidence interval for βⱼ is:
β̂ⱼ ± t_{α/2, n−p−1} · SE(β̂ⱼ)
Sections 12.4 – 12.6
Categorical Predictors, Indicator Variables, and Interactions
Indicator Variables · Reference Category · Parallel Lines · Interaction Terms
Section 12.4
Categorical Predictors
Many research questions involve categorical grouping variables: treatment group, region, educational attainment, or species. These variables carry real information but cannot be entered as raw numbers without imposing an arbitrary numerical structure. Coding "Northeast = 1, South = 2, Midwest = 3, West = 4" falsely implies that the Midwest is exactly halfway between South and West.
Multiple regression accommodates categorical predictors through indicator variables (also called dummy variables), which encode group membership using binary (0/1) variables.
The k − 1 Rule
For a categorical predictor with k categories, create exactly k − 1 indicator variables. Choose one category as the reference category, represented by setting all indicators to zero. Including all k indicators creates perfect multicollinearity and makes estimation impossible.
Example 12.4.1
Seasonal indicator variables (k = 4 categories)
- SetupSeason: Spring, Summer, Fall, Winter. Spring is the reference (k − 1 = 3 indicators created).
- Model
Ŷ = β̂₀ + β̂_Summer · X_Summer + β̂_Fall · X_Fall + β̂_Winter · X_Winter
- Spring obs.All three indicators = 0 → Ŷ = β̂₀ (estimated mean for reference group)
- Summer obs.X_Summer = 1, others = 0 → Ŷ = β̂₀ + β̂_Summer
- Interpretationβ̂_Summer is the estimated difference in mean Y between Summer and Spring.
Example 12.4.2
Blood pressure model with treatment indicator
- Model
Ŷ = 130.2 − 8.4·X_DrugA − 5.1·X_DrugB (reference: Placebo)
- Drug A coeffDrug A is associated with an estimated 8.4 mmHg reduction in mean blood pressure compared to Placebo.
- Drug B coeffDrug B is associated with an estimated 5.1 mmHg reduction compared to Placebo.
- Drug A vs BThe difference Drug A − Drug B = 8.4 − 5.1 = 3.3 mmHg. This is not directly reported but computed from the output.
Always identify the reference category when reporting indicator coefficients.
Section 12.5
Indicator Variables
When a categorical variable has exactly k = 2 categories, one indicator variable suffices. For a binary predictor X coded 0 or 1, the estimated slope β̂₁ is the estimated difference in mean Y between the two groups. The t-test for H₀: β₁ = 0 tests whether the group means differ. When there are no other predictors, this is equivalent to a two-sample t-test.
Example 12.5.1
Mixing numerical and categorical predictors — parallel lines
- Model
Ŷ_Salary = 42.0 + 2.3·Experience + 9.8·Private (Private = 1 for private sector)
- Public sectorX₂ = 0: Ŷ = 42.0 + 2.3·Experience
- Private sectorX₂ = 1: Ŷ = 51.8 + 2.3·Experience
- Key featureThe slope on experience is the same (2.3) in both sectors. Only the intercept differs by 9.8. These are parallel lines.
The parallel lines assumption: the additive model assumes the effect of each predictor is the same across all levels of the other predictors. An interaction term is needed when this assumption fails.
Section 12.6
Interactions as an Introduction
An interaction between two predictors means the effect of one predictor on the response depends on the value of the other. Interactions arise naturally: a drug's benefit may depend on another medication; fertilizer effects depend on soil type; the education-income relationship may differ by gender.
An interaction between X₁ and X₂ is represented by their product, X₁ · X₂, entered as a new predictor. The model remains linear in its parameters:
Y = β₀ + β₁X₁ + β₂X₂ + β₃(X₁ · X₂) + ε
Example 12.6.1
Interpreting an interaction coefficient
- Model
Ŷ = 42.0 + 2.3·Experience + 9.8·Private + 1.1·(Experience × Private)
- Public (X₂ = 0)
Ŷ = 42.0 + 2.3·Experience
- Private (X₂ = 1)
Ŷ = 51.8 + 3.4·Experience
- Interpretationβ̂₃ = 1.1 is the difference in slopes: each additional year of experience is associated with an extra $1,100 per year in the private sector compared to the public sector.
Hierarchy Principle
When an interaction is included, both main effects (the individual terms X₁ and X₂) must also be included, even if their individual p-values are not significant. Removing a main effect while keeping its interaction changes the meaning of the interaction coefficient in scientifically unjustifiable ways.
Interaction Models Require Care in Interpretation
In models with interactions, no single number summarizes the effect of a predictor. The effect of X₁ depends on X₂ and vice versa. Report effects at meaningful or representative values of the moderating variable. Graphing predicted values as a function of one predictor across several levels of the other is often the clearest communication tool.
Sections 12.7 – 12.9
Model Fit, Adjusted R², and Model Selection
R² · Adjusted R² · AIC and BIC · Stepwise Selection Warnings
Section 12.7
Model Fit and R²
Just as in simple regression, we decompose total variation in Y:
SST = SSR + RSS
SST = Σ(yᵢ − ȳ)² Total Sum of Squares
SSR = Σ(ŷᵢ − ȳ)² Regression Sum of Squares
RSS = Σ(yᵢ − ŷᵢ)² Residual Sum of Squares
Coefficient of Determination
R² = SSR / SST = 1 − RSS / SST
R² measures the proportion of total variation in Y explained by the model. It ranges from 0 to 1. An R² of 0.72 means 72% of the variation in Y is explained by the included predictors.
What R² Does Not Mean
A high R² does not mean the model is correct. A low R² does not mean the model is wrong — in fields with high natural variability, R² values of 0.20 or 0.30 may be entirely reasonable. Most critically: R² cannot decrease when predictors are added. Adding random noise as a predictor still increases R², making it an unreliable criterion for comparing models with different numbers of predictors.
Section 12.8
Adjusted R² as an Introduction
Adjusted R² penalizes for the number of predictors:
R²_adj = 1 − (RSS / (n − p − 1)) / (SST / (n − 1))
= 1 − (1 − R²) · (n − 1) / (n − p − 1)
The factor (n−1)/(n−p−1) increases as p increases, so adjusted R² increases only when a new predictor improves model fit enough to overcome the penalty. Unlike R², adjusted R² can decrease when an uninformative predictor is added. It can even be negative when the model fits worse than the grand mean line Ŷ = ȳ.
Example 12.8.1
Adjusted R² detecting an uninformative predictor
- 3-predictor modelR² = 0.614, R²_adj = 0.601
- 4-predictor modelR² = 0.617 (increased), R²_adj = 0.598 (decreased)
- ConclusionThe fourth predictor did not contribute meaningfully. R² rose by construction; adjusted R² correctly signaled that the model worsened.
Section 12.9
Model Selection as an Introduction
Information Criteria
AIC and BIC penalize model fit (measured by RSS) for the number of parameters. For regression models with normally distributed errors:
AIC = n·ln(RSS/n) + 2(p + 2)
BIC = n·ln(RSS/n) + (p + 2)·ln(n)
| Criterion | Goal | Penalty | Tendency |
|---|
| AIC | Predictive accuracy | 2(p + 2) | Selects more complex models |
| BIC | Model identification | (p + 2)·ln(n) | More conservative; prefers simpler models |
Lower values of AIC or BIC indicate better models.
A Warning About Data-Driven Model Selection
Post-Selection Inference Problem
When the same data are used to select predictors and to perform inference, reported standard errors and p-values no longer have their nominal properties. A coefficient that appears significant partly because it was selected for significance cannot be tested at the nominal level using standard formulas. Stepwise selection followed by standard inference is a common and serious error.
Best Practices for Model Selection
→ Specify the model before looking at the data, based on subject-matter knowledge.
→ Use held-out test data or cross-validation to evaluate prediction accuracy.
→ Report all candidate models considered, not just the selected one.
→ Use information criteria rather than significance thresholds as selection criteria.
→ Treat automated stepwise procedures with skepticism.
Sections 12.10 – 12.12
Comparing Group Means: One-Way ANOVA and F-Tests
ANOVA Setup · SST = SSB + SSW · Mean Squares · F-Distribution · Partial F-Test
Section 12.10
Comparing Group Means
Chapter 9 introduced two-sample t-tests for comparing two group means. When there are three or more groups, applying multiple two-sample t-tests inflates the overall Type I error rate. If we perform all pairwise comparisons at the α = 0.05 level, the probability of at least one false rejection far exceeds 0.05.
Analysis of Variance (ANOVA) tests equality of means across multiple groups simultaneously, controlling the overall Type I error rate at the nominal level.
The One-Way ANOVA Setup
k groups, nⱼ observations in group j, total n = Σnⱼ observations. Let Yᵢⱼ = the i-th observation in group j.
Model: Yᵢⱼ = μⱼ + εᵢⱼ, εᵢⱼ ~ N(0, σ²) i.i.d.
H₀: μ₁ = μ₂ = ··· = μₖ (all group means equal)
H₁: at least two group means differ
Section 12.11
One-Way ANOVA
The Logic: Comparing Two Estimates of σ²
ANOVA compares two estimates of the population variance σ²: within-group variance (always a valid estimate of σ²) and between-group variance (estimates σ² when H₀ is true, estimates something larger when group means differ). If the ratio is close to 1, data are consistent with H₀. If the ratio is much greater than 1, H₀ is suspect.
Partitioning Total Variation
SST = SSB + SSW
SST = ΣⱼΣᵢ (yᵢⱼ − ȳ)² Total variation around grand mean ȳ
SSB = Σⱼ nⱼ(ȳⱼ − ȳ)² Between-group variation (weighted group means vs grand mean)
SSW = ΣⱼΣᵢ (yᵢⱼ − ȳⱼ)² Within-group variation (observations vs their group mean)
| Source | Sum of Squares | Degrees of Freedom | Mean Square |
|---|
| Between groups | SSB | k − 1 | MSB = SSB / (k − 1) |
| Within groups | SSW | n − k | MSW = SSW / (n − k) |
| Total | SST | n − 1 | |
What Each Mean Square Estimates
MSW is an unbiased estimator of σ² regardless of whether H₀ is true. MSB estimates σ² when H₀ is true; it estimates σ² + (extra term reflecting group mean differences) when H₀ is false. The F-statistic MSB/MSW exploits this contrast.
Example 12.11.1
Complete one-way ANOVA — fertilizer experiment
- SetupThree fertilizer formulations (A, B, C), 10 plots each, n = 30. Response: crop yield (kg/plot).
- Data
Group A: n = 10, ȳ_A = 48.2, s²_A = 12.4
Group B: n = 10, ȳ_B = 52.6, s²_B = 15.1
Group C: n = 10, ȳ_C = 55.9, s²_C = 11.8
Grand mean: ȳ = (48.2 + 52.6 + 55.9) / 3 = 52.23
- SSB
SSB = 10(48.2 − 52.23)² + 10(52.6 − 52.23)² + 10(55.9 − 52.23)²
= 10(16.24) + 10(0.137) + 10(13.49)
= 162.4 + 1.37 + 134.9 = 298.7
- SSW
SSW = 9(12.4) + 9(15.1) + 9(11.8) = 111.6 + 135.9 + 106.2 = 353.7
| Source | SS | df | MS | F | p-value |
|---|
| Between | 298.7 | 2 | 149.35 | 11.37 | < 0.001 |
| Within | 353.7 | 27 | 13.10 | | |
| Total | 652.4 | 29 | | | |
F = 149.35 / 13.10 ≈ 11.4. With 2 and 27 degrees of freedom, the p-value is very small (< 0.001), providing strong evidence against H₀. Not all fertilizer formulations produce the same mean yield.
Section 12.12
F-Tests
The F-Distribution
If U ~ χ²(d₁) and V ~ χ²(d₂) are independent, then:
F = (U/d₁) / (V/d₂) ~ F(d₁, d₂)
The F-distribution has two parameters — numerator df d₁ and denominator df d₂ — and is right-skewed, taking only non-negative values. Under the ANOVA normality assumption, when H₀ is true: F = MSB/MSW ~ F(k − 1, n − k). The ANOVA F-test is always one-sided in the upper tail.
The Overall Regression F-Test
The overall regression F-test tests H₀: β₁ = β₂ = ··· = βₚ = 0 against the alternative that at least one βⱼ ≠ 0.
F = (SSR / p) / (RSS / (n−p−1)) = MSR / MSE ~ F(p, n−p−1) under H₀
| Source | SS | df | MS | F | p-value |
|---|
| Regression | SSR | p | MSR = SSR/p | MSR/MSE | P(F(p, n−p−1) ≥ F_obs) |
| Error | RSS | n−p−1 | MSE = RSS/(n−p−1) | | |
| Total | SST | n−1 | | | |
Partial F-Tests
The partial F-test compares a full model to a reduced model that omits q specific predictors. It tests whether those q predictors collectively contribute beyond the remaining predictors.
F = [(RSS_Reduced − RSS_Full) / q] / [RSS_Full / (n − p − 1)] ~ F(q, n−p−1) under H₀
When to Use Partial F-Tests
Use a partial F-test to test whether all k − 1 indicator variables for a categorical factor are jointly significant, or whether all interaction terms for a pair of predictors are jointly significant. Testing groups of related predictors together controls the Type I error rate better than testing each coefficient individually.
Sections 12.13 – 12.14
Regression and ANOVA in Research and Common Mistakes
Unified Linear Model · Multiple Comparisons · Diagnostics · Ten Common Errors
Section 12.13
Regression and ANOVA in Research
ANOVA as a Special Case of Regression
One-way ANOVA is mathematically equivalent to a regression model with k − 1 indicator variables for group membership and no other predictors. The F-test for equality of group means in ANOVA is identical to the overall F-test for that regression. Both are special cases of the general linear model. Adding numerical covariates to an ANOVA model produces analysis of covariance (ANCOVA), which tests group mean differences after adjusting for the covariates.
Multiple Comparisons
When the ANOVA F-test rejects H₀, we conclude that not all group means are equal — but not which pairs differ. Post-hoc pairwise comparisons are needed. Performing all pairwise t-tests without adjustment inflates the family-wise error rate (the probability of at least one false rejection).
| Procedure | Approach | When to Use |
|---|
| Bonferroni | Divide α by the number of comparisons | Any set of planned comparisons; conservative |
| Tukey HSD | Controls family-wise error rate exactly for all pairwise comparisons | All pairwise comparisons after ANOVA |
| Dunnett | Compares each group to a single control | Control-vs-treatment comparisons |
ANOVA Assumptions and Diagnostics
| Assumption | Diagnostic Tool | Alternative if Violated |
|---|
| Independence | Study design review | Mixed models, repeated measures ANOVA |
| Equal variances (homoscedasticity) | Levene's test; Brown-Forsythe test | Welch ANOVA |
| Normality within groups | Shapiro-Wilk test; QQ plots of residuals | Kruskal-Wallis test (nonparametric) |
Regression Diagnostics in Multiple Regression
| Diagnostic | Purpose | Flag |
|---|
| Residual vs fitted plot | Check linearity and equal variance | Patterns, fanning, curvature |
| Normal QQ plot of residuals | Check normality assumption | Systematic departure from diagonal |
| Leverage | Unusual predictor values | High-leverage obs. may dominate fits |
| Cook's distance | Overall influence on all coefficients | Cook's D > 1 (or > 4/n) warrants review |
| Variance inflation factor (VIF) | Degree of multicollinearity | VIF > 5 or 10 signals inflated SE |
Section 12.14
Common Multiple Regression and ANOVA Mistakes
Mistake 1 — Interpreting Coefficients Without "Holding Others Fixed"
β̂₁ in a multiple regression is the same as the slope from a simple regression of Y on X₁.
The partial slope β̂₁ describes the relationship among observations with the same values of all other predictors. When predictors are correlated, simple and partial slopes can differ substantially in magnitude and sign.
Mistake 2 — Concluding Causation from Regression
"The significant coefficient on X₁ means that changing X₁ causes Y to change."
Correct language: "X₁ is associated with Y after adjusting for X₂, …, Xₚ." Causal language requires randomization, theoretical justification, and careful design — not a significant p-value.
Mistake 3 — Ignoring the Reference Category
Reporting "the coefficient for Drug A is −8.4" without naming the reference group.
A coefficient of −8.4 is meaningless without knowing the comparison is to Placebo. Always identify the reference category when reporting indicator variable results.
Mistake 4 — Using R² to Compare Models with Different Numbers of Predictors
"Model B has R² = 0.79 and Model A has R² = 0.72, so Model B is clearly better."
R² increases with every predictor added, regardless of whether the predictor is meaningful. Use R²_adj, AIC, BIC, or cross-validation to compare models of different complexity.
Mistake 5 — Including All k Indicators for a k-Category Variable
Creating 4 indicator variables for a 4-category predictor and including all 4 plus an intercept.
This creates perfect multicollinearity. The design matrix is singular. Always include exactly k − 1 indicator variables, omitting one category as the reference.
Mistake 6 — Performing Post-Hoc Comparisons Without Adjustment
After a significant ANOVA F-test, performing all 6 pairwise t-tests at α = 0.05 without correction.
The family-wise error rate is inflated. Use Tukey's HSD, Bonferroni, or another appropriate correction for all pairwise comparisons after ANOVA.
Mistake 7 — Neglecting Assumption Checks
Running ANOVA or regression and reporting p-values without examining residual plots or testing equal variances.
F-tests and t-tests are only valid under the stated assumptions. Residual plots, QQ plots, and variance tests are mandatory, not optional.
Mistake 8 — Treating P-Values as Measures of Effect Size
"p < 0.001, so the effect is large and practically important."
A small p-value indicates the effect is unlikely to be zero. With large samples, even trivially small effects produce very small p-values. Always report effect sizes (coefficient estimates, group mean differences) alongside p-values.
Mistake 9 — Stepwise Selection Followed by Standard Inference
Using stepwise selection to choose the model, then reporting the selected coefficients' p-values as valid tests.
The selected model was chosen partly because its coefficients happened to look significant. Standard errors and p-values from the final model are biased. Pre-specify models or use out-of-sample evaluation.
Mistake 10 — Forgetting that ANOVA Tests the Global Null
"The ANOVA p-value was 0.003, so Drug B is better than Placebo."
The ANOVA F-test only rejects H₀: μ₁ = ··· = μₖ. It does not identify which pairs differ. Post-hoc comparisons with multiplicity correction are required to make specific group comparisons.
Chapter 12 — Back Matter
Chapter Summary
- Sec 12.1Multiple regression is needed when confounding, precision concerns, or the structure of the research question requires accounting for several predictors simultaneously. Omitting a confounder creates omitted variable bias.
- Sec 12.2The model Y = β₀ + β₁X₁ + ··· + βₚXₚ + ε. Least squares yields β̂ = (XᵀX)⁻¹Xᵀy. Assumptions extend simple regression with the addition of no perfect multicollinearity. The Gauss-Markov theorem applies.
- Sec 12.3Each partial slope β̂ⱼ estimates the change in mean Y per unit increase in Xⱼ, holding all other predictors fixed. Coefficients can change magnitude and sign when additional predictors are added. Individual t-tests use the t(n−p−1) distribution.
- Sec 12.4Categorical predictors with k categories require k − 1 indicator variables. One category is designated the reference. Each indicator coefficient measures the estimated mean difference between that category and the reference.
- Sec 12.5Binary predictors are the k = 2 special case. Additive models with indicator variables produce parallel regression lines — one per group — with the same slope but different intercepts.
- Sec 12.6Interactions between predictors are modeled by their product. The interaction coefficient measures how the slope of one predictor changes across values of the other. The hierarchy principle requires main effects whenever their interaction is included.
- Sec 12.7R² = SSR/SST measures the proportion of variation in Y explained by the model. It increases mechanically with every predictor added and cannot be used to compare models with different numbers of predictors.
- Sec 12.8Adjusted R² penalizes for model complexity and can decrease when uninformative predictors are added. It is a better basis for model comparison than R², though not a formal inferential criterion.
- Sec 12.9AIC and BIC are information-theoretic criteria; lower values indicate better models. Stepwise selection followed by standard inference is invalid. Best practice is to specify models before examining data.
- Sec 12.10Comparing k ≥ 3 group means with multiple t-tests inflates the Type I error rate. One-way ANOVA tests H₀: μ₁ = ··· = μₖ simultaneously while controlling the error rate at the nominal level.
- Sec 12.11SST = SSB + SSW. MSB and MSW are estimates of σ²: MSW is always valid; MSB is inflated when H₀ is false. The ANOVA table organizes the partition, degrees of freedom, mean squares, and the F-statistic.
- Sec 12.12The F-distribution is the distribution of a ratio of scaled chi-squared variables. The ANOVA F-test, the overall regression F-test, and the partial F-test all use upper-tail F critical values.
- Sec 12.13ANOVA is equivalent to regression with indicator variables — both are special cases of the general linear model. Post-hoc comparisons after ANOVA require multiple comparison corrections. Assumptions must be checked before inference is valid.
- Sec 12.14Ten common mistakes: omitting "holding others fixed"; causal language; missing reference category; using R² for model comparison; including all k indicators; no post-hoc correction; no assumption checks; conflating p-value with effect size; stepwise then standard inference; misreading the global ANOVA null.
Key Terms Review
multiple regression
A regression model relating a response variable to two or more predictor variables simultaneously.
design matrix
The n × (p+1) matrix X of predictor values with a column of ones for the intercept, used in matrix least squares.
partial regression coefficient
β̂ⱼ: the estimated change in mean Y per unit increase in Xⱼ, holding all other predictors fixed.
multicollinearity
High correlation among predictors, inflating standard errors and destabilizing coefficient estimates.
indicator variable
A binary (0/1) variable encoding group membership of a categorical predictor. Also called a dummy variable.
reference category
The category represented by setting all indicator variables to zero; the baseline for interpreting indicator coefficients.
interaction
A product of two predictors whose coefficient measures how the effect of one predictor depends on the value of the other.
hierarchy principle
The rule that main effects must be included whenever their interaction is included in the model.
R²
The proportion of total variation in Y explained by the regression model. Ranges from 0 to 1; increases with every predictor added.
adjusted R²
A modified R² that penalizes for the number of predictors; can decrease when an uninformative predictor is added.
AIC
Akaike Information Criterion. An information-theoretic model selection criterion targeting predictive accuracy; lower is better.
BIC
Bayesian Information Criterion. Similar to AIC but with a heavier penalty; targets model identification and tends to select simpler models.
one-way ANOVA
Analysis of variance with a single categorical factor; tests equality of means across k ≥ 2 groups.
SSB
Between-group Sum of Squares. Variability among group sample means around the grand mean; measures the between-group signal.
SSW
Within-group Sum of Squares. Total variability of observations within groups around their group means; measures unexplained variability.
mean square
A sum of squares divided by its degrees of freedom. MSB = SSB/(k−1); MSW = SSW/(n−k).
F-statistic
The ratio MSB/MSW (or MSR/MSE in regression); follows an F-distribution under H₀.
F-distribution
The distribution of a ratio of two scaled chi-squared random variables; indexed by numerator and denominator degrees of freedom.
overall regression F-test
Tests whether all regression slopes equal zero simultaneously; uses the F(p, n−p−1) distribution.
partial F-test
Tests whether a specific subset of q predictors collectively contributes to the model beyond the remaining predictors.
post-hoc comparisons
Pairwise group mean comparisons performed after a significant ANOVA F-test; require multiple comparison corrections.
family-wise error rate
The probability of at least one false rejection when conducting multiple hypothesis tests simultaneously.
Tukey's HSD
A post-hoc multiple comparison procedure designed to control the family-wise error rate for all pairwise comparisons after ANOVA.
Cook's distance
A measure of the overall influence of one observation on all coefficient estimates simultaneously.
variance inflation factor
A measure of how much the variance of a coefficient estimate is inflated due to multicollinearity. VIF > 5–10 signals concern.
ANCOVA
Analysis of covariance: a model combining ANOVA group comparisons with numerical covariates.
omitted variable bias
Bias in a regression coefficient caused by failing to include a relevant confounding variable in the model.
general linear model
The unified framework encompassing simple regression, multiple regression, ANOVA, and ANCOVA as special cases.
Practice Exercises
A researcher models systolic blood pressure (mmHg) on age (years), weekly exercise hours, and smoking status (1 = smoker, 0 = non-smoker):
Ŷ = 15.3 + 0.82·Age − 3.1·Exercise + 6.4·Smoking
- Interpret β̂₁ = 0.82 in context, including the phrase "holding all other predictors fixed."
- Interpret β̂₂ = −3.1 in context.
- Interpret β̂₃ = 6.4 in context. What is the reference category for the smoking variable?
- Estimate the mean blood pressure for a 45-year-old non-smoker who exercises 3 hours per week.
- A house price model gives Ŷ = 120.4 + 0.085·SqFt + 12.3·Bedrooms − 4.7·Distance. Estimate price for a 1,800 sq ft, 3-bedroom home 6 miles from downtown. Interpret each coefficient.
- Why can we not interpret the SqFt coefficient above as the simple regression slope of price on square footage?
A regression with 3 predictors, n = 60 gives:
| Predictor | β̂ | SE | t | p-value |
|---|
| Intercept | 42.1 | 5.3 | 7.94 | <0.001 |
| X₁ | 1.8 | 0.6 | 3.00 | 0.004 |
| X₂ | −0.4 | 0.9 | −0.44 | 0.659 |
| X₃ | 5.2 | 1.1 | 4.73 | <0.001 |
R² = 0.612, R²_adj = 0.592, Residual SE = 8.3
- Which predictors have statistically significant coefficients at α = 0.05?
- Construct a 95% confidence interval for β₁. Use t₀.₀₂₅,₅₆ ≈ 2.00.
- Interpret the residual SE = 8.3 in context.
- If X₂ is removed from the model, would R² increase, decrease, or stay the same? What about R²_adj? Explain.
- A partial F-test comparing full model (RSS = 1,102.3, df = 44) to reduced (RSS = 1,284.6, df = 47) yields F = ? Compute it and state the conclusion at α = 0.05 (p = 0.032).
Three teaching methods (Lecture, Video, Active Learning) are tested with 12 students each on a 20-point quiz:
| Method | n | Mean | SD |
|---|
| Lecture | 12 | 13.2 | 2.4 |
| Video | 12 | 14.8 | 2.1 |
| Active | 12 | 16.3 | 2.7 |
- Compute the grand mean.
- Compute SSB and SSW.
- Construct the one-way ANOVA table with all columns.
- Compute the F-statistic. Using F(2,33), assess significance at α = 0.05.
- State the conclusion in context. If H₀ is rejected, what additional analysis is needed?
- Complete the partial ANOVA table: Between SS = 284.6, Within SS = 619.4, df_total = 40. Find all missing values and compute F.
Four models for predicting recovery time (n = 80, SST = 5,200):
| Model | Predictors | p | RSS |
|---|
| M1 | Age | 1 | 4,210 |
| M2 | Age, Severity | 2 | 3,580 |
| M3 | Age, Severity, Treatment | 3 | 3,140 |
| M4 | Age, Severity, Treatment, BMI | 4 | 3,105 |
- Compute R² and R²_adj for each model.
- Compute the partial F-statistic for each consecutive pair. Assess significance at α = 0.05.
- Using R²_adj and partial F-tests, which model would you select? Justify.
- Why might AIC or BIC give a different recommendation than R²_adj?
A model of income (thousands) on education (years) with an interaction for gender:
Ŷ = 12.4 + 3.8·Education + 10.2·Male + 1.9·(Education × Male)
- Write out the estimated regression equation for women (Male = 0).
- Write out the estimated regression equation for men (Male = 1).
- Interpret the interaction coefficient 1.9 in context.
- Among individuals with 16 years of education, estimate the gender gap in income.
- Why does the hierarchy principle require that both Education and Male remain in this model even if their individual p-values were large?
Error Analysis
Each item contains an error. Identify and correct it.
Problem 1
"Because R² = 0.88, our multiple regression model is correct."
What is wrong with this statement? What does R² actually measure, and what does it not guarantee?
Problem 2
"The coefficient on years of experience is 2.3 and the coefficient on education is 1.7, so experience has a bigger effect on salary."
Identify the two errors in this reasoning. How should coefficients be compared across predictors?
Problem 3
"We included all four indicator variables for the four-category variable to be comprehensive."
What mathematical problem does this create? How many indicator variables should be included?
Problem 4
"The ANOVA p-value was 0.003, so Drug B is significantly better than Placebo."
What does the ANOVA F-test actually tell us? What analysis is needed to support the conclusion about Drug B?
Problem 5
"We ran stepwise regression and the final model has only significant predictors, so all the p-values are trustworthy."
Explain the post-selection inference problem. Why are p-values from a stepwise-selected model not valid at the nominal level?
Problem 6
Model includes X₁·X₂ interaction but removes X₁ main effect because "its p-value was not significant."
Which principle is violated? What should be done instead?
Chapter 12 Checkpoint
Chapter 12 Checkpoint
Complete without assistance — assesses all key skills and concepts from Chapter 12
Part A — Multiple Choice
- In a multiple regression model, β̂₂ is best interpreted as: (A) the correlation between X₂ and Y; (B) the expected change in Y per unit increase in X₂, ignoring all other predictors; (C) the expected change in Y per unit increase in X₂, holding all other predictors fixed; (D) the proportion of variation in Y explained by X₂.
- A researcher adds a completely irrelevant predictor (pure random noise) to a regression model. What will necessarily happen to R²? (A) Decrease; (B) Stay the same; (C) Increase; (D) Become undefined.
- A categorical variable has 5 categories. How many indicator variables should be included in the regression? (A) 5; (B) 4; (C) 3; (D) 6.
- The one-way ANOVA F-test with k = 4 groups and n = 40 total observations has degrees of freedom: (A) F(3, 36); (B) F(4, 36); (C) F(3, 37); (D) F(4, 40).
- A partial F-test compares: (A) two non-nested models; (B) a full model to a reduced model omitting a subset of predictors; (C) adjusted R² values for different models; (D) group means in ANOVA.
Part B — Short Answer
- Explain in one paragraph why adding more predictors always increases R² but does not always increase R²_adj.
- A model with 4 groups has SSB = 180, SSW = 540, n = 44. Construct the full ANOVA table. Compute the F-statistic. Using the critical value F₀.₀₅(3, 40) ≈ 2.84, state your conclusion.
- A model gives Ŷ = 22.4 + 3.1X₁ − 5.8X₂ + 2.4(X₁·X₂) where X₂ is binary (0 or 1). Write out the equation for each level of X₂ and interpret the interaction coefficient 2.4.
Part C — Applied Problem
- A study examines whether three diets (Low-Carb, Mediterranean, Low-Fat) produce different mean weight loss over 12 weeks. Each group has 15 participants (n = 45).
| Diet | Mean (kg) | SD (kg) |
|---|
| Low-Carb | 8.4 | 3.2 |
| Mediterranean | 6.7 | 2.9 |
| Low-Fat | 5.3 | 3.5 |
- State the null and alternative hypotheses for one-way ANOVA.
- Compute SSB and SSW. Construct the ANOVA table.
- Compute the F-statistic. Using F₀.₀₅(2, 42) ≈ 3.22, state your conclusion.
- If you reject H₀, what additional analysis is needed, and what correction must be applied?
- A colleague suggests adding baseline weight as a numerical covariate. What method does this create, and why might it increase precision?
Part D — Conceptual
- Explain in your own words why ANOVA and multiple regression with indicator variables are mathematically equivalent. What does this equivalence reveal about the general linear model?
- A researcher uses stepwise selection on 20 candidate predictors and reports that the final 6 predictors are all statistically significant at p < 0.05. Describe two specific problems with the subsequent inference and explain what should have been done instead.
Bridge to Chapter 13
Chapter 13: Stochastic Processes and Time-Dependent Models
Chapter 12 completes the inferential core of this textbook. The methods of regression and ANOVA assume that observations are independent — the error structure is static, not sequential. But many phenomena of scientific and practical importance unfold over time: stock prices, disease spread, queue lengths, weather systems, ecological populations. For these systems, the independence assumption is violated by construction.
Chapter 13 introduces stochastic processes: mathematical models for how randomness evolves dynamically. Key concepts will include Bernoulli processes, random walks, Markov chains, transition matrices, stationary distributions, and Poisson processes. The probability theory developed throughout this textbook — density functions, expected value, variance, distributions — provides the tools to analyze these time-dependent models. Students continuing to data science, time series analysis, machine learning, finance, reliability engineering, or epidemiology will find stochastic processes a natural and essential extension.
MGU Library Connection
- Regression and ANOVA Reference — Appendix P of this textbook
- Linear Algebra Readiness Review — Appendix C of this textbook (matrix notation for regression)
- Chapter 11: Linear Regression and Correlation — simple regression, least squares, residuals
- Chapter 9: Hypothesis Testing — two-sample t-tests, Type I error, power
- Linear Algebra, Chapter 3: Matrix Operations and Inverses — for the normal equations
- MGU Dictionary entries: partial regression coefficient, indicator variable, F-distribution, ANOVA, general linear model