Purpose and Overview

Learning Objectives

1. 1Explain the statistical model for simple linear regression and distinguish it from a deterministic linear equation.
2. 2Construct and interpret scatter plots; describe direction, form, strength, and outliers in bivariate data.
3. 3Compute and interpret the Pearson correlation coefficient; explain its properties and limitations.
4. 4Apply the least squares criterion to derive formulas for the regression slope and intercept.
5. 5Use calculus to derive least squares estimates by minimizing the residual sum of squares.
6. 6Interpret the slope and intercept of a fitted regression line in context; use the line for prediction within the observed range.
7. 7Compute and analyze residuals; construct residual plots; interpret R² as a measure of explained variability.
8. 8State and check the four regression assumptions; describe consequences of violations.
9. 9Compute the standard error of the regression and the standard error of the slope.
10. 10Perform hypothesis tests and construct confidence intervals for the slope.
11. 11Distinguish confidence intervals for the mean response from prediction intervals for a new observation.
12. 12Explain why correlation does not imply causation; identify confounders, reverse causation, and spurious associations.

Sections at a Glance

• 11.1Modeling Relationships with LinesThe simple linear regression model; deterministic vs. statistical lines; population parameters vs. sample estimates; when linear models are appropriate
• 11.2Scatter Plots and AssociationPlotting bivariate data; direction, form, strength, outliers; choosing axes; heteroscedasticity; when regression is appropriate
• 11.3CorrelationPearson correlation coefficient; formula and computation; properties; interpreting magnitude; population correlation ρ
• 11.4Least Squares CriterionFitted values and residuals; residual sum of squares; why squared residuals; least squares formulas; connection to r and standard deviations
• 11.5Deriving Least Squares as an IntroductionCalculus derivation via partial derivatives; normal equations; minimum confirmed; connection to covariance; matrix notation preview
• 11.6Regression Line InterpretationSlope as rate of change; intercept meaning and limitations; prediction within range; extrapolation; predictions as averages
• 11.7ResidualsResidual properties; SST = SSR + RSS decomposition; R²; residual plots; detecting curvature, heteroscedasticity, outliers
• 11.8Regression AssumptionsLinearity, independence, homoscedasticity, normality; how to check each; consequences of violation; fixed-x assumption
• 11.9Standard Error of the RegressionEstimating σ²; residual mean square; standard error s; standard error of the slope; what increases precision
• 11.10Inference for the Slopet-distribution for slope estimate; hypothesis test H₀: β₁ = 0; p-value; confidence interval for β₁; connection to F-test
• 11.11Confidence and Prediction Intervals as an IntroductionTwo distinct questions at x*; confidence interval for mean response; prediction interval for new observation; formulas; comparison
• 11.12Correlation Is Not CausationConfounding variables; reverse causation; spurious correlation; selection bias; what causation requires; description vs. explanation
• 11.13Common Regression MistakesEight recurring errors; ignoring scatter plots, extrapolating, misreading R², assuming causation, influential observations
• 11.14Preparing for Multiple Regression and ANOVAMultiple predictors; adjusted coefficients; categorical predictors; ANOVA as regression; unified general linear model framework

Modeling Relationships with Lines

Scatter Plots and Association

Correlation

Least Squares Criterion

Deriving Least Squares as an Introduction

Regression Line Interpretation

Residuals

Reading Residual Plots

Regression Assumptions

• Linearity E[y | x] = β₀ + β₁x. The true relationship between x and y is linear. Check with the scatter plot and residual plot; systematic curvature indicates a violation. If linearity fails, the fitted line describes the wrong relationship and predictions will be systematically biased.
• Independence ε₁, ε₂, …, εₙ are independent. Knowing the error for one observation gives no information about another. Check by examining how data were collected — random sampling supports independence; time series or clustered data may not. Violation produces incorrect standard errors and p-values.
• Constant Variance Var(εᵢ) = σ² for all i. Homoscedasticity: the spread of y around the regression line is the same at every value of x. Check with the residual plot for a fan shape. Violation means OLS is no longer the most efficient estimator.
• Normality εᵢ ~ N(0, σ²). The errors follow a normal distribution. Check with a Q-Q plot of residuals; points should fall approximately on a straight line. Normality supports the t-tests and F-tests used in inference. For large samples, the central limit theorem provides some protection against non-normality.

Standard Error of the Regression

Inference for the Slope

Confidence and Prediction Intervals as an Introduction

Correlation Is Not Causation

Common Regression Mistakes

Preparing for Multiple Regression and ANOVA

Chapter Summary

• Sec 11.1The simple linear regression model y = β₀ + β₁x + ε adds a random error term to account for variability not explained by the linear relationship. Population parameters β₀ and β₁ are unknown and estimated from data as β̂₀ and β̂₁.
• Sec 11.2Scatter plots are the foundational visualization for bivariate data. We look for direction, form, strength, and outliers. A linear pattern with roughly constant spread around the trend supports a linear regression model.
• Sec 11.3The Pearson correlation coefficient r measures the strength and direction of a linear association; −1 ≤ r ≤ 1. It is unit-free and symmetric. It measures only linear association — a strong curved relationship can have r near zero.
• Sec 11.4The least squares criterion minimizes RSS = Σeᵢ². The formulas are β̂₁ = Σ(xᵢ−x̄)(yᵢ−ȳ)/Σ(xᵢ−x̄)² and β̂₀ = ȳ − β̂₁x̄. The line always passes through (x̄, ȳ) and satisfies β̂₁ = r · sᵧ/sₓ.
• Sec 11.5Least squares estimates are derived by setting partial derivatives of Q(β₀, β₁) to zero. This gives the normal equations, whose solution yields the slope and intercept. The matrix formula β̂ = (XᵀX)⁻¹Xᵀy generalizes to multiple regression.
• Sec 11.6The slope β̂₁ is the predicted change in y per one-unit increase in x, on average. The intercept β̂₀ is the predicted y when x = 0, which may not be meaningful. Predictions should remain within the observed range of x.
• Sec 11.7Residuals eᵢ = yᵢ − ŷᵢ sum to zero for OLS. The decomposition SST = SSR + RSS gives R² = SSR/SST, the proportion of variability in y explained by the regression. Residual plots reveal assumption violations that R² alone cannot detect.
• Sec 11.8The four regression assumptions are linearity, independence, constant variance (homoscedasticity), and normality of errors. Violations have specific consequences: failed linearity biases predictions; failed independence corrupts inference; failed homoscedasticity reduces efficiency; failed normality affects small-sample tests.
• Sec 11.9The residual standard error s = √(RSS/(n−2)) estimates σ, the standard deviation of errors. The standard error of the slope SE(β̂₁) = s/√Σ(xᵢ−x̄)² measures slope estimation precision. It decreases with better fit, wider x range, and larger n.
• Sec 11.10Under regression assumptions, (β̂₁−β₁)/SE(β̂₁) ~ t_{n−2}. The hypothesis test H₀: β₁ = 0 uses t = β̂₁/SE(β̂₁). A confidence interval is β̂₁ ± t*_{n−2}·SE(β̂₁). A non-significant result means the data are consistent with β₁ = 0, not that β₁ is zero.
• Sec 11.11At x*, a confidence interval for the mean response quantifies uncertainty about E[y|x*]. A prediction interval for a new observation also incorporates individual variability (σ²) and is always wider. Both intervals are widest when x* is far from x̄.
• Sec 11.12Correlation does not imply causation. Confounders, reverse causation, spurious correlation, and selection bias can produce associations without causal relationships. Causation requires randomized experiments or explicit causal assumptions. Regression describes associations; it does not establish causes.
• Sec 11.13Eight recurring errors: ignoring the scatter plot, extrapolating, misreading R², treating the line as fixed, confusing non-significance with no relationship, assuming causation, ignoring influential observations, and confusing the two regression lines.
• Sec 11.14Multiple regression adds predictors; each slope is interpreted holding others fixed. ANOVA compares group means using the same F-test. Both are faces of the general linear model. All Chapter 11 ideas — assumptions, decomposition, residual analysis, causation caveats — carry forward to Chapter 12.

Key Terms Review

Scatter Plot Practice

1. Describe in words what the scatter plot would look like if two variables had (a) a strong positive linear association; (b) a weak negative linear association; (c) a moderate positive association with one clear outlier; (d) no linear association but a clear curved relationship.
2. For each scenario, identify which variable you would place on the horizontal axis and which on the vertical axis, and explain your reasoning. (a) Studying whether advertising spending predicts sales revenue. (b) Studying whether birth weight is associated with gestational age. (c) Studying whether altitude is associated with average temperature.
3. A researcher plots mean temperature (horizontal) against annual ice cream consumption per capita (vertical) across 30 countries and finds a positive, moderate association. (a) Describe the association. (b) Should the researcher conclude warmer temperatures cause more ice cream consumption? Explain. (c) Name one plausible confounding variable.
4. A scatter plot of hours of sleep and reaction time shows a negative association with moderate spread. (a) What direction of association does this suggest? Interpret in context. (b) What additional features of the scatter plot would you examine before fitting a linear model?

Correlation Practice

1. Compute the correlation coefficient for the data: (1, 5), (3, 9), (5, 11), (7, 15), (9, 17). Show all steps.
2. For each value of r, describe the strength and direction of the association: (a) r = 0.95; (b) r = −0.42; (c) r = 0.08; (d) r = −0.87.
3. Two variables have r = 0.70. (a) What is R² for the regression of y on x? (b) If the units of x are changed from inches to centimeters (multiplied by 2.54), what happens to r? (c) If y is replaced by y' = 2y + 10, what happens to r?
4. Summary statistics: x̄ = 20, ȳ = 50, sₓ = 4, sᵧ = 10, sₓᵧ = 28 (where sₓᵧ is the sample covariance). (a) Compute r. (b) Compute β̂₁. (c) Compute β̂₀.
5. A scatter plot shows a clear U-shaped relationship between x and y. Explain why r might be close to zero even though an obvious relationship exists in the data.

Least Squares Practice

Use the following data for all parts.

1. Compute x̄, ȳ, Σ(xᵢ−x̄)(yᵢ−ȳ), and Σ(xᵢ−x̄)².
2. Compute β̂₁ and β̂₀. Write the equation of the fitted regression line.
3. Predict y when x = 7. Is this a legitimate prediction? Explain.
4. A student claims: "The regression line always passes through the origin if the data is centered." Is this correct? Under what condition does the line pass through the origin?

For a data set with n = 12: x̄ = 8, ȳ = 35, Σ(xᵢ−x̄)² = 180, Σ(xᵢ−x̄)(yᵢ−ȳ) = −270.

1. Compute β̂₁ and β̂₀.
2. Interpret the slope in context, given that x is weekly screen time (hours) and y is a reading proficiency score.
3. What does the sign of β̂₁ indicate about the direction of the association?

Residual Analysis

1. For the data in Problem LS-1: compute fitted values ŷᵢ and residuals eᵢ. Verify Σeᵢ = 0. Compute RSS, s², and s.
2. A residual plot for a regression of car price (y) on age (x) shows a fan-shaped pattern — small spread for low ŷ, large spread for high ŷ. (a) Which regression assumption does this violate? (b) What might a researcher do to address it?
3. A residual plot shows a curved arc (positive residuals at low ŷ, negative in the middle, positive at high ŷ). (a) What does this indicate? (b) Does a high R² necessarily mean a linear model is appropriate here?
4. True or False. Justify each answer. (a) A residual plot with random scatter confirms the model is correct. (b) Residuals always sum to zero. (c) A large residual always means an influential observation. (d) A residual of zero means the model fits that observation perfectly.

Inference Problems

1. A regression of exam score on study hours (n = 25) gives β̂₁ = 4.5, SE(β̂₁) = 1.8. (a) Compute the t-statistic for H₀: β₁ = 0. (b) State the degrees of freedom. (c) Do you expect the p-value to be significant at α = 0.05? Explain. (d) Construct a 95% CI using t*₂₃ ≈ 2.069.
2. A researcher reports: β̂₁ = 1.2 mmHg/year with 95% CI (0.8, 1.6) for a regression of blood pressure on age. (a) Interpret the CI in context. (b) Does this provide evidence of a positive association? (c) Without computing, what can you conclude from the CI about the test of H₀: β₁ = 0?
3. For n = 18, β̂₁ = −2.3, s = 5.6, Σ(xᵢ−x̄)² = 320: (a) Compute SE(β̂₁). (b) Construct a 90% CI using t*₁₆ ≈ 1.746. (c) Interpret in context with x = temperature (°C) and y = number of defective units.
4. A 95% CI for the slope is (−3.5, 0.4). (a) Can we reject H₀: β₁ = 0 at α = 0.05? Explain. (b) Can we reject at α = 0.10? What additional information would you need?
5. Explain in your own words the difference between a confidence interval for the mean response and a prediction interval for a new observation. Give a practical example where each is more appropriate.

Applications

1. (Biology) A biologist measures length (x, cm) and weight (y, g) of n = 30 fish. The regression line is ŷ = −22.4 + 5.8x, with R² = 0.88 and s = 8.3 g. (a) Interpret the slope. (b) Interpret R². (c) Predict weight for a fish of length 18 cm. (d) Should we predict for a fish of length 55 cm? Why or why not?
2. (Economics) An economist regresses income (y, thousands of dollars) on education (x, years) for n = 150 adults: β̂₀ = 12.4, β̂₁ = 3.2, SE(β̂₁) = 0.7, R² = 0.41. (a) Interpret the slope. (b) Test H₀: β₁ = 0 at α = 0.01. (c) Evaluate the claim: "More education causes higher income."
3. (Engineering) A quality engineer studies curing temperature (x, °C) and tensile strength (y, MPa) for n = 20 specimens: β̂₁ = 0.45, RSS = 120, Σ(xᵢ−x̄)² = 500. (a) Compute s and SE(β̂₁). (b) Construct a 95% CI using t*₁₈ ≈ 2.101. (c) The engineer says temperature "explains" tensile strength. Critique this statement.
4. (Finance) An analyst regresses monthly stock returns (y, %) on market index returns (x, %) for n = 60 months (the market model; the slope is the stock's beta). (a) If β̂₁ = 1.4, interpret this value. What does it suggest about how the stock moves relative to the market? (b) If β̂₁ = 0.6, what would that suggest? (c) If r = 0.55, what percentage of the stock's return variability is explained by the market?
5. (Public Health) A researcher regresses life expectancy (y, years) on per capita GDP (x, thousands of dollars) across 50 countries. The scatter plot shows a positive association with noticeable curvature: steep for low GDP, flattening for high GDP. (a) Why might a linear model be inadequate? (b) What transformation of x might linearize the relationship? (c) If the researcher fits a linear model anyway, what would the residual plot likely reveal?

Error Analysis

Each problem shows a student's error. Identify it and write the correct statement or calculation.

Chapter 11 Checkpoint