The Central Idea
Simple linear regression uses one predictor:
\[ \hat{y}=b_0+b_1x. \]Multiple regression extends the fitted mean to several predictors:
\[ \boxed{ \hat{y} = b_0+b_1x_1+b_2x_2+\cdots+b_px_p }. \]The model combines the predictors' contributions to produce one fitted response for each case.
Why Use Several Predictors?
Many outcomes depend on more than one measured feature. House price may be associated with floor area, location, age, and condition. Business revenue may be associated with customer traffic, pricing, advertising, and market size.
Multiple regression can help us
- predict a response from several available measurements;
- compare cases at similar values of other included predictors;
- separate overlapping linear associations;
- evaluate whether one predictor adds information beyond others.
Response and Predictors
The model has one quantitative response \(y\) and \(p\) predictors:
\[ x_1,x_2,\ldots,x_p. \]A predictor may be quantitative, an indicator representing a category, or a constructed term such as a product or square. The way a predictor is encoded determines how its coefficient must be interpreted.
The Population Model
A population multiple linear regression model can be written
\[ \boxed{ Y = \beta_0+\beta_1x_1+\beta_2x_2+\cdots+\beta_px_p+\varepsilon }. \]The conditional mean is
\[ E(Y\mid X_1=x_1,\ldots,X_p=x_p) = \beta_0+\beta_1x_1+\cdots+\beta_px_p. \]The error \(\varepsilon\) represents variation not captured by the modeled conditional mean.
The Fitted Sample Model
Sample data estimate the population coefficients:
\[ \boxed{ \hat{y} = b_0+b_1x_1+b_2x_2+\cdots+b_px_p }. \]Each \(b_j\) is a fitted coefficient. The fitted model produces one \(\hat{y}_i\) for every observed predictor profile.
The Adjusted Interpretation
The most important interpretive phrase in multiple regression is holding the other included predictors fixed.
The coefficient \(b_1\) is the fitted change in \(y\) associated with a one-unit increase in \(x_1\), comparing cases with the same modeled values of \(x_2,\ldots,x_p\).
\[ \boxed{ \Delta\hat{y}=b_1 \quad\text{when }x_1\text{ increases by }1 \text{ and the other predictors remain fixed} }. \]Holding Fixed Is a Model Comparison
“Holding fixed” does not mean that the researcher physically freezes the other predictors. It describes a comparison made by the fitted equation.
If two cases differ by one unit in \(x_1\) but have the same values of all other included predictors, their fitted responses differ by \(b_1\).
Partial or Adjusted Associations
A multiple-regression coefficient describes a partial or adjusted linear association. It need not match the slope from a simple regression using that predictor alone.
The simple slope mixes the predictor's association with relationships involving omitted variables. The multiple-regression slope describes the association remaining after linear adjustment for the included predictors.
Why a Coefficient Can Change
Suppose larger homes tend to be newer, and both size and age are associated with price. A simple regression of price on size blends the size relationship with the fact that larger homes in the sample are newer.
Adding age can change the size coefficient because the model now compares homes of the same modeled age. A coefficient change is often substantive information, not a computational defect.
The Intercept
The intercept \(b_0\) is the fitted response when every predictor equals zero:
\[ \hat{y}=b_0 \quad\text{when}\quad x_1=x_2=\cdots=x_p=0. \]This value has a practical interpretation only if the all-zero predictor profile is meaningful and relevant to the data.
Centering Predictors
A quantitative predictor can be centered by subtracting a reference value, often its mean:
\[ x_j^{\,c}=x_j-\bar{x}_j. \]With all quantitative predictors centered at their sample means, the intercept represents the fitted response at the mean predictor profile. Centering changes the intercept but does not change fitted values in an additive model when performed consistently.
Fitted Values
For case \(i\), the fitted response is
\[ \hat{y}_i = b_0+b_1x_{i1}+b_2x_{i2}+\cdots+b_px_{ip}. \]Every predictor for that case contributes to the fitted value.
Residuals
The residual remains observed minus fitted:
\[ \boxed{e_i=y_i-\hat{y}_i}. \]A positive residual means the observed response is higher than the model predicted for that case's entire predictor profile. A negative residual means it is lower.
Least Squares with Several Predictors
Ordinary least squares selects all coefficients together to minimize
\[ \boxed{ \operatorname{SSE} = \sum_{i=1}^{n} \left[ y_i- \left( b_0+b_1x_{i1}+\cdots+b_px_{ip} \right) \right]^2 }. \]Each coefficient depends on the other predictors in the model because the complete set is chosen jointly.
A Two-Predictor Geometry
With two predictors, the fitted equation is
\[ \hat{y}=b_0+b_1x_1+b_2x_2. \]This equation describes a plane in three-dimensional \((x_1,x_2,y)\)-space. With more predictors, the fitted object is a hyperplane in a higher-dimensional space.
An Introductory Worked Example
Suppose \(y\) is a task-performance score. Predictor \(x_1\) is training hours centered at the sample mean, and \(x_2\) is prior experience centered at its sample mean. The small dataset is constructed to make the calculations transparent.
| Case | Centered training \(x_1\) | Centered experience \(x_2\) | Observed score \(y\) |
|---|---|---|---|
| A | \(-2\) | 1 | 70 |
| B | \(-1\) | \(-2\) | 61 |
| C | 0 | 1 | 73 |
| D | 0 | \(-1\) | 67 |
| E | 1 | 2 | 77 |
| F | 2 | \(-1\) | 72 |
Example: Predictor Structure
Both predictors have mean zero:
\[ \bar{x}_1=0, \qquad \bar{x}_2=0. \]In this constructed dataset, their cross-product sum is also zero:
\[ \sum_{i=1}^{6}x_{i1}x_{i2}=0. \]This orthogonality makes the two slope calculations separate cleanly. Real predictors are often correlated, so software solves the coefficients jointly.
Example: Calculate the Intercept
Because both predictors are centered and the model includes an intercept, the fitted intercept equals the response mean:
\[ b_0=\bar{y}=70. \]It represents the fitted task score for a case at the sample mean of both training and experience.
Example: Calculate the Training Coefficient
Orthogonality allows
\[ b_1= \frac{\sum x_{i1}(y_i-\bar{y})} {\sum x_{i1}^2}. \]For the data,
\[ \sum x_{i1}(y_i-\bar{y})=20, \qquad \sum x_{i1}^2=10. \]Therefore,
\[ b_1=\frac{20}{10}=2. \]Example: Calculate the Experience Coefficient
Similarly,
\[ b_2= \frac{\sum x_{i2}(y_i-\bar{y})} {\sum x_{i2}^2}. \]Here,
\[ \sum x_{i2}(y_i-\bar{y})=36, \qquad \sum x_{i2}^2=12. \]Thus,
\[ b_2=\frac{36}{12}=3. \]Example: The Fitted Equation
The fitted model is
\[ \boxed{\hat{y}=70+2x_1+3x_2}. \]- Holding centered experience fixed, one additional training hour is associated with a \(2\)-point increase in fitted performance.
- Holding centered training fixed, one additional year-equivalent unit of experience is associated with a \(3\)-point increase in fitted performance.
These are adjusted associations within the modeled data, not automatic causal effects.
Example: A Fitted Value
Case E has \(x_1=1\) and \(x_2=2\). Its fitted score is
\[ \hat{y} = 70+2(1)+3(2) = 78. \]Its observed score is \(77\), so its residual is
\[ e=77-78=-1. \]Example: Fitted Values and Residuals
| Case | Observed \(y_i\) | Fitted \(\hat{y}_i\) | Residual \(e_i\) | \(e_i^2\) |
|---|---|---|---|---|
| A | 70 | 69 | 1 | 1 |
| B | 61 | 62 | \(-1\) | 1 |
| C | 73 | 73 | 0 | 0 |
| D | 67 | 67 | 0 | 0 |
| E | 77 | 78 | \(-1\) | 1 |
| F | 72 | 71 | 1 | 1 |
| Sums | 0 | 4 | ||
The residual sum of squares is
\[ \operatorname{SSE}=4. \]Residual Properties
With an intercept, ordinary least-squares residuals satisfy
\[ \sum e_i=0. \]They are also orthogonal to each included predictor column:
\[ \sum x_{ij}e_i=0 \qquad \text{for each fitted predictor }j. \]These are algebraic properties of the fit. They do not guarantee that the model is substantively correct.
Total Variation and \(R^2\)
The total response variation is
\[ \operatorname{SST} = \sum(y_i-\bar{y})^2. \]For the example,
\[ \operatorname{SST}=152. \]Therefore,
\[ R^2 = 1-\frac{\operatorname{SSE}}{\operatorname{SST}} = 1-\frac{4}{152} \approx 0.974. \]Interpreting Multiple \(R^2\)
In the constructed sample, the complete two-predictor model accounts for about \(97.4\%\) of the observed variation in task score around its mean.
This unusually high value reflects a deliberately transparent teaching example. It is not a benchmark that real models should be expected to reach.
Adding Predictors Cannot Lower Training \(R^2\)
In ordinary least squares, adding a predictor gives the model another option. It can always assign that predictor a coefficient of zero. Therefore, on the same training data,
\[ R^2_{\text{larger model}} \ge R^2_{\text{smaller model}}. \]This mechanical property means that a higher training \(R^2\) alone does not prove that the larger model is better.
Adjusted \(R^2\)
Adjusted \(R^2\) penalizes model size:
\[ \boxed{ R_{\mathrm{adj}}^2 = 1- \frac{\operatorname{SSE}/(n-p-1)} {\operatorname{SST}/(n-1)} }. \]Unlike ordinary \(R^2\), adjusted \(R^2\) can decrease when an unhelpful predictor is added.
Example: Adjusted \(R^2\)
The example has \(n=6\) cases and \(p=2\) predictors:
\[ R_{\mathrm{adj}}^2 = 1- \frac{4/(6-2-1)} {152/(6-1)} \approx 0.956. \]The adjustment is larger when the sample is small relative to the number of fitted coefficients.
Residual Standard Error
A common estimate of the typical residual scale is
\[ \boxed{ s = \sqrt{ \frac{\operatorname{SSE}}{n-p-1} } }. \]For the example,
\[ s=\sqrt{\frac{4}{3}}\approx 1.155 \]response points. This quantity uses the residual degrees of freedom \(n-p-1\).
Comparing Nested Models
A smaller model is nested within a larger model when the smaller model can be obtained by setting one or more coefficients in the larger model to zero.
Model comparisons can examine the reduction in SSE, changes in adjusted fit, predictive validation, and inferential tests. No single number should replace the model's purpose and diagnostic evidence.
Multicollinearity
Multicollinearity occurs when predictors contain overlapping linear information. If two predictors move together closely, the model may struggle to separate their individual adjusted relationships.
Consequences can include unstable coefficients, large standard errors, sensitivity to small data changes, and signs that are difficult to interpret.
Perfect Collinearity
Perfect collinearity occurs when one predictor is an exact linear combination of others. For example, including temperature in Celsius, temperature in Fahrenheit, and an intercept creates exact redundancy.
The ordinary coefficient solution is then not unique. One redundant term must be removed or the model must be reformulated.
Prediction Can Be Stable While Coefficients Are Not
With strongly correlated predictors, several coefficient combinations may produce similar fitted values in the observed data. Prediction within that data region may remain acceptable even while individual coefficient interpretations are unstable.
Prediction and explanation are related goals, but they do not always favor the same model.
Marginal and Adjusted Coefficients Can Differ
A predictor may have a positive simple-regression slope and a small, zero, or negative adjusted coefficient after other predictors are included. Such a reversal can arise from confounding, suppression, subgroup structure, or collinearity.
Always state which predictors are included when interpreting a coefficient.
Categorical Predictors
Categories can enter a regression through indicator variables. For a two-level category, one common encoding uses
\[ D= \begin{cases} 1,&\text{case belongs to the designated group},\\ 0,&\text{case belongs to the reference group}. \end{cases} \]Its coefficient compares fitted responses between the groups while the other included predictors are fixed. A later lesson develops categorical predictors in detail.
Additive Models
The basic model
\[ \hat{y}=b_0+b_1x_1+b_2x_2 \]is additive. The contribution \(b_1x_1\) does not depend on \(x_2\), and the contribution \(b_2x_2\) does not depend on \(x_1\).
Additivity is a substantive assumption, not merely a convenient notation.
Interactions
An interaction allows the association between one predictor and the response to depend on another predictor:
\[ \hat{y} = b_0+b_1x_1+b_2x_2+b_3x_1x_2. \]With the product term present, the fitted change associated with one unit of \(x_1\) is
\[ b_1+b_3x_2. \]Main-effect coefficients must then be interpreted at the reference value of the interacting predictor.
Nonlinear Terms
A multiple regression can include constructed terms such as \(x^2\):
\[ \hat{y}=b_0+b_1x+b_2x^2. \]The graph is curved in \(x\), but the equation is linear in the unknown coefficients \(b_0,b_1,b_2\). The coefficients must be interpreted as a group because the fitted change in \(y\) is no longer constant across \(x\).
Linearity in the Parameters
“Linear regression” means that the model is linear in its coefficients. Predictors may be transformed or combined, provided the coefficients enter additively:
\[ b_0+b_1g_1(x)+b_2g_2(x). \]Whether such a form is appropriate remains an empirical and substantive question.
Residual Diagnostics Still Matter
More predictors do not remove the need to inspect residuals. A model can have a high \(R^2\) and still show curvature, unequal variance, dependence, clusters, or influential observations.
Diagnostic plots should be examined against fitted values and relevant predictors.
Linearity and Additivity
The modeled conditional mean should be reasonably represented by the chosen combination of predictor terms. Systematic residual patterns may indicate missing curvature, interactions, group effects, or transformations.
Independent Observations
Standard regression inference commonly assumes independent errors across cases. Repeated measures, time series, geographic clusters, classrooms, firms within industries, or family relationships can create dependence.
Independence comes mainly from how data were generated and sampled.
Constant Conditional Variance
Conventional least-squares uncertainty formulas often assume approximately constant error variance across predictor profiles. A widening residual spread can make standard errors and intervals unreliable unless the analysis adapts.
Normality for Small-Sample Inference
Normal error assumptions support familiar small-sample tests and intervals. They are not required merely to calculate least-squares coefficients.
The appropriate inferential method depends on sample size, design, error structure, and the severity of departures.
Outliers, Leverage, and Influence
In multiple regression, leverage depends on the complete predictor profile. A case may have ordinary values on each predictor separately but an unusual combination of those values.
An influential case materially changes fitted coefficients, predictions, or conclusions. Such cases require verification and contextual investigation.
Sample Size and Model Complexity
A model with \(p\) predictors and an intercept estimates \(p+1\) coefficients. If the model is too complex for the available information, it may fit accidental patterns and perform poorly on new cases.
The residual degrees of freedom are
\[ n-p-1. \]Adequacy depends on more than a fixed cases-per-predictor rule; effect sizes, noise, collinearity, missingness, validation, and inferential goals all matter.
Overfitting
Overfitting occurs when a model learns noise or sample-specific detail rather than a stable relationship. It can produce excellent training fit and weak performance on new data.
Simpler specifications, subject-matter constraints, penalization, and validation data are common safeguards. Later lessons examine overfitting and validation directly.
Prediction at New Predictor Profiles
For a new case with predictors \((x_1^\ast,\ldots,x_p^\ast)\), the point prediction is
\[ \hat{y}^{\,\ast} = b_0+b_1x_1^\ast+\cdots+b_px_p^\ast. \]Prediction is most defensible when the new profile resembles profiles used to fit the model.
Multivariable Extrapolation
A new case can be unusual because one predictor is outside its observed range or because its combination of predictor values was not represented in the training data.
Checking each range separately is necessary but not always sufficient. Multiple regression requires attention to the joint predictor region.
Prediction Is Not Causation
A predictor can improve forecasts without being a cause of the response. It may act as a proxy, share a common cause, or simply precede the outcome.
Predictive usefulness and causal identification are distinct objectives.
Adjustment Does Not Automatically Remove Confounding
Adding variables does not automatically create a causal estimate. The model may omit important confounders, adjust for variables that should not be controlled, use poor measurements, impose the wrong form, or suffer from selection bias.
Causal adjustment requires a defensible causal structure and appropriate data, not merely a long predictor list.
A Matrix Preview
Multiple regression can be written compactly as
\[ \mathbf{y}=\mathbf{X}\boldsymbol{\beta}+\boldsymbol{\varepsilon}. \]When the design matrix \(\mathbf{X}\) has full column rank, the least-squares estimate is
\[ \boxed{ \hat{\boldsymbol{\beta}} = (\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} \mathbf{X}^{\mathsf T}\mathbf{y} }. \]This matrix expression is the higher-dimensional version of choosing coefficients that minimize SSE.
A Reliable Modeling Workflow
- Define the cases, response, predictors, units, population, and purpose.
- Inspect data quality, missingness, distributions, and pairwise displays.
- Choose predictor forms using subject knowledge and the analytical goal.
- Fit the model and record the complete specification.
- Interpret every coefficient conditionally on the other included terms.
- Examine residuals, leverage, influence, and collinearity.
- Compare fit with model complexity and predictive validation.
- Limit predictions to supported predictor profiles.
- Quantify uncertainty using methods appropriate to the design.
- Separate associational, predictive, and causal claims.
Reporting a Multiple Regression
A useful report identifies the response, predictors, units, sample, coefficient meanings, fit measures, uncertainty, diagnostic findings, and limitations.
Coefficients should never be reported as context-free numbers. State the reference categories, transformations, interactions, and variables held fixed.
Common Error: Forgetting “Holding Fixed”
Interpreting \(b_1\) as the overall relationship between \(x_1\) and \(y\) ignores the other predictors. It is the adjusted relationship within the stated model.
Common Error: Comparing Raw Coefficient Sizes
A coefficient measured per dollar cannot be ranked directly against one measured per year. Raw coefficient magnitude depends on units and variable scaling.
Common Error: Treating a Sign Change as Impossible
A coefficient can change sign when correlated predictors are added. The simple and adjusted coefficients answer different comparison questions. The change should be investigated, not automatically rejected.
Common Error: Selecting by \(R^2\) Alone
Training \(R^2\) cannot decrease when predictors are added. Choosing the largest possible model by \(R^2\) rewards complexity even when it does not generalize.
Common Error: Ignoring Collinearity
Strongly overlapping predictors can produce unstable adjusted coefficients. A high overall \(R^2\) does not make each coefficient precise or separately interpretable.
Common Error: Assuming Additivity
If the relationship between \(x_1\) and \(y\) changes across values of \(x_2\), a model without an interaction can give an incomplete average description.
Common Error: Interpreting Adjustment as Causal Control
Statistical adjustment is not automatically valid causal adjustment. Variable choice must follow a defensible causal question and design.
Practice 1: Write the Fitted Model
A model predicts weekly sales \(y\) from store area \(x_1\) and customer traffic \(x_2\). The fitted coefficients are \(b_0=12\), \(b_1=1.5\), and \(b_2=0.08\). Write the equation.
Practice 2: Interpret a Coefficient
In the model from Practice 1, store area is measured in thousands of square feet and sales in thousands of dollars. Interpret \(b_1=1.5\).
Practice 3: Calculate a Prediction
Use
\[ \hat{y}=12+1.5x_1+0.08x_2 \]to predict weekly sales when \(x_1=10\) and \(x_2=200\).
Practice 4: Calculate a Residual
If the observed sales for the case in Practice 3 are \(40\) thousand dollars, find the residual.
Practice 5: Interpret an Indicator
A model includes \(D=1\) for urban stores and \(D=0\) for rural stores. Its coefficient is \(b_D=4.2\). Interpret the coefficient.
Practice 6: Calculate Adjusted \(R^2\)
A model has \(n=20\), \(p=3\), \(\operatorname{SSE}=80\), and \(\operatorname{SST}=200\). Find adjusted \(R^2\).
Practice 7: Identify Collinearity
A model includes distance in miles and the same distance in kilometers. What problem arises?
Practice 8: Interpret an Interaction
In
\[ \hat{y}=5+2x_1+3x_2+0.4x_1x_2, \]what is the fitted change in \(y\) for a one-unit increase in \(x_1\) when \(x_2=5\)?
Practice 9: Evaluate a Coefficient Change
The simple slope for \(x_1\) is positive, but its multiple-regression coefficient becomes negative after \(x_2\) is added. Is this automatically an error? What should be investigated?
Practice 10: Critique a Causal Claim
An analyst adds twenty available predictors to an observational model and says the coefficient of \(x_1\) must now be causal because “everything was controlled.” Identify the problem.
Practice Answers
Practice 1
\[ \boxed{\hat{y}=12+1.5x_1+0.08x_2}. \]Practice 2
Holding customer traffic fixed, each additional thousand square feet of store area is associated with \(1.5\) thousand dollars in higher fitted weekly sales, on average within the modeled data.
Practice 3
\[ \hat{y} = 12+1.5(10)+0.08(200) = 12+15+16 = 43. \]Predicted weekly sales are \(43\) thousand dollars.
Practice 4
\[ e=y-\hat{y}=40-43=-3. \]Observed sales are \(3\) thousand dollars below the fitted value.
Practice 5
Holding the other included predictors fixed, urban stores have fitted responses \(4.2\) response units higher than rural stores, the reference group.
Practice 6
\[ R_{\mathrm{adj}}^2 = 1- \frac{80/(20-3-1)}{200/(20-1)} = 1-\frac{5}{200/19} = 0.525. \]Practice 7
The predictors are perfectly collinear because kilometers are a constant multiple of miles. The individual coefficient solution is not unique; one redundant predictor should be removed.
Practice 8
The fitted change associated with one unit of \(x_1\) is
\[ b_1+b_3x_2=2+0.4(5)=4. \]Practice 9
No. The simple and adjusted coefficients answer different questions. Investigate the relationship between the predictors, confounding or suppression, subgroup patterns, multicollinearity, influential cases, and whether the model form is appropriate.
Practice 10
A long predictor list does not guarantee valid causal adjustment. Some confounders may be missing or poorly measured, and some included variables may be mediators, colliders, consequences of the outcome, or otherwise inappropriate controls. Causal interpretation requires a justified causal structure and design.
What to Remember
The fitted multiple-regression model is
\[ \boxed{ \hat{y}=b_0+b_1x_1+\cdots+b_px_p }. \]- Each slope describes a fitted change while the other included predictors are held fixed.
- All coefficients are estimated jointly by minimizing SSE.
- Residuals are \(e_i=y_i-\hat{y}_i\).
- Training \(R^2\) cannot decrease when predictors are added.
- Adjusted \(R^2\) accounts partly for model size.
- Collinearity can destabilize individual coefficients.
- Interactions allow one predictor's association to depend on another.
- Prediction, association, and causation are distinct objectives.
Where We Go Next
Fitting coefficients is only the beginning of regression analysis. A model must also be checked for systematic errors, unusual cases, unstable variance, and observations that exert disproportionate influence.
The next lesson, Residuals and Model Diagnostics, develops the graphical and numerical tools used to determine whether a fitted model is behaving as intended.