The Central Idea
For observation \(i\), the residual is
\[ \boxed{e_i=y_i-\hat{y}_i}. \]A well-behaved model leaves residuals that look like unstructured variation around zero. Visible structure in the residuals is evidence that the model, uncertainty calculation, or data-generating assumptions need attention.
\[ \boxed{ \text{model structure removed} \quad\Longrightarrow\quad \text{residuals show no systematic structure} }. \]Observed, Fitted, and Residual Values
Every modeled case has three connected quantities:
\[ y_i=\hat{y}_i+e_i. \]- \(y_i\) is the observed response.
- \(\hat{y}_i\) is the response fitted by the model.
- \(e_i\) is the remaining vertical difference.
Diagnostics study the residuals in relation to fitted values, predictors, observation order, and leverage.
The Sign of a Residual
- If \(e_i>0\), the model underpredicts that observation.
- If \(e_i<0\), the model overpredicts that observation.
- If \(e_i=0\), the observed point lies exactly on the fitted value.
A residual's sign describes one case. A diagnostic conclusion usually depends on the pattern across many cases.
Residuals Are Estimates of Errors
In a population model,
\[ Y_i=E(Y_i\mid\mathbf{X}_i)+\varepsilon_i, \]the error \(\varepsilon_i\) is the unobserved deviation from the true conditional mean. The residual \(e_i\) is calculated after fitting a sample model and is therefore observable.
Residuals estimate aspects of the errors, but they are not identical to the unobserved population errors.
Why Fit Statistics Are Not Enough
Two models can have similar \(R^2\) and SSE values while behaving very differently. One may have random residual scatter. Another may contain a strong curve, a funnel-shaped spread, or one point that determines the fitted slope.
A model should therefore be evaluated through both numerical summaries and graphical diagnostics.
What Diagnostics Can Reveal
Diagnostics can identify evidence of
- incorrect functional form;
- nonconstant variance;
- dependent observations;
- non-normal residual behavior relevant to inference;
- response outliers;
- high-leverage predictor profiles;
- influential observations;
- unmodeled groups or data-quality problems.
Diagnostics Are Investigative Tools
A diagnostic flag is not an automatic verdict. A large residual may be a data error, a valid rare case, evidence of a missing predictor, or ordinary sampling variation.
Diagnostics should lead to verification, contextual investigation, and sensitivity analysis—not automatic deletion or mechanical model changes.
The Residual-versus-Fitted Plot
The most common diagnostic plot places fitted values on the horizontal axis and residuals on the vertical axis:
\[ (\hat{y}_i,e_i). \]The horizontal reference line is \(e=0\). A satisfactory pattern is an unstructured cloud centered around zero with reasonably stable vertical spread.
Residuals versus Predictors
Plotting residuals against each quantitative predictor can reveal structure that is less visible against fitted values. In multiple regression, a separate residual plot should be considered for each important predictor and for scientifically meaningful groups.
Residuals versus Observation Order
If cases have a natural order in time, space, production sequence, or data collection, plot residuals in that order. Long runs of positive or negative residuals, cycles, drift, or abrupt shifts can indicate dependence or a changing process.
Independence is mainly a property of the study design and data-generating process, but ordered residual plots can expose violations.
A Pattern-Free Residual Plot
A useful residual plot generally shows
- points centered around zero;
- no systematic curve;
- roughly stable vertical spread;
- no unexplained clusters or bands;
- no single point dominating the scale.
“Pattern-free” does not prove that every assumption is true. It means that the plot did not reveal an obvious violation.
Curvature in Residuals
A U-shaped or inverted-U pattern indicates that the fitted mean is missing curvature. The line systematically overpredicts in some regions and underpredicts in others.
Possible responses include a transformation, polynomial term, spline, interaction, or a different model family. The appropriate remedy depends on the mechanism and the analytical purpose.
A Funnel-Shaped Pattern
Residuals that spread out or contract as fitted values increase suggest nonconstant conditional variance, called heteroscedasticity.
The fitted mean may still be useful, but conventional standard errors, confidence intervals, and prediction intervals may be unreliable if they assume constant variance.
Clusters and Bands
Separate residual clusters may indicate unmodeled categories, different operating regimes, repeated measurements, or data from distinct populations.
Horizontal bands can arise when the response takes only a few values. A model designed for continuous outcomes may then be inappropriate.
Gaps and Boundaries
Gaps may reflect missing portions of the predictor space, sampling rules, censoring, or structural separation. Sharp residual boundaries can occur when a response is limited below, above, or to an interval.
Model conclusions should not be extended across unsupported regions without justification.
Example Model
Return to the simple-regression example
\[ (1,2),\ (2,3),\ (3,5),\ (4,4),\ (5,6), \]with fitted equation
\[ \hat{y}=1.3+0.9x. \]The fitted values are \(2.2,3.1,4.0,4.9,5.8\), and the raw residuals are
\[ -0.2,\ -0.1,\ 1.0,\ -0.9,\ 0.2. \]Example Residual Plot
Residual Scale Depends on the Response
Raw residuals retain the response variable's units. A residual of \(5\) dollars may be negligible, while a residual of \(5\) million dollars may not be.
Raw residual size is therefore difficult to compare across models with different response scales.
The Residual Standard Error
For a model with \(p\) predictors and an intercept, the residual mean square is
\[ \operatorname{MSE} = \frac{\operatorname{SSE}}{n-p-1}. \]The residual standard error is
\[ \boxed{s=\sqrt{\operatorname{MSE}}}. \]It estimates the typical error scale under the model and retains the units of the response.
Example: Residual Standard Error
In the example, \(n=5\), \(p=1\), and \(\operatorname{SSE}=1.90\). Therefore,
\[ \operatorname{MSE} = \frac{1.90}{5-1-1} = \frac{1.90}{3} \approx 0.6333, \]and
\[ s=\sqrt{0.6333}\approx 0.7958. \]Why Residual Variances Differ
Residuals near unusual predictor profiles are constrained differently by the fitted model. Under the standard linear model,
\[ \operatorname{Var}(e_i) = \sigma^2(1-h_{ii}), \]where \(h_{ii}\) is the observation's leverage. Raw residuals therefore do not all have the same model-based variance.
Standardized Residuals
An internally studentized residual adjusts for the estimated error scale and leverage:
\[ \boxed{ r_i= \frac{e_i}{s\sqrt{1-h_{ii}}} }. \]Software sometimes calls \(r_i\) a standardized residual and sometimes an internally studentized residual. Always check the software's definition.
Reading Standardized Residuals
Rough screening conventions often inspect observations with
\[ |r_i|>2 \]and treat
\[ |r_i|>3 \]as more unusual. These are not automatic deletion rules. With many observations, some large values can occur by chance.
Externally Studentized Residuals
An externally studentized residual estimates the error scale after leaving observation \(i\) out:
\[ \boxed{ t_i= \frac{e_i}{s_{(i)}\sqrt{1-h_{ii}}} }. \]Because the observation does not help estimate its own comparison scale, externally studentized residuals are useful for formal outlier assessment.
Leverage
Leverage measures how unusual an observation's predictor profile is relative to the fitted data. It is the diagonal element \(h_{ii}\) of the hat matrix.
\[ \hat{\mathbf{y}}=\mathbf{H}\mathbf{y}, \qquad \mathbf{H} = \mathbf{X} (\mathbf{X}^{\mathsf T}\mathbf{X})^{-1} \mathbf{X}^{\mathsf T}. \]Leverage depends on the predictors, not on the observed response.
Leverage in Simple Regression
With one predictor and an intercept,
\[ \boxed{ h_{ii} = \frac{1}{n} + \frac{(x_i-\bar{x})^2} {\sum_{j=1}^{n}(x_j-\bar{x})^2} }. \]Points far from \(\bar{x}\) have greater leverage.
Average Leverage
With \(p\) predictors and an intercept,
\[ \sum_{i=1}^{n}h_{ii}=p+1. \]Therefore, average leverage is
\[ \boxed{\frac{p+1}{n}}. \]Rough screens often flag leverage above \(2(p+1)/n\) or \(3(p+1)/n\), but the design and sample size determine what deserves attention.
A High-Leverage Point Is Not Necessarily Bad
A high-leverage observation may be accurately measured, scientifically important, and consistent with the fitted relationship. Leverage says that the predictor profile has unusual potential to affect the model.
Whether that potential becomes actual influence also depends on the response and residual.
Outlier, Leverage, and Influence
| Concept | Main question | Typical evidence |
|---|---|---|
| Response outlier | Is the response unusual given the fitted model? | Large studentized residual |
| High leverage | Is the predictor profile unusual? | Large \(h_{ii}\) |
| Influential case | Does the case materially change the fitted analysis? | Large deletion-based change or influence measure |
Influence
An observation is influential when removing it materially changes coefficients, fitted values, predictions, standard errors, or conclusions.
Influence usually requires some combination of leverage and disagreement with the fitted pattern. A case with low leverage has limited ability to rotate or reposition the fitted surface.
Cook's Distance
Cook's distance combines residual size and leverage into a measure of how much the fitted values would change if observation \(i\) were removed:
\[ \boxed{ D_i = \frac{e_i^2}{(p+1)\operatorname{MSE}} \frac{h_{ii}}{(1-h_{ii})^2} }. \]Larger values indicate greater influence on the fitted model.
Screening Cook's Distance
Rules such as
\[ D_i>\frac{4}{n} \]or \(D_i>1\) are rough screens, not universal cutoffs. Influence should be assessed relative to the analysis, the other cases, and how conclusions change when the observation is examined.
Other Influence Measures
Several related diagnostics answer more specific questions:
- DFBETAS measure how deleting a case changes individual coefficients.
- DFFITS measures how deleting a case changes its fitted value.
- COVRATIO measures how deletion changes coefficient uncertainty.
These measures supplement, rather than replace, graphs and contextual investigation.
Example: Leverage Values
For \(x=1,2,3,4,5\), \(\bar{x}=3\), and \(\sum(x_i-\bar{x})^2=10\). The leverage values are
\[ 0.6,\ 0.3,\ 0.2,\ 0.3,\ 0.6. \]They sum to
\[ 2=p+1, \]because the model has one predictor and an intercept.
Example: Diagnostic Table
| Case | Raw residual \(e_i\) | Leverage \(h_{ii}\) | Internally studentized \(r_i\) | Cook's \(D_i\) |
|---|---|---|---|---|
| 1 | \(-0.2\) | 0.60 | \(-0.397\) | 0.118 |
| 2 | \(-0.1\) | 0.30 | \(-0.150\) | 0.005 |
| 3 | 1.0 | 0.20 | 1.405 | 0.247 |
| 4 | \(-0.9\) | 0.30 | \(-1.352\) | 0.392 |
| 5 | 0.2 | 0.60 | 0.397 | 0.118 |
Example: Interpret the Diagnostics
None of the internally studentized residuals has magnitude above \(2\), so no response outlier is strongly flagged by that rough screen. Cases \(1\) and \(5\) have the largest leverage because they are farthest from \(\bar{x}=3\).
Case \(4\) has the largest Cook's distance, approximately \(0.392\), but no case exceeds \(1\) or \(4/n=0.8\). The sample is far too small for strong diagnostic reassurance; the numbers illustrate the calculations rather than establish model adequacy.
The Normal Quantile–Quantile Plot
A normal Q–Q plot compares ordered residuals or studentized residuals with the quantiles expected under a normal distribution.
Approximate alignment along a straight reference line supports the normal approximation. Systematic curvature, heavy tails, skewness, or isolated points indicate departures.
What Normality Is Used For
Normality is not required to calculate ordinary least-squares coefficients. It supports exact small-sample \(t\)- and \(F\)-based inference under the classical model.
Large-sample methods may tolerate moderate non-normality, but outliers, heavy tails, dependence, and heteroscedasticity can still materially affect inference.
Do Not Diagnose Normality with a Histogram Alone
Residual histograms can be useful, but their appearance depends heavily on bin choices and sample size. A Q–Q plot often reveals tail behavior and asymmetry more directly.
Neither display should be interpreted without considering the data design and the inferential method.
Scale–Location Plots
A scale–location plot often places
\[ \sqrt{|r_i|} \]against fitted values. A changing upward or downward spread can make nonconstant variance easier to see than a raw residual plot.
Residuals versus Leverage
A residual-versus-leverage plot combines two ingredients of influence. Cook's-distance contours are often added to highlight cases that have both unusual predictor profiles and substantial residual disagreement.
Partial Residual Plots
In multiple regression, a partial residual plot examines the modeled form of one predictor while retaining information from its fitted coefficient. For predictor \(x_j\), a common component-plus-residual quantity is
\[ e_i+b_jx_{ij}. \]Such plots can suggest curvature, but they must be interpreted carefully when predictors are strongly related or interactions are present.
Added-Variable Plots
An added-variable plot compares the portion of \(y\) not linearly explained by the other predictors with the portion of one predictor not linearly explained by those predictors.
Its slope equals the corresponding multiple-regression coefficient. It can reveal leverage and influence in the adjusted relationship.
Diagnostics for Categorical Predictors
Residuals should be compared across predictor groups. Differences in center, spread, shape, or outlier frequency may indicate missing group effects, interactions, or unequal variance.
Side-by-side plots are often more informative than forcing categories onto a numerical axis.
Diagnostics for Time-Ordered Data
Time plots of residuals can reveal trend, seasonality, cycles, and shifts. An autocorrelation plot measures how residuals relate to lagged residuals.
A conventional regression that ignores strong time dependence can produce misleading standard errors and predictions.
The Durbin–Watson Statistic
For ordered residuals, the Durbin–Watson statistic is
\[ D= \frac{\sum_{i=2}^{n}(e_i-e_{i-1})^2} {\sum_{i=1}^{n}e_i^2}. \]Values near \(2\) suggest little first-order residual autocorrelation; smaller values suggest positive autocorrelation, and larger values suggest negative autocorrelation. Formal interpretation depends on the model, sample, and test framework.
Data Quality Before Model Repair
When a diagnostic is unusual, first verify
- data entry and decimal placement;
- measurement units and conversions;
- case identifiers and row alignment;
- duplicate or impossible records;
- missing-value codes;
- whether the case belongs to the intended population.
Do Not Delete a Point Automatically
Removing a valid observation because it weakens a result biases the analysis. Deletion is defensible when supported by a documented data error, an explicit population rule, or a principled analysis plan.
When a valid case is influential, report sensitivity with and without it if that comparison helps explain model dependence.
Responding to Curvature
Possible responses to curvature include
- transforming a predictor or response;
- adding polynomial or spline terms;
- including a scientifically motivated interaction;
- using a different model family;
- restricting conclusions to a region with approximately linear behavior.
The revised model should be checked again.
Responding to Nonconstant Variance
Depending on the objective and mechanism, possible responses include
- transforming the response;
- modeling the variance;
- weighted least squares;
- heteroscedasticity-robust standard errors;
- a model designed for the response distribution.
Robust standard errors adjust uncertainty estimates; they do not repair a wrong mean structure.
Responding to Dependence
Repeated, clustered, spatial, or time-ordered observations may require a multilevel model, generalized estimating equation, cluster-robust uncertainty estimate, time-series model, or another method that represents dependence.
The correct response follows from how the dependence arose.
Responding to Collinearity
Collinearity is primarily a predictor-structure problem rather than a residual pattern. Responses may include removing redundant encodings, combining measures, redesigning data collection, using subject-matter constraints, or applying regularization for prediction.
Removing a scientifically necessary confounder only to make another coefficient more stable can create a more serious interpretive problem.
Diagnostics and Validation Answer Different Questions
Diagnostics ask whether the fitted model leaves recognizable problems in the observed data. Validation asks how well the complete modeling process performs on new or withheld data.
A model can pass familiar residual checks and still generalize poorly. Conversely, a predictive model may perform well while failing assumptions needed for a specific coefficient interpretation.
Multiplicity in Diagnostic Screening
In a large dataset, some residuals will appear unusual simply because many cases are being screened. A fixed threshold applied to thousands of cases will generate flags even under a reasonable model.
Consider the number of observations, the expected tail frequency, and the purpose of the screen.
Diagnostics Do Not Prove Causation
Clean residual plots do not establish that regression coefficients are causal. Confounding, selection bias, reverse causation, and measurement error may remain invisible to standard residual diagnostics.
Model adequacy and causal identification are separate requirements.
A Reliable Diagnostic Workflow
- Verify cases, units, pairing, coding, and missing-data treatment.
- Inspect the original response and predictor displays.
- Fit the prespecified or substantively justified model.
- Plot residuals against fitted values and important predictors.
- Inspect residuals in time, spatial, or collection order when relevant.
- Examine Q–Q and scale–location plots for the intended inference.
- Review studentized residuals, leverage, and influence measures.
- Investigate flagged cases in their original context.
- Revise the model only with a defensible rationale.
- Refit, rediagnose, validate, and report sensitivity transparently.
Reporting Diagnostic Results
A diagnostic report should state which plots and measures were examined, what patterns were found, which cases were investigated, what model changes followed, and whether conclusions were sensitive to those changes.
Avoid unsupported statements such as “all assumptions were met.” A more accurate report identifies the evidence and its limits.
Common Error: Using the Wrong Residual Sign
The residual is
\[ e_i=y_i-\hat{y}_i. \]A point above the fitted model has a positive residual.
Common Error: Looking Only at \(R^2\)
A high \(R^2\) can coexist with curvature, unequal variance, dependence, or a model controlled by one observation. Fit magnitude does not replace diagnostics.
Common Error: Treating Every Large Residual as Influential
A large residual identifies an unusual response conditional on the model. Influence also depends on leverage and the resulting change in the fitted analysis.
Common Error: Treating High Leverage as an Error
High leverage means that a predictor profile is unusual. The case may be valid and valuable. Verify and investigate it before deciding how to proceed.
Common Error: Applying Cutoffs Mechanically
Thresholds for studentized residuals, leverage, and Cook's distance are screening conventions. They do not determine deletion or validity by themselves.
Common Error: Repairing One Plot and Stopping
Adding a term to remove curvature may affect variance, influence, and extrapolation. Every revised model requires a new diagnostic cycle.
Common Error: Searching Until a Model Looks Clean
Repeatedly modifying a model to satisfy visual preferences can overfit the data and invalidate conventional inference. Model changes should be substantively justified and evaluated with validation when prediction is the goal.
Practice 1: Calculate a Residual
A model predicts \(\hat{y}=42\), and the observed response is \(y=47\). Find and interpret the residual.
Practice 2: Interpret a Curved Pattern
A residual-versus-fitted plot has positive residuals at both ends and negative residuals in the middle. What does this suggest?
Practice 3: Interpret a Funnel
Residual spread grows as fitted values increase. Name the likely issue and one possible response.
Practice 4: Calculate Residual Standard Error
A model has \(n=30\), \(p=4\), and \(\operatorname{SSE}=250\). Find \(s\).
Practice 5: Calculate Simple-Regression Leverage
Suppose \(n=10\), \(\bar{x}=5\), \(\sum(x_j-\bar{x})^2=40\), and \(x_i=9\). Find \(h_{ii}\).
Practice 6: Distinguish the Concepts
A case has an ordinary predictor profile but a very unusual response. Which diagnostic concept is most directly implicated: leverage, response outlier, or influence?
Practice 7: Calculate an Internally Studentized Residual
If \(e_i=3\), \(s=2\), and \(h_{ii}=0.36\), find \(r_i\).
Practice 8: Interpret Cook's Distance
One case has the largest Cook's distance in a dataset. What does that fact establish, and what does it not establish?
Practice 9: Ordered Residuals
Residuals alternate in a seasonal pattern over time. Why is an ordinary independence assumption questionable?
Practice 10: Evaluate a Deletion Decision
An analyst removes every case with \(|r_i|>2\) without checking the records and refits the model. Explain why this procedure is unsound.
Practice Answers
Practice 1
\[ e=y-\hat{y}=47-42=5. \]The observed response is \(5\) response units above the fitted value; the model underpredicted the case by \(5\).
Practice 2
The U-shaped residual pattern suggests missing curvature. The current model underpredicts at the ends and overpredicts in the middle.
Practice 3
The funnel suggests heteroscedasticity. Depending on the mechanism and goal, responses may include transforming the response, modeling the variance, weighted least squares, robust standard errors, or a different model family.
Practice 4
\[ s = \sqrt{\frac{250}{30-4-1}} = \sqrt{10} \approx 3.162. \]Practice 5
\[ h_{ii} = \frac{1}{10} + \frac{(9-5)^2}{40} = 0.1+0.4 = 0.5. \]Practice 6
Response outlier is most directly implicated because the response is unusual conditional on an ordinary predictor profile. Whether the case is influential requires an additional change-in-fit assessment.
Practice 7
\[ r_i = \frac{3}{2\sqrt{1-0.36}} = \frac{3}{2(0.8)} = 1.875. \]Practice 8
It establishes that the case has the greatest measured influence by Cook's distance relative to the other cases. It does not establish that the case is erroneous, should be deleted, exceeds a meaningful threshold, or changes the substantive conclusion.
Practice 9
The repeating pattern indicates that residuals at different times are structurally related. Conventional independent-error standard errors and forecasts may therefore be misleading.
Practice 10
The threshold is only a screen. Some flags arise by chance, and valid unusual cases may carry important information. Mechanical deletion changes the target population, biases results, hides model failure, and ignores data verification, leverage, influence, multiplicity, and context.
What to Remember
Residuals connect observed and fitted responses:
\[ \boxed{e_i=y_i-\hat{y}_i}. \]- Residual plots should be centered near zero without systematic structure.
- Curvature suggests an incomplete mean form.
- A funnel suggests nonconstant variance.
- Studentized residuals help identify unusual responses.
- Leverage measures unusual predictor profiles.
- Influence measures how much a case changes the fitted analysis.
- Cook's distance combines residual size and leverage.
- Diagnostic cutoffs are screens, not deletion rules.
- Every model revision requires new diagnostics and transparent reporting.
Where We Go Next
Regression diagnostics evaluate whether a fitted model behaves as intended. The next step is to expand how predictors can enter the model.
The next lesson, Categorical Predictors, develops indicator variables, reference groups, group-specific intercepts, and interactions between categorical and quantitative predictors.