The Central Idea
A scatterplot may show that a response variable \(y\) tends to change as an explanatory variable \(x\) changes. Simple linear regression represents that tendency with a fitted line:
\[ \boxed{\hat{y}=b_0+b_1x}. \]For a given \(x\)-value, the line produces a fitted or predicted response \(\hat{y}\). The difference between the observed response and that fitted value is a residual.
Why “Simple” and Why “Linear”?
The model is called simple because it uses one explanatory variable. It is called linear because the mean structure is a straight line in \(x\):
\[ b_0+b_1x. \]A model with several explanatory variables is multiple regression. A model with a curved mean structure is nonlinear in its displayed relationship, even though some curved models can still be estimated using linear regression methods after suitable construction.
Explanatory and Response Variables
The roles of the variables matter.
- The explanatory variable \(x\) is used to explain or predict.
- The response variable \(y\) is the outcome being modeled.
Reversing these roles generally produces a different regression line. Regression is therefore not symmetric, even though correlation is.
The Population Model
A common population model is
\[ \boxed{Y=\beta_0+\beta_1x+\varepsilon}. \]Here, \(\beta_0\) and \(\beta_1\) are unknown population parameters, and \(\varepsilon\) represents variation not captured by the linear mean. Equivalently, the conditional mean is
\[ E(Y\mid X=x)=\beta_0+\beta_1x. \]The Fitted Sample Line
Sample data are used to estimate the unknown population line. The fitted equation is
\[ \boxed{\hat{y}=b_0+b_1x}, \]where \(b_0\) estimates \(\beta_0\) and \(b_1\) estimates \(\beta_1\).
The hat in \(\hat{y}\) distinguishes a value produced by the fitted model from an observed response \(y\).
Fitted Values
For observation \(i\), substitute \(x_i\) into the fitted equation:
\[ \hat{y}_i=b_0+b_1x_i. \]The result \(\hat{y}_i\) is the response value located on the fitted line directly above or below \(x_i\).
Residuals
A residual is the observed response minus the fitted response:
\[ \boxed{e_i=y_i-\hat{y}_i}. \]- If \(e_i>0\), the point lies above the fitted line.
- If \(e_i<0\), the point lies below the fitted line.
- If \(e_i=0\), the point lies exactly on the fitted line.
Residuals are vertical deviations because the model predicts \(y\) from \(x\).
The Least-Squares Criterion
Many lines could be drawn through a point cloud. The ordinary least-squares line chooses \(b_0\) and \(b_1\) to minimize the sum of squared residuals:
\[ \boxed{ \operatorname{SSE} = \sum_{i=1}^{n}e_i^2 = \sum_{i=1}^{n} \left[y_i-(b_0+b_1x_i)\right]^2 }. \]SSE stands for sum of squared errors, also called the residual sum of squares.
Why Square the Residuals?
Simply adding residuals would allow positive and negative deviations to cancel. Squaring makes every contribution nonnegative:
\[ e_i^2\ge 0. \]Squaring also gives greater weight to large misses and produces a smooth mathematical objective with a direct solution. This choice is useful, but it also makes least squares sensitive to outliers.
The Least-Squares Slope
When the \(x\)-values are not all identical, the fitted slope is
\[ \boxed{ b_1= \frac{\sum_{i=1}^{n}(x_i-\bar{x})(y_i-\bar{y})} {\sum_{i=1}^{n}(x_i-\bar{x})^2} }. \]The numerator measures paired co-movement. The denominator measures variation in the explanatory variable.
The Least-Squares Intercept
Once the slope is known, the fitted intercept is
\[ \boxed{b_0=\bar{y}-b_1\bar{x}}. \]This formula makes the fitted line pass through the point of sample means:
\[ \boxed{(\bar{x},\bar{y})}. \]Connection to Correlation
The least-squares slope can also be written
\[ \boxed{b_1=r\frac{s_y}{s_x}}. \]The sign of \(b_1\) is therefore the sign of \(r\). Their magnitudes are not generally equal because slope contains units and correlation does not.
Example Data
Continue with the five observations used in the correlation lesson:
\[ (1,2),\ (2,3),\ (3,5),\ (4,4),\ (5,6). \]The means and deviation sums are
\[ \bar{x}=3, \qquad \bar{y}=4, \] \[ \sum(x_i-\bar{x})(y_i-\bar{y})=9, \qquad \sum(x_i-\bar{x})^2=10. \]Example: Calculate the Slope
Substitute the deviation sums:
\[ b_1=\frac{9}{10}=0.9. \]For each one-unit increase in \(x\), the fitted response increases by \(0.9\) units on average.
Example: Calculate the Intercept
Use the means and the fitted slope:
\[ b_0 = \bar{y}-b_1\bar{x} = 4-(0.9)(3) = 1.3. \]The fitted equation is therefore
\[ \boxed{\hat{y}=1.3+0.9x}. \]Example: The Fitted Line
Example: Calculate Fitted Values
Substitute each \(x\)-value into \(\hat{y}=1.3+0.9x\). For example, at \(x=4\),
\[ \hat{y}=1.3+0.9(4)=4.9. \]The model predicts a response of \(4.9\) when \(x=4\).
Example: Calculate Residuals
| \(x_i\) | Observed \(y_i\) | Fitted \(\hat{y}_i\) | Residual \(e_i=y_i-\hat{y}_i\) | Squared residual \(e_i^2\) |
|---|---|---|---|---|
| 1 | 2 | 2.2 | \(-0.2\) | 0.04 |
| 2 | 3 | 3.1 | \(-0.1\) | 0.01 |
| 3 | 5 | 4.0 | \(1.0\) | 1.00 |
| 4 | 4 | 4.9 | \(-0.9\) | 0.81 |
| 5 | 6 | 5.8 | \(0.2\) | 0.04 |
| Sums | 0.0 | 1.90 | ||
Therefore,
\[ \operatorname{SSE}=1.90. \]Residuals Sum to Zero
When a least-squares model includes an intercept, its residuals satisfy
\[ \boxed{\sum_{i=1}^{n}e_i=0}. \]The positive and negative vertical deviations balance. This property is a consequence of the fitting method, not evidence that the model is automatically appropriate.
The Line Passes Through the Means
Substitute \(x=\bar{x}\) into the fitted equation:
\[ \hat{y} = b_0+b_1\bar{x} = (\bar{y}-b_1\bar{x})+b_1\bar{x} = \bar{y}. \]Thus, every least-squares line with an intercept passes through \((\bar{x},\bar{y})\).
Interpreting the Slope
The slope \(b_1\) is the fitted change in the response for a one-unit increase in the explanatory variable.
If a model for weekly sales \(y\), measured in thousands of dollars, and advertising \(x\), measured in thousands of dollars, has \(b_1=2.4\), then each additional thousand dollars of advertising is associated with an estimated increase of \(2.4\) thousand dollars in weekly sales, on average.
“Associated with” is appropriate unless the design supports a causal claim.
Units of the Slope
The slope's units are
\[ \boxed{ \frac{\text{units of }y}{\text{units of }x} }. \]Changing dollars to thousands of dollars or miles to kilometers changes the numerical slope. This contrasts with correlation, which is unitless.
Interpreting the Intercept
The intercept \(b_0\) is the fitted response when \(x=0\):
\[ \hat{y}=b_0+b_1(0)=b_0. \]This interpretation is useful only when \(x=0\) is meaningful and reasonably connected to the observed data. If zero is impossible or far outside the observed range, the intercept may function mainly as a mathematical anchor for the line.
Prediction by Interpolation
Interpolation predicts at an \(x\)-value within the observed range. If the example data contain \(1\le x\le 5\), then a prediction at \(x=2.5\) is an interpolation:
\[ \hat{y}=1.3+0.9(2.5)=3.55. \]Interpolation is generally more defensible than prediction far beyond the data, provided the linear model fits the observed range.
The Danger of Extrapolation
Extrapolation predicts beyond the observed \(x\)-range. The relationship may flatten, curve, reverse, encounter a physical limit, or enter a new operating regime outside the data.
A fitted line is not evidence that its pattern continues indefinitely. Every prediction should be compared with the range used to fit the model.
A Prediction Is Not a Guarantee
The fitted value \(\hat{y}\) is a point prediction, not a promise that a new response will equal that number. Individual outcomes vary around the regression line.
Later methods construct intervals that distinguish uncertainty about the mean response from uncertainty about an individual future response.
Total Variation in the Response
Before using \(x\), variation in \(y\) can be measured around its mean:
\[ \operatorname{SST} = \sum_{i=1}^{n}(y_i-\bar{y})^2. \]SST is the total sum of squares. For the example,
\[ \operatorname{SST}=10. \]The Coefficient of Determination
The coefficient of determination is
\[ \boxed{ R^2 = 1-\frac{\operatorname{SSE}}{\operatorname{SST}} }. \]It measures the proportion of the observed sample variation in \(y\), around \(\bar{y}\), accounted for by the fitted regression on \(x\).
Example: Calculate \(R^2\)
For the example,
\[ R^2 = 1-\frac{1.90}{10} = 0.81. \]Within this sample, the fitted linear relationship with \(x\) accounts for \(81\%\) of the observed variation in \(y\) around its mean. The remaining \(19\%\) is residual variation in the fitted sample.
In Simple Regression, \(R^2=r^2\)
For ordinary least-squares regression with one explanatory variable and an intercept,
\[ \boxed{R^2=r^2}. \]The example correlation was \(r=0.9\), so
\[ R^2=(0.9)^2=0.81. \]Squaring removes direction. The slope or \(r\) is still needed to determine whether the association is positive or negative.
Explained and Residual Variation
With an intercept, total variation separates into explained and residual components:
\[ \boxed{ \operatorname{SST} = \operatorname{SSR} + \operatorname{SSE} }. \]Here, SSR is the regression sum of squares:
\[ \operatorname{SSR} = \sum_{i=1}^{n}(\hat{y}_i-\bar{y})^2. \]Do Not Overinterpret \(R^2\)
A large \(R^2\) does not prove that the model is causal, unbiased, correctly specified, or useful outside the observed range. A small \(R^2\) does not automatically make a relationship unimportant.
Model quality depends on purpose, residual behavior, data collection, uncertainty, and consequences—not on one threshold.
A Residual Plot
A residual plot places \(x_i\), or sometimes \(\hat{y}_i\), on the horizontal axis and \(e_i\) on the vertical axis. The reference line is
\[ e=0. \]A useful linear fit has residuals scattered around zero without a systematic curve, widening funnel, separated group structure, or dominating outlier.
Linearity
The conditional mean of \(Y\) should be reasonably represented by a line over the range being modeled. A curved residual pattern indicates that a straight line systematically overpredicts in some regions and underpredicts in others.
Adding more data does not repair a structurally wrong straight-line form.
Independent Observations
Standard regression inference commonly assumes that the errors for different observations are independent. Repeated measurements on one person, nearby locations, related firms, or time-ordered observations may violate this condition.
Independence is primarily a design question. It cannot be guaranteed by the fitted equation alone.
Constant Error Variance
Standard simple regression methods commonly assume that the vertical variability around the line is approximately constant across \(x\). This is called homoscedasticity.
A residual plot that widens or narrows like a funnel suggests heteroscedasticity. The fitted line may still summarize the mean, but conventional uncertainty calculations can become unreliable.
Normality and Inference
Normal error assumptions support familiar small-sample \(t\)-tests, confidence intervals, and prediction intervals. Normality is not required merely to calculate the least-squares line.
The importance of the assumption depends on sample size, the inferential method, and the severity of departures.
Outliers, Leverage, and Influence
A point with an unusual response can have a large residual. A point with an \(x\)-value far from the rest has high leverage. A point is influential when removing it materially changes the fitted equation or conclusions.
Least squares is sensitive to such observations because large residuals are squared. Investigate influential cases rather than deleting them automatically.
Subgroups and Omitted Structure
A pooled line can conceal different intercepts, slopes, or directions among subgroups. For example, departments, regions, product classes, or treatment groups may follow different patterns.
Plot and label relevant groups. A single line should not replace a more appropriate grouped or multiple-regression analysis.
Regression Does Not Establish Causation
A fitted slope measures association conditional on the chosen data and model. It does not establish that intervening on \(x\) will change \(y\) by the slope amount.
Confounding, reverse causation, selection, measurement error, and study design all affect causal interpretation.
Correlation and Regression Compared
| Feature | Correlation | Simple linear regression |
|---|---|---|
| Purpose | Measure linear direction and strength | Model or predict a response from an explanatory variable |
| Variable roles | Symmetric | Explanatory and response roles differ |
| Units | Unitless | Slope has response units per explanatory unit |
| Output | A number from \(-1\) to \(1\) | A fitted equation and residuals |
When the Slope Is Zero
If \(b_1=0\), the fitted equation becomes
\[ \hat{y}=\bar{y}. \]The explanatory variable provides no sample linear improvement over using the response mean. This does not rule out a nonlinear relationship.
A Reliable Fitting Procedure
- Identify the cases, response, explanatory variable, and units.
- Inspect the scatterplot and verify the paired data.
- Check whether a linear summary is reasonable for the intended range.
- Calculate \(b_1\) and \(b_0\), or fit them with reliable software.
- Write the fitted equation with meaningful variable names and units.
- Interpret the slope and, when meaningful, the intercept.
- Calculate fitted values, residuals, SSE, and \(R^2\).
- Inspect residual plots and influential observations.
- Limit predictions to defensible cases and ranges.
- Report uncertainty and causal limitations appropriate to the design.
Reporting a Fitted Model
A useful report names the variables and units, provides the equation and sample size, interprets the slope, summarizes fit, and states material limitations.
For example:
Among \(80\) sampled stores, the fitted weekly-sales model was \(\widehat{\text{sales}}=18.4+2.1(\text{floor area})\), with sales measured in thousands of dollars and floor area in thousands of square feet. Each additional thousand square feet was associated with \(2.1\) thousand dollars in higher fitted weekly sales. The model had \(R^2=0.67\), and residual checks showed no strong curvature.
Common Error: Reversing the Residual
The residual is
\[ e=y-\hat{y}, \]not \(\hat{y}-y\). A point above the line must have a positive residual.
Common Error: Calling the Slope a Correlation
A slope of \(0.8\) does not mean the correlation is \(0.8\). The slope depends on units through
\[ b_1=r\frac{s_y}{s_x}. \]Common Error: Forcing Meaning onto the Intercept
An intercept can be mathematically necessary even when \(x=0\) is impossible or far beyond the data. In that situation, state that the intercept has limited practical interpretation.
Common Error: Treating \(R^2\) as Accuracy
\(R^2=0.80\) does not mean that predictions are “\(80\%\) accurate.” It is a sample variation ratio. Prediction error must be evaluated through residual size, validation, and uncertainty appropriate to the prediction target.
Common Error: Extrapolating the Line
A regression equation is not a universal law. Predictions far beyond the observed range depend on unverified continuation of the linear pattern.
Common Error: Ignoring Residual Patterns
A high \(R^2\) can coexist with curvature, unequal variance, clusters, or an influential point. Inspecting residuals is part of fitting the model, not an optional afterthought.
Common Error: Using the Line as Causal Proof
The fitted coefficient describes an association under the model. Causal wording requires a design and identification argument that justify it.
Practice 1: Identify the Model Components
A fitted model for monthly electricity use \(y\), in kilowatt-hours, from average temperature \(x\), in degrees Celsius, is
\[ \hat{y}=920-18x. \]Identify the slope and intercept.
Practice 2: Interpret the Slope
Interpret the slope \(-18\) from Practice 1 in context without making a causal claim.
Practice 3: Calculate a Prediction
Use \(\hat{y}=920-18x\) to predict monthly electricity use when \(x=15^\circ\text{C}\).
Practice 4: Calculate a Residual
At \(x=15^\circ\text{C}\), the observed electricity use is \(700\) kilowatt-hours. Find and interpret the residual.
Practice 5: Find a Slope
Suppose
\[ \sum(x_i-\bar{x})(y_i-\bar{y})=24 \]and
\[ \sum(x_i-\bar{x})^2=30. \]Find \(b_1\).
Practice 6: Find an Intercept
If \(\bar{x}=10\), \(\bar{y}=50\), and \(b_1=0.8\), find \(b_0\).
Practice 7: Calculate \(R^2\)
A fitted model has \(\operatorname{SSE}=36\) and \(\operatorname{SST}=120\). Find and interpret \(R^2\).
Practice 8: Diagnose a Residual Pattern
A residual plot has positive residuals at low and high \(x\)-values and negative residuals in the middle. What does this suggest?
Practice 9: Evaluate Extrapolation
A sales model was fitted using advertising budgets between \(\$5{,}000\) and \(\$40{,}000\). Explain why a prediction at \(\$500{,}000\) is risky.
Practice 10: Critique a Causal Claim
An observational regression of employee training hours on productivity has a positive slope. A manager concludes that requiring ten additional hours will cause the predicted productivity increase. What is missing?
Practice Answers
Practice 1
The slope is \(b_1=-18\) kilowatt-hours per degree Celsius. The intercept is \(b_0=920\) kilowatt-hours.
Practice 2
Each one-degree Celsius increase in average temperature is associated with an estimated decrease of \(18\) kilowatt-hours in fitted monthly electricity use, on average, within the modeled range.
Practice 3
\[ \hat{y}=920-18(15)=920-270=650. \]The predicted monthly electricity use is \(650\) kilowatt-hours.
Practice 4
\[ e=y-\hat{y}=700-650=50. \]The household used \(50\) kilowatt-hours more than the model predicted.
Practice 5
\[ b_1=\frac{24}{30}=0.8. \]Practice 6
\[ b_0=\bar{y}-b_1\bar{x}=50-(0.8)(10)=42. \]Practice 7
\[ R^2=1-\frac{36}{120}=1-0.30=0.70. \]The fitted regression accounts for \(70\%\) of the observed sample variation in the response around its mean.
Practice 8
The systematic U-shaped residual pattern suggests curvature. A straight line underpredicts at the ends and overpredicts in the middle, so the linear mean structure is incomplete.
Practice 9
\(\$500{,}000\) is far outside the observed budget range. Market saturation, capacity constraints, changing strategy, or other mechanisms may prevent the fitted linear pattern from continuing.
Practice 10
The regression is observational and may reflect confounding, selection, or reverse causation. A causal conclusion requires an appropriate design and a defensible argument that the slope identifies the effect of additional training.
What to Remember
The fitted simple linear regression model is
\[ \boxed{\hat{y}=b_0+b_1x}. \]Its coefficients are
\[ \boxed{ b_1= \frac{\sum(x_i-\bar{x})(y_i-\bar{y})} {\sum(x_i-\bar{x})^2} = r\frac{s_y}{s_x} } \]and
\[ \boxed{b_0=\bar{y}-b_1\bar{x}}. \]- Residuals are \(e_i=y_i-\hat{y}_i\).
- Least squares minimizes \(\sum e_i^2\).
- The fitted line passes through \((\bar{x},\bar{y})\).
- In simple regression with an intercept, \(R^2=r^2\).
- Residual patterns can reveal model failure.
- Extrapolation and causal interpretation require separate justification.
Where We Go Next
Simple linear regression uses one explanatory variable. Many practical outcomes depend on several variables at the same time, and an apparent two-variable relationship can change after another predictor is considered.
The next lesson, Introductory Multiple Regression, extends the fitted mean to several predictors and develops the idea of interpreting one coefficient while holding the others fixed.