The Central Idea
Regression needs numerical columns, but a category such as region, product type, or treatment group is not itself a numerical measurement. We represent category membership with indicator variables.
For two groups, A and B, define
\[ D= \begin{cases} 0,&\text{case belongs to group A},\\ 1,&\text{case belongs to group B}. \end{cases} \]The regression model can then estimate a fitted difference between the groups.
Categorical Predictor, Quantitative Response
In this lesson, the response \(y\) is quantitative and one or more predictors are categorical.
A categorical response requires a classification or categorical response model rather than ordinary least squares. That distinction matters: the same variable can require different methods depending on whether it is used as a predictor or as the response.
Indicators Are Numerical Encodings
An indicator variable, sometimes called a dummy variable, records whether a condition is present:
\[ D_i= \begin{cases} 1,&\text{condition is present for case }i,\\ 0,&\text{condition is absent for case }i. \end{cases} \]The numbers \(0\) and \(1\) are coding devices. They do not claim that the category labeled \(1\) is larger, better, or more important.
A Binary Predictor Model
With one binary predictor, the fitted model is
\[ \boxed{\hat{y}=b_0+b_1D}. \]Substituting the two possible indicator values produces the fitted group means.
The Reference Group
Group A has \(D=0\), so its fitted response is
\[ \hat{y}_{A}=b_0+b_1(0)=b_0. \]Group A is the reference group. The intercept is its fitted mean when no other predictors are present.
The Comparison Group
Group B has \(D=1\), so its fitted response is
\[ \hat{y}_{B}=b_0+b_1. \]The fitted B-minus-A difference is
\[ \boxed{\hat{y}_{B}-\hat{y}_{A}=b_1}. \]A positive \(b_1\) places group B above group A; a negative \(b_1\) places group B below group A.
Example: Two Group Means
Suppose
\[ \hat{y}=72+6D, \]where \(D=0\) for the standard process and \(D=1\) for the revised process. Then
\[ \hat{y}_{\text{standard}}=72 \]and
\[ \hat{y}_{\text{revised}}=72+6=78. \]The revised process has a fitted response \(6\) units higher than the standard process in the modeled data.
Regression and the Two-Group Comparison
With only an intercept and one binary indicator, ordinary least squares reproduces the two sample means:
\[ b_0=\bar{y}_{A}, \qquad b_1=\bar{y}_{B}-\bar{y}_{A}. \]Regression therefore connects the familiar difference in group means to a model that can later include additional predictors and interactions.
Adding a Quantitative Predictor
Suppose \(x\) is quantitative and \(D\) identifies two groups. An additive model is
\[ \boxed{\hat{y}=b_0+b_1x+b_2D}. \]This model produces two parallel fitted lines.
The Reference-Group Line
For the reference group, \(D=0\):
\[ \hat{y}_{A}=b_0+b_1x. \]The reference group has intercept \(b_0\) and slope \(b_1\).
The Comparison-Group Line
For group B, \(D=1\):
\[ \hat{y}_{B} = b_0+b_1x+b_2 = (b_0+b_2)+b_1x. \]Group B has intercept \(b_0+b_2\) and the same slope \(b_1\).
The Adjusted Group Difference
At the same \(x\)-value,
\[ \boxed{ \hat{y}_{B}-\hat{y}_{A}=b_2 }. \]Thus, \(b_2\) is the fitted B-minus-A difference while \(x\) is held fixed. Because the lines are parallel, this difference is constant across \(x\).
A Worked Dataset
Let \(x_c=x-2\) be a centered quantitative predictor. Group A is the reference group, and \(D=1\) identifies group B.
| Case | Group | \(D\) | Centered predictor \(x_c\) | Observed \(y\) |
|---|---|---|---|---|
| A1 | A | 0 | \(-1\) | 53 |
| A2 | A | 0 | 0 | 53 |
| A3 | A | 0 | 1 | 56 |
| B1 | B | 1 | \(-1\) | 59 |
| B2 | B | 1 | 0 | 63 |
| B3 | B | 1 | 1 | 64 |
Example: The Fitted Model
Ordinary least squares gives
\[ \boxed{\hat{y}=54+2x_c+8D}. \]The coefficients have direct meanings:
- \(54\) is the fitted group A response at \(x_c=0\), or \(x=2\).
- \(2\) is the common fitted slope across both groups.
- \(8\) is the fitted B-minus-A difference at the same \(x_c\).
Example: The Two Fitted Lines
For group A,
\[ \hat{y}_{A}=54+2x_c. \]For group B,
\[ \hat{y}_{B}=62+2x_c. \]Both lines rise by \(2\) response units per unit of \(x_c\), and group B is fitted \(8\) units above group A throughout the modeled range.
Example: Parallel Group Lines
Example: Fitted Values and Residuals
| Case | Observed \(y_i\) | Fitted \(\hat{y}_i\) | Residual \(e_i\) | \(e_i^2\) |
|---|---|---|---|---|
| A1 | 53 | 52 | 1 | 1 |
| A2 | 53 | 54 | \(-1\) | 1 |
| A3 | 56 | 56 | 0 | 0 |
| B1 | 59 | 60 | \(-1\) | 1 |
| B2 | 63 | 62 | 1 | 1 |
| B3 | 64 | 64 | 0 | 0 |
| Sums | 0 | 4 | ||
The model has \(\operatorname{SSE}=4\).
Example: Model Fit
The observed response mean is \(\bar{y}=58\), and
\[ \operatorname{SST} = \sum(y_i-\bar{y})^2 = 116. \]Therefore,
\[ R^2 = 1-\frac{4}{116} \approx 0.966. \]This small, constructed example is designed for transparent interpretation, not as a realistic benchmark for fit.
Centering Makes the Group Comparison Useful
Because \(x_c=x-2\), the value \(x_c=0\) corresponds to the meaningful reference \(x=2\). The group-specific intercepts \(54\) and \(62\) are fitted responses at that reference value.
Centering does not change the fitted values of the additive model. It changes the intercept's reference point and can make interpretation more practical.
Changing the Reference Group
If group B becomes the reference, define \(A=1\) for group A and \(A=0\) for group B. The same fitted values can be written
\[ \hat{y}=62+2x_c-8A. \]The intercept and group coefficient change, but the fitted lines and residuals do not. The coefficient now reports A minus B instead of B minus A.
Reference Choice Is Not a Ranking
A reference group provides a comparison baseline. It is not declared normal, preferred, superior, or causally prior.
Choose a reference that makes the intended comparisons readable, such as a standard treatment, existing policy, largest group, or scientifically meaningful baseline.
More Than Two Categories
A categorical predictor with \(k\) levels requires \(k-1\) indicators when the model includes an intercept and uses reference coding.
\[ \boxed{k\text{ categories}\longrightarrow k-1\text{ indicator columns}}. \]The omitted level is the reference category.
Example: Three Regions
Suppose region has levels North, South, and West. Choose North as the reference and define
\[ D_S= \begin{cases} 1,&\text{South},\\ 0,&\text{otherwise}, \end{cases} \qquad D_W= \begin{cases} 1,&\text{West},\\ 0,&\text{otherwise}. \end{cases} \]The Three-Region Model
With a quantitative predictor \(x\), an additive model is
\[ \hat{y} = b_0+b_1x+b_SD_S+b_WD_W. \]- \(b_0+b_1x\) is the North fitted line.
- \(b_S\) is the South-minus-North fitted difference at the same \(x\).
- \(b_W\) is the West-minus-North fitted difference at the same \(x\).
Why Not Include All \(k\) Indicators?
If an intercept and indicators for all \(k\) categories are included, every row satisfies
\[ D_1+D_2+\cdots+D_k=1. \]The intercept column is therefore an exact linear combination of the indicator columns. This perfect collinearity is sometimes called the dummy-variable trap.
Full-Rank Coding
A valid coding system must avoid exact redundancy. Common options include
- an intercept plus \(k-1\) indicators;
- no intercept plus \(k\) group indicators;
- a set of \(k-1\) linearly independent contrasts.
Different full-rank codings can represent the same fitted group means.
Treatment or Reference Coding
Reference coding compares each non-reference level directly with one chosen baseline. It is often easiest to interpret when those specific comparisons answer the study question.
Coefficients change when the baseline changes, but the represented fitted values remain the same.
Effect and Contrast Coding
Other coding systems compare levels with a grand-style average, successive levels, polynomial trends, or planned scientific contrasts.
The numerical coefficients depend on the chosen contrast matrix. Report the coding explicitly so that every coefficient has an identifiable comparison.
Fitted Values Are Coding-Invariant
If two full-rank coding systems span the same model space, they produce the same fitted values, residuals, SSE, and overall \(R^2\).
Coding changes the coordinate description of the model, not the fitted group predictions it represents.
A Joint Test for a Categorical Predictor
For the three-region reference-coded model, the complete region effect is assessed through the joint hypothesis
\[ H_0:b_S=b_W=0. \]Testing only one indicator answers one comparison, not whether the categorical predictor as a whole adds information.
Regression and One-Way ANOVA
A one-way analysis of variance can be written as a regression with categorical indicators. Both frameworks compare group means by decomposing response variation into between-group and within-group components.
Regression notation makes it straightforward to add quantitative predictors, planned contrasts, and interactions.
Adjusted Group Comparisons
When additional predictors enter the model, category coefficients compare fitted group responses at common values of those included predictors.
This adjusted difference need not equal the unadjusted difference in sample means because the groups may have different predictor distributions.
Estimated Marginal Means
Estimated marginal means, sometimes called adjusted means or least-squares means, summarize fitted group responses over a specified reference distribution of the other predictors.
Their interpretation depends on how predictors are fixed or averaged. Always state the averaging rule and target population.
Interactions with a Quantitative Predictor
The parallel-line model assumes the quantitative slope is the same for both groups. To allow different slopes, include an interaction:
\[ \boxed{ \hat{y} = b_0+b_1x+b_2D+b_3xD }. \]The Reference-Group Interaction Line
For group A, \(D=0\):
\[ \hat{y}_{A}=b_0+b_1x. \]Group A has intercept \(b_0\) and slope \(b_1\).
The Comparison-Group Interaction Line
For group B, \(D=1\):
\[ \hat{y}_{B} = (b_0+b_2)+(b_1+b_3)x. \]Group B has intercept \(b_0+b_2\) and slope \(b_1+b_3\).
The Group Difference Now Depends on \(x\)
With an interaction,
\[ \boxed{ \hat{y}_{B}-\hat{y}_{A}=b_2+b_3x }. \]The coefficient \(b_2\) is the group difference specifically at \(x=0\). Centering \(x\) at a meaningful reference makes this comparison useful.
Example: A Group-by-\(x\) Interaction
Suppose
\[ \hat{y}=54+2x_c+8D+1.5x_cD. \]Group A has slope \(2\). Group B has slope
\[ 2+1.5=3.5. \]At \(x_c=0\), the fitted B-minus-A difference is \(8\). At other values, the difference is \(8+1.5x_c\).
Main Effects in an Interaction Model
When \(xD\) is present,
- \(b_1\) is the \(x\)-slope for the reference group;
- \(b_2\) is the group difference at \(x=0\);
- \(b_3\) is the difference between the group slopes.
It is generally inappropriate to describe \(b_1\) or \(b_2\) as an overall effect without naming the reference conditions.
The Hierarchy Principle
When an interaction such as \(xD\) is included, the corresponding lower order terms \(x\) and \(D\) are usually retained even if their individual tests are not significant.
This preserves the model's meaning across coding and makes the interaction interpretable.
Interactions Between Categories
Two categorical predictors can also interact. A treatment difference, for example, may vary by region. The model then requires product terms between the corresponding indicators.
Interpret such models through fitted cell means or planned contrasts rather than reading isolated coefficients without their reference structure.
Ordered Categories Are Still Categories
Levels such as low, medium, and high have an order, but the spacing between them is not automatically equal.
Coding them as \(1,2,3\) and entering one numerical column imposes a constant linear step. Use indicators or scientifically justified contrasts unless that equal-step assumption is intended.
Numerical Labels Can Be Misleading
Postal codes, department numbers, product IDs, and category codes may be written with digits while remaining nominal categories.
Treating code \(300\) as three times code \(100\) creates a false numerical structure.
Rare Categories
A category with few observations can produce an unstable coefficient and wide uncertainty. Its fitted comparison may depend heavily on one case.
Combining rare categories should follow substantive similarity and a documented plan, not merely a desire for a smaller standard error.
Unbalanced Groups
Groups need not contain equal numbers of cases, but severe imbalance can reduce precision and leave some adjusted comparisons weakly supported.
With interactions, sparse combinations of categories and quantitative ranges can make fitted contrasts especially fragile.
Empty Cells
When combinations of categorical predictors are absent, some interaction effects may be unidentifiable or depend on extrapolation across unsupported cells.
Inspect the cross-tabulation of categorical predictors before fitting a complex model.
Missing Category Values
A missing value is not automatically a substantive category. Coding “missing” as a level changes the model and may mix several reasons for nonresponse.
Missing-data treatment should follow the data-generating process and the analytical target.
New Categories at Prediction Time
A fitted model cannot automatically estimate a category level that was absent from the training data. A deployment plan should specify how unknown, new, or misspelled levels are validated and handled.
Residual Diagnostics by Group
Examine residual centers, spreads, shapes, and unusual cases separately by category. A constant group offset can suggest a missing indicator; different slopes can suggest an interaction; different spreads can suggest group-specific variance.
Group-Specific Variance
The mean model may fit different groups while their residual variability differs. Conventional equal-variance standard errors may then be inappropriate.
Depending on the design and goal, the analysis may use robust uncertainty estimates, explicit variance modeling, or a more suitable model family.
Adjusted Differences Are Not Automatically Causal
A category coefficient describes a conditional association in the fitted model. It is not automatically the effect of changing a case from one group to another.
Some categories cannot be meaningfully manipulated, and observational group differences may reflect confounding, selection, measurement, or structural conditions.
Categories and Fairness
Including or excluding a sensitive category can affect predictions, subgroup errors, interpretation, and legal or ethical obligations.
A technically valid indicator model does not resolve questions of fairness, appropriate use, disparate impact, or whether the prediction target itself is legitimate.
A Reliable Coding Workflow
- Identify the response, categorical predictor, levels, and population.
- Inspect level counts, missingness, rare groups, and cross-tabulations.
- Choose a coding system that matches the intended comparisons.
- Record the reference group and every indicator definition.
- Center quantitative predictors at meaningful reference values.
- Fit lower-order terms before interpreting interactions.
- Translate coefficients into group-specific fitted equations.
- Use joint tests or planned contrasts for multi-level predictors.
- Inspect residuals, influence, and support within each group.
- Report coding, uncertainty, limitations, and causal status explicitly.
Reporting Categorical Predictors
A clear report states
- the category levels and sample sizes;
- the reference group or contrast system;
- the values at which quantitative predictors are held fixed;
- whether interactions are included;
- the fitted group comparisons and their uncertainty;
- diagnostic, support, and causal limitations.
Common Error: Treating Codes as Measurements
Entering nominal codes \(1,2,3\) as one numerical predictor imposes order and equal spacing that may not exist.
Common Error: Including Every Indicator with an Intercept
All \(k\) indicators plus an intercept create perfect collinearity. Use \(k-1\) reference-coded indicators or another full-rank parameterization.
Common Error: Forgetting the Reference Group
A category coefficient has no complete interpretation until the comparison direction and reference group are named.
Common Error: Interpreting an Interaction Main Effect Globally
With \(xD\) in the model, the \(D\) coefficient is the group difference at \(x=0\), not at every \(x\).
Common Error: Comparing Coefficients Across Codings
Reference and contrast changes alter individual coefficient meanings. Check whether the represented fitted comparisons are actually the same before declaring a result inconsistent.
Common Error: Ignoring Sparse Groups
A fitted difference supported by only a few cases or a narrow predictor range may be unstable even when software returns a coefficient.
Common Error: Calling Adjustment a Causal Effect
Holding included predictors fixed is a statistical comparison. Causal interpretation requires an appropriate causal structure, design, and measurement process.
Practice 1: Build an Indicator
Define an indicator \(D\) using “standard” as the reference and “premium” as the comparison category.
Practice 2: Interpret a Binary Model
For
\[ \hat{y}=40+7D, \]find the fitted response for each group and the comparison-minus-reference difference.
Practice 3: Interpret an Adjusted Difference
A model is
\[ \hat{y}=18+3x-5D. \]Interpret the coefficient of \(D\).
Practice 4: Count Indicator Columns
A categorical predictor has five levels. How many reference-coded indicator columns are required in a model with an intercept?
Practice 5: Identify the Dummy-Variable Trap
A model includes an intercept and indicators for North, South, and West, where every case belongs to exactly one region. Explain the problem.
Practice 6: Change the Reference Group
A two-group model with A as reference has a B-minus-A coefficient of \(9\). What is the A-minus-B coefficient when B becomes the reference?
Practice 7: Interpret an Interaction
For
\[ \hat{y}=10+2x+6D-0.5xD, \]find the fitted slope for each group and the B-minus-A difference at \(x=4\).
Practice 8: Ordered Categories
Why might coding low, medium, and high as \(1,2,3\) in one numerical column be inappropriate?
Practice 9: Joint Testing
A four-level categorical predictor is represented by three indicators. What null hypothesis tests whether the predictor contributes as a whole?
Practice 10: Critique a Causal Claim
An observational model finds a group coefficient of \(12\) after adjusting for age. An analyst says group membership causes a \(12\)-unit response change. What is missing?
Practice Answers
Practice 1
\[ D= \begin{cases} 0,&\text{standard},\\ 1,&\text{premium}. \end{cases} \]Practice 2
The reference group has fitted response \(40\). The comparison group has fitted response \(40+7=47\). The comparison-minus-reference difference is \(7\).
Practice 3
Holding \(x\) fixed, the \(D=1\) group has a fitted response \(5\) units lower than the \(D=0\) reference group.
Practice 4
\[ 5-1=4 \]indicator columns are required.
Practice 5
The three indicators sum to the intercept column for every row. This exact linear dependence creates perfect collinearity, so one region must be the omitted reference or the model must use another full-rank coding.
Practice 6
Reversing the comparison reverses the sign. The A-minus-B coefficient is \(-9\).
Practice 7
Group A has slope \(2\). Group B has slope
\[ 2-0.5=1.5. \]At \(x=4\), the B-minus-A difference is
\[ 6-0.5(4)=4. \]Practice 8
The numerical coding imposes equally spaced, constant changes from low to medium and from medium to high. Order alone does not justify that spacing or linearity.
Practice 9
If the three indicator coefficients are \(b_1,b_2,b_3\), the joint null is
\[ H_0:b_1=b_2=b_3=0. \]Practice 10
Adjustment for age alone does not establish causal identification. The groups may differ on other confounders, selection processes, measurements, or structural conditions. A causal interpretation requires a justified design and causal model.
What to Remember
A binary category can enter a regression through
\[ D\in\{0,1\}. \]In the additive model
\[ \hat{y}=b_0+b_1x+b_2D, \]\(b_2\) is the fitted comparison-minus-reference difference while \(x\) is held fixed.
- Use \(k-1\) indicators for \(k\) levels with an intercept.
- Name the reference group and coding system.
- Different full-rank codings can produce identical fitted values.
- Use joint tests for a multi-level predictor as a whole.
- Interactions allow group differences or slopes to vary.
- Do not treat nominal codes as quantitative measurements.
- Inspect support, residuals, and uncertainty within each group.
- Adjusted group differences are not automatically causal effects.
Where We Go Next
Indicator variables expand regression across groups. The fitted relationship within a group may still curve, level off, accelerate, or change shape in ways that a straight-line term cannot capture.
The next lesson, Nonlinear Relationships, develops transformations, polynomial terms, piecewise structure, and flexible methods for representing relationships whose fitted change is not constant.