Classification

The Central Idea

A classifier uses a feature vector \(\mathbf{x}\) to estimate how plausible each class is, then applies a rule that selects a class.

Probability estimation and class assignment are related, but they are not the same task. The distinction is essential because different decisions can use the same probability model with different thresholds.

Classification versus Regression

Binary Classification

A binary response has two classes. We often encode the event of interest as \(Y=1\) and the other outcome as \(Y=0\). Examples include

• default versus no default;
• fraud versus legitimate transaction;
• disease present versus disease absent;
• customer leaves versus customer remains.

The labels \(0\) and \(1\) are codes. They do not imply that one class is intrinsically more valuable than the other.

The Positive Class

The class designated \(Y=1\) is commonly called the positive class. Positive does not necessarily mean desirable. Fraud, equipment failure, and disease may all be coded positive because they are the events an analysis is designed to detect.

State the positive class before reporting sensitivity, specificity, precision, or recall. These measures change meaning if the class labels are reversed.

Conditional Class Probability

For binary classification, the central statistical quantity is

This is the probability of the positive class among cases having features \(\mathbf{x}\), under the model and the population represented by the data.

Why Ordinary Linear Regression Is Problematic

If a \(0\)-\(1\) response is modeled by

the fitted values can fall below \(0\) or above \(1\). The error variance also changes with the probability because

A probability model should respect the probability scale and its changing variance.

Odds

A probability \(p\) can be expressed as odds:

If \(p=0.75\), the odds are

This means three expected positive outcomes for every one expected negative outcome among comparable cases. It does not mean a probability of \(3\).

From Odds Back to Probability

If the odds are \(o\), then

Odds of \(4\) correspond to

Log-Odds

The logarithm of the odds is called the logit:

Probabilities between \(0\) and \(1\) map to log-odds anywhere on the real line. This makes it possible to model log-odds with a linear predictor.

Logistic Regression

Logistic regression assumes that the log-odds of the positive class are linear in the predictors:

The right side may also include indicator variables, interactions, polynomial terms, or spline terms. “Linear” refers to the coefficients, not necessarily to a straight relationship on the probability scale.

The Logistic Function

Let

Solving the logit equation for \(p\) gives

The logistic function converts any real value of \(\eta\) into a valid probability.

The Logistic Curve

Interpreting a Logistic Coefficient

Holding all other predictors fixed, a one-unit increase in \(x_j\) changes the log-odds by \(\beta_j\) and multiplies the odds by

If \(\beta_j=0.40\), the odds ratio is

The fitted odds are about \(49.2\%\) higher for a one-unit increase in \(x_j\), conditional on the other predictors. This is not a constant \(49.2\)-percentage-point increase in probability.

Probability Changes Are Not Constant

In a one-predictor logistic model, the fitted probability slope is

The same coefficient can produce a small probability change near \(p=0\) or \(p=1\) and a larger change near \(p=0.5\). Report probability changes at relevant predictor values when stakeholders think in probabilities.

Indicator Predictors

Suppose \(G=1\) identifies a group and \(G=0\) is the reference group:

Then \(e^{\beta_1}\) is the fitted odds ratio comparing \(G=1\) with \(G=0\). With additional predictors, the comparison holds those predictors fixed.

Interactions and Nonlinear Terms

Classification models can represent changing effects:

The interaction allows the association between \(x\) and the log-odds to differ by group. The quadratic term permits curvature. Preserve lower-order terms when including interactions or powers unless a strong structural reason supports a restriction.

Estimating the Coefficients

Logistic regression is ordinarily fitted by maximum likelihood. For observations \(i=1,\ldots,n\), with \(y_i\in\{0,1\}\), the likelihood is

The fitted coefficients make the observed class outcomes as likely as possible under the model. Numerical algorithms perform the optimization.

Log Loss

The average negative log-likelihood is called log loss or cross-entropy:

Smaller values are better. Log loss rewards accurate probabilities and penalizes confident errors severely.

A Worked Probability Model

Suppose a fitted model for late payment uses debt-to-income percentage \(x\):

For an applicant with \(x=30\),

The model estimates a \(26.9\%\) probability of late payment for a case with this predictor value.

Interpret the Worked Coefficient

A one-percentage-point increase in debt-to-income percentage multiplies the fitted odds of late payment by

Thus, the fitted odds increase by about \(10.5\%\) per percentage point, within the model and holding any other included predictors fixed.

The \(50\%\) Probability Point

The logistic probability is \(0.5\) when the linear predictor is \(0\). In the worked model,

so

This point describes the model. It is not automatically the best decision threshold or an operational boundary.

From Probability to a Class

A threshold \(t\) converts the fitted probability into a label:

The conventional threshold \(t=0.5\) is only one option. The appropriate threshold depends on the purpose of the decision and the consequences of each error.

Changing the Threshold

• Lowering \(t\) predicts more positives.
• Lowering \(t\) usually catches more actual positives.
• Lowering \(t\) also usually creates more false alarms.
• Raising \(t\) produces the opposite tradeoff.

Threshold selection is a decision-policy choice, not a hidden property of the fitted model.

The Confusion Matrix

For a fixed threshold, compare predicted and actual classes:

A Worked Confusion Matrix

Suppose a screening classifier is evaluated on \(1{,}000\) cases:

Sensitivity or Recall

Sensitivity is the proportion of actual positives correctly identified:

In the example,

The classifier detects \(80\%\) of the cases with the condition.

Specificity

Specificity is the proportion of actual negatives correctly identified:

In the example,

Precision or Positive Predictive Value

Precision is the proportion of predicted positives that are actually positive:

In the example,

Forty percent of positive classifications are true positives in this evaluation population.

Negative Predictive Value

Negative predictive value is the proportion of predicted negatives that are actually negative:

In the example,

Accuracy

Accuracy is the overall proportion classified correctly:

In the example,

Why Accuracy Can Mislead

Only \(90\) of the \(1{,}000\) cases in the example are positive, so the positive prevalence is \(9\%\). A rule that predicts every case negative would have

accuracy while detecting none of the positive cases. Always compare a classifier with simple baselines and examine class-specific performance.

False-Positive and False-Negative Rates

In the worked example, the false-positive rate is about \(0.119\), and the false-negative rate is \(0.20\).

Balanced Accuracy

Balanced accuracy gives equal weight to performance in the two actual classes:

For the worked example,

The \(F_1\) Score

The \(F_1\) score is the harmonic mean of precision and recall:

For the worked example,

\(F_1\) omits true negatives, so it should not be treated as a universal performance measure.

Conditioning Directions Matter

Sensitivity and precision answer different questions:

Reversing the condition reverses the question. High sensitivity does not guarantee high precision.

Prevalence Affects Predictive Values

Sensitivity and specificity condition on the actual class. Precision and negative predictive value condition on the predicted class and therefore change when class prevalence changes.

A screening rule can have the same sensitivity and specificity in two populations but a lower precision where the condition is rarer.

Choose Metrics from the Decision

Decision Costs

Let \(C_{\mathrm{FP}}\) be the cost of a false positive and \(C_{\mathrm{FN}}\) the cost of a false negative. A decision policy can be evaluated by expected cost:

with additional terms when correct decisions also have material benefits or costs. Cost assumptions should be documented and sensitivity-tested.

A Cost-Based Threshold

In a simplified setting with correct calibrated probabilities and zero cost for correct decisions, predict positive when

Solving gives

If a false negative costs four times as much as a false positive, the simplified threshold is \(1/(1+4)=0.20\), not \(0.50\).

The ROC Curve

A receiver operating characteristic curve plots

as the classification threshold varies. Each point represents a different sensitivity–specificity tradeoff. The curve evaluates ranking behavior across thresholds, not the quality of one selected operating threshold.

Area under the ROC Curve

The ROC area, or AUC, can be interpreted as the probability that a randomly selected positive case receives a higher score than a randomly selected negative case, with ties handled appropriately.

• \(\mathrm{AUC}=0.5\) corresponds to random ranking.
• \(\mathrm{AUC}=1\) corresponds to perfect ranking.
• A high AUC does not guarantee calibrated probabilities.
• A high AUC does not identify the best operational threshold.

Precision–Recall Curves

A precision–recall curve plots precision against recall as the threshold changes. It focuses attention on positive predictions and can be especially informative when the positive class is rare.

Unlike ROC analysis, the baseline precision depends on positive-class prevalence. Comparisons across populations therefore require care.

Ranking Is Not Calibration

A model has strong discrimination if it tends to rank positive cases above negative cases. A model is calibrated if predicted probabilities agree with observed frequencies.

A model can rank cases perfectly while reporting probabilities that are systematically too high or too low. Evaluate both capabilities when the probabilities themselves guide decisions.

Calibration

Informally, a calibrated model should satisfy

Among cases assigned probabilities near \(0.70\), about \(70\%\) should be positive over repeated comparable cases. Calibration plots compare observed frequencies with predicted probabilities across ranges of risk.

The Brier Score

For binary outcomes, the Brier score is the mean squared probability error:

Smaller values are better. Unlike label accuracy, the Brier score uses the full probability prediction and reflects both calibration and resolution.

Training and Test Performance

Performance calculated on the same data used to fit the model is usually optimistic. A credible evaluation uses observations not used to estimate the model:

All feature selection, transformations, imputation, scaling, and threshold tuning must be learned without using the final test outcomes.

Data Leakage

Leakage occurs when information unavailable at prediction time, or information from the evaluation outcomes, enters model training. Examples include

• using a post-outcome variable as a predictor;
• scaling all observations before splitting the data;
• choosing features after examining test performance;
• including duplicate or closely related cases across splits.

Leakage can produce excellent reported metrics and poor real-world performance.

Respect the Data Structure

Random row splitting is not always appropriate. Use time-based splits for future prediction, group-based splits when observations from the same person or organization are related, and location-based evaluation when geographic transfer matters.

The evaluation design should reproduce the intended deployment boundary.

Class Imbalance

When one class is rare, a model may achieve high accuracy by favoring the common class. Responses include

• using class-specific and probability-based metrics;
• adjusting the operational threshold;
• using case weights when they match the target objective;
• resampling only within training folds;
• collecting more informative positive examples when feasible.

Artificially changing class proportions can distort probability calibration unless the shift is explicitly corrected.

Complete and Quasi-Complete Separation

In logistic regression, separation occurs when a predictor or combination of predictors perfectly distinguishes the observed classes. Ordinary maximum-likelihood coefficient estimates may then grow without a finite limit.

Diagnose separation rather than interpreting enormous coefficients literally. Penalized estimation, revised predictors, or additional information may be appropriate.

Other Classification Methods

Classification Boundaries

A decision boundary is the set of feature values where the selected class changes. For a logistic model with threshold \(0.5\),

is a straight boundary in the \(x_1,x_2\) plane. Interactions, polynomial terms, splines, kernels, and trees can create curved or segmented boundaries.

Multiclass Classification

A multiclass response has \(K>2\) possible classes. The model estimates

subject to

The highest-probability class may be selected, but costs or abstention rules can justify another policy.

Softmax Probabilities

Multinomial logistic regression assigns each class a score \(\eta_k\) and converts the scores to probabilities using

The probabilities are nonnegative and sum to \(1\). One class is commonly treated as a reference for identifiable coefficient interpretation.

Multiclass Evaluation

A \(K\times K\) confusion matrix records actual and predicted classes. Class-specific precision and recall can be combined in several ways:

• macro averaging gives each class equal weight;
• weighted macro averaging weights by class frequency;
• micro averaging pools decisions before calculating the metric.

State the averaging convention because the same model can receive materially different summaries.

Ordinal Classification

Some categories have a natural order, such as low, medium, and high. Treating them as unrelated classes discards that order; treating them as equally spaced numbers adds an assumption that may be false.

Ordinal regression models cumulative probabilities while respecting the ranking of the categories.

Abstention and Human Review

A system need not force a label for every case. It can defer uncertain or high-impact cases:

A review option can improve risk control, but its workload, delay, and reviewer consistency must be measured.

Fairness Requires More Than One Metric

Performance may differ across groups because of sample composition, measurement quality, historical decisions, prevalence, or model behavior. Evaluate relevant error rates, calibration, coverage, and downstream effects by group.

Fairness criteria can conflict mathematically when group prevalences differ. The appropriate governance standard depends on the application, rights at stake, and legal and ethical requirements.

Association Is Not Causation

A classifier predicts an observed label from available features. A coefficient, feature importance, or accurate prediction does not establish that changing a predictor would change the outcome.

Causal decisions require a defined intervention, suitable study design, and defensible assumptions beyond predictive accuracy.

Dataset Shift

A model can degrade when deployment differs from development:

• covariate shift: predictor distributions change;
• prior shift: class prevalence changes;
• concept drift: the relationship between features and outcome changes;
• measurement shift: variables are recorded differently.

Predeployment validation cannot eliminate the need for postdeployment monitoring.

Deployment Monitoring

A production classification system should track

• input quality and missingness;
• score and class distributions;
• delayed outcome metrics when labels arrive;
• calibration and group-specific performance;
• review volume, overrides, and downstream consequences.

Define alert thresholds, ownership, escalation paths, and retraining criteria before deployment.

A Practical Classification Workflow

1. Define the outcome, positive class, prediction time, and unit of analysis.
2. Specify the decision, stakeholders, error costs, and constraints.
3. Audit label quality, feature availability, missingness, and class balance.
4. Create training, validation, and test partitions that match deployment.
5. Build a transparent baseline before adding complexity.
6. Fit models without leaking evaluation information.
7. Evaluate discrimination, calibration, class-specific errors, and uncertainty.
8. Select and document the threshold or abstention policy.
9. Test robustness across groups, time periods, and plausible shifts.
10. Monitor the deployed system and maintain a rollback path.

Reporting Checklist

• What exactly are the classes, and which is positive?
• When is the prediction made, and what information is available then?
• How were observations partitioned for evaluation?
• What baseline and competing models were considered?
• Which probability and label metrics were used, and why?
• How was the operating threshold selected?
• How stable are results across relevant subgroups and time periods?
• What monitoring and intervention controls govern use?

Common Error: Treating \(0.5\) as Universal

A \(0.5\) threshold assumes a particular decision structure. It can be unsuitable when classes are imbalanced, error costs differ, resources are constrained, or review is available.

Common Error: Reporting Only Accuracy

Accuracy hides which class is being missed and can reward trivial majority-class predictions. Report metrics aligned with the operational decision and include the confusion matrix at the selected threshold.

Common Error: Calling Scores Probabilities

Some algorithms produce ranking scores that are not calibrated probabilities. A number between \(0\) and \(1\) is not automatically a trustworthy risk estimate. Calibration must be evaluated on relevant unseen data.

Common Error: Tuning on the Test Set

Repeatedly selecting models or thresholds based on test performance turns the test set into training information. Reserve the final test set for one honest evaluation after modeling decisions are fixed.

Practice 1: Convert Probability and Odds

A model assigns probability \(p=0.20\) to the positive class. Find the odds. Then convert odds of \(3\) back to a probability.

Practice 2: Calculate a Logistic Probability

For

find \(\hat{p}\) when \(x=4\).

Practice 3: Interpret an Odds Ratio

A logistic coefficient is \(\beta=0.25\). Interpret \(e^\beta\).

Practice 4: Change the Threshold

What generally happens to sensitivity and the false-positive rate when the decision threshold is lowered?

Practice 5: Calculate Metrics

A classifier produces \(\mathrm{TP}=45\), \(\mathrm{FP}=15\), \(\mathrm{FN}=5\), and \(\mathrm{TN}=135\). Find sensitivity, specificity, precision, and accuracy.

Practice 6: Diagnose Accuracy

In a dataset, \(99\%\) of cases are negative. A classifier predicts every case negative. Explain why its \(99\%\) accuracy is inadequate evidence of usefulness.

Practice 7: Select a Cost Threshold

Under the simplified cost model, a false positive costs \(2\) units and a false negative costs \(8\) units. Find the probability threshold.

Practice 8: Distinguish Ranking and Calibration

A model ranks nearly every positive case above every negative case, but cases assigned probability \(0.80\) are positive only \(50\%\) of the time. What is strong, and what is weak?

Practice 9: Identify Leakage

An analyst imputes missing values using the full dataset and then creates training and test sets. What is the problem?

Practice 10: Evaluate a Causal Claim

A classifier accurately predicts employee turnover from commute length. A manager concludes that shortening commutes will reduce turnover by the amount implied by the model. What is wrong with this conclusion?

Practice Answers

Practice 1

Odds of \(3\) correspond to

Practice 2

Practice 3

Holding the other included predictors fixed, a one-unit increase in the predictor multiplies the fitted odds by about \(1.284\), an increase of about \(28.4\%\). It is not a constant probability increase.

Practice 4

More cases are labeled positive. Sensitivity generally rises, and the false-positive rate generally also rises.

Practice 5

Practice 6

The classifier has zero sensitivity and detects none of the positive cases. Its accuracy merely reflects the class imbalance and matches the trivial majority-class baseline.

Practice 7

Practice 8

Discrimination or ranking is strong, but calibration is poor because the stated probabilities exceed the observed event frequency.

Practice 9

The imputation values use information from the test data. Imputation should be fitted on the training data and then applied unchanged to the test data, ordinarily within a complete modeling pipeline.

Practice 10

Predictive association does not identify the causal effect of changing commute length. Confounding, selection, reverse causation, and the feasibility of the intervention require separate causal analysis.

What to Remember

• Classification predicts categories; many classifiers first estimate probabilities.
• Logistic regression models log-odds and keeps fitted probabilities between \(0\) and \(1\).
• Exponentiating a logistic coefficient gives a conditional odds ratio.
• Probability estimation and threshold-based class assignment are distinct.
• A confusion matrix separates the four kinds of binary decisions.
• Sensitivity, specificity, precision, and accuracy answer different questions.
• Class prevalence affects precision and negative predictive value.
• Discrimination and calibration are separate model capabilities.
• Evaluation must use unseen data and prevent leakage.
• Deployment requires monitoring, governance, and attention to dataset shift.

Where We Go Next

Classification emphasizes categorical outcomes and probability-based decisions. For quantitative outcomes, a point prediction alone does not communicate how uncertain a future response may be.

The next lesson, Prediction Intervals, develops intervals designed to contain future individual outcomes and distinguishes them from confidence intervals for a mean response.