The Central Idea
A loss function assigns a numerical penalty to the result of a decision or prediction.
In general, we write
\[ \boxed{L(a,\theta)} \]where \(a\) is the action taken and \(\theta\) represents the true state, target, or quantity of interest.
Smaller values are preferred.
Loss Turns Consequences into an Objective
A decision problem may involve many kinds of undesirable outcomes:
- prediction error;
- financial cost;
- delay;
- misclassification;
- failure;
- shortage;
- excess inventory;
- or another measurable penalty.
A loss function puts those consequences onto a numerical scale that can be compared and minimized.
Loss and Utility Point in Opposite Directions
Utility and loss are closely related but use opposite conventions.
With utility:
\[ \text{larger is better}. \]With loss:
\[ \text{smaller is better}. \]In simple cases, a loss function can be defined as the negative of utility, but the two concepts are not always introduced on the same numerical scale.
Zero Loss Represents a Perfect Result
Many loss functions are designed so that the best possible outcome has loss \(0\).
For example, if \(\hat y\) is a prediction and \(y\) is the true value, a common structure is
\[ L(\hat y,y)=0 \]whenever
\[ \hat y=y. \]Loss Need Not Be Monetary
A loss function may measure dollars, but it does not have to.
Loss can represent:
- squared error;
- absolute error;
- number of mistakes;
- time;
- energy;
- risk score;
- or another decision-relevant penalty.
Prediction Error
Suppose a model predicts \(\hat y\) and the true value is \(y\).
The prediction error is often written
\[ e=\hat y-y. \]A loss function converts this error into a nonnegative penalty.
Absolute Loss
The absolute loss is
\[ \boxed{L(\hat y,y)=|\hat y-y|}. \]It measures the distance between the prediction and the true value.
Example: Absolute Loss
Suppose the true demand is
\[ y=120 \]and the forecast is
\[ \hat y=135. \]Then
\[ L(\hat y,y)=|135-120| \] \[ =|15| \] \[ \boxed{15}. \]Absolute Loss Treats Equal-Sized Errors Equally
Under absolute loss,
\[ |135-120|=15 \]and
\[ |105-120|=15. \]Overprediction and underprediction by the same amount receive the same loss.
Squared Loss
The squared loss is
\[ \boxed{L(\hat y,y)=(\hat y-y)^2}. \]Squaring makes the loss nonnegative and penalizes large errors more heavily than small ones.
Example: Squared Loss
If
\[ y=120,\qquad \hat y=135, \]then
\[ L(\hat y,y)=(135-120)^2 \] \[ =15^2 \] \[ \boxed{225}. \]Why Squared Loss Emphasizes Large Errors
Compare errors of \(2\) and \(10\).
Their absolute losses are
\[ 2\qquad\text{and}\qquad10. \]Their squared losses are
\[ 4\qquad\text{and}\qquad100. \]The larger error receives a disproportionately larger penalty.
Zero-One Loss
For classification decisions, a simple loss function is zero-one loss:
\[ \boxed{ L(\hat y,y)= \begin{cases} 0, & \hat y=y,\\ 1, & \hat y\ne y. \end{cases}} \]A correct classification receives loss \(0\), and an incorrect classification receives loss \(1\).
Example: Zero-One Loss
Suppose a classifier predicts either approved or not approved.
If the prediction matches the actual category, loss is
\[ 0. \]If it does not match, loss is
\[ 1. \]Zero-One Loss Counts Mistakes
Across many classifications, average zero-one loss equals the fraction of cases classified incorrectly.
If \(18\) of \(100\) classifications are wrong, the average zero-one loss is
\[ \frac{18}{100}=0.18. \]The corresponding accuracy is
\[ 1-0.18=0.82. \]Not All Mistakes Have the Same Consequence
Zero-one loss treats every incorrect classification equally.
In many real problems, however, different errors have different costs.
For example, a false alarm may be inconvenient while a missed critical failure may be much more serious.
Asymmetric Loss
An asymmetric loss function assigns different penalties to different directions or types of error.
For example,
\[ L(\hat y,y)= \begin{cases} 2|\hat y-y|, & \hat y<y,\\ |\hat y-y|, & \hat y\ge y. \end{cases} \]Here, underprediction is penalized twice as heavily as overprediction.
Why Asymmetric Loss Matters
In an inventory problem, ordering too little may cause lost sales while ordering too much may create storage costs.
If the consequences are not equally severe, a symmetric loss function can misrepresent the decision.
Piecewise Loss Functions
Loss functions are often piecewise because different regions of an outcome have different consequences.
A simple threshold loss might be
\[ L(x)= \begin{cases} 0, & x\le c,\\ x-c, & x>c. \end{cases} \]No loss is incurred below the threshold \(c\), while excess above \(c\) generates a penalty.
Loss Matrices for Classification
Classification losses can be organized in a table.
| Decision | True State: Negative | True State: Positive |
|---|---|---|
| Predict Negative | 0 | 8 |
| Predict Positive | 2 | 0 |
In this example, a false negative has loss \(8\), while a false positive has loss \(2\).
Expected Loss
When the true state is uncertain, evaluate a decision by its expected loss.
\[ \boxed{E[L]=\sum_i p_iL_i}. \]The preferred decision is the one with the smallest expected loss.
Example: Expected Classification Loss
Suppose the probability of a positive state is
\[ P(+)=0.3 \]and the probability of a negative state is
\[ P(-)=0.7. \]Use the loss matrix:
| Decision | Negative | Positive |
|---|---|---|
| Predict Negative | 0 | 8 |
| Predict Positive | 2 | 0 |
Expected Loss of Predicting Negative
If we predict negative,
\[ E[L\mid\text{predict negative}] =0.7(0)+0.3(8) \] \[ =2.4. \]Therefore,
\[ \boxed{E[L\mid\text{negative}]=2.4}. \]Expected Loss of Predicting Positive
If we predict positive,
\[ E[L\mid\text{predict positive}] =0.7(2)+0.3(0) \] \[ =1.4. \]Therefore,
\[ \boxed{E[L\mid\text{positive}]=1.4}. \]Since \(1.4<2.4\), predicting positive has the smaller expected loss.
Risk as Expected Loss
In statistical decision theory, the expected loss associated with a decision rule is often called its risk.
In a simplified form,
\[ \boxed{R=E[L]}. \]The goal is to choose a decision rule with small risk.
Decision Rules
A decision rule tells us what action to take for each possible observation or information state.
If data are denoted by \(x\), a decision rule may be written
\[ \delta(x). \]The resulting loss depends on both the rule's chosen action and the true state.
Risk Function
A more formal risk function is
\[ \boxed{R(\theta,\delta)=E_\theta[L(\delta(X),\theta)]}. \]This means: when the true state is \(\theta\), average the loss produced by decision rule \(\delta\) over the possible data \(X\).
Empirical Loss
In data analysis and machine learning, we often evaluate a model using the average loss over observed cases.
\[ \boxed{ \frac{1}{n}\sum_{i=1}^{n}L(\hat y_i,y_i) }. \]This is commonly called average empirical loss or empirical risk.
Example: Average Absolute Loss
Suppose prediction errors have absolute values
\[ 2,\qquad5,\qquad1,\qquad4. \]The average absolute loss is
\[ \frac{2+5+1+4}{4} \] \[ =\frac{12}{4} \] \[ \boxed{3}. \]Example: Average Squared Loss
Suppose the prediction errors are
\[ 2,\qquad -1,\qquad3. \]Squared losses are
\[ 4,\qquad1,\qquad9. \]Their average is
\[ \frac{4+1+9}{3} \] \[ =\frac{14}{3} \] \[ \boxed{\frac{14}{3}\approx4.67}. \]Mean Squared Error
When squared loss is averaged over observations, the result is the mean squared error, or MSE:
\[ \boxed{ \operatorname{MSE} =\frac{1}{n}\sum_{i=1}^{n}(\hat y_i-y_i)^2 }. \]MSE is widely used because it strongly penalizes large errors and works well with many optimization methods.
Mean Absolute Error
When absolute loss is averaged over observations, the result is the mean absolute error, or MAE:
\[ \boxed{ \operatorname{MAE} =\frac{1}{n}\sum_{i=1}^{n}|\hat y_i-y_i| }. \]MAE is usually less sensitive to extreme errors than MSE.
Squared Loss and the Mean
Suppose we must choose a single number \(a\) to predict a random variable \(X\).
Under squared loss,
\[ L(a,X)=(a-X)^2. \]The value of \(a\) that minimizes expected squared loss is the mean:
\[ \boxed{a=E[X]}. \]Absolute Loss and the Median
Under absolute loss,
\[ L(a,X)=|a-X|. \]A median of \(X\) minimizes expected absolute loss.
This helps explain why the mean and median naturally arise from different optimization criteria.
Zero-One Loss and the Mode
If the action is to choose one category and zero-one loss is used, expected loss is minimized by selecting the most probable category.
In other words, choose a mode of the relevant probability distribution.
Different Loss Functions Produce Different Best Decisions
There is no universal best predictor independent of the loss function.
For a distribution:
- squared loss points toward the mean;
- absolute loss points toward the median;
- zero-one loss points toward the mode.
The objective determines what counts as optimal.
The Loss Function Encodes What Matters
Choosing a loss function is a modeling decision.
It tells the mathematics which errors or outcomes should matter most.
A poorly chosen loss function can optimize the wrong objective even when the calculations are performed perfectly.
Scaling a Loss Function
Multiplying every loss by the same positive constant does not change which decision minimizes loss.
If
\[ L_2=aL_1,\qquad a>0, \]then minimizing \(L_1\) and minimizing \(L_2\) produce the same ranking of decisions.
Adding a Constant to Every Loss
Adding the same constant to every possible loss also preserves the ranking.
If
\[ L_2=L_1+b, \]then every decision receives the same additional amount \(b\), so the minimizing decision is unchanged.
Loss and Decision Tables
A decision table can contain losses instead of payoffs.
| Decision | Low Demand | High Demand |
|---|---|---|
| Small Run | 2 | 6 |
| Large Run | 12 | 1 |
The direction of preference is reversed from a payoff table: smaller entries are better.
Loss and Decision Trees
A decision tree can also be evaluated with losses.
At chance nodes, compute expected loss:
\[ \sum_i p_iL_i. \]At decision nodes, choose the branch with the smallest expected loss.
Expected Loss and Expected Utility Are Parallel Ideas
Expected utility chooses the largest expected utility:
\[ \max E[U]. \]Expected loss chooses the smallest expected loss:
\[ \min E[L]. \]These are two different numerical languages for ranking uncertain alternatives.
Common Error: Forgetting What the Loss Measures
A numerical loss is meaningful only if we know what it represents.
A loss of \(10\) could mean:
- $10;
- 10 squared units;
- 10 minutes;
- 10 penalty points;
- or something else entirely.
Always identify the units and interpretation.
Common Error: Confusing Error with Loss
Error and loss are not necessarily the same quantity.
If
\[ e=\hat y-y, \]then squared loss is
\[ e^2, \]while absolute loss is
\[ |e|. \]The loss function transforms the error.
Common Error: Assuming Symmetry
A symmetric loss assigns the same penalty to equal-sized errors in opposite directions.
Many practical decisions are asymmetric, so this assumption should be checked rather than taken for granted.
Common Error: Minimizing the Wrong Quantity
If the table contains payoffs, we usually maximize.
If the table contains losses, we usually minimize.
Confusing these conventions reverses the decision.
Common Error: Ignoring Rare but Severe Losses
A low-probability event may still contribute materially to expected loss when its consequence is very large.
For example,
\[ 0.001(1{,}000{,}000)=1{,}000. \]Small probability does not automatically mean negligible expected impact.
Common Error: Using Training Loss as the Entire Objective
In predictive modeling, a low loss on observed data does not guarantee low loss on new data.
The purpose of a predictive model is usually to perform well beyond the observations used to fit it.
This is one reason validation and generalization matter.
A Reliable Procedure for Building a Loss Function
- Identify the decision or prediction being evaluated.
- Identify the possible true states or outcomes.
- Determine what kinds of errors or consequences matter.
- Decide whether different errors should be treated symmetrically.
- Assign numerical penalties.
- Check that smaller values represent better outcomes.
- Use probabilities to compute expected loss when uncertainty is present.
- Choose the action or rule that minimizes the relevant expected loss.
Practice 1: Absolute Loss
A forecast is \(72\), and the true value is \(65\).
Find the absolute loss.
Practice 2: Squared Loss
A forecast is \(72\), and the true value is \(65\).
Find the squared loss.
Practice 3: Compare Two Errors
Compare errors of \(3\) and \(9\) under:
- absolute loss;
- squared loss.
Practice 4: Zero-One Loss
A classifier makes \(12\) incorrect predictions out of \(80\).
Find its average zero-one loss.
Practice 5: Expected Loss
A decision produces loss \(4\) with probability \(0.7\) and loss \(15\) with probability \(0.3\).
Find the expected loss.
Practice 6: Compare Decisions by Expected Loss
Decision A has losses \(2\) and \(10\) with probabilities \(0.6\) and \(0.4\).
Decision B has losses \(5\) and \(6\) with probabilities \(0.6\) and \(0.4\).
Which decision has smaller expected loss?
Practice 7: Asymmetric Loss
Use
\[ L(\hat y,y)= \begin{cases} 3|\hat y-y|, & \hat y<y,\\ |\hat y-y|, & \hat y\ge y. \end{cases} \]Find the loss when \(y=50\) and \(\hat y=46\).
Practice 8: Mean Absolute Error
The absolute errors are
\[ 1,\qquad4,\qquad2,\qquad5. \]Find the MAE.
Practice 9: Mean Squared Error
The errors are
\[ 1,\qquad-2,\qquad3. \]Find the MSE.
Practice 10: Choose the Appropriate Center
Which measure minimizes each loss when choosing one constant prediction for a distribution?
- squared loss;
- absolute loss;
- zero-one loss for categories.
Practice Answers
Practice 1
\[ |72-65|=|7|=\boxed{7}. \]Practice 2
\[ (72-65)^2=7^2=\boxed{49}. \]Practice 3
Under absolute loss:
\[ 3,\qquad9. \]Under squared loss:
\[ 3^2=9,\qquad9^2=81. \]Squared loss magnifies the larger error much more strongly.
Practice 4
\[ \frac{12}{80}=0.15. \]Therefore,
\[ \boxed{0.15}. \]Practice 5
\[ E[L]=0.7(4)+0.3(15) \] \[ =2.8+4.5 \] \[ \boxed{7.3}. \]Practice 6
Decision A:
\[ E[L_A]=0.6(2)+0.4(10) \] \[ =1.2+4 \] \[ =5.2. \]Decision B:
\[ E[L_B]=0.6(5)+0.4(6) \] \[ =3+2.4 \] \[ =5.4. \]Therefore,
\[ \boxed{\text{Decision A has the smaller expected loss}}. \]Practice 7
Because \(46<50\), use the underprediction penalty:
\[ L=3|46-50| \] \[ =3(4) \] \[ \boxed{12}. \]Practice 8
\[ \operatorname{MAE} =\frac{1+4+2+5}{4} \] \[ =\frac{12}{4} \] \[ \boxed{3}. \]Practice 9
\[ \operatorname{MSE} =\frac{1^2+(-2)^2+3^2}{3} \] \[ =\frac{1+4+9}{3} \] \[ =\boxed{\frac{14}{3}\approx4.67}. \]Practice 10
- Squared loss: the mean.
- Absolute loss: a median.
- Zero-one categorical loss: a mode.
What to Remember
A loss function tells the mathematics what counts as a bad outcome and how severely different mistakes should be penalized.
- Loss is minimized rather than maximized.
- Absolute loss measures error magnitude directly.
- Squared loss penalizes large errors more strongly.
- Zero-one loss records whether a classification is correct.
- Asymmetric loss allows different errors to carry different penalties.
- Expected loss weights each loss by its probability.
- Risk is often defined as expected loss.
- Squared loss naturally leads to the mean.
- Absolute loss naturally leads to the median.
- The choice of loss function determines what the model or decision rule is optimizing.
The most important modeling principle is: choose a loss function that reflects the consequences that actually matter in the decision.
Where We Go Next
Loss functions provide a numerical language for mistakes and undesirable outcomes. Once losses are defined, we can compare alternatives by the risks they create.
The next lesson, Risk and Uncertainty, develops the broader distinction between known probabilities, unknown conditions, variability, exposure, and the ways decision makers reason when outcomes cannot be predicted with certainty.