The Central Idea
A hypothesis test can produce a correct decision or an incorrect decision.
There are two basic ways to be wrong:
- reject a true null hypothesis;
- fail to reject a false null hypothesis.
These are called Type I and Type II errors.
The Four Possible Outcomes
A statistical decision can be compared with the actual state of the world.
| Decision | \(H_0\) true | \(H_0\) false |
|---|---|---|
| Fail to reject \(H_0\) | Correct decision | Type II error |
| Reject \(H_0\) | Type I error | Correct decision |
Type I Error
A Type I error occurs when we reject the null hypothesis even though it is true.
Symbolically,
\[ \boxed{ \text{Type I error} = \text{reject }H_0 \text{ when }H_0\text{ is true} }. \]Type I Error as a False Positive
A Type I error is often called a false positive.
The test reports evidence of a departure from the null when no such departure actually exists in the modeled population.
Probability of a Type I Error
The probability of a Type I error is controlled by the significance level
\[ \alpha. \]Thus,
\[ \boxed{ \alpha = P(\text{reject }H_0\mid H_0\text{ true}) }. \]Example: Type I Error
Suppose
\[ H_0: \text{the new treatment has no effect}. \]A Type I error occurs if the study concludes that the treatment has an effect when, in fact, the null hypothesis is true.
Type II Error
A Type II error occurs when we fail to reject the null hypothesis even though a specified alternative is true.
Symbolically,
\[ \boxed{ \text{Type II error} = \text{fail to reject }H_0 \text{ when }H_A\text{ is true} }. \]Type II Error as a False Negative
A Type II error is often called a false negative.
A real effect exists, but the statistical procedure fails to detect it.
Probability of a Type II Error
The probability of a Type II error is denoted
\[ \beta. \]Thus,
\[ \boxed{ \beta = P(\text{fail to reject }H_0\mid H_A\text{ true}) }. \]Beta Depends on the Alternative
Unlike \(\alpha\), which is usually fixed by the testing procedure, \(\beta\) generally depends on which specific alternative value is true.
For example, when testing
\[ H_0:\mu=100, \]the probability of missing
\[ \mu=101 \]is usually different from the probability of missing
\[ \mu=120. \]Statistical Power
The probability of correctly rejecting a false null hypothesis is called power.
\[ \boxed{ \text{Power} = 1-\beta }. \]Equivalently,
\[ \boxed{ \text{Power} = P(\text{reject }H_0\mid H_A\text{ true}) }. \]Correct Decisions
There are also two ways to be correct:
- fail to reject a true null hypothesis;
- reject a false null hypothesis.
The second probability is the statistical power of the test at the specified alternative.
Alpha Is Not the Probability the Null Is True
The significance level
\[ \alpha \]does not mean there is an \(\alpha\) probability that the null hypothesis is true.
It is conditional on the null already being true.
Beta Is Not the Probability the Alternative Is True
Likewise,
\[ \beta \]is not the probability that the alternative hypothesis is true.
It is the probability of failing to reject the null given a specified alternative is true.
Errors Are Conditional Probabilities
The key formulas are:
\[ \boxed{ \alpha = P(\text{reject }H_0\mid H_0) } \] \[ \boxed{ \beta = P(\text{fail to reject }H_0\mid H_A) }. \]The conditions after the vertical bar are essential.
Example: Medical Screening Analogy
Suppose the null represents
\[ H_0: \text{no disease}. \]Then:
- Type I error: healthy person classified as diseased;
- Type II error: diseased person classified as healthy.
This analogy is useful, although formal diagnostic testing terminology may involve sensitivity and specificity rather than hypothesis-testing language.
Example: Manufacturing
Suppose
\[ H_0: \text{the production process is correctly calibrated}. \]Then:
- Type I error: stop or adjust a properly calibrated process;
- Type II error: continue operating a miscalibrated process.
Example: Fraud Detection
Suppose
\[ H_0: \text{the transaction is legitimate}. \]Then:
- Type I error: legitimate transaction flagged as fraud;
- Type II error: fraudulent transaction allowed through.
Example: Scientific Discovery
Suppose
\[ H_0: \text{no treatment effect}. \]Then:
- Type I error: claim a treatment effect that is not real;
- Type II error: miss a treatment effect that is real.
The Costs of the Two Errors Can Differ
Type I and Type II errors do not necessarily have equal practical consequences.
In one application, a false positive may be very costly.
In another, missing a real effect may be far more serious.
Statistical thresholds should be informed by these consequences.
Lower Alpha Reduces Type I Error Probability
Reducing
\[ \alpha \]makes the test more conservative against false positives.
For example,
\[ \alpha=0.01 \]permits fewer false rejections under the null than
\[ \alpha=0.05. \]Lower Alpha Usually Increases Beta
Holding sample size and effect size fixed, lowering \(\alpha\) makes rejection harder.
This usually increases
\[ \beta \]and therefore lowers power.
The Alpha-Beta Tradeoff
With all other design features fixed, reducing one error probability can increase the other.
This creates a tradeoff:
\[ \boxed{ \text{smaller }\alpha \Rightarrow \text{larger }\beta \text{ in many fixed designs} }. \]Sample Size Can Improve the Tradeoff
Increasing the sample size can reduce standard errors.
This can improve the ability to distinguish the null from meaningful alternatives without necessarily increasing \(\alpha\).
Larger samples can therefore increase power while maintaining the same Type I error rate.
Effect Size Changes Beta
Large departures from the null are easier to detect.
Therefore,
\[ \beta \]is usually smaller for large effects than for small effects.
Example: Small versus Large Alternatives
Suppose
\[ H_0:\mu=50. \]A test may have low power to detect
\[ \mu=51 \]but very high power to detect
\[ \mu=65. \]The Type II error probability depends on how far the true mean lies from the null.
Variability Changes Beta
Larger population variability makes null and alternative sampling distributions overlap more strongly.
This makes real effects harder to distinguish and generally increases \(\beta\).
Measurement Precision Matters
Reducing measurement noise can decrease the standard error of an estimate.
Better measurement can therefore reduce Type II error probability and increase power.
One-Sided versus Two-Sided Testing
For a fixed \(\alpha\), a one-sided test concentrates the rejection probability in one direction.
If the true effect lies in that pre-specified direction, the one-sided test can have greater power there than a two-sided test.
The direction must be justified before inspecting the data.
Two-Sided Tests Protect against Both Directions
A two-sided test sacrifices some directional power in order to detect departures on either side of the null value.
This is appropriate when effects in both directions matter.
Decision Thresholds
In a test statistic framework, choosing \(\alpha\) determines a critical boundary.
Moving that boundary farther into the tail lowers false-positive probability but makes some real alternatives harder to detect.
Visualizing the Two Errors
Imagine two overlapping sampling distributions:
- one generated under \(H_0\);
- one generated under a particular \(H_A\).
The rejection threshold divides the possible test-statistic values.
The tail of the null distribution beyond the threshold corresponds to \(\alpha\), while the portion of the alternative distribution on the nonrejection side corresponds to \(\beta\).
Greater Separation Improves Detection
If the null and alternative distributions are farther apart, their overlap decreases.
This reduces the probability of Type II error and increases power.
Greater separation can come from:
- larger true effects;
- larger sample sizes;
- lower variability;
- better measurement.
Type I Error Is Defined under the Null
We evaluate
\[ \alpha \]using the distribution of the test statistic under the null hypothesis.
This is why critical values and rejection regions are constructed from the null distribution.
Type II Error Is Defined under an Alternative
To calculate
\[ \beta, \]we need a specific alternative value or distribution.
Saying only
\[ H_A:\mu\neq50 \]is not enough to produce one unique \(\beta\) value.
Power Functions
A power function gives the probability of rejection as a function of the true parameter:
\[ \boxed{ \pi(\theta) = P_\theta(\text{reject }H_0) }. \]Under null values, this function describes Type I error behavior.
Under alternative values, it describes statistical power.
Power at the Null Boundary
For a correctly calibrated continuous test at the null boundary,
\[ \pi(\theta_0) \approx \alpha. \]As the true parameter moves farther into the alternative region, the power function usually rises.
Discrete Tests May Be Conservative
In discrete tests, it may be impossible to construct a rejection region with probability exactly equal to the desired \(\alpha\).
The actual Type I error probability may therefore be less than the nominal significance level.
Exact Tests and Error Control
Exact tests calculate probabilities from the exact null distribution.
They can provide controlled Type I error without relying on large-sample approximations.
They may still be conservative in discrete settings.
Approximate Tests and Error Rates
Tests based on normal, chi-square, or other asymptotic approximations are designed to achieve approximately the desired Type I error rate.
Poor approximations can cause actual error probabilities to differ from nominal values.
Model Violations Can Distort Both Errors
If assumptions are seriously violated, neither the nominal \(\alpha\) nor the calculated power may describe the actual behavior of the procedure.
Dependence, heteroskedasticity, misspecification, or poor sampling can change error rates.
Bias Is Different from Type I and Type II Error
Statistical bias concerns systematic behavior of an estimator or study design.
Type I and Type II errors concern decisions made by a hypothesis test.
A biased study can distort both error rates, but the concepts are distinct.
Multiple Testing Increases False Positives
If many hypotheses are tested independently at level
\[ \alpha=0.05, \]some false positives are expected even when every null hypothesis is true.
The familywise probability of at least one Type I error can become much larger than \(0.05\).
Familywise Error Rate
The familywise error rate is
\[ P(\text{at least one Type I error in a family of tests}). \]Procedures such as Bonferroni adjustment can control this probability.
Bonferroni and Power
Bonferroni adjustment lowers the threshold for each individual test.
This reduces false-positive risk across the family but also makes individual discoveries harder.
Thus multiplicity control can reduce power.
False Discovery Rate
In large-scale testing, one may control the expected proportion of false positives among all rejected hypotheses.
This is the false discovery rate.
It is a different error criterion from the probability of any Type I error.
Type I Error and False Discovery Are Not Identical
A Type I error is defined for a particular null hypothesis.
False discovery rate describes behavior across a collection of rejected hypotheses.
The concepts are related but not interchangeable.
Diagnostic False Positives and Statistical Type I Errors
The phrase false positive appears in both diagnostic classification and hypothesis testing.
The mathematical structures are similar, but the underlying probabilities and conditioning can differ.
Terminology should therefore be interpreted within the specific framework.
Prevalence Does Not Change Alpha Directly
In a classical fixed hypothesis test,
\[ \alpha \]is defined conditionally on the null being true.
How often the null is true across a broader collection of cases is a separate question.
Decision Theory Perspective
Statistical decisions can be framed using costs or losses.
Let
\[ C_I \]be the cost of a Type I error and
\[ C_{II} \]the cost of a Type II error.
A rational decision rule can depend on both costs rather than treating all errors as equally serious.
High Cost of Type I Error
If false positives are especially harmful, a smaller significance level may be appropriate.
Examples may include:
- claiming a major scientific discovery;
- approving an unsafe intervention;
- accusing a legitimate transaction of fraud;
- stopping a reliable process unnecessarily.
High Cost of Type II Error
If missing a real effect is especially harmful, the study may need:
- greater power;
- a larger sample;
- better measurements;
- a less stringent threshold where justified;
- a design specifically optimized for detection.
Example: Safety Monitoring
Suppose
\[ H_0: \text{no safety problem}. \]A Type II error could mean failing to detect a genuine safety hazard.
In such a context, high power may be especially important.
Example: Quality Control
Suppose
\[ H_0: \text{process is in control}. \]A Type I error may cause unnecessary shutdown.
A Type II error may allow defective production to continue.
The appropriate thresholds depend on the relative costs of these outcomes.
Sample Size and Both Errors
With a fixed significance level, increasing sample size usually leaves \(\alpha\) approximately unchanged while reducing \(\beta\) for meaningful alternatives.
This is one reason sample-size planning is central to experimental design.
Small Samples Increase False Negatives
Small samples generally produce larger standard errors.
Real effects can therefore be obscured by sampling variability.
This increases Type II error probability and lowers power.
Large Samples Can Detect Tiny Effects
Larger samples reduce Type II error for small effects.
However, they can make effects of negligible practical importance statistically detectable.
Power and practical significance must therefore be considered together.
Practical Significance and Error Definitions
Traditional Type II error is defined relative to any specified alternative.
In practice, it is often more meaningful to focus power calculations on alternatives large enough to matter.
Missing a trivially small effect may be statistically a Type II error but practically unimportant.
Minimum Important Effect
Let
\[ \Delta_{\text{important}} \]denote the smallest effect that matters.
A study can be planned so that power is high at this effect size rather than at every mathematically nonzero alternative.
Power Is Usually Chosen before Data Collection
Desired power is a design target.
Common values include
\[ 0.80 \]and
\[ 0.90. \]These correspond to Type II error probabilities of
\[ 0.20 \]and
\[ 0.10, \]respectively.
Example: Convert Power to Beta
Suppose a study has
\[ 90\% \]power at a particular alternative.
Then
\[ \beta = 1-0.90 = \boxed{0.10}. \]Example: Convert Beta to Power
Suppose
\[ \beta=0.25. \]Then
\[ \text{Power} = 1-0.25 = \boxed{0.75}. \]Repeated-Sampling Interpretation of Alpha
If the null hypothesis is true and the same valid testing procedure is repeated many times at
\[ \alpha=0.05, \]approximately \(5\%\) of the tests are expected to reject the null under ideal calibration.
Repeated-Sampling Interpretation of Power
If a specific alternative is true and the same study is repeated many times with power
\[ 0.80, \]approximately \(80\%\) of those studies would reject the null under the assumed model and design.
Error Rates Are Long-Run Properties
After a single study has been completed, the decision is either correct or incorrect.
The values \(\alpha\), \(\beta\), and power describe repeated-sampling behavior of the testing procedure.
A Nonsignificant Result Can Be Ambiguous
Failing to reject the null may mean:
- the null is approximately correct;
- the true effect is too small to matter;
- the study has low power;
- the estimate is too noisy;
- the sample is too small.
A nonsignificant result alone cannot distinguish these possibilities.
Confidence Intervals Help Diagnose Nonsignificance
A narrow confidence interval near the null can rule out large effects.
A wide confidence interval can include both the null and large meaningful effects.
Confidence intervals therefore provide more context than a binary rejection decision.
A Significant Result Can Still Be Wrong
Statistical significance does not eliminate the possibility of a Type I error.
A test at
\[ \alpha=0.05 \]is deliberately designed to produce false rejections at a nonzero rate when the null is true.
Replication Reduces Reliance on One Decision
Repeating studies under independent data collection can help determine whether a statistically significant finding is reproducible.
Replication does not erase the original error probabilities, but it provides additional evidence.
Pre-Registration and Error Control
Pre-specifying hypotheses, outcomes, sample sizes, and analysis methods helps preserve intended Type I error properties.
Data-dependent choices can inflate false-positive rates.
Optional Stopping Can Inflate Type I Error
Repeatedly checking significance and stopping data collection as soon as
\[ p<0.05 \]occurs can increase the actual Type I error rate when ordinary fixed-sample methods are used.
Sequential Methods Can Control Error Properly
Group-sequential and other sequential testing procedures can allow repeated interim analyses while controlling the overall Type I error rate.
The stopping rule must be included in the design.
Common Error: Type I Means the Null Is Probably False
A Type I error is defined under a true null.
It does not measure the posterior probability that the null is false or true.
Common Error: Alpha Is the Probability of Making Any Mistake
Alpha controls one specific error:
\[ \text{rejecting a true null}. \]It does not include Type II errors.
Common Error: Beta Is Fixed by Alpha Alone
Beta depends on more than the significance level.
It also depends on:
- effect size;
- sample size;
- population variability;
- test direction;
- study design;
- the specific alternative considered.
Common Error: Failure to Reject Means No Type II Risk
Failure to reject can itself be the outcome of a Type II error when a meaningful alternative is true.
This is why power analysis matters.
Common Error: Significant Means No Type I Risk
A statistically significant result can still be a false positive.
The significance level describes the long-run rate of such false rejections under the null.
Common Error: Smaller Alpha Is Always Better
Lowering \(\alpha\) reduces false positives but can increase false negatives unless sample size or other design features compensate.
Error thresholds should reflect the consequences of both types of mistakes.
Common Error: Larger Sample Fixes Bias
A larger sample can reduce Type II error caused by sampling variability.
It does not automatically correct:
- selection bias;
- measurement bias;
- confounding;
- nonresponse;
- model misspecification.
Common Error: Power Is a Property of the Observed P-Value
Power is primarily a property of a planned test under specified alternative conditions.
It is not determined simply by whether one observed study produced a small or large p-value.
A Reliable Procedure for Thinking about Testing Errors
- State the null and alternative hypotheses.
- Define what a false positive means in context.
- Define what a false negative means in context.
- Choose a justified significance level \(\alpha\).
- Identify a practically meaningful alternative effect.
- Determine the desired power \(1-\beta\).
- Estimate variability and other design parameters.
- Choose a sample size large enough for the desired power.
- Account for clustering, attrition, or multiplicity when relevant.
- Interpret nonsignificant results in light of precision and power.
- Interpret significant results in light of Type I error risk and effect size.
- Report the practical consequences of both error types.
Example: Full Error Analysis
Suppose a factory tests
\[ H_0: \mu=100 \]for a process target.
The alternative is
\[ H_A: \mu\neq100. \]A Type I error means concluding the process mean has shifted when it is actually still \(100\).
A Type II error means failing to detect a real shift away from \(100\).
If unnecessary shutdowns are costly, the factory may prefer a small \(\alpha\).
If undetected shifts are also costly, it must compensate with adequate sample size and power.
Example: Error Tradeoff
Suppose a test originally uses
\[ \alpha=0.05. \]Management changes the threshold to
\[ \alpha=0.01 \]without changing the sample size.
False-positive probability decreases.
But the rejection threshold becomes more difficult to cross, so \(\beta\) generally increases for the same alternative effect.
Practice 1: Identify Type I Error
Suppose
\[ H_0: \text{a treatment has no effect}. \]What would a Type I error mean?
Practice 2: Identify Type II Error
Using the same null hypothesis, what would a Type II error mean?
Practice 3: Alpha
Write the conditional probability represented by
\[ \alpha. \]Practice 4: Beta
Write the conditional probability represented by
\[ \beta. \]Practice 5: Power
If
\[ \beta=0.15, \]what is the power?
Practice 6: Effect of Sample Size
Holding \(\alpha\) and the alternative effect fixed, what generally happens to \(\beta\) when sample size increases?
Practice 7: Effect of Lower Alpha
Holding sample size fixed, what generally happens to \(\beta\) when \(\alpha\) is reduced?
Practice 8: False Positive or False Negative?
A fraud-detection test flags a legitimate transaction as fraudulent.
If legitimacy is the null state, which error type is this?
Practice 9: Nonsignificant Result
Does failing to reject the null prove that no meaningful effect exists?
Practice 10: Identify the Error
A student says:
“Because the test uses \(\alpha=0.05\), there is a \(5\%\) probability that the null hypothesis is true.”
Explain the mistake.
Practice Answers
Practice 1
A Type I error means concluding that the treatment has an effect when the null hypothesis of no effect is actually true.
Practice 2
A Type II error means failing to reject the no-effect null when a real treatment effect exists at the specified alternative.
Practice 3
\[ \boxed{ \alpha = P(\text{reject }H_0\mid H_0\text{ true}) }. \]Practice 4
\[ \boxed{ \beta = P(\text{fail to reject }H_0\mid H_A\text{ true}) }. \]Practice 5
\[ \text{Power} = 1-0.15 = \boxed{0.85}. \]Practice 6
Beta generally decreases because the test gains power.
Practice 7
Beta generally increases because rejection becomes harder.
Practice 8
This is a Type I error, or false positive.
Practice 9
No.
Failure to reject can occur because the study has low power or high uncertainty. Confidence intervals and power considerations are needed for interpretation.
Practice 10
Alpha is conditional on the null being true.
It is the probability of rejecting a true null under the testing procedure, not the probability that the null hypothesis itself is true.
What to Remember
Type I and Type II errors describe the two fundamental ways a hypothesis test can make a wrong decision.
- A Type I error rejects a true null hypothesis.
- A Type II error fails to reject a false null hypothesis.
- \(\alpha\) is the probability of a Type I error under the null.
- \(\beta\) is the probability of a Type II error at a specified alternative.
- Power is \(1-\beta\).
- Lowering \(\alpha\) usually lowers false positives but increases \(\beta\) when sample size is fixed.
- Larger samples generally reduce \(\beta\) and increase power while holding \(\alpha\) fixed.
- Effect size, variability, test direction, and study design all influence power.
- Multiple testing can inflate false-positive risk.
- The practical costs of false positives and false negatives should influence study design.
The most important habit is: define both possible errors in the context of the real decision, then choose significance level, sample size, and power targets that reflect the consequences of those mistakes.
Where We Go Next
Type II error and statistical power depend strongly on effect size, variability, sample size, and the significance threshold.
The next lesson, Power and Sample Size, develops these relationships directly and shows how to plan a study so that meaningful effects have a high probability of being detected.