Chapter 9 — Front Matter
Purpose and Overview
"The null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation."
— Ronald A. Fisher
Every claim about the world invites a question: how do you know? A pharmaceutical company announces that a new drug lowers blood pressure. An engineer reports that a manufacturing process produces components within tolerance. A data scientist claims that a new recommendation algorithm increases user engagement. Each of these claims rests on data — but data alone do not speak for themselves. Data are subject to variability. Even if a drug had no real effect, a small clinical trial might show blood pressure differences simply by chance.
The fundamental question in statistical inference is always this: is what we observe in the data evidence of a real phenomenon, or could it plausibly be explained by ordinary randomness? Hypothesis testing is the formal statistical framework for answering that question. It is not a machine that produces truth. It is a disciplined method for quantifying how surprising data would be if a specific claim were true.
This chapter develops hypothesis testing from the ground up. We begin with why it exists. We build the vocabulary carefully: null hypothesis, alternative hypothesis, test statistic, p-value, significance level, and decision rule. We examine one-sided and two-sided tests, the errors a test can make, and the concept of statistical power. We then apply these ideas to tests for means, proportions, and differences. We close by confronting the critical distinction between statistical significance and practical importance.
Opening Question
A coin is flipped 1000 times and lands heads 700 times. A coin is also flipped 10 times and lands heads 7 times. Both results show 70% heads — but one seems far more suspicious than the other. Why? By the end of this chapter, you will have the mathematical framework to answer this question precisely, and to answer it consistently for any claim about any population.
14Sections
13Objectives
20Key Terms
15+Worked Examples
Learning Objectives
- 1Explain why hypothesis testing exists and what problem it solves in statistical reasoning.
- 2Formulate null and alternative hypotheses for a given research question.
- 3Compute test statistics for means, proportions, and differences.
- 4Compute and correctly interpret p-values for one-sided and two-sided tests.
- 5Apply the decision rule using a significance level to reach a conclusion.
- 6Choose between one-sided and two-sided tests based on the research question.
- 7Define and distinguish Type I errors, Type II errors, and their probability relationships.
- 8Explain power and identify the factors that increase it.
- 9Conduct one-sample t-tests and proportion tests with full six-step procedure.
- 10Conduct two-sample t-tests and paired t-tests for comparing two groups.
- 11Distinguish statistical significance from practical importance and use effect sizes.
- 12Identify and correct the most common hypothesis testing mistakes.
- 13Connect hypothesis testing to estimation and preview the likelihood framework.
Sections 9.1 – 9.3
The Framework: Questions, Claims, and Hypotheses
Why Testing Exists · Claims and Evidence · Null and Alternative Hypotheses
Section 9.1
Why Hypothesis Testing Exists
Imagine you flip a coin ten times and get seven heads. Should you conclude the coin is unfair? Seven heads in ten flips happens with meaningful probability for a fair coin. Now imagine 1000 flips with 700 heads. The proportion is identical — 70 percent — but the evidence is overwhelming. With 1000 flips, 700 heads is essentially impossible for a fair coin.
This contrast captures the core challenge that hypothesis testing addresses. We observe data that differ from some expectation. How much difference is enough to conclude that something real is going on, rather than that chance alone produced the result? The answer depends on how large the difference is, how much variability we expect, and how much data we have. Hypothesis testing packages all three into a systematic procedure.
The Historical Foundations
Ronald A. Fisher introduced the concept of the p-value and the null hypothesis in the 1920s as a way to evaluate experimental results. Jerzy Neyman and Egon Pearson extended Fisher's ideas in the 1930s by formalizing the alternative hypothesis, the binary decision framework, and the concept of error rates. Modern practice blends both approaches.
Core Principle
Hypothesis testing assesses evidence against a default claim. It does not confirm a theory. We assume a null model is true, compute what data would look like under that assumption, observe actual data, and measure how consistent the observation is with the default. Hypothesis testing is about falsification, not confirmation.
Connection to Chapter 7
Hypothesis testing is built on sampling distribution theory. When we assume a population mean is μ₀, we know the sample mean X̄ follows an approximately normal distribution with mean μ₀ and standard error σ/√n by the Central Limit Theorem. The test asks: given this distribution, how far from μ₀ is our observed x̄?
Section 9.2
Claims and Statistical Evidence
Before a hypothesis test can begin, a vague question must become a precise, testable claim about a population parameter. "Does the drug work?" must become "is the population mean change in blood pressure greater than zero?" "Is the process on target?" must become "does the population mean diameter equal 50 mm?"
Statistical evidence must account for variability. A difference of 3 units from a claimed value is strong evidence if the standard error is 0.5, and weak evidence if the standard error is 10. This is why the test statistic — introduced in Section 9.4 — expresses the observed difference in units of standard error.
| Research Question | Parameter | Testable Claim |
|---|
| Does a supplement increase energy? | μ = population mean change | Mean change > 0 |
| Is the process producing correct diameters? | μ = population mean diameter | Mean diameter ≠ 50 mm |
| Do vaccination rates differ by region? | p₁, p₂ = two proportions | p₁ ≠ p₂ |
Critical Limitation
A hypothesis test evaluates one specific claim against one specific alternative. If we reject the null, we conclude only that the data are inconsistent with that claim — not that any particular alternative is true. Confounding variables, measurement error, and model misspecification remain as alternative explanations. Hypothesis testing is a diagnostic tool, not a proof machine.
Section 9.3
Null and Alternative Hypotheses
The null hypothesis, denoted H₀, is the default claim assumed to be true unless the data provide sufficient evidence against it. "Null" suggests nothing unusual is happening — no effect, no difference, no departure from baseline. In most standard tests, H₀ specifies a single value for the parameter.
The alternative hypothesis, denoted Hₐ, is the claim we would accept if we reject the null. It must be specified before the data are examined. Choosing an alternative after looking at the data — sometimes called HARKing (Hypothesizing After Results are Known) — undermines the integrity of the test.
| Type | Null Hypothesis H₀ | Alternative Hypothesis Hₐ | Direction |
|---|
| Two-sided | μ = μ₀ | μ ≠ μ₀ | Either direction |
| One-sided (upper) | μ = μ₀ | μ > μ₀ | Above null only |
| One-sided (lower) | μ = μ₀ | μ < μ₀ | Below null only |
Student Warning — Parameters, Not Statistics
Hypotheses are always written in terms of population parameters, not sample statistics. Never write H₀: x̄ = 100. Write H₀: μ = 100. The sample statistic x̄ is observed data; μ is the unknown quantity we are learning about.
Example 9.3.1
Coffee shop fill volume — one-sided lower test
- ProblemA coffee shop claims large cups average at least 16 oz. A consumer group suspects underfilling. Write hypotheses.
- Parameterμ = population mean fill volume in ounces.
- Hypotheses
H₀: μ = 16 (default: cups are properly filled)
Hₐ: μ < 16 (consumer group suspects underfilling)
One-sided lower test: the alternative specifies values below the null.
Example 9.3.2
Drug effect on blood pressure — two-sided test
- ProblemA researcher wants to know whether a new drug changes mean systolic blood pressure compared to placebo. Write hypotheses.
- Parameterμ_d = mean difference in blood pressure (drug minus placebo) in the population.
- Hypotheses
H₀: μ_d = 0 (no difference)
Hₐ: μ_d ≠ 0 (any change, either direction)
Two-sided test: the researcher is looking for any change, not predicting a direction.
Sections 9.4 – 9.7
Test Statistics, p-Values, and Decision Rules
Test Statistics · p-Values · Significance Levels · One- and Two-Sided Tests
Section 9.4
Test Statistics
A test statistic is a numerical summary of the data measuring how far the observed value falls from what the null hypothesis predicts. A large test statistic (in absolute value) indicates data far from the null. A small statistic indicates data consistent with the null. The test statistic is designed to have a known distribution when the null is true — called the null distribution.
General Form — All Test Statistics
Test Statistic = (Observed Value − Null Hypothesis Value) / Standard Error
This converts the raw difference into a dimensionless measure of how many standard errors the observation falls from the null value — the same standardization used for z-scores.
Example 9.4.1
Bolt length — z-test statistic (σ known)
- ProblemA manufacturer claims bolts have mean length 10 cm with known σ = 0.2 cm. A sample of n = 36 gives x̄ = 10.07 cm. Compute the test statistic.
- Solve
z = (10.07 − 10) / (0.2 / √36)
= 0.07 / (0.2 / 6)
= 0.07 / 0.0333
≈ 2.10
z ≈ 2.10. The sample mean falls 2.10 standard errors above the claimed mean.
Example 9.4.2
Patient recovery — t-test statistic (σ unknown)
- ProblemA sample of n = 20 patients has x̄ = 8.3 days and s = 2.1 days. Claimed standard recovery is μ₀ = 7.5 days. Compute the test statistic.
- Solve
t = (8.3 − 7.5) / (2.1 / √20)
= 0.8 / (2.1 / 4.47)
= 0.8 / 0.470
≈ 1.70
t ≈ 1.70 with df = 19. This follows a t-distribution with 19 degrees of freedom under H₀.
Example 9.4.3
Website conversion rate — proportion test statistic
- ProblemA website historically converts 12% of visitors. A new page is tested on n = 300 visitors, and 45 convert. Compute the test statistic.
- Solve
p̂ = 45 / 300 = 0.15
z = (0.15 − 0.12) / √(0.12 × 0.88 / 300)
= 0.03 / √(0.000352)
= 0.03 / 0.01877
≈ 1.60
z ≈ 1.60. The sample proportion falls 1.60 standard errors above the null value of 0.12.
Section 9.5
p-Values
The p-value is the probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one computed from the sample.
A small p-value means: if H₀ were true, data this extreme would be very unlikely. This is evidence against the null. A large p-value means: data like what we observed would not be unusual under H₀. The data are consistent with the null.
Example 9.5.1
Bolt length — two-sided p-value
- SetupFrom Example 9.4.1: z = 2.10, two-sided alternative Hₐ: μ ≠ 10.
- Solve
p-value = 2 · P(Z ≥ 2.10)
= 2 · (1 − Φ(2.10))
= 2 · (1 − 0.9821)
= 2 · 0.0179
= 0.0358
p ≈ 0.036. If the true mean were 10 cm, a sample mean this far from 10 (in either direction) would occur about 3.6% of the time by chance.
What a p-Value Is NOT
Not the probability that H₀ is true. Not the probability the results occurred by chance. Not the probability the alternative is true. Not a measure of effect size or practical importance.
A p-value IS a conditional probability: given that H₀ is true, how probable is data at least this extreme? A p-value of 0.03 does not mean "there is a 3% chance the null is true." It means "if the null were true, data this extreme would occur 3% of the time."
Section 9.6
Significance Levels
Decision Rule
Let α be the pre-specified significance level (common choices: 0.05, 0.01, 0.10).
If p-value ≤ α: Reject H₀. The data are statistically significant at level α.
If p-value > α: Fail to reject H₀. The data are not statistically significant at level α.
The significance level α is the probability the test will reject a true null hypothesis. At α = 0.05, if H₀ were true and the experiment were repeated many times, 5% of tests would produce a rejection purely by chance. This is the Type I error rate.
An equivalent approach uses critical values. For a z-test with two-sided alternative at α = 0.05, reject H₀ when |z| > z_{α/2} = z_{0.025} = 1.96.
Example 9.6.1
Bolt length — decision at α = 0.05
- SetupFrom Example 9.5.1: p-value = 0.0358, α = 0.05, two-sided test. Critical values ±1.96.
- Decision
p-value = 0.0358 < 0.05 = α → Reject H₀
Equivalently: |z| = 2.10 > 1.96 → Reject H₀
- ConclusionAt the 5% significance level, there is sufficient evidence to reject the claim that the mean bolt length is 10 cm.
Reject H₀. Both the p-value and critical-value approaches agree.
Language Precision Matters
Say "reject H₀" — not "prove Hₐ." Say "fail to reject H₀" — not "accept H₀." Say "statistically significant at level α" — not simply "significant." Statistical significance is always relative to a threshold and says nothing, by itself, about practical importance.
Section 9.7
One-Sided and Two-Sided Tests
A two-sided test is appropriate when you are interested in detecting a departure from the null in either direction. This is the correct default for most research questions. A one-sided test is appropriate only when scientific knowledge makes deviations in only one direction relevant — and this must be determined before data collection.
One-sided tests are more powerful when the true effect is in the specified direction (the critical value is less extreme), but will miss effects in the opposite direction entirely.
Example 9.7.1
Drug lowering blood pressure — one-sided lower test
- Hypotheses
H₀: μ = 80 mmHg
Hₐ: μ < 80 mmHg
- Datan = 50, x̄ = 77.4 mmHg, s = 8.5 mmHg.
- Test Stat
t = (77.4 − 80) / (8.5 / √50) = −2.6 / 1.202 ≈ −2.16
- p-ValueOne-sided (left tail) with df = 49: p ≈ 0.018
- Decisionp = 0.018 < 0.05. Reject H₀.
- ConclusionThere is evidence at the 5% level that the drug lowers mean diastolic blood pressure below 80 mmHg.
Reject H₀ at α = 0.05. For reference, a two-sided p-value would be ≈ 0.036 — also significant here, but with less margin.
Student Warning — Choosing the Side After Seeing Data
A common and serious mistake is to look at the data, notice which direction the sample falls, and then choose a one-sided alternative in that direction. This halves the p-value and dramatically inflates the actual Type I error rate. The alternative must be chosen before data are examined. This is a form of p-hacking (see Section 9.14).
Sections 9.8 – 9.9
Errors, Power, and Sample Size
Type I and Type II Errors · Power · Sample Size Planning
Section 9.8
Type I and Type II Errors
A hypothesis test produces one of two decisions: reject H₀ or fail to reject H₀. The truth about the population is either that H₀ is true or H₀ is false. This creates a 2×2 table of possible outcomes.
| H₀ True |
H₀ False |
Type I Error
False Positive
Probability = α |
Correct Rejection
True Positive
Probability = Power = 1−β |
Correct Retention
True Negative
Probability = 1−α |
Type II Error
False Negative
Probability = β |
Error Definitions
Type I Error — Rejecting a true null hypothesis. Probability = α. The significance level directly sets the Type I error rate.
Type II Error — Failing to reject a false null hypothesis. Probability = β. Depends on the true parameter value, sample size, and variability.
Power — Correctly rejecting a false null hypothesis. Probability = 1 − β.
For a fixed sample size and variance, α and β trade off against each other. Lowering α makes the test more conservative, which increases β. The only way to reduce both simultaneously is to collect more data.
Example 9.8.1
Balancing error types — factory calibration
- ScenarioA factory tests whether mean bolt length equals 10 cm. The inspector sets α = 0.01 to avoid falsely concluding the machine is miscalibrated (Type I error).
- Trade-offBy setting a very conservative α, the inspector reduces false positives but increases β — the chance of missing a real calibration problem (Type II error).
- QuestionIf a miscalibrated machine going undetected is costly, is α = 0.01 appropriate?
The right balance depends on the relative costs of each error type — a judgment that statistics provides a framework for, but subject-matter expertise must resolve.
Section 9.9
Power of a Test
Power is the probability of correctly rejecting a false null hypothesis: Power = 1 − β = P(Reject H₀ | H₀ false). Power is not a single number — it is a function of where the truth lies relative to the null.
What Increases Power
| Factor | Direction | Reason |
|---|
| Effect size |μₐ − μ₀| | Larger → more power | Larger departures from null are easier to detect |
| Sample size n | Larger → more power | Reduces standard error; tighter sampling distribution |
| Significance level α | Larger → more power | Easier to reject; but increases Type I errors |
| Population SD σ | Smaller → more power | Less variability makes differences more visible |
Example 9.9.1
Sample size planning for 80% power
- SetupDetect δ = 2 points from μ₀ = 100, with σ = 10, at α = 0.05 (two-sided) and 80% power.
- Valuesz_{0.025} = 1.96, z_{0.20} = 0.842 (80th percentile of standard normal).
- Solve
n = [(1.96 + 0.842) × 10 / 2]²
= [2.802 × 10 / 2]²
= [14.01]²
≈ 196
A sample of approximately 196 participants is needed to detect a 2-point difference with 80% power at α = 0.05.
Underpowered Studies
A study with low power is unlikely to detect a real effect even when one exists. Failing to reject H₀ in an underpowered study should not be interpreted as evidence that the null is true — the study may simply be too small to see the effect. Much published research has been criticized for underpowered designs, contributing to the reproducibility crisis.
Sections 9.10 – 9.12
Tests for Means, Proportions, and Differences
One-Sample t-Test · Proportion Test · Two-Sample and Paired Tests
Section 9.10
Tests for a Mean
One-Sample t-Test — Conditions
1. Simple random sample from the population.
2. Population approximately normal, or n large enough for CLT (n < 15: assume normality; 15 ≤ n < 30: roughly symmetric, no extreme outliers; n ≥ 30: CLT adequate).
3. Population standard deviation σ is unknown (use s; this is almost always the case).
Six-Step Procedure
Step 1 State H₀ and Hₐ in terms of the population parameter μ.
Step 2 Check conditions (random sample, normality/CLT, σ unknown).
Step 3 Compute the test statistic: t = (x̄ − μ₀) / (s / √n).
Step 4 Compute the p-value using T ~ t_{n−1} under H₀.
Step 5 Compare p-value to α and state the decision.
Step 6 Write a conclusion in context.
Example 9.10.1
Sleep study — full six-step procedure
- SetupA claim states adults sleep an average of 7 hours per night. A health researcher samples n = 25 adults. The sample yields x̄ = 6.5 hr and s = 1.8 hr. Test at α = 0.05 whether mean sleep time differs from 7 hours.
- Step 1
H₀: μ = 7 Hₐ: μ ≠ 7
- Step 2Simple random sample assumed. Sleep hours are approximately normal. σ unknown. Conditions met.
- Step 3
t = (6.5 − 7) / (1.8 / √25)
= −0.5 / (1.8 / 5)
= −0.5 / 0.36
= −1.39
- Step 4
p-value = 2 · P(T₂₄ ≤ −1.39) ≈ 2 · 0.089 = 0.178
- Step 5p-value = 0.178 > 0.05 = α. Fail to reject H₀.
- Step 6At the 5% significance level, these data do not provide sufficient evidence that mean adult sleep time differs from 7 hours.
Fail to reject H₀. The data are consistent with a population mean of 7 hours.
Section 9.11
Tests for a Proportion
The one-sample proportion z-test evaluates whether a population proportion p equals a specified value p₀. The standard error uses p₀ (not p̂), because we compute probabilities under H₀. Conditions: simple random sample; np₀ ≥ 10 and n(1 − p₀) ≥ 10.
Example 9.11.1
Transit plan support — six-step proportion test
- SetupA city council claims 60% of residents support a new transit plan. A reporter surveys n = 200 residents and finds 108 support it. Test at α = 0.05.
- Step 1
H₀: p = 0.60 Hₐ: p ≠ 0.60
- Step 2Conditions: np₀ = 120 ≥ 10 ✓ and n(1−p₀) = 80 ≥ 10 ✓.
- Step 3
p̂ = 108/200 = 0.54
z = (0.54 − 0.60) / √(0.60 × 0.40 / 200)
= −0.06 / √(0.0012)
= −0.06 / 0.03464
≈ −1.73
- Step 4
p-value = 2 · P(Z ≤ −1.73) = 2 · 0.0418 = 0.0836
- Step 5p-value = 0.0836 > 0.05 = α. Fail to reject H₀.
- Step 6At the 5% level, the survey does not provide sufficient evidence that the proportion supporting the plan differs from 60%. (Note: a p-value of 0.084 is not negligible — more data could change this conclusion.)
Fail to reject H₀. Consistent with a 95% confidence interval for p that includes 0.60.
Duality: Tests and Confidence Intervals
A two-sided test at level α rejects H₀: p = p₀ if and only if p₀ falls outside the 100(1−α)% confidence interval for p. Report confidence intervals alongside test results — they provide information about the size of the effect, not just its detectability.
Section 9.12
Tests for Differences
Two-Sample t-Test (Independent Groups)
When comparing two independent group means, use the two-sample t-test with Welch degrees of freedom (computed by software; conservative approximation: min(n₁−1, n₂−1)).
Example 9.12.1
Teaching methods — two-sample t-test
- DataMethod A: n₁ = 30, x̄₁ = 82, s₁ = 8. Method B: n₂ = 28, x̄₂ = 77, s₂ = 10. Test at α = 0.05.
- HypothesesH₀: μ₁ − μ₂ = 0 Hₐ: μ₁ − μ₂ ≠ 0
- Test Stat
t = (82 − 77) / √(64/30 + 100/28)
= 5 / √(2.133 + 3.571)
= 5 / √5.704
= 5 / 2.388
≈ 2.09
- p-ValueWith Welch df ≈ 49: p-value = 2 · P(T₄₉ ≥ 2.09) ≈ 0.042.
- Decision0.042 < 0.05. Reject H₀.
- ConclusionThere is evidence at the 5% level of a difference in mean scores between the two teaching methods.
Reject H₀. Method A produced a significantly higher mean score than Method B.
Paired t-Test (Dependent Samples)
When two measurements come from the same individuals (before/after, matched pairs), compute differences dᵢ = x_{i,1} − x_{i,2} for each pair, then conduct a one-sample t-test on the differences.
Example 9.12.2
Tutoring program — paired t-test
- DataTen students, differences (after − before): 5, −1, 8, 4, 6, 2, 7, 3, 5, 1. Test at α = 0.05 whether tutoring changed scores.
- HypothesesH₀: μ_d = 0 Hₐ: μ_d ≠ 0
- Statisticsd̄ = 4.0, s_d = 2.79, n = 10.
- Test Stat
t = 4.0 / (2.79 / √10) = 4.0 / 0.882 = 4.53
- p-Valuep-value = 2 · P(T₉ ≥ 4.53) ≈ 0.001
- Decision0.001 < 0.05. Reject H₀.
Strong evidence (p ≈ 0.001) that the tutoring program increased mean test scores.
Two-Sample z-Test for Proportions
To compare two population proportions, use the pooled proportion under H₀: p₁ = p₂, where p̂_pool = (x₁ + x₂) / (n₁ + n₂).
Example 9.12.3
Vaccine trial — two-sample proportion test
- DataVaccine group: n₁ = 500, 30 infections. Placebo group: n₂ = 500, 70 infections. Test at α = 0.01.
- HypothesesH₀: p₁ = p₂ Hₐ: p₁ ≠ p₂
- Pooled
p̂₁ = 0.06, p̂₂ = 0.14
p̂_pool = (30 + 70) / 1000 = 0.10
- Test Stat
z = (0.06 − 0.14) / √(0.10 × 0.90 × (1/500 + 1/500))
= −0.08 / √(0.00036)
= −0.08 / 0.01897
≈ −4.22
- p-Valuep = 2 · P(Z ≤ −4.22) ≈ 0.000025
- Decisionp ≪ 0.01. Reject H₀.
Extremely strong evidence (p ≈ 0.000025) that the vaccine reduces infection rates compared to placebo.
Sections 9.13 – 9.14
Significance, Importance, and Common Mistakes
Statistical vs. Practical Significance · Effect Sizes · Ten Recurring Errors
Section 9.13
Statistical Significance Versus Practical Importance
Statistical significance tells us that an effect is detectable in data. It does not tell us the effect is large, meaningful, or worth caring about. With a large enough sample, even a trivially small difference from the null will produce a statistically significant result. This is because the test statistic depends on both the effect size and the sample size n: as n grows, the standard error shrinks, and tiny differences produce large t-values.
Example 9.13.1
Educational software — large n inflates significance
- SetupTesting if software improves scores: μ₀ = 75, x̄ = 75.2, s = 12.
- n = 10,000
t = 0.2 / (12/√10000) = 0.2 / 0.12 = 1.67
p-value ≈ 0.095 (not significant at α = 0.05)
- n = 40,000
t = 0.2 / (12/√40000) = 0.2 / 0.06 = 3.33
p-value ≈ 0.001 (highly significant at α = 0.05)
- Effect Sized = 0.2/12 = 0.017 — an extremely small effect in either case. The significance came from the sample size, not the effect.
A 0.2-point difference on a 100-point scale is practically meaningless regardless of p-value. Always report effect sizes.
Effect Sizes
Cohen's d — Standardized Effect Size
One-sample: d = (x̄ − μ₀) / s
Two-sample: d = (x̄₁ − x̄₂) / s_pooled
Conventional benchmarks (apply with judgment): |d| ≈ 0.2 → small; |d| ≈ 0.5 → medium; |d| ≈ 0.8 → large.
Best Practice
Provide both a test result (with p-value) and a confidence interval for the effect. Confidence intervals show the plausible range of true effect size — not just whether it is detectable. Many journals now require this combination. A narrow interval around a small value is far more informative than a p-value alone.
Section 9.14
Common Hypothesis Testing Mistakes
Mistake 1 — Confusing "Fail to Reject" with "Proving the Null"
"The p-value is 0.40, so we have proven the null is true."
Correct: Failing to reject H₀ means the data are not inconsistent with H₀ — not that H₀ is proven. Low power or small sample size may be responsible. The null is never "proved" in the frequentist framework.
Mistake 2 — Misinterpreting the p-Value
"p = 0.03 means there is only a 3% chance the null hypothesis is true."
Correct: p = 0.03 means that if H₀ were true, data this extreme would occur 3% of the time. The p-value is P(data | H₀), not P(H₀ | data). The latter requires Bayesian reasoning (Chapter 10).
Mistake 3 — Choosing Hypotheses After Seeing Data (p-Hacking)
"The data showed the mean was above the null, so I switched to a one-sided upper test to get p = 0.04 instead of p = 0.08."
Correct: The alternative hypothesis must be chosen before data collection. Switching direction after observing data halves the p-value artificially, inflating the actual Type I error rate far above α.
Mistake 4 — Multiple Comparisons Without Correction
Running 40 hypothesis tests at α = 0.05 and reporting 3 "significant" findings as discoveries.
Correct: With 40 tests at α = 0.05 where all nulls are true, we expect 2 rejections by chance. Apply corrections (e.g., Bonferroni: α/number of tests). Always report all tests, not only significant ones.
Mistake 5 — Equating Statistical and Practical Significance
"The effect was highly significant (p = 0.0001), so it is practically important."
Correct: With large n, even trivially small effects become statistically significant. Report Cohen's d or a confidence interval to convey the actual size of the effect.
Mistake 6 — Writing Hypotheses with Sample Statistics
H₀: x̄ = 42 or Hₐ: p̂ > 0.5
Correct: Hypotheses are always about population parameters. Write H₀: μ = 42 and Hₐ: p > 0.5. Sample statistics are observed data, not unknown quantities being tested.
Mistake 7 — Treating α = 0.049 and p = 0.051 as Fundamentally Different
"This result (p = 0.049) is real; that result (p = 0.051) is not."
Correct: The significance threshold is a convention, not a bright line between "proven" and "not proven." A p-value of 0.049 and 0.051 represent essentially the same level of evidence. Report the actual p-value and let readers judge.
Mistake 8 — Ignoring Assumptions
Applying a t-test to a small sample (n = 8) drawn from a highly skewed population without checking normality.
Correct: Every test rests on assumptions. The one-sample t-test requires approximate normality for small samples. Check with a histogram or QQ plot, or use non-parametric methods when assumptions are violated.
Chapter 9 — Back Matter
Chapter Summary
- Sec 9.1Hypothesis testing exists to quantify how surprising data would be under a default model. It is built on sampling distribution theory and implements falsification rather than confirmation.
- Sec 9.2Vague research questions must become precise claims about population parameters. Statistical evidence must account for variability — the same numerical difference is stronger evidence with smaller standard error or larger sample size.
- Sec 9.3H₀ is the default claim about a parameter; Hₐ specifies the departure we wish to detect. Hypotheses are written in terms of parameters, not statistics, and must be specified before data are collected.
- Sec 9.4The test statistic standardizes the observed departure: (observed − null) / standard error. For means with σ unknown, this gives a t-statistic with n−1 degrees of freedom. For proportions, a z-statistic using p₀ in the denominator.
- Sec 9.5The p-value is P(data this extreme | H₀ true). It is not the probability the null is true, nor the probability of a chance result. For two-sided tests, the p-value is the two-tail area; for one-sided tests, a single tail.
- Sec 9.6The significance level α is the pre-specified threshold for the p-value. The decision rule: reject H₀ if p-value ≤ α. Use precise language: "reject," "fail to reject," and "statistically significant at level α."
- Sec 9.7Two-sided tests detect departures in either direction; one-sided tests detect departures in one specified direction. The direction must be chosen before data collection. Switching sides after seeing data inflates Type I error.
- Sec 9.8Type I error = rejecting a true null; probability = α. Type II error = failing to reject a false null; probability = β. For fixed n, α and β trade off: reducing one increases the other. More data reduces both.
- Sec 9.9Power = 1 − β. Power increases with effect size, sample size, larger α, and smaller σ. Sample size planning uses n = [(z_{α/2} + z_β)σ/δ]². Underpowered studies that fail to reject H₀ do not prove the null.
- Sec 9.10The one-sample t-test uses t = (x̄ − μ₀)/(s/√n) with df = n−1. The six-step procedure: state hypotheses, check conditions, compute statistic, compute p-value, state decision, write conclusion in context.
- Sec 9.11The proportion z-test uses p₀ (not p̂) in the standard error, because we compute probabilities under H₀. Conditions: np₀ ≥ 10 and n(1−p₀) ≥ 10. Test and confidence interval are in duality.
- Sec 9.12Two-sample t-test compares independent group means using Welch degrees of freedom. Paired t-test reduces to a one-sample test on differences. Two-sample proportion test uses a pooled proportion under H₀.
- Sec 9.13With large n, trivial effects become statistically significant. Effect sizes (Cohen's d) measure practical magnitude independently of sample size. Confidence intervals complement p-values by showing the plausible range of the effect.
- Sec 9.14Eight major mistakes: failing to distinguish "not reject" from "prove null"; misinterpreting p-values; p-hacking; ignoring multiple comparisons; confusing statistical with practical significance; writing hypotheses in terms of statistics; treating significance thresholds as bright lines; ignoring assumptions.
Key Terms Review
null hypothesis (H₀)
The default claim assumed true unless data provide sufficient evidence against it. Specifies a single parameter value.
alternative hypothesis (Hₐ)
The claim accepted if H₀ is rejected. Must be specified before data collection. Can be one-sided or two-sided.
test statistic
A standardized summary of the data measuring departure from H₀: (observed − null value) / standard error.
null distribution
The sampling distribution of the test statistic when H₀ is true. Enables calculation of p-values.
p-value
P(test statistic as extreme or more extreme | H₀ true). A conditional probability — not the probability H₀ is true.
significance level (α)
Pre-specified threshold for the p-value. The probability of Type I error. Common values: 0.05, 0.01, 0.10.
critical value
Boundary of the rejection region in test-statistic units. For a two-sided z-test at α = 0.05: ±1.96.
rejection region
The set of test statistic values that lead to rejection of H₀. Determined by α and the null distribution.
Type I error
Rejecting a true null hypothesis (false positive). Probability equals α by design.
Type II error
Failing to reject a false null hypothesis (false negative). Probability = β; depends on true parameter, n, and σ.
power
The probability of correctly rejecting a false null. Power = 1 − β. Increases with effect size and sample size.
effect size
A measure of the magnitude of an effect, independent of sample size. Cohen's d = (difference in means) / standard deviation.
one-sample t-test
Test for a population mean when σ is unknown; uses T ~ t_{n−1} under H₀.
two-sample t-test
Test comparing means from two independent populations using Welch degrees of freedom.
paired t-test
Test on the mean of paired differences; reduces dependent two-sample data to a one-sample problem.
pooled proportion
Combined estimate of a common proportion under H₀: p₁ = p₂. Used in the standard error for two-sample proportion tests.
Cohen's d
Standardized effect size: difference in means divided by a standard deviation. Benchmarks: 0.2 small, 0.5 medium, 0.8 large.
p-hacking
Manipulating analysis choices — including test direction, threshold, or which tests to report — to achieve a significant result. Seriously distorts error rates.
multiple comparisons
The inflation of Type I error when many tests are conducted simultaneously. Bonferroni correction: use α/k for k tests.
HARKing
Hypothesizing After Results are Known — choosing hypotheses to match the data after examination. Invalidates the test's error rate guarantees.
Skill Practice
Write appropriate null and alternative hypotheses for each scenario. State whether the test is one-sided or two-sided and identify the parameter.
- A manufacturer states average battery lifetime is 500 hours. A consumer group believes the batteries last less.
- A clinical trial tests whether a new medication changes mean recovery time from 14 days.
- A social scientist believes more than 40% of adults in a city read a daily newspaper.
- An engineer suspects mean tensile strength of a new alloy exceeds 300 MPa.
- A researcher compares the proportion of voters supporting a candidate in rural vs. urban areas.
- Identify the error: H₀: x̄ = 42. Rewrite correctly.
- After collecting data, a student switches from two-sided to one-sided Hₐ to reduce the p-value. Explain the problem.
- n = 16, x̄ = 23.5, s = 4.0, μ₀ = 25. Compute t and state df.
- n = 100, p̂ = 0.44, p₀ = 0.50. Compute z.
- Group 1: n₁ = 40, x̄₁ = 115, s₁ = 15. Group 2: n₂ = 35, x̄₂ = 109, s₂ = 18. Compute t for H₀: μ₁ = μ₂.
- Paired: n = 10 pairs, d̄ = −3.2, s_d = 2.9. Compute t.
- n = 50, x̄ = 97.8, σ = 8 (known), μ₀ = 100. Compute z.
- Trial: 24/200 vs. 40/200 infections. Compute pooled proportion and z for H₀: p₁ = p₂.
- A test gives p = 0.03 with α = 0.05. Write a correct one-sentence interpretation of this p-value.
- A manager says "p = 0.42 means there is a 42% chance the training works." Correct this misinterpretation.
- z = 1.85 for a two-sided test. Compute the p-value using the standard normal.
- t = −2.30 with df = 24, one-sided (Hₐ: μ < μ₀). Find the approximate p-value.
- True or False: A smaller p-value means the alternative hypothesis is more likely to be true. Explain.
- Study A: p = 0.04, n = 40. Study B: p = 0.04, n = 400. Do both represent the same strength of evidence?
- t = 2.58, df = 30, two-sided, α = 0.05 (critical value ≈ 2.042). State decision and conclusion: test whether mean blood pressure differs from 120 mmHg.
- z = −1.55, one-sided (Hₐ: p < p₀), p-value ≈ 0.061, α = 0.10. State decision and interpret.
- t = 1.98, df = 50, two-sided, α = 0.01 (critical value ≈ 2.68). State decision. Would it change at α = 0.05?
- Vaccine study: p = 0.0003 for two-sided test. Decide at α = 0.05 and α = 0.01. Write a conclusion.
- A researcher gets p = 0.051 and a colleague says "just round it to 0.05 and reject." Discuss why this is problematic.
- Define Type I error, Type II error, and power for testing whether a new engine design improves fuel efficiency above 35 mpg.
- A researcher uses α = 0.01 instead of α = 0.05. How does this affect β, holding everything else fixed?
- A drug study uses n = 25 patients. A statistician warns it may be underpowered. What does this mean if the drug is genuinely effective?
- Find required n to detect δ = 5 with σ = 20, α = 0.05 (two-sided), 80% power. (z_{0.025} = 1.96, z_{0.20} = 0.842.)
- Describe the power curve for a two-sided test of H₀: μ = 50. How does power change as the true μ moves away from 50?
- A company sets α = 0.001 for a safety test. Is this wise if Type II errors (missing a real hazard) are far more costly than Type I?
Use the six-step procedure for each application test.
- Medication Study. Historical average improvement for standard treatment: 10.0. New treatment sample: n = 45, x̄ = 12.3, s = 4.1. Test at α = 0.05 whether the new treatment differs from the historical average.
- Manufacturing. Target diameter: 25 mm. Sample: n = 36, x̄ = 25.4, s = 0.9 mm. Test for miscalibration (two-sided) at α = 0.05.
- Voting. In 600 voters, 318 support a ballot measure. Test whether the proportion differs from 50% at α = 0.05.
- Educational Intervention. Previous mean score: 68. After new curriculum, n = 40, x̄ = 71.5, s = 9.2. Test whether the mean increased at α = 0.05.
- Two Drugs. Drug A: n₁ = 50, x̄₁ = 28, s₁ = 10. Drug B: n₂ = 50, x̄₂ = 23, s₂ = 12. Test for difference in cholesterol reduction at α = 0.01.
- Runners. Differences (standard − cushioned shoes, seconds): 12, −3, 8, 15, 6, 10, 2, 9. Test whether cushioned shoes change mean race time at α = 0.05.
Error Analysis
Each item contains an error or misconception. Identify it and write the correct interpretation or procedure.
Problem G1
"The p-value was 0.07, so there is a 7% probability the null hypothesis is true."
Identify the error. What does p = 0.07 actually mean?
Problem G2
H₀: x̄ = 100 and Hₐ: x̄ ≠ 100
What is wrong with writing hypotheses this way? Write correct versions.
Problem G3
After seeing x̄ > μ₀, a researcher switches from Hₐ: μ ≠ μ₀ to Hₐ: μ > μ₀, reporting p = 0.04 instead of 0.08.
Why is this problematic? What is the actual effect on error rates?
Problem G4
A study with n = 50,000 finds p = 0.0001 for a 0.5-unit difference. "This is highly significant, meaning the effect is large and practically important."
Explain the error. How would you assess practical importance?
Problem G5
A study with n = 10 fails to reject H₀. "We have proven that μ = 50."
What is wrong? What should the conclusion say?
Problem G6
A team runs 50 tests at α = 0.05, finds 3 significant results, and publishes only those 3.
What phenomenon is this? How many significant results would be expected by chance if all nulls are true?
Chapter 9 Checkpoint
Chapter 9 Checkpoint
Complete without assistance — assesses all key skills and concepts from Chapter 9
Checkpoint 1 — One-Sample t-Test (Full Procedure)
- A nutritionist claims average daily sugar intake for adults is 90 grams. A survey of n = 64 adults gives x̄ = 94.5 g and s = 22 g.
- (a) State null and alternative hypotheses for a two-sided test.
- (b) Compute the test statistic.
- (c) Find the approximate p-value using df = 63.
- (d) State your decision at α = 0.05.
- (e) Write a conclusion in context.
- (f) Compute Cohen's d. Is this a large, medium, or small effect?
Checkpoint 2 — Proportion Test
- A factory claims 5% of bulbs are defective (p₀ = 0.05). A sample of 400 bulbs yields 28 defectives.
- (a) State null and alternative hypotheses (two-sided).
- (b) Check the conditions for the proportion test.
- (c) Compute the test statistic.
- (d) Compute the p-value.
- (e) Make a decision at α = 0.01.
Checkpoint 3 — Two-Sample t-Test
- Protocol A: n₁ = 30, x̄₁ = 21.4 days, s₁ = 5.2 days. Protocol B: n₂ = 30, x̄₂ = 24.1 days, s₂ = 6.8 days.
- (a) State the hypotheses for a two-sided test.
- (b) Compute the two-sample t-test statistic.
- (c) Using df ≈ 55, find the approximate p-value.
- (d) At α = 0.05, state your decision.
- (e) Compute the 95% confidence interval for μ₁ − μ₂ and verify it is consistent with the test decision.
Checkpoint 4 — Short Answer
- Explain in two sentences why failing to reject H₀ does not mean H₀ is true.
- Plan a study to test H₀: μ = 50 with σ = 10, α = 0.05 (two-sided), for 90% power to detect μ = 55. Compute required n. (z_{0.025} = 1.96, z_{0.10} = 1.282.)
- If a test at α = 0.05 is repeated 200 times where H₀ is always true, how many rejections are expected? What phenomenon does this illustrate?
- One researcher finds p = 0.049; another finds p = 0.051. Comment on whether this distinction is as meaningful as it appears.
Looking Ahead
Chapter 10: Likelihood, Maximum Likelihood, and Bayesian Foundations
Chapter 9 completed the classical frequentist framework for inference. In this framework, probability is a long-run frequency. The p-value is P(data this extreme | H₀ true) — a conditional probability given the null model. Crucially, it says nothing about the probability that any hypothesis is true.
Chapter 10 opens two alternative frameworks. Likelihood-based inference asks: given the data we observed, which parameter values are most consistent with what we see? The likelihood function compares parameter values in light of data, and maximum likelihood estimation finds the value that makes the observed data most probable.
Bayesian inference goes further: it treats the parameter itself as a random quantity with a prior distribution, updates that distribution using the likelihood of the data, and produces a posterior distribution representing updated beliefs. This directly answers the question that p-values cannot: what should we believe about the parameter after seeing this data?
MGU Library Connection
- Chapter 7: Limit Theorems and Sampling Distributions — the Central Limit Theorem and sampling distributions that underlie all tests in this chapter
- Chapter 8: Estimation and Confidence Intervals — the duality between tests and intervals; confidence intervals as complements to p-values
- Chapter 10: Likelihood and Bayesian Foundations — the frameworks that answer what p-values cannot: the probability of hypotheses given data
- Chapter 12: Multiple Regression and ANOVA — hypothesis tests for regression coefficients and group comparisons as extensions of the framework here
- Appendix N of this textbook: Hypothesis Testing Reference — decision rules, critical values, and test summaries