The Central Idea
A hypothesis test begins with a proposed statement about a population parameter.
We then ask:
If that statement were true, how unusual would the observed sample result be?
The observed statistic is compared with a reference distribution generated under the null hypothesis.
If the result is sufficiently incompatible with that reference model, we reject the null hypothesis.
Statistical Hypotheses
A statistical hypothesis is a claim about a population parameter or probability model.
Examples include:
\[ \mu=50, \] \[ p=0.40, \] \[ \mu_1-\mu_2=0. \]Hypothesis testing gives a formal procedure for evaluating such claims using sample evidence.
Null and Alternative Roles
Every classical hypothesis test distinguishes between:
- the null hypothesis, which supplies the reference model;
- the alternative hypothesis, which describes the departure of interest.
These roles will be developed in more detail in the next lesson.
A Typical Test Structure
A hypothesis test often follows this pattern:
- state the null and alternative hypotheses;
- choose a test statistic;
- derive or identify its reference distribution under the null;
- calculate the observed test statistic;
- measure how extreme that value is under the null;
- compare the evidence with a decision threshold;
- state the conclusion in context.
Test Statistics
A test statistic is a numerical summary designed to measure disagreement between the observed data and the null hypothesis.
A common structure is
\[ \boxed{ \text{test statistic} = \frac{ \text{observed estimate} - \text{null value} }{ \text{standard error under the null} } }. \]This expresses the discrepancy in standard-error units.
Why Standardization Matters
A raw difference does not tell us whether a result is large relative to its expected sampling variability.
A difference of \(5\) may be enormous when the standard error is \(1\), but modest when the standard error is \(20\).
Standardization converts the discrepancy to a comparable probability scale.
Reference Distribution
The reference distribution describes how the test statistic behaves when the null hypothesis is true under the assumptions of the test.
Common reference distributions include:
- standard normal distributions;
- Student's \(t\) distributions;
- chi-square distributions;
- \(F\) distributions;
- exact discrete distributions;
- permutation or randomization distributions.
Observed Test Statistic
Once the sample is observed, the test statistic becomes a specific number.
For example,
\[ z=2.35. \]This means the observed estimate lies \(2.35\) standard errors above the null value according to the standardization used by the test.
Extremeness under the Null
The more extreme the observed statistic is relative to its null distribution, the stronger the evidence against the null model.
“Extreme” depends on the alternative hypothesis.
A right-tailed test treats unusually large values as evidence.
A left-tailed test treats unusually small values as evidence.
A two-sided test treats unusually large magnitudes in either direction as evidence.
Decision Rules
A hypothesis test uses a decision rule to determine whether the observed evidence is strong enough to reject the null hypothesis.
Two equivalent common approaches are:
- compare the test statistic with a critical value;
- compare the p-value with the significance level.
Critical-Value Approach
In the critical-value approach, the reference distribution is divided into:
- a rejection region;
- a nonrejection region.
If the observed test statistic falls in the rejection region, the null hypothesis is rejected.
Example: Two-Sided z Critical Values
For a two-sided standard normal test at significance level
\[ \alpha=0.05, \]the critical values are approximately
\[ -1.96 \]and
\[ 1.96. \]The null is rejected when
\[ |z|>1.96. \]One-Sided Critical Values
For a right-tailed standard normal test at
\[ \alpha=0.05, \]the critical value is approximately
\[ 1.645. \]Reject when
\[ z>1.645. \]For a left-tailed test, reject when
\[ z<-1.645. \]Two-Sided Tests
A two-sided alternative examines departures in either direction.
For example,
\[ H_0:\mu=50 \]versus
\[ H_A:\mu\neq50. \]Both unusually small and unusually large sample means can count as evidence against the null.
Right-Tailed Tests
A right-tailed test considers values larger than the null value as evidence.
For example,
\[ H_0:\mu=50 \]versus
\[ H_A:\mu>50. \]Only sufficiently large positive deviations support rejection.
Left-Tailed Tests
A left-tailed test considers values smaller than the null value as evidence.
For example,
\[ H_0:\mu=50 \]versus
\[ H_A:\mu<50. \]Only sufficiently large negative deviations support rejection.
The Direction Should Be Chosen Before Seeing the Data
The alternative hypothesis should reflect the scientific or operational question, not whichever direction the observed sample happens to favor.
Choosing a one-sided alternative after inspecting the data changes the error properties of the procedure.
Hypothesis Test for a Mean with Known \(\sigma\)
Suppose the null hypothesis is
\[ H_0:\mu=\mu_0, \]the population standard deviation \(\sigma\) is known, and the normal approximation is appropriate.
The z statistic is
\[ \boxed{ z= \frac{ \bar{x}-\mu_0 }{ \sigma/\sqrt{n} } }. \]Example: z Test for a Mean
Suppose
\[ H_0:\mu=100, \]and the sample gives
\[ \bar{x}=104,\qquad \sigma=12,\qquad n=36. \]The standard error is
\[ \frac{12}{\sqrt{36}} = 2. \]Therefore,
\[ z= \frac{104-100}{2} = \boxed{2}. \]Unknown Population Standard Deviation
In most real problems, \(\sigma\) is unknown.
We estimate it using the sample standard deviation \(s\).
Under a normal population model, the resulting statistic follows Student's \(t\) distribution.
One-Sample t Test
To test
\[ H_0:\mu=\mu_0, \]use
\[ \boxed{ t= \frac{ \bar{x}-\mu_0 }{ s/\sqrt{n} } } \]with
\[ n-1 \]degrees of freedom under the standard normal-population model.
Example: One-Sample t Test
Suppose
\[ H_0:\mu=50, \]and
\[ \bar{x}=54,\qquad s=10,\qquad n=25. \]Then
\[ t= \frac{54-50}{10/\sqrt{25}} = \frac{4}{2} = \boxed{2}. \]The reference distribution has
\[ 24 \]degrees of freedom.
Conditions for a One-Sample t Test
The test is most directly justified when:
- the sampling process supports the intended inference;
- observations are independent or appropriately modeled;
- the population is normal, or the sample mean has an adequate approximate normal distribution;
- extreme skewness and influential outliers are not severe relative to the sample size.
Hypothesis Test for a Population Proportion
Suppose the null hypothesis is
\[ H_0:p=p_0. \]Under a large-sample normal approximation, the test statistic is
\[ \boxed{ z= \frac{ \hat{p}-p_0 }{ \sqrt{ p_0(1-p_0)/n } } }. \]Notice that the standard error uses the null value \(p_0\).
Why the Null Proportion Appears in the Standard Error
A hypothesis test asks how the statistic would behave if the null were true.
Therefore, when testing
\[ p=p_0, \]the null reference variance is
\[ \frac{p_0(1-p_0)}{n}. \]This differs from the plug-in standard error often used for a basic confidence interval.
Example: One-Proportion z Test
Suppose
\[ H_0:p=0.50, \]and a sample of
\[ n=400 \]produces
\[ \hat{p}=0.55. \]Under the null,
\[ \operatorname{SE}_0 = \sqrt{ \frac{0.5(0.5)}{400} } = 0.025. \]Therefore,
\[ z= \frac{0.55-0.50}{0.025} = \boxed{2}. \]Success-Failure Condition for Proportion Tests
A common large-sample check uses the null expected counts:
\[ np_0\ge10 \]and
\[ n(1-p_0)\ge10. \]These are practical approximation guidelines, not exact mathematical boundaries.
Two-Sample Tests for Means
To compare two population means, a common null hypothesis is
\[ H_0:\mu_1-\mu_2=0. \]The observed difference is
\[ \bar{x}_1-\bar{x}_2. \]The difference is standardized using a standard error appropriate to the two-sample design.
Welch Two-Sample t Test
For independent samples, a widely used statistic is
\[ \boxed{ t= \frac{ (\bar{x}_1-\bar{x}_2)-\Delta_0 }{ \sqrt{ s_1^2/n_1+s_2^2/n_2 } } }. \]Here \(\Delta_0\) is the null difference, often \(0\).
Degrees of freedom are approximated using the Welch-Satterthwaite formula.
Why Welch's Test Is Useful
Welch's two-sample \(t\) test does not require equal population variances.
It is therefore commonly preferred over the pooled equal-variance test when equal variance is not strongly justified.
Paired t Test
For paired observations, define
\[ D_i=X_i-Y_i. \]To test a null mean difference
\[ \mu_D=\Delta_0, \]use
\[ \boxed{ t= \frac{ \bar{d}-\Delta_0 }{ s_d/\sqrt{n} } }. \]This is a one-sample \(t\) test applied to the paired differences.
Two-Sample Test for Proportions
A common null hypothesis is
\[ H_0:p_1-p_2=0. \]When the null says the two proportions are equal, a pooled proportion is commonly used in the null standard error.
Pooled Proportion under the Null
If sample 1 has \(x_1\) successes out of \(n_1\) and sample 2 has \(x_2\) successes out of \(n_2\), the pooled estimate is
\[ \boxed{ \hat{p}_{\text{pool}} = \frac{x_1+x_2}{n_1+n_2} }. \]Under the null of equal population proportions, both samples are treated as arising from a common success probability.
Two-Proportion z Statistic
The test statistic is
\[ \boxed{ z= \frac{ \hat{p}_1-\hat{p}_2 }{ \sqrt{ \hat{p}_{\text{pool}} (1-\hat{p}_{\text{pool}}) \left( \frac{1}{n_1}+\frac{1}{n_2} \right) } } }. \]Exact Tests
When large-sample approximations are poor, exact probability calculations may be available.
Examples include:
- exact binomial tests;
- Fisher's exact test for certain contingency-table problems;
- exact permutation tests.
Exact Binomial Test
Suppose
\[ X\sim\operatorname{Binomial}(n,p_0) \]under the null hypothesis.
Instead of using a normal approximation, an exact test calculates tail probabilities directly from the binomial distribution.
Permutation Tests
A permutation test constructs a null reference distribution by rearranging labels or assignments in a way justified by the null hypothesis and study design.
The observed statistic is compared with the distribution of statistics generated by those rearrangements.
Randomization Tests
In randomized experiments, a randomization test can use the actual treatment assignment mechanism to generate the null distribution.
This directly connects inference to the random assignment used in the experiment.
Hypothesis Tests and Confidence Intervals
For many standard two-sided procedures, confidence intervals and hypothesis tests are dual.
A parameter value outside a
\[ 100(1-\alpha)\% \]confidence interval is typically rejected by the corresponding two-sided test at level
\[ \alpha. \]Example: Confidence Interval and Test Agreement
Suppose a \(95\%\) confidence interval for
\[ \mu_1-\mu_2 \]is
\[ (2.1,\;7.4). \]Since
\[ 0 \]is not in the interval, the corresponding two-sided test of
\[ H_0:\mu_1-\mu_2=0 \]would reject at
\[ \alpha=0.05, \]assuming compatible methods are used.
Statistical Evidence Is Not Proof
Rejecting a null hypothesis does not prove the alternative with logical certainty.
Statistical testing operates under uncertainty.
The conclusion means that the observed data are sufficiently incompatible with the null model according to the chosen procedure.
Failure to Reject Is Not Acceptance
If a test does not reject the null, we should usually say
fail to reject the null hypothesis.
This does not prove that the null is true.
The data may simply be insufficiently informative to distinguish the null from nearby alternatives.
Sample Size Affects Detectability
Larger samples generally reduce standard errors.
For a fixed true effect, a smaller standard error makes the standardized test statistic larger in magnitude on average.
Thus larger samples can detect smaller departures from the null.
Very Large Samples Can Detect Tiny Effects
With enough observations, even a very small difference from the null can produce a large test statistic.
Statistical significance therefore does not automatically imply practical importance.
Effect size and confidence intervals should accompany hypothesis-test conclusions.
Practical Significance
A difference can be statistically detectable but operationally trivial.
For example, a process change might reduce average completion time by only \(0.2\) seconds.
With millions of observations, that difference could be statistically significant while having little practical value.
Effect Size
An effect size describes the magnitude of a difference or relationship.
Examples include:
- a raw difference in means;
- a difference in proportions;
- a standardized mean difference;
- a correlation coefficient;
- an odds ratio;
- a risk ratio.
Hypothesis tests answer an evidence question; effect sizes answer a magnitude question.
Model Assumptions Matter
A test statistic can be computed even when its assumptions are badly violated.
That does not make its p-value or rejection rule valid.
The reference distribution depends on the assumed sampling and probability structure.
Independence Matters
Many textbook tests assume observations are independent or approximately independent.
Clustered data, repeated measurements, time series, spatial dependence, and network dependence can invalidate naive standard errors.
The testing procedure must match the actual dependence structure.
Random Sampling and Generalization
A test can be mathematically correct for the observed sample while still having limited population generalizability.
Generalization depends on how the sample relates to the target population.
Hypothesis testing does not repair undercoverage, voluntary response, or other representation problems.
Random Assignment and Causal Testing
In experiments, random assignment helps support causal comparisons by controlling systematic treatment-group differences through chance.
A statistically significant difference in a randomized experiment can therefore support stronger causal interpretation than the same association in an uncontrolled observational study.
Multiple Testing
Performing many hypothesis tests increases the chance of obtaining at least one apparently significant result by chance.
For example, if many independent null hypotheses are tested at level
\[ \alpha=0.05, \]some rejections are expected even when all null hypotheses are true.
Familywise Error Rate
The familywise error rate is the probability of making at least one Type I error within a designated family of tests.
Procedures such as Bonferroni adjustment can control this probability.
Bonferroni Adjustment
If \(m\) tests are performed and the desired familywise error rate is \(\alpha\), a simple Bonferroni rule tests each hypothesis at
\[ \boxed{ \frac{\alpha}{m} }. \]This method is simple and broadly valid but can be conservative.
False Discovery Rate
In large-scale testing, another goal is to control the expected proportion of false rejections among all rejected hypotheses.
This is called the false discovery rate.
It addresses a different error criterion from familywise error control.
Pre-Specification
Testing procedures are most interpretable when major choices are made before examining the outcomes.
Important choices include:
- the primary outcome;
- the null and alternative hypotheses;
- the significance level;
- the direction of the test;
- the analysis population;
- the planned statistical model.
Data-Dependent Testing Can Inflate Evidence
If many outcomes, subgroups, transformations, or model specifications are tried and only the most favorable result is reported, nominal significance levels no longer describe the full search process.
This can create exaggerated evidence against the null.
Replication
A statistically significant result in one study is not the end of scientific evaluation.
Replication examines whether similar findings appear in new data collected under comparable or deliberately varied conditions.
Reproducible evidence is stronger than a single isolated rejection.
Common Error: Treating a p-Value as the Probability the Null Is True
A p-value is calculated assuming the null hypothesis is true.
It does not directly equal
\[ P(H_0\mid \text{data}). \]That is a different probability statement requiring a different inferential framework.
Common Error: Accepting the Null after a Nonsignificant Result
Failure to reject can result from:
- the null being approximately true;
- small sample size;
- high variability;
- weak measurement;
- a small but real effect.
A nonsignificant result does not automatically prove equality.
Common Error: Choosing a One-Sided Test after Seeing the Data
A one-sided test allocates the rejection probability to one direction.
Choosing that direction after observing the sample distorts the intended significance level.
Direction should be justified in advance.
Common Error: Ignoring Practical Importance
A tiny effect can be statistically significant in a very large sample.
Statistical significance should therefore be interpreted alongside effect magnitude, uncertainty, and practical thresholds.
Common Error: Ignoring Assumptions
A test result is only as trustworthy as the assumptions connecting the data to the reference distribution.
Independence, sampling design, distributional conditions, and model specification must be checked.
Common Error: Treating Statistical Significance as Causation
A small p-value indicates incompatibility with a null model.
It does not by itself establish that one variable caused another.
Causal interpretation depends on study design, especially random assignment or other credible identification strategies.
Common Error: Running Many Tests and Reporting Only One
Searching across many hypotheses increases the chance of finding an apparently significant result by chance.
Multiple-testing procedures, preregistration, replication, and transparent reporting help manage this problem.
A Reliable Procedure for Hypothesis Testing
- State the population parameter of interest.
- State the null hypothesis.
- State the alternative hypothesis.
- Choose the significance level before examining the result.
- Identify the appropriate test statistic.
- Check sampling, independence, and model assumptions.
- Compute the observed statistic.
- Determine the reference distribution under the null.
- Compute the p-value or compare with a critical value.
- Make the reject or fail-to-reject decision.
- State the conclusion in context.
- Report effect size and uncertainty when possible.
- Discuss practical significance and study limitations.
Example: Full One-Sample Mean Test
Suppose we test
\[ H_0:\mu=40 \]against
\[ H_A:\mu\neq40. \]A sample gives
\[ \bar{x}=44,\qquad s=12,\qquad n=36. \]The standard error is
\[ \frac{12}{\sqrt{36}} = 2. \]The test statistic is
\[ t= \frac{44-40}{2} = 2. \]The reference distribution is a \(t\) distribution with
\[ 35 \]degrees of freedom.
The final decision is made by comparing this statistic with the chosen significance threshold or by computing its two-sided p-value.
Example: Full One-Proportion Test
Suppose we test
\[ H_0:p=0.60 \]against
\[ H_A:p\neq0.60. \]A sample of
\[ n=500 \]produces
\[ \hat{p}=0.65. \]Under the null,
\[ \operatorname{SE}_0 = \sqrt{ \frac{0.60(0.40)}{500} } \approx0.02191. \]Therefore,
\[ z = \frac{0.65-0.60}{0.02191} \approx \boxed{2.28}. \]The statistic is then compared with the standard normal reference distribution.
Practice 1: Identify the Test Statistic Structure
Complete the general form:
\[ \text{test statistic} = \frac{ \text{observed estimate}-\text{null value} }{ \text{?} }. \]Practice 2: One-Sample z Test
Suppose
\[ H_0:\mu=50, \]and
\[ \bar{x}=54,\qquad \sigma=10,\qquad n=25. \]Find the z statistic.
Practice 3: One-Sample t Test
Suppose
\[ H_0:\mu=20, \]and
\[ \bar{x}=23,\qquad s=8,\qquad n=16. \]Find the t statistic.
Practice 4: One-Proportion z Test
Suppose
\[ H_0:p=0.50, \]with
\[ \hat{p}=0.56,\qquad n=400. \]Find the z statistic.
Practice 5: Critical-Value Decision
A two-sided z test at
\[ \alpha=0.05 \]produces
\[ z=2.40. \]Using critical values \(\pm1.96\), should the null be rejected?
Practice 6: Fail to Reject
A test fails to reject the null hypothesis.
Does this prove the null hypothesis is true?
Practice 7: One-Sided or Two-Sided?
A researcher wants to determine whether a new process changes the mean output in either direction.
Should the alternative be one-sided or two-sided?
Practice 8: Paired or Independent?
The same \(30\) patients are measured before and after a treatment.
Should the analysis use a paired test or an independent two-sample test?
Practice 9: Statistical versus Practical Significance
A study with one million observations detects a mean difference of \(0.01\) units with a very small p-value.
Does the small p-value alone show the difference is practically important?
Practice 10: Multiple Testing
A researcher performs \(100\) unrelated tests at
\[ \alpha=0.05 \]and reports only the smallest p-value.
What major statistical concern should be raised?
Practice Answers
Practice 1
The denominator is the standard error under the null model.
Practice 2
\[ z = \frac{54-50}{10/\sqrt{25}} = \frac{4}{2} = \boxed{2}. \]Practice 3
\[ t = \frac{23-20}{8/\sqrt{16}} = \frac{3}{2} = \boxed{1.5}. \]Practice 4
Under the null,
\[ \operatorname{SE}_0 = \sqrt{ \frac{0.5(0.5)}{400} } = 0.025. \]Therefore,
\[ z = \frac{0.56-0.50}{0.025} = \boxed{2.4}. \]Practice 5
Yes.
Because
\[ |2.40|>1.96, \]the observed statistic falls in the two-sided rejection region.
Practice 6
No.
Failure to reject means the evidence is not strong enough under the chosen procedure to reject the null. It does not prove the null is true.
Practice 7
The alternative should be two-sided because changes in either direction matter.
Practice 8
Use a paired test because the before-and-after observations come from the same patients.
Practice 9
No.
The p-value addresses statistical evidence against the null. Practical importance requires examining the magnitude of the effect and its relevance in context.
Practice 10
The major concern is multiple testing.
Searching across many tests increases the chance of obtaining an apparently significant result by chance unless the multiplicity is accounted for.
What to Remember
Hypothesis testing compares observed sample evidence with what would be expected under a specified null model.
- A test statistic measures discrepancy from the null in standardized units.
- The reference distribution describes the test statistic under the null hypothesis.
- Critical values and p-values are two ways to assess extremeness.
- One-sided and two-sided tests answer different questions.
- Mean tests often use z or Student's \(t\) statistics depending on whether \(\sigma\) is known.
- Proportion tests use null-based binomial variance.
- Paired data should usually be analyzed through paired differences.
- Failure to reject is not proof that the null is true.
- Statistical significance does not automatically imply practical importance or causation.
- Multiple testing, model assumptions, sampling design, and dependence all affect valid interpretation.
The most important habit is: ask how unusual the observed result would be under the null model, then interpret that evidence in light of the study design, effect size, and assumptions.
Where We Go Next
Hypothesis testing depends on two competing statements that define the reference model and the departure being investigated.
The next lesson, Null and Alternative Hypotheses, develops how those hypotheses are written, how equality and direction are assigned, how one-sided and two-sided alternatives differ, and how the hypotheses connect to the scientific question.