The Central Idea
A hypothesis test compares the observed data with what would be expected if the null hypothesis were true.
Two quantities organize the decision:
- the significance level \(\alpha\), chosen before the result is examined;
- the p-value, calculated from the observed data.
The basic decision rule is
\[ \boxed{ \text{reject }H_0 \text{ when } p\text{-value}\le\alpha }. \]The Significance Level
The significance level, denoted
\[ \alpha, \]is the threshold used to define how much evidence against the null hypothesis is required before rejection.
Common choices are
\[ 0.10,\qquad0.05,\qquad0.01. \]Alpha Is Chosen Before the Result
The value of \(\alpha\) should be selected before examining the observed test statistic or p-value.
This keeps the decision rule from being adjusted after the evidence is known.
Choosing \(\alpha\) after seeing the p-value undermines the intended error properties of the test.
What Alpha Controls
In a standard hypothesis test, \(\alpha\) controls the probability of rejecting the null hypothesis when the null is true, under the assumptions of the testing procedure.
This is the probability of a Type I error.
The next lesson will examine Type I and Type II errors directly.
The Rejection Region
The significance level determines a region of sufficiently extreme values under the null distribution.
This is the rejection region.
If the observed test statistic falls inside that region, the null hypothesis is rejected.
Example: Two-Sided Standard Normal Test
For a two-sided z test with
\[ \alpha=0.05, \]approximately \(2.5\%\) of the null distribution lies in each tail.
The critical values are approximately
\[ -1.96 \]and
\[ 1.96. \]The rejection rule is
\[ \boxed{|z|>1.96}. \]Example: Right-Tailed Standard Normal Test
For a right-tailed z test with
\[ \alpha=0.05, \]the rejection region contains the upper \(5\%\) of the standard normal distribution.
The critical value is approximately
\[ 1.645. \]Reject when
\[ \boxed{z>1.645}. \]Example: Left-Tailed Standard Normal Test
For a left-tailed z test with
\[ \alpha=0.05, \]the rejection region contains the lower \(5\%\) of the standard normal distribution.
Reject when
\[ \boxed{z<-1.645}. \]The P-Value
The p-value is the probability, calculated under the null hypothesis, of obtaining a test statistic at least as extreme as the observed statistic in the direction or directions specified by the alternative hypothesis.
Informally:
\[ \boxed{ p\text{-value} = P(\text{result at least as extreme as observed}\mid H_0) }. \]The P-Value Is Conditional on the Null
The p-value is calculated assuming the null hypothesis is true.
It measures how surprising the observed test statistic would be under that null reference model.
It does not directly give the probability that the null itself is true.
Small P-Values
A small p-value means the observed result is unusual under the null model.
The smaller the p-value, the stronger the statistical evidence against the null hypothesis, assuming the testing model and study design are valid.
Large P-Values
A large p-value means the observed result is not especially unusual under the null model.
This is not proof that the null hypothesis is true.
The data may simply be insufficiently informative to distinguish the null from plausible alternatives.
P-Value Decision Rule
The standard decision rule is
\[ \boxed{ p\le\alpha \quad\Longrightarrow\quad \text{reject }H_0 }. \]If
\[ p>\alpha, \]we fail to reject \(H_0\).
Example: Comparing p with Alpha
Suppose
\[ \alpha=0.05 \]and the p-value is
\[ p=0.032. \]Since
\[ 0.032<0.05, \]the result is statistically significant at the \(5\%\) level, and we reject the null hypothesis.
Example: Failing to Reject
Suppose
\[ \alpha=0.05 \]and
\[ p=0.21. \]Since
\[ 0.21>0.05, \]we fail to reject the null hypothesis.
Statistical Significance
A result is commonly called statistically significant at level \(\alpha\) when
\[ p\le\alpha. \]Statistical significance is therefore a comparison between the observed p-value and a pre-specified threshold.
Significance Is Threshold-Dependent
Suppose
\[ p=0.03. \]This result is statistically significant at
\[ \alpha=0.05 \]but not at
\[ \alpha=0.01. \]The evidence is the same; only the decision threshold changes.
Do Not Treat 0.05 as a Law of Nature
The choice
\[ \alpha=0.05 \]is conventional in many fields, but it is not mathematically privileged.
Appropriate thresholds depend on the consequences of false positives and false negatives, scientific norms, regulatory requirements, and the purpose of the analysis.
Alpha and Critical Values
Lowering \(\alpha\) makes the rejection criterion more stringent.
For a two-sided standard normal test:
\[ \alpha=0.10 \]uses critical values approximately
\[ \pm1.645, \]while
\[ \alpha=0.05 \]uses approximately
\[ \pm1.96, \]and
\[ \alpha=0.01 \]uses approximately
\[ \pm2.576. \]One-Sided P-Values
For a right-tailed test, the p-value is the probability under the null of a test statistic at least as large as the observed value.
If the observed statistic is \(z_{\text{obs}}\),
\[ \boxed{ p=P(Z\ge z_{\text{obs}}) }. \]Left-Sided P-Values
For a left-tailed test,
\[ \boxed{ p=P(Z\le z_{\text{obs}}) }. \]Large negative test statistics produce small left-tail p-values.
Two-Sided P-Values
For a symmetric reference distribution, a common two-sided p-value is
\[ \boxed{ p= 2P(Z\ge|z_{\text{obs}}|) }. \]Both tails count because departures in either direction are relevant.
Example: Right-Tailed P-Value
Suppose a right-tailed z test produces
\[ z=2.00. \]Then
\[ p=P(Z\ge2.00). \]From the standard normal distribution,
\[ p\approx0.0228. \]Example: Two-Sided P-Value
Suppose a two-sided z test produces
\[ z=2.00. \]Then
\[ p = 2P(Z\ge2.00) \]so
\[ p\approx2(0.0228)=0.0456. \]Same Test Statistic, Different Alternative
The same observed statistic can produce different p-values depending on the alternative hypothesis.
For
\[ z=2, \]the right-tailed p-value is about
\[ 0.0228, \]while the two-sided p-value is about
\[ 0.0456. \]The alternative determines which tail probabilities count as extreme.
Direction Must Be Specified in Advance
A one-sided p-value is not simply a convenient way to make a result appear more significant.
The directional alternative must be justified by the research question before the result is examined.
P-Values from t Distributions
When the test statistic follows Student's \(t\) distribution under the null, p-values are computed using that distribution and the appropriate degrees of freedom.
For a two-sided test,
\[ p = 2P(T_{\nu}\ge|t_{\text{obs}}|), \]where
\[ \nu \]denotes the degrees of freedom.
P-Values from Chi-Square Distributions
Chi-square tests use the chi-square reference distribution.
Since chi-square statistics are nonnegative, larger values commonly indicate stronger disagreement with the null model.
A typical p-value has the form
\[ P(\chi^2_{\nu}\ge \chi^2_{\text{obs}}). \]P-Values from F Distributions
Tests involving variance ratios and analysis of variance often use an \(F\) distribution.
A typical upper-tail p-value is
\[ P(F_{\nu_1,\nu_2}\ge F_{\text{obs}}). \]Exact P-Values
An exact p-value is calculated directly from the exact null distribution rather than from a large-sample approximation.
Examples include exact binomial tests and some permutation tests.
Approximate P-Values
Many common tests use an approximation to the null distribution.
Examples include:
- normal approximations for proportions;
- large-sample likelihood tests;
- asymptotic chi-square tests;
- large-sample regression tests.
The quality of the p-value depends on the quality of the approximation.
Discrete P-Values
In discrete tests, not every p-value between \(0\) and \(1\) is attainable.
The p-value can change in jumps because the test statistic itself takes only discrete values.
This is common in exact binomial and contingency-table procedures.
P-Values Are Not Effect Sizes
A p-value does not measure how large an effect is.
It measures compatibility between the observed statistic and the null model, relative to sampling variability.
Effect magnitude should be reported separately.
Large Samples Can Produce Tiny P-Values
As sample size increases, standard errors often shrink.
A very small departure from the null can therefore generate a large test statistic and a very small p-value.
Statistical significance can coexist with negligible practical importance.
Small Samples Can Produce Large P-Values
A practically meaningful effect can fail to reach statistical significance when the sample is small or variability is high.
A large p-value can therefore reflect limited power rather than evidence that the effect is exactly zero.
P-Values Depend on the Standard Error
Consider the generic statistic
\[ \frac{ \text{estimate}-\text{null value} }{ \text{standard error} }. \]For the same observed difference, a smaller standard error produces a larger standardized statistic in magnitude and usually a smaller p-value.
P-Values Depend on the Model
A p-value is not a property of the raw data alone.
It depends on:
- the null hypothesis;
- the alternative hypothesis;
- the test statistic;
- the sampling design;
- the assumed reference distribution;
- the handling of nuisance parameters.
P-Values and Confidence Intervals
In many compatible two-sided procedures, testing at level
\[ \alpha \]and constructing a
\[ 100(1-\alpha)\% \]confidence interval give equivalent conclusions about whether the null value is rejected.
Example: Confidence Interval Connection
Suppose a \(95\%\) confidence interval for a mean difference is
\[ (1.2,\;4.8). \]Since the null value
\[ 0 \]lies outside the interval, the corresponding two-sided test would reject equality at
\[ \alpha=0.05 \]when compatible methods are used.
Reporting Exact P-Values
When practical, report the actual p-value rather than only saying “significant” or “not significant.”
For example,
\[ p=0.018 \]gives more information than simply writing
“\(p<0.05\).”
Very Small P-Values
Software sometimes displays p-values as
\[ 0.000. \]This should not be interpreted as a literal probability of zero.
A better report is something like
\[ p<0.001 \]when the actual value is smaller than the displayed precision.
Borderline P-Values
A p-value such as
\[ 0.049 \]and one such as
\[ 0.051 \]represent very similar evidence.
Treating one as a dramatic success and the other as a complete failure exaggerates the importance of the threshold.
Evidence Is Continuous
The p-value is a continuous measure of compatibility with the null model, even though decision rules often use a binary cutoff.
Statistical reporting should distinguish the evidence measure from the operational decision threshold.
Alpha Is a Decision Threshold, Not an Evidence Measure
The significance level \(\alpha\) is chosen by the analyst or procedure.
The p-value is calculated from the data.
They play different roles:
\[ \boxed{ \alpha = \text{pre-specified threshold} } \] \[ \boxed{ p = \text{observed evidence measure under }H_0 }. \]Changing Alpha Does Not Change the P-Value
Suppose the observed data produce
\[ p=0.04. \]Changing the significance level from
\[ 0.05 \]to
\[ 0.01 \]changes the decision, but the p-value remains
\[ 0.04. \]Changing the Alternative Can Change the P-Value
The alternative determines which outcomes count as at least as extreme as the observed statistic.
Therefore, changing from a two-sided to a one-sided alternative can change the p-value.
That change is legitimate only when the alternative itself is substantively justified.
Multiple Testing
If many null hypotheses are tested, some small p-values can occur by chance even when all null hypotheses are true.
The interpretation of a single p-value must therefore consider the broader testing process.
Bonferroni Threshold
If \(m\) hypotheses are tested and the desired familywise error rate is \(\alpha\), the Bonferroni method compares each p-value with
\[ \boxed{ \frac{\alpha}{m} }. \]For example, with
\[ \alpha=0.05 \]and
\[ m=10, \]each test uses
\[ 0.005. \]Adjusted P-Values
Some multiple-testing methods report adjusted p-values.
These are modified so that comparison with the original familywise or false discovery threshold accounts for the multiplicity procedure.
Adjusted and unadjusted p-values should not be confused.
False Discovery Rate Procedures
In large-scale testing, procedures such as Benjamini-Hochberg control the expected proportion of false discoveries among rejected hypotheses.
This is a different objective from controlling the probability of any Type I error in the entire family.
Selective Reporting Distorts P-Values
If many analyses are tried but only the smallest p-value is reported, the nominal p-value does not reflect the full search process.
Transparent reporting and pre-specification help preserve valid interpretation.
P-Hacking
P-hacking refers to data-dependent analytic choices made in ways that increase the chance of obtaining a conventionally significant p-value.
Examples can include repeatedly changing:
- outcome definitions;
- subgroups;
- covariates;
- stopping rules;
- transformations;
- inclusion criteria.
The problem is not exploration itself, but presenting exploratory results as if they came from a single pre-specified test.
Optional Stopping
Repeatedly checking the p-value as data accumulate and stopping when significance first appears can inflate the Type I error rate under ordinary fixed-sample procedures.
Sequential testing requires methods designed for repeated looks at the data.
Pre-Specified Sequential Methods
Sequential and group-sequential designs can allow interim testing while controlling overall error rates.
The key is that the stopping rule is incorporated into the inferential design rather than improvised after data collection begins.
Practical Significance
A small p-value does not tell us whether an effect matters in practice.
Decisions should consider:
- effect size;
- confidence interval width;
- costs and benefits;
- clinical or operational thresholds;
- scientific relevance.
Example: Tiny Effect, Tiny P-Value
Suppose a very large study estimates a treatment effect of
\[ 0.05 \]units with
\[ p<0.001. \]The evidence against a zero-effect null may be strong, but an effect of \(0.05\) units may still be too small to matter operationally.
Model Assumptions Still Matter
A p-value is only as trustworthy as the model used to compute it.
Violations involving dependence, sampling design, distributional assumptions, or model misspecification can make a nominal p-value misleading.
Sampling Bias Is Not Fixed by a Small P-Value
A highly biased sample can produce a very small p-value.
Statistical significance does not repair undercoverage, voluntary response, nonresponse, measurement bias, or confounding.
Causal Interpretation Requires Design
A small p-value for an association does not automatically imply causation.
Causal conclusions require a study design or identification strategy capable of supporting causal inference.
Common Error: P-Value as Probability the Null Is True
Incorrect:
“\(p=0.03\) means there is a \(3\%\) probability that the null hypothesis is true.”
A p-value is calculated assuming the null is true. It does not directly compute
\[ P(H_0\mid\text{data}). \]Common Error: P-Value as Probability the Result Was Due to Chance
Saying
“There is a \(4\%\) chance the result was due to random chance”
is generally too vague and misleading.
The p-value is specifically a tail probability under a stated null model.
Common Error: Large P-Value Proves the Null
A large p-value indicates that the observed result is not especially incompatible with the null model.
It does not prove the null hypothesis is exactly correct.
Common Error: Small P-Value Means Large Effect
P-values combine effect magnitude with standard error.
A tiny effect can generate a tiny p-value in a very large sample.
Effect size must be examined separately.
Common Error: P = 0.049 and P = 0.051 Are Fundamentally Different
These two p-values represent nearly the same statistical evidence.
A binary decision threshold may classify them differently, but the evidence itself changes only slightly.
Common Error: Choosing Alpha after Seeing P
Suppose
\[ p=0.08. \]Changing the significance level to
\[ \alpha=0.10 \]only after seeing the result undermines the intended pre-specified decision process.
Common Error: Halving a Two-Sided P-Value Automatically
A one-sided p-value should not be created after the fact merely by dividing a two-sided p-value by \(2\).
The one-sided alternative must have been substantively justified in advance, and the observed statistic must lie in the specified direction.
Common Error: Ignoring Multiple Testing
A p-value of
\[ 0.04 \]can have a different interpretation when it is the only pre-specified test than when it is the smallest among hundreds of exploratory tests.
A Reliable Procedure for Using Alpha and P-Values
- State the null and alternative hypotheses.
- Choose the significance level before examining the result.
- Identify the appropriate test statistic and null distribution.
- Check sampling, independence, and model assumptions.
- Compute the observed test statistic.
- Determine which tail or tails count as extreme.
- Compute the p-value.
- Compare the p-value with \(\alpha\).
- Reject or fail to reject the null.
- Report the actual p-value when practical.
- Report effect size and uncertainty.
- Account for multiple testing when relevant.
- State practical and design limitations.
Example: Full Right-Tailed Workflow
Suppose
\[ H_0:\mu=100 \]versus
\[ H_A:\mu>100. \]A z test produces
\[ z=2.10. \]The right-tail p-value is
\[ p=P(Z\ge2.10)\approx0.0179. \]If
\[ \alpha=0.05, \]then
\[ 0.0179<0.05. \]Reject \(H_0\).
The data provide statistically significant evidence that
\[ \mu>100 \]under the assumptions of the test.
Example: Full Two-Sided Workflow
Suppose
\[ H_0:\mu=50 \]versus
\[ H_A:\mu\neq50. \]The observed statistic is
\[ z=-1.70. \]The two-sided p-value is
\[ p = 2P(Z\ge1.70) \approx 2(0.0446) = 0.0892. \]At
\[ \alpha=0.05, \]we fail to reject \(H_0\).
Practice 1: Compare p and Alpha
Suppose
\[ p=0.041, \qquad \alpha=0.05. \]What is the decision?
Practice 2: More Stringent Threshold
Which significance level is more stringent:
\[ 0.05 \]or
\[ 0.01? \]Practice 3: Right-Tailed P-Value
A right-tailed z test produces
\[ z=1.50. \]Express the p-value using probability notation.
Practice 4: Two-Sided P-Value
A symmetric two-sided z test produces
\[ z=-2.20. \]Express the p-value using probability notation.
Practice 5: Significance at Different Levels
Suppose
\[ p=0.03. \]Is the result significant at
\[ \alpha=0.05? \]Is it significant at
\[ \alpha=0.01? \]Practice 6: Interpret a Large P-Value
Suppose
\[ p=0.62. \]Does this prove the null hypothesis is true?
Practice 7: P-Value versus Effect Size
A study reports
\[ p<0.001. \]Does this tell you the effect is large?
Practice 8: Change Alpha
A study produces
\[ p=0.04. \]If the analyst changes \(\alpha\) from \(0.05\) to \(0.01\), does the p-value itself change?
Practice 9: Multiple Testing
Ten hypotheses are tested with a Bonferroni familywise level of
\[ \alpha=0.05. \]What threshold is used for each individual test?
Practice 10: Identify the Error
A student says:
“The p-value is \(0.02\), so there is a \(2\%\) chance that the null hypothesis is true.”
Explain the error.
Practice Answers
Practice 1
Since
\[ 0.041<0.05, \]reject the null hypothesis.
Practice 2
The more stringent threshold is
\[ \boxed{0.01}. \]It requires stronger evidence before rejection.
Practice 3
\[ \boxed{ p=P(Z\ge1.50) }. \]Practice 4
\[ \boxed{ p=2P(Z\ge2.20) }. \]Practice 5
At
\[ \alpha=0.05, \]the result is significant because
\[ 0.03<0.05. \]At
\[ \alpha=0.01, \]it is not significant because
\[ 0.03>0.01. \]Practice 6
No.
A large p-value indicates that the observed result is not unusually incompatible with the null model. It does not prove the null is true.
Practice 7
No.
A small p-value measures evidence against the null relative to sampling variability. It does not directly measure effect magnitude.
Practice 8
No.
The observed p-value remains
\[ \boxed{0.04}. \]Only the decision threshold changes.
Practice 9
\[ \frac{0.05}{10} = \boxed{0.005}. \]Practice 10
The p-value is calculated under the assumption that the null is true.
It is not the posterior probability that the null hypothesis is true.
What to Remember
Significance levels and p-values play different but complementary roles in hypothesis testing.
- \(\alpha\) is a pre-specified decision threshold.
- The p-value is calculated from the observed data under the null model.
- Reject \(H_0\) when \(p\le\alpha\).
- A small p-value indicates stronger incompatibility with the null model.
- A large p-value does not prove the null hypothesis is true.
- One-sided and two-sided alternatives produce different tail probabilities.
- Changing \(\alpha\) changes the decision threshold but not the observed p-value.
- P-values do not measure effect size, practical importance, or the probability that the null is true.
- Multiple testing and data-dependent analysis can distort nominal p-value interpretation.
- Model assumptions, sampling design, and data quality remain essential.
The most important habit is: treat \(\alpha\) as a pre-specified decision threshold and the p-value as a null-based measure of extremeness, not as the probability that the null hypothesis is true.
Where We Go Next
The significance level gives a rule for rejecting the null hypothesis, but every decision made under uncertainty can be wrong in more than one way.
The next lesson, Type I and Type II Errors, develops false positives, false negatives, the relationship between \(\alpha\) and \(\beta\), and the tradeoffs involved in statistical decision-making.