The Central Idea
A hypothesis test can fail to detect a real effect.
Power is the probability that the test correctly rejects the null hypothesis when a specified alternative is true.
If
\[ \beta \]is the probability of a Type II error, then
\[ \boxed{ \text{Power}=1-\beta }. \]Higher power means a greater chance of detecting an effect of the size being considered.
Type II Error
A Type II error occurs when the null hypothesis is not rejected even though the specified alternative is true.
Its probability is denoted
\[ \beta. \]Therefore,
\[ P(\text{fail to reject }H_0\mid H_A\text{ true}) = \beta. \]Power as a Probability
Power is
\[ \boxed{ P(\text{reject }H_0\mid H_A\text{ true}) }. \]This probability depends on which specific alternative value is considered.
A test does not usually have one universal power value for all possible departures from the null.
Power Depends on the True Parameter
Suppose we test
\[ H_0:\mu=100. \]The power to detect
\[ \mu=101 \]is usually smaller than the power to detect
\[ \mu=110. \]Larger departures from the null are generally easier to detect.
Effect Size
An effect size describes the magnitude of the departure from the null hypothesis.
For a mean,
\[ \Delta = \mu-\mu_0. \]For a proportion,
\[ \Delta = p-p_0. \]Larger absolute effect sizes generally produce greater power.
Standardized Effect Size
For a mean, a common standardized effect size is
\[ \boxed{ d = \frac{\mu-\mu_0}{\sigma} }. \]This expresses the effect in population standard deviation units.
Standardized effects allow comparisons across measurement scales.
Power Depends on Sample Size
Larger samples generally reduce standard errors.
For a sample mean,
\[ \operatorname{SE}(\bar{X}) = \frac{\sigma}{\sqrt{n}}. \]A smaller standard error makes a fixed real effect larger in standardized units.
Therefore, larger samples usually increase power.
Power Depends on Variability
Greater population variability makes real effects harder to distinguish from ordinary sampling fluctuation.
Holding all else fixed, larger
\[ \sigma \]produces a larger standard error and lower power.
More precise measurement can therefore increase power even without increasing sample size.
Power Depends on the Significance Level
Increasing
\[ \alpha \]makes the rejection region larger.
This generally increases power.
However, it also increases the probability of a Type I error when the null is true.
Statistical design therefore involves a tradeoff between Type I and Type II error probabilities.
Lower Alpha Usually Reduces Power
Suppose a test changes from
\[ \alpha=0.05 \]to
\[ \alpha=0.01. \]The evidence required for rejection becomes stronger.
For the same sample size and effect size, power generally decreases.
One-Sided Tests Can Have More Directional Power
When the true effect lies in a pre-specified direction, a one-sided test can have greater power in that direction than a two-sided test at the same \(\alpha\).
This occurs because the rejection probability is concentrated in one tail.
The directional hypothesis must be justified before seeing the data.
Two-Sided Tests Protect Both Directions
A two-sided test allocates rejection probability to both tails.
It therefore protects against meaningful departures in either direction.
This broader protection comes at the cost of somewhat less directional power for a fixed \(\alpha\).
Power Is a Design Property
Power is not usually something we discover only after data collection.
It is a property of a proposed test under specified assumptions about:
- the null value;
- the alternative value;
- the significance level;
- the sample size;
- the population variability;
- the sampling design;
- the statistical model.
Prospective Power Analysis
Prospective power analysis is performed before data collection.
It can answer questions such as:
- How many observations are needed?
- What effect sizes can this study detect?
- What power will a planned design have?
- How does changing \(\alpha\) affect the design?
Post Hoc Observed Power
Calculating “observed power” after a study by plugging the observed effect into a power formula is often of limited inferential value.
Once the data are observed, confidence intervals and effect estimates usually communicate uncertainty more directly.
Prospective power is most useful for planning.
Power Curves
A power curve shows power as a function of the true parameter value.
At values very close to the null, power is relatively low.
As the true parameter moves farther into the alternative region, power generally rises.
Power at the Null Boundary
At the null boundary, the probability of rejection is approximately the significance level:
\[ P(\text{reject }H_0\mid H_0\text{ true}) = \alpha. \]As the parameter moves away from the null into the alternative region, the rejection probability increases.
Common Power Targets
Common design targets include:
\[ 0.80 \]and
\[ 0.90. \]A target power of \(0.80\) corresponds to
\[ \beta=0.20. \]These are conventions, not universal requirements.
Why 80% Power Is Common
A target of
\[ 80\% \]has become common because it balances study size and the risk of missing meaningful effects in many applications.
Higher-stakes settings may require greater power.
The appropriate target depends on the consequences of missed effects.
Sample Size Planning
Sample size planning chooses \(n\) so that the study achieves a desired precision or power.
In hypothesis testing, the usual ingredients are:
- significance level \(\alpha\);
- desired power \(1-\beta\);
- effect size to detect;
- variability;
- test direction;
- design structure.
Sample Size for a One-Sample Mean
For a simple normal-theory one-sample z test, a common approximate sample size formula for detecting a mean difference
\[ \Delta=|\mu_1-\mu_0| \]with two-sided significance level \(\alpha\) and power \(1-\beta\) is
\[ \boxed{ n \approx \left( \frac{ z_{1-\alpha/2}+z_{1-\beta} }{ \Delta/\sigma } \right)^2 }. \]Equivalent Mean Sample Size Formula
The same formula can be written
\[ \boxed{ n \approx \left( \frac{ (z_{1-\alpha/2}+z_{1-\beta})\sigma }{ \Delta } \right)^2 }. \]After calculation, round upward.
Example: One-Sample Mean Planning
Suppose we want:
\[ \alpha=0.05, \] \[ 1-\beta=0.80, \]and we want to detect
\[ \Delta=5 \]when
\[ \sigma=12. \]Use
\[ z_{0.975}\approx1.96 \]and
\[ z_{0.80}\approx0.842. \]Then
\[ n \approx \left( \frac{(1.96+0.842)(12)}{5} \right)^2 \approx45.23. \]Round upward:
\[ \boxed{n=46}. \]One-Sided Mean Sample Size
For a one-sided test, the critical term typically uses
\[ z_{1-\alpha} \]instead of
\[ z_{1-\alpha/2}. \]A corresponding approximation is
\[ \boxed{ n \approx \left( \frac{ (z_{1-\alpha}+z_{1-\beta})\sigma }{ \Delta } \right)^2 }. \]Effect Size and Required Sample Size
The required sample size is inversely proportional to the square of the effect size.
In simplified form,
\[ n \propto \frac{1}{\Delta^2}. \]Detecting an effect half as large requires approximately four times as many observations, all else equal.
Variability and Required Sample Size
The required sample size is proportional to the variance:
\[ n\propto\sigma^2. \]If the population standard deviation doubles, the required sample size is approximately multiplied by
\[ 2^2=4 \]for the same effect, power, and significance level.
Desired Power and Sample Size
Increasing the desired power increases the required sample size.
A study designed for
\[ 90\% \]power generally requires more observations than one designed for
\[ 80\% \]power, all else equal.
Alpha and Required Sample Size
Lowering the significance level makes rejection more difficult.
Therefore, maintaining the same desired power generally requires a larger sample size.
Minimum Detectable Effect
Instead of choosing a sample size to detect a target effect, we can ask what effect size a fixed sample can detect with specified power.
This is called the minimum detectable effect, or MDE.
Approximate MDE for a Mean
Rearranging the one-sample formula gives
\[ \boxed{ \Delta_{\min} \approx \frac{ (z_{1-\alpha/2}+z_{1-\beta})\sigma }{ \sqrt{n} } }. \]A larger sample reduces the minimum effect that can be detected with the specified power.
Example: Minimum Detectable Effect
Suppose
\[ \alpha=0.05, \qquad 1-\beta=0.80, \qquad \sigma=10, \qquad n=100. \]Then
\[ \Delta_{\min} \approx \frac{ (1.96+0.842)(10) }{ 10 } = 2.802. \]The design can detect an effect of roughly
\[ \boxed{2.8} \]units with about \(80\%\) power under the approximation.
Two-Sample Mean Planning
For two independent groups of equal size and a common standard deviation \(\sigma\), a common approximation is
\[ \boxed{ n_{\text{per group}} \approx 2 \left( \frac{ (z_{1-\alpha/2}+z_{1-\beta})\sigma }{ \Delta } \right)^2 }. \]Here
\[ \Delta=|\mu_1-\mu_2|. \]Why Two Groups Need More Observations
For two independent sample means,
\[ \operatorname{SE}(\bar{X}_1-\bar{X}_2) = \sqrt{ \frac{\sigma_1^2}{n_1} + \frac{\sigma_2^2}{n_2} }. \]Both groups contribute sampling variability.
Therefore, detecting a given mean difference generally requires more total observations than estimating a single mean against a fixed known benchmark.
Equal Allocation Is Often Efficient
When per-observation cost and variability are similar in two independent groups, dividing the total sample approximately equally often minimizes the variance of the difference in means.
Thus
\[ n_1\approx n_2 \]is a common efficient design.
Unequal Allocation
Equal allocation is not always optimal.
Unequal group sizes may be appropriate when:
- one group is more expensive to recruit;
- one population is rarer;
- variances differ;
- ethical constraints limit one treatment group;
- historical controls are available.
Paired Designs Can Increase Power
Pairing or repeated measurement can reduce unexplained variability when the paired observations are strongly related.
For paired data, power depends on the variability of the differences
\[ D_i=X_i-Y_i, \]rather than on the marginal variability of the two raw measurements separately.
Correlation Helps Paired Designs
If paired measurements have positive correlation, the variance of the difference can be smaller than the sum of the separate variances.
This can substantially reduce the sample size needed for a given power.
Sample Size for a One-Sample Proportion
Planning a test for a population proportion requires specifying:
- the null proportion \(p_0\);
- the alternative proportion \(p_1\);
- \(\alpha\);
- desired power;
- test direction.
The exact formula depends on the chosen approximation.
Approximate One-Proportion Planning
For a two-sided normal approximation, one common planning formula is
\[ \boxed{ n \approx \frac{ \left[ z_{1-\alpha/2}\sqrt{p_0(1-p_0)} + z_{1-\beta}\sqrt{p_1(1-p_1)} \right]^2 }{ (p_1-p_0)^2 } }. \]Different methods can produce slightly different sample-size requirements.
Two-Proportion Sample Size
To compare two proportions, sample size depends on:
\[ p_1, \qquad p_2, \qquad \alpha, \qquad 1-\beta, \]and the group allocation.
Smaller differences
\[ |p_1-p_2| \]require larger samples.
Power for Rare Outcomes
Rare outcomes can require large samples because only a small number of events are expected.
For example, detecting a change from
\[ 0.5\% \]to
\[ 0.7\% \]may require far more observations than detecting a change from
\[ 40\% \]to
\[ 50\%. \]Event Counts Matter
In many binary-outcome studies, the number of observed events is as important as the total sample size.
A large dataset with very few events may provide limited information about the event process.
Regression Power
In regression, power can depend on:
- the size of the coefficient being tested;
- residual variability;
- sample size;
- correlations among predictors;
- the number of parameters estimated;
- the distribution of predictor values.
Multicollinearity Reduces Precision
Strong correlation among predictors can inflate the standard errors of regression coefficients.
Larger standard errors reduce the power to detect individual predictor effects.
ANOVA Power
Analysis of variance power depends on:
- the number of groups;
- group mean differences;
- within-group variability;
- group sample sizes;
- the significance level.
More groups and more complex contrasts can change the required total sample size.
Clustered Data Require Larger Samples
Observations within the same cluster are often correlated.
This reduces the amount of independent information in the data.
Therefore, clustered studies often require more total observations than simple-random-sample calculations suggest.
Design Effect in Cluster Sampling
A simple approximation for equal cluster size \(m\) and intracluster correlation \(\rho\) is
\[ \boxed{ \operatorname{DEFF} \approx 1+(m-1)\rho }. \]The required simple-random-sample size can be multiplied by this design effect as a first planning adjustment.
Example: Cluster Design Effect
Suppose
\[ m=20 \]and
\[ \rho=0.05. \]Then
\[ \operatorname{DEFF} \approx 1+(20-1)(0.05) = 1.95. \]A clustered design may therefore require nearly twice as many observations as the corresponding simple-random-sample design for similar precision.
Attrition and Nonresponse
Planned sample size should account for expected loss of observations.
If the analysis requires
\[ n_{\text{complete}} \]complete observations and the expected retention rate is \(r\), a simple adjustment is
\[ \boxed{ n_{\text{recruit}} \approx \frac{n_{\text{complete}}}{r} }. \]Example: Attrition Adjustment
Suppose the required complete-case sample size is
\[ 400 \]and expected retention is
\[ 80\%. \]Then
\[ n_{\text{recruit}} = \frac{400}{0.80} = \boxed{500}. \]Attrition Can Also Create Bias
Increasing the initial sample to compensate for attrition addresses the loss of precision.
It does not automatically remove bias if dropout is systematically related to the outcome.
Power planning and missing-data quality are separate issues.
Finite Population Sampling
When a large fraction of a finite population is sampled without replacement, the finite population correction reduces variance.
This can lower the sample size needed for a target precision relative to an infinite-population calculation.
Finite Population Correction
The correction is
\[ \boxed{ \sqrt{ \frac{N-n}{N-1} } }. \]When
\[ n\ll N, \]the factor is close to \(1\).
Multiplicity Can Reduce Power
If many hypotheses are tested and the significance threshold is adjusted to control familywise error, each individual test may use a smaller effective \(\alpha\).
This makes rejection more difficult and can reduce power unless sample size is increased.
Primary Outcomes Matter
A study with one clearly defined primary outcome can often be planned more efficiently than a study that tries to maintain high power across many co-primary outcomes.
Pre-specifying priorities helps align sample size with the main scientific objective.
Power for Interaction Effects
Interaction effects are often harder to detect than main effects.
Studies designed only for strong main-effect power may be underpowered for subgroup interactions or effect modification.
If interactions are important, they should be included explicitly in the planning model.
Subgroup Analyses Require Information within Subgroups
A large overall sample can still provide weak power within small subgroups.
Planned subgroup comparisons may require oversampling or larger total sample size.
Power Is Not the Probability the Alternative Is True
A design with
\[ 80\% \]power does not mean there is an \(80\%\) probability that the alternative hypothesis is true.
Power is conditional on a specified alternative already being true.
Power Is Not Confidence
A
\[ 95\% \]confidence level and
\[ 80\% \]power describe different properties.
Confidence concerns long-run interval coverage.
Power concerns the probability of rejecting a false null at a specified alternative.
Power Is Not the Significance Level
The significance level is
\[ \alpha = P(\text{reject }H_0\mid H_0\text{ true}), \]while power is
\[ 1-\beta = P(\text{reject }H_0\mid H_A\text{ true}). \]The conditions are different.
Effect Size Must Be Chosen Before Sample Size
Sample size planning requires a target effect.
That effect should represent a difference worth detecting, not merely the smallest mathematically nonzero value.
A practically meaningless target can produce an unnecessarily large study.
Clinically or Practically Important Difference
Many fields define a minimum important difference.
This is the smallest effect large enough to matter scientifically, clinically, operationally, or economically.
Power analysis should often be anchored to this meaningful threshold.
Uncertain Variability
Sample size formulas for means often require an estimate of
\[ \sigma. \]If this value is underestimated, the resulting study may have less power than planned.
Pilot data, prior studies, or conservative assumptions can help.
Pilot Studies
A pilot study can help estimate:
- variability;
- event rates;
- retention rates;
- recruitment feasibility;
- measurement reliability.
Very small pilots can estimate these quantities imprecisely, so uncertainty in planning inputs should be acknowledged.
Sensitivity Analysis for Sample Size
Rather than relying on one assumed effect size or standard deviation, it is often useful to calculate required sample sizes across several plausible scenarios.
This shows how sensitive the design is to uncertain planning assumptions.
Example: Sensitivity to Effect Size
Suppose the required sample size is calculated for target effects
\[ \Delta=2,\quad3,\quad4. \]Because
\[ n\propto\frac{1}{\Delta^2}, \]the required sample size drops quickly as the detectable effect becomes larger.
Underpowered Studies
An underpowered study has too little probability of detecting effects that matter.
Consequences can include:
- high Type II error probability;
- wide confidence intervals;
- unstable estimates;
- failure to detect practically important effects.
Overly Large Studies
More data are often useful, but unnecessarily large studies can consume resources without proportional benefit.
They can also make tiny, practically irrelevant effects statistically significant.
Good design aims for adequate information, not merely the largest possible sample.
Power and Confidence Interval Width Are Related
Both power and confidence interval width depend strongly on standard error.
Larger samples:
- increase power for fixed alternatives;
- reduce standard errors;
- produce narrower confidence intervals.
Therefore, planning for precision and planning for power are closely related but not identical tasks.
Precision-Based Sample Size
Sometimes the study goal is not hypothesis testing but estimation with a desired margin of error.
For a mean with known \(\sigma\),
\[ \boxed{ n = \left( \frac{z^\ast\sigma}{E} \right)^2 }. \]This is a precision calculation rather than a power calculation.
Power-Based and Precision-Based Planning
Power-based planning asks:
“How many observations are needed to detect a specified effect?”
Precision-based planning asks:
“How many observations are needed to estimate the parameter within a specified margin of error?”
Both approaches may be useful in the same study.
Common Error: Saying 80% Power Means 80% Chance the Result Will Be Significant
That statement is incomplete.
An \(80\%\) power calculation is conditional on a particular assumed alternative effect being true.
Different true effects produce different power values.
Common Error: Planning for an Unrealistic Effect
If the assumed effect size is much larger than what is plausible, the calculated sample size can be too small.
The study may then have poor power for the effects that actually matter.
Common Error: Ignoring Attrition
A calculation may show that \(300\) complete observations are required.
Recruiting exactly \(300\) participants is risky if meaningful dropout is expected.
Recruitment targets should account for anticipated loss.
Common Error: Ignoring Clustering
Treating \(500\) highly clustered observations as though they were \(500\) independent observations can substantially overstate power.
Intracluster correlation must be incorporated into planning.
Common Error: Using Post Hoc Observed Power as the Main Result
After data collection, confidence intervals and effect estimates generally provide more direct information about what the data support.
Power is most valuable prospectively for design.
Common Error: Assuming Larger Sample Size Removes Bias
Larger samples reduce many forms of sampling variability.
They do not automatically fix:
- undercoverage;
- nonresponse bias;
- measurement bias;
- confounding;
- model misspecification.
Common Error: Ignoring Multiple Comparisons
A study powered for one primary test may not have adequate power after strong multiplicity adjustments across many tests.
The planned analysis structure should be incorporated into the sample-size calculation.
Common Error: Treating More Power as Always Better
Higher power is useful, but increasing it requires more information, usually through larger sample size or improved measurement.
The optimal design balances power, cost, ethics, feasibility, and the importance of the effect being studied.
A Reliable Procedure for Power Analysis
- State the primary hypothesis or estimation goal.
- Identify the parameter and test statistic.
- Choose the significance level \(\alpha\).
- Choose a meaningful target effect size.
- Specify the desired power \(1-\beta\).
- Estimate variability or event rates.
- Choose one-sided or two-sided testing as justified.
- Specify group allocation and design structure.
- Adjust for clustering, weighting, or repeated measures when needed.
- Adjust recruitment targets for expected attrition or nonresponse.
- Perform sensitivity analyses across plausible assumptions.
- Round the required sample size upward.
- Document all assumptions used in the calculation.
Example: Full Power-Planning Workflow
A study will test
\[ H_0:\mu=50 \]against
\[ H_A:\mu\neq50. \]The study wants to detect a meaningful difference of
\[ \Delta=4 \]with
\[ \alpha=0.05, \qquad 1-\beta=0.80, \qquad \sigma=10. \]Using
\[ z_{0.975}=1.96 \]and
\[ z_{0.80}\approx0.842, \]we obtain
\[ n \approx \left( \frac{ (1.96+0.842)(10) }{ 4 } \right)^2. \]Therefore,
\[ n \approx49.07. \]Round upward:
\[ \boxed{n=50}. \]Example: Full Recruitment Adjustment
Suppose the analysis requires
\[ 250 \]complete observations.
If the expected retention rate is
\[ 85\%=0.85, \]then the recruitment target is
\[ \frac{250}{0.85} \approx294.12. \]Round upward:
\[ \boxed{295}. \]Practice 1: Power and Beta
A study has
\[ \beta=0.20. \]What is its power?
Practice 2: Effect of Sample Size
Other things equal, what generally happens to power when the sample size increases?
Practice 3: Effect of Variability
Other things equal, what generally happens to power when population variability increases?
Practice 4: Effect of Alpha
Other things equal, which test generally has greater power:
\[ \alpha=0.05 \]or
\[ \alpha=0.01? \]Practice 5: Sample Size Scaling
If the target detectable effect is cut in half, approximately how does the required sample size change in a design with
\[ n\propto\frac{1}{\Delta^2}? \]Practice 6: Attrition Adjustment
A study needs \(360\) complete observations and expects a retention rate of \(90\%\).
Approximately how many participants should be recruited?
Practice 7: Cluster Design Effect
Suppose average cluster size is
\[ m=15 \]and intracluster correlation is
\[ \rho=0.04. \]Estimate the design effect using
\[ 1+(m-1)\rho. \]Practice 8: Power Interpretation
A design has \(90\%\) power to detect a specified effect.
Does this mean there is a \(90\%\) probability that the alternative hypothesis is true?
Practice 9: Minimum Detectable Effect
Other things equal, what happens to the minimum detectable effect when the sample size increases?
Practice 10: Identify the Error
A researcher calculates sample size using an effect size that is much larger than any effect observed in prior studies.
What risk does this create?
Practice Answers
Practice 1
\[ \text{Power} = 1-\beta = 1-0.20 = \boxed{0.80}. \]Practice 2
Power generally increases because standard errors usually decrease as sample size grows.
Practice 3
Power generally decreases because greater variability makes the effect harder to distinguish from sampling noise.
Practice 4
The test with
\[ \boxed{\alpha=0.05} \]generally has greater power because its rejection region is larger.
Practice 5
Halving
\[ \Delta \]multiplies the required sample size by approximately
\[ \left(\frac{1}{1/2}\right)^2 = \boxed{4}. \]Practice 6
\[ n_{\text{recruit}} = \frac{360}{0.90} = \boxed{400}. \]Practice 7
\[ \operatorname{DEFF} = 1+(15-1)(0.04) = 1+0.56 = \boxed{1.56}. \]Practice 8
No.
Power is the probability of rejecting the null conditional on the specified alternative being true.
Practice 9
The minimum detectable effect generally decreases.
Larger samples can detect smaller effects with the same target power.
Practice 10
An unrealistically large assumed effect can make the required sample size appear too small.
The resulting study may be underpowered for realistic effects.
What to Remember
Power analysis connects statistical decision-making to study design.
- Power is \(1-\beta\), the probability of rejecting a false null for a specified alternative.
- Power increases as the true effect moves farther from the null.
- Larger samples generally increase power.
- Greater variability generally decreases power.
- Larger \(\alpha\) generally increases power but also increases Type I error probability.
- Higher target power requires more information, often through larger sample size.
- Smaller target effects require much larger samples because sample size often scales with \(1/\Delta^2\).
- Clustered designs, attrition, unequal allocation, and multiplicity can change sample-size requirements.
- Power calculations depend on assumptions about effect size, variability, and design.
- Power is most useful prospectively for planning studies before data collection.
The most important habit is: choose a practically meaningful effect, a justified error threshold, and a realistic design before calculating the sample size.
Where We Go Next
Power calculations depend on the statistical model and test chosen for the data. Some procedures make explicit distributional assumptions, while others rely less heavily on a particular parametric family.
The next lesson, Parametric and Nonparametric Methods, compares these two broad approaches, explains when each is useful, and develops the tradeoffs among assumptions, efficiency, robustness, and interpretability.