Internal and External Validity | Malone Math

The Central Idea

Internal validity is the degree to which a study's result supports the intended causal or descriptive conclusion for the studied units, treatments, measurements, and follow-up period.

External validity is the degree to which that result applies to a defined target beyond the immediate study—other people, organizations, locations, treatment implementations, outcomes, or times.

\[ \boxed{ \begin{aligned} &\text{internal validity: Is the study conclusion credible here?}\\ &\text{external validity: Does the conclusion apply there?} \end{aligned} } \]

These questions are connected but not interchangeable. A biased study cannot become externally valid by using a broad sample, and a highly controlled, internally valid study does not automatically generalize everywhere.

Validity Belongs to a Claim

It is imprecise to call an entire study simply “valid” or “invalid.” Validity depends on the claim being made.

A study might estimate the effect of assignment to a program accurately for participating schools but fail to estimate the effect of actual participation. It might estimate a short-term biological outcome accurately but not a long-term quality-of-life effect. It might generalize to similar urban clinics but not to rural hospitals.

Every validity assessment should name:

the treatment or exposure contrast;
the outcome and measurement time;
the study population;
the target population or setting; and
the causal or descriptive estimand.

Internal Validity

For a causal study, internal validity asks whether the observed treatment-group difference estimates the intended counterfactual contrast for the study units.

\[ E[Y(1)-Y(0)\mid S=1], \]

where \(S=1\) indicates participation in the study.

Strong internal validity requires that treatment groups be comparable, outcomes be measured appropriately, study procedures preserve the comparison, and the analysis match the design.

Threats to Internal Validity

Threat	How It Distorts the Study
Confounding	Treatment or exposure groups differ on causes of the outcome
Selection during enrollment	Knowledge of assignment or prognosis changes who enters each group
Differential attrition	Loss after assignment changes group comparability
Measurement bias	Outcomes or exposures are recorded differently from the truth or across groups
Noncompliance and crossover	Received treatment differs from assigned treatment in outcome-related ways
Contamination	Control units receive part of the treatment or treatment knowledge spreads
Co-interventions	Groups receive different additional care, resources, or attention
Selective analysis or reporting	Outcome, exclusion, model, or reporting choices respond to observed results

Validity and Precision Are Different

A small study can be internally well designed yet imprecise. Its estimate may fluctuate substantially because of chance. A large study can be highly precise yet internally biased.

Mean squared error can be decomposed as

\[ \operatorname{MSE}(\widehat{\theta}) = \operatorname{Var}(\widehat{\theta}) + \operatorname{Bias}(\widehat{\theta})^2. \]

Replication primarily reduces variance. Valid design and measurement primarily protect against bias. Credible evidence needs both.

History, Maturation, and Secular Change

In before-and-after studies, outcomes may change because of events unrelated to the intervention.

History: another event occurs during the study period.
Maturation: units naturally develop, age, recover, or tire.
Seasonality: outcomes vary with recurring calendar patterns.
Secular trend: a broader long-term trend changes the outcome.
Regression toward the mean: extreme baseline values become less extreme on repetition.

A concurrent comparison group exposed to the same time period helps estimate what would have happened without the intervention.

Testing and Instrumentation Effects

Repeated testing can change performance because participants learn the test, become sensitized to the study purpose, or alter behavior after baseline measurement.

Instrumentation change occurs when the device, observer, rubric, laboratory method, software, or case definition changes during the study. An apparent outcome trend may then reflect a measurement-system change rather than a treatment effect.

Stable instruments, calibration, common rubrics, parallel test forms, blinded assessment, and records of version changes protect the comparison.

Treatment Fidelity

Treatment fidelity concerns whether the intervention was delivered as defined. A study of “weekly tutoring” may include sessions that differ in duration, curriculum, tutor preparation, attendance, and instructional quality.

Fidelity data distinguish:

the effect of assignment to a program;
the effect of offering access;
the effect of treatment as actually delivered; and
the effect of receiving a specified dose or quality.

These are different counterfactual contrasts. Internal validity requires the reported estimand to match the implementation and analysis.

External Validity

External validity asks whether the study effect applies to a specified target. The target may differ from the study in several dimensions:

Population: age, risk, culture, resources, or eligibility.
Setting: clinic, school, industry, region, or institution.
Treatment: dose, provider, delivery, adherence, or co-interventions.
Outcome: measurement instrument, definition, or follow-up horizon.
Time: technology, policy, behavior, baseline risk, or historical period.

Generalization is not one jump from “sample” to “everyone.” It is a set of specific transport questions across these dimensions.

Generalizability and Transportability

Usage varies by field, but a useful distinction is:

Generalizability extends a study result from participants to a target population of which they are considered a sample.
Transportability extends a result to a distinct population or setting with known differences from the study.

Both require defining the target and explaining why the causal effect would or would not change across the study-target differences.

Random Sampling and Random Assignment

Random sampling and random assignment support different validity goals.

	Random Assignment	No Random Assignment
Random Sample	Strong basis for causal inference and population generalization, if execution is sound	Representative association, but causal interpretation may remain confounded
Nonrandom Sample	Strong causal basis for study units, but target-population transport needs justification	Both causal inference and population generalization require stronger assumptions

Many randomized experiments use volunteers or selected sites rather than probability samples. Random assignment can protect internal validity without making participants representative.

Effect Modification Drives External-Validity Questions

External validity becomes difficult when treatment effects differ across a characteristic \(X\):

\[ E[Y(1)-Y(0)\mid X=x] \]

changes with \(x\), and the distribution of \(X\) differs between study and target populations.

For example, a program may have larger effects in schools with more staff capacity. If the trial includes mostly high-capacity schools but the target includes many low-capacity schools, the trial average may not equal the target average.

Standardizing to a Target Population

Suppose treatment effects are transportable within levels of baseline characteristics \(X\). A target-population effect can be written conceptually as

\[ \operatorname{ATE}_{\text{target}} = E_{\text{target}} \left[ E[Y\mid T=1,X,S=1] - E[Y\mid T=0,X,S=1] \right]. \]

The inner treatment comparison is estimated from the study. The outer average uses the target population's distribution of \(X\).

This approach requires the important effect modifiers and selection variables to be measured in both study and target data, adequate target-study overlap, valid internal estimates within \(X\), and well-defined treatment and outcome conditions.

Exchangeability Across Study Participation

Let \(S=1\) indicate study participation and \(S=0\) indicate membership in the target outside the study. A strong transport assumption is

\[ Y(t)\perp S\mid X. \]

Conditional on \(X\), the potential outcome distribution under treatment \(t\) is assumed comparable between participants and target nonparticipants.

The assumption is more credible when \(X\) includes the characteristics that both influence participation and modify treatment effects. It cannot generally be verified from observed outcomes outside the study because those counterfactuals remain missing.

Target-Study Overlap

The target population must contain profiles represented adequately in the study. If the study excludes all people over age \(70\), data from the study alone cannot estimate age-specific effects for an elderly target without extrapolation or additional evidence.

\[ 0 must hold for target-relevant \(x\) under a basic transport-positivity condition. Weak overlap produces unstable weights and model-dependent conclusions.

Eligibility Criteria and External Validity

Eligibility criteria can protect safety, improve measurement, define a coherent treatment contrast, and reduce avoidable outcome variability. They can also exclude the people or settings for whom a decision must eventually be made.

Every criterion should have a reason:

Is exclusion required for safety?
Does it remove a condition that prevents treatment implementation?
Does it merely create a cleaner but less relevant study population?
Will the excluded group receive the intervention in practice?
What additional evidence is needed for excluded groups?

Efficacy and Effectiveness

Efficacy asks whether an intervention works under controlled or ideal conditions. Effectiveness asks how it performs under ordinary practice conditions.

Feature	Efficacy Emphasis	Effectiveness Emphasis
Participants	Narrowly eligible and closely monitored	Broadly representative of intended users
Delivery	Specialists and high protocol fidelity	Usual providers and operational variation
Adherence	Supported and monitored intensively	Reflects routine use
Outcome	Often proximal or mechanistic	Often practical and decision-centered

Efficacy and effectiveness are not competing labels of study quality. They answer different causal questions under different implementation conditions.

Explanatory and Pragmatic Designs

An explanatory trial emphasizes biological or mechanistic effects under controlled conditions. A pragmatic trial emphasizes consequences of offering or using an intervention in routine practice.

Pragmatic features may broaden sites, allow normal provider variation, use routine records, and include diverse participants. These choices can improve relevance while introducing operational variability and measurement challenges.

The design should match the decision. A mechanism question and an implementation question need not use the same population, treatment protocol, or outcome.

Treatment Versions and Implementation

A treatment effect travels only as well as the treatment definition. A program tested with expert staff, small groups, extensive training, and generous resources may produce a different effect when scaled with ordinary staff and constrained budgets.

Relevant implementation variables include:

provider skill and training;
dose, duration, and schedule;
quality and fidelity;
participant uptake and adherence;
organizational capacity;
co-interventions and competing services; and
technology, infrastructure, and supply constraints.

Transporting a result may require transporting the supporting system, not merely the treatment label.

Outcome and Time-Horizon Validity

A short-term surrogate effect may not generalize to a long-term final outcome. A six-week improvement may fade, grow, or reverse after one year. Adverse effects may emerge later than benefits.

External claims should preserve the study's measurement:

\[ \text{effect at time }t \ne \text{effect at every later time}. \]

Extending the time horizon requires follow-up data, mechanistic justification, or explicit modeling assumptions.

Effect Scale Matters

An effect can be expressed as a risk difference, risk ratio, odds ratio, mean difference, standardized difference, or another measure. An effect that is stable on one scale may vary on another.

If treatment reduces risk from \(20\%\) to \(10\%\), then

\[ \text{risk difference}=10\%-20\%=-10 \text{ percentage points}, \]

while

\[ \text{risk ratio}=\frac{10\%}{20\%}=0.5. \]

In a population with a different baseline risk, a stable risk ratio produces a different risk difference. External-validity claims should state the effect scale relevant to the decision.

Transport Across Time

Historical results may change in relevance because of:

new competing treatments;
technology and infrastructure changes;
population aging or demographic change;
shifts in baseline risk;
learning and adaptation;
policy or market changes; and
changes in pathogen, environment, or behavior.

“The study was rigorous” does not establish that its effect remains current. Temporal transport requires examining which effect-modifying conditions changed.

Multisite Studies and Replication

Including multiple sites can reveal whether an effect persists across settings and whether site characteristics modify it. Independent replications conducted by different teams add evidence about robustness to investigators and procedures.

A multisite average can still conceal substantial variation. Reports should show site-specific estimates, uncertainty, implementation differences, and whether the distribution of sites resembles the intended target.

Many similar sites do not automatically establish transport to a fundamentally different setting.

Mechanisms Can Support Transport

Understanding how a treatment produces an outcome helps identify the conditions required for the effect to persist.

\[ T\longrightarrow M\longrightarrow Y. \]

If the treatment changes mechanism \(M\) similarly across settings and the mechanism-outcome relationship is stable, transport may be more credible. If the target lacks the resources needed to activate \(M\), the original effect may not travel.

Mechanistic evidence supports external reasoning but does not replace direct evidence in important target populations.

Internal Validity Comes First Logically

An internally biased estimate has no stable causal quantity to transport. If treatment groups were confounded or outcomes measured differentially, broad sampling cannot rescue the causal effect.

\[ \boxed{ \text{first identify the effect credibly, then determine where it applies} } \]

This does not mean external validity is an afterthought. The target population and implementation should shape eligibility, sites, outcomes, and data collection from the beginning.

Internal and External Validity Are Not a Zero-Sum Tradeoff

Greater control can improve internal validity, while broader participation and routine implementation can improve relevance. But studies need not choose one and abandon the other.

Designs can combine:

random assignment with broad eligibility;
standardized primary outcomes with routine delivery;
high-fidelity implementation data with diverse sites;
an internally focused trial followed by target-population replication; and
careful causal identification with reweighting to a defined target.

Example: Strong External Reach, Weak Internal Validity

A national survey includes a probability sample of adults and finds that people who voluntarily use a wellness app have lower blood pressure than nonusers.

External Strength

The probability sample may support describing app use and blood pressure associations in the national adult population.

Internal Weakness

App users may differ in health motivation, income, medical care, exercise, and baseline risk. The study may not identify the causal effect of app use.

Conclusion

Representativeness does not convert an observational association into a causal effect.

Example: Strong Internal Validity, Limited External Reach

A carefully randomized experiment tests a learning program in one highly resourced private school. Assignment, adherence, outcome measurement, and follow-up are excellent.

Internal Strength

The experiment can credibly estimate the program's effect for the participating students under that implementation.

External Questions

Public schools may differ in class size, staffing, technology, schedule, student preparation, and available support. Those differences may modify the effect or prevent the tested treatment version from being delivered.

Next Evidence

Replicate across diverse schools, measure implementation and effect modifiers, and compare the study and target populations before making a system-wide claim.

Define the Target Before Interpreting the Result

A transport question should specify:

Target units: which people, organizations, or systems?
Target treatment: what exact intervention or policy version?
Comparator: compared with what alternative?
Outcome: measured how and at what time?
Setting: under which resources, providers, and conditions?
Effect scale: mean difference, risk difference, ratio, or another measure?

Without a target, “generalizable” has no operational meaning.

A Two-Stage Validity Checklist

Stage 1: Internal Validity

Was treatment assignment or exposure comparison credibly controlled?
Were allocation, adherence, contamination, and co-interventions documented?
Were outcomes measured consistently and with appropriate blinding?
Was attrition limited, explained, and handled appropriately?
Did the analysis preserve the design and prespecified estimand?
Is uncertainty reported separately from bias concerns?

Stage 2: External Validity

What target population and setting are intended?
How do participants and sites differ from the target?
Which differences plausibly modify the effect?
Is the same treatment version feasible in the target?
Are outcome definition and follow-up horizon preserved?
Is there adequate study-target overlap?
What replication, standardization, weighting, or mechanistic evidence supports transport?

Common Errors

Calling a Representative Association Causal

Random sampling supports description and population generalization. It does not remove confounding from a nonrandom exposure comparison.

Calling a Randomized Trial Universally Generalizable

Random assignment supports internal causal inference. It does not make study participants, sites, or treatment delivery representative of every target.

Treating External Validity as Demographic Similarity Alone

Settings, treatment versions, providers, outcomes, time, resources, and mechanisms can matter as much as demographic composition.

Generalizing Beyond the Outcome and Time Measured

A short-term surrogate result does not automatically establish a long-term practical benefit.

Assuming More Natural Conditions Guarantee Better Evidence

Routine settings can improve relevance but may introduce confounding, inconsistent treatment delivery, missingness, and measurement problems. Relevance does not substitute for identification.

Guided Practice

Problem 1

A randomized study has excellent allocation and measurement but includes only healthy adults ages \(18\) to \(30\). Which validity question arises when applying the result to adults over \(70\)?

Show the solution

External validity or transportability is the main question. Age, baseline risk, metabolism, coexisting conditions, and treatment tolerance may modify the effect. The original randomization protects the study comparison but does not supply elderly outcomes.

Problem 2

A representative national sample shows a strong association between a self-chosen diet and health. Does representativeness establish the diet's causal effect?

Show the solution

No. Representativeness may support generalizing the association, but self-chosen diet can be confounded by health behavior, income, access, prior health, and other causes of the outcome.

Problem 3

A treatment lowers risk from \(30\%\) to \(15\%\). Compute the risk difference and risk ratio.

Show the solution

The risk difference is \(15\%-30\%=-15\) percentage points. The risk ratio is \(15\%/30\%=0.5\).

Problem 4

Trial participants and the target population differ greatly in a variable that strongly modifies treatment effect. What information is needed to standardize the trial effect to the target?

Show the solution

The variable must be measured comparably in both trial and target data. The trial needs treatment-effect information across its levels, the target needs its distribution, and the two populations need adequate overlap. Conditional transportability must also be defensible.

Problem 5

A program succeeds when delivered by highly trained specialists but is planned for routine delivery by general staff. Name the primary external-validity concern.

Show the solution

Treatment implementation differs. Provider training and fidelity may modify the effect, so the tested treatment version may not match the target version. Effectiveness evidence under routine delivery is needed.

A Final Validity Check

Complete two sentences:

\[ \boxed{ \begin{aligned} &\text{This design identifies the claimed effect here because }\ldots\\ &\text{The effect applies to the target there because }\ldots \end{aligned} } \]

The first explanation should address assignment, comparability, measurement, follow-up, and analysis. The second should address population, setting, treatment version, outcome, time, effect modification, and overlap.

Summary

Internal validity concerns whether a study credibly supports its intended conclusion for the studied units and conditions.
External validity concerns whether that conclusion applies to a defined target population, setting, treatment version, outcome, and time.
Random assignment primarily supports internal causal validity; random sampling primarily supports population generalization.
Confounding, attrition, contamination, noncompliance, differential measurement, and selective analysis threaten internal validity.
Effect modification, study-target differences, weak overlap, implementation, outcome scale, and historical change govern external validity.
Efficacy and effectiveness studies answer different questions under controlled and routine conditions.
A causal effect must first be identified credibly and then transported using explicit target definitions, evidence, and assumptions.

Next Step

Validity determines whether a study supports its scientific claim and where that claim applies. The final lesson in this section develops ethical design and interpretation—the obligation to protect participants, use resources responsibly, report uncertainty honestly, and align conclusions with the evidence.