The Central Idea
Every empirical result depends on two pipelines:
\[ \text{target population} \longrightarrow \text{observed units} \]and
\[ \text{true characteristics} \longrightarrow \text{recorded variables}. \]Selection bias arises when the process determining who enters, remains in, or is analyzed in a study creates a systematically distorted comparison or estimate.
Measurement bias arises when the process used to observe or record a variable systematically differs from the truth or differs across the groups being compared.
\[ \boxed{ \text{selection asks who is observed; measurement asks what is recorded} } \]Bias Is Not the Same as Random Variability
Random variability causes estimates to fluctuate around a target from one sample to another. Bias moves the expected estimate systematically away from the target.
If an estimator \(\widehat{\theta}\) targets a quantity \(\theta\), its bias is
\[ \operatorname{Bias}(\widehat{\theta}) = E(\widehat{\theta})-\theta. \]Increasing the sample size usually reduces random variability. It does not automatically reduce systematic bias. A large, consistently distorted sample may produce a very precise answer to the wrong question.
The Selection Pipeline
It is useful to distinguish several populations and samples:
- Target population: the group to which the final claim is intended to apply.
- Source population: the accessible population from which study units can be identified.
- Invited or eligible sample: units meeting the study rules and approached for participation.
- Enrolled sample: units that enter the study.
- Followed sample: units for whom later data are collected.
- Analytic sample: units actually included in the reported calculation.
Selection can occur at every transition. A transparent study reports the counts, reasons for loss, and relevant group characteristics at each stage.
How Selection Creates Bias
Let \(S=1\) indicate that a unit is observed or included. Selection bias can arise when inclusion depends on exposure \(X\), outcome \(Y\), or causes of both. Conditioning on the selected group may open a noncausal association:
\[ X\longrightarrow S\longleftarrow Y. \]Because \(S\) is caused by both \(X\) and \(Y\), it is a collider. Restricting the analysis to \(S=1\) can make \(X\) and \(Y\) associated even if they were unrelated before selection.
Not every nonrepresentative sample creates the same bias for every question. The critical issue is whether selection changes the distribution or relationship needed for the target estimate.
Undercoverage
Undercoverage occurs when some parts of the target population are missing or poorly represented in the sampling frame.
Examples include:
- an online survey that excludes people without reliable internet access;
- a telephone frame that omits certain service types;
- a customer database that excludes people who considered but did not purchase;
- a school survey that misses students absent on the survey day; or
- a health registry that captures only people who seek care.
Undercoverage creates bias when the omitted groups differ on the quantity or relationship being estimated.
Self-Selection and Volunteer Bias
Self-selection bias occurs when participation is voluntary and the decision to participate is related to the study variables.
People with unusually strong opinions may be more likely to answer an open poll. Health-conscious people may be more likely to join a wellness study. Highly motivated learners may enroll in an optional training program.
A large volunteer sample does not become representative merely because many volunteers participate. The mechanism producing the sample remains voluntary.
Nonresponse Bias
Nonresponse occurs when selected units do not provide requested data. Nonresponse bias arises when respondents and nonrespondents differ in ways relevant to the estimate.
The response rate is
\[ \text{response rate} = \frac{\text{eligible completed responses}} {\text{eligible units invited}} \]under a stated eligibility and disposition rule. A low response rate raises concern but does not determine the amount of bias by itself. A high response rate can still be biased if the remaining nonresponse is strongly related to the outcome.
Attrition and Loss to Follow-Up
Attrition is the loss of enrolled units before the study is complete. In longitudinal research it is often called loss to follow-up.
Attrition is especially threatening when it differs by treatment or is related to outcomes. If participants experiencing side effects leave one treatment group, the remaining participants may make that treatment appear safer or more effective than it is.
Random assignment protects the groups at baseline. Differential post-assignment loss can weaken that comparability. Reporting only complete cases may therefore undo part of the benefit of randomization.
Survivorship Bias
Survivorship bias occurs when attention is restricted to units that remain visible after a process, while failures or departures disappear.
Studying only existing businesses omits businesses that failed. Evaluating only funds still operating omits closed funds. Interviewing only employees who remain after a difficult policy change omits those who left because of it.
The surviving group can make success look more common, risk look smaller, or a process look more stable than it was for the full starting population.
The Healthy-Worker Pattern
Employed populations often appear healthier than the general population because people must be sufficiently healthy to enter and remain in employment. Comparing workers with the full public can therefore underestimate occupational risks.
This pattern combines selection, employment eligibility, and continued workforce participation. A more meaningful comparison may use workers in similar jobs with different exposure levels, while still addressing other differences.
Convenience Samples
A convenience sample includes units that are easiest to reach. It can be useful for pilot work, instrument testing, or exploratory analysis, but its relationship to a target population must be justified rather than assumed.
Students in one course, customers at one store, users active during one hour, or records from one clinic may differ systematically from the broader population. Precision within the convenience sample does not establish external validity.
Missing Data Mechanisms
Missing values are closely connected to selection because an analysis often includes only units with observed variables. Three broad descriptions are useful:
| Mechanism | Informal Meaning | Example |
|---|---|---|
| Missing completely at random | Missingness is unrelated to recorded or unrecorded study values | A random equipment failure destroys some forms |
| Missing at random | After conditioning on observed variables, missingness does not depend on the missing value itself | Follow-up depends on recorded age and clinic but not the outcome after accounting for them |
| Missing not at random | Missingness still depends on unobserved values or unmeasured causes | People with the most severe symptoms are least likely to report them |
These are assumptions about the missingness process, not labels established by a simple test. Analysis should examine several plausible mechanisms when the conclusion depends strongly on missing data.
The Measurement Pipeline
Measurement translates a construct or true value into recorded data:
\[ \text{construct} \longrightarrow \text{operational definition} \longrightarrow \text{instrument or observer} \longrightarrow \text{recorded value}. \]Bias can enter through a vague definition, an uncalibrated instrument, an inconsistent observer, leading questions, faulty recall, data processing, or a rule that differs across groups.
Accuracy and Precision
Accuracy describes closeness to the true or reference value. Precision describes closeness among repeated measurements.
A scale that always reads \(2\) kilograms too high may be highly precise but inaccurate. Repeating the measurement and averaging can reduce random reading noise, but it does not remove the \(2\)-kilogram calibration bias.
\[ \boxed{ \text{repeatability does not guarantee validity} } \]Random and Systematic Measurement Error
Let \(X\) be a true value and \(X^*\) its recorded measurement. A simple model is
\[ X^*=X+U, \]where \(U\) is measurement error.
- Random measurement error varies unpredictably around the true value and often reduces precision.
- Systematic measurement error tends to shift measurements in a consistent direction or according to a consistent group-dependent pattern.
Real measurement error can combine both components and may change across the range of values being measured.
Measurement Error Can Weaken Associations
Under a classical linear-error model where \(U\) is independent of \(X\) and the outcome, measurement error in an explanatory variable often attenuates a regression slope toward zero.
The attenuation factor is
\[ \lambda = \frac{\operatorname{Var}(X)} {\operatorname{Var}(X)+\operatorname{Var}(U)}, \qquad 0<\lambda\le 1. \]With nonzero error variance, \(\lambda<1\). If the true slope is \(\beta\), the observed slope may be approximately \(\lambda\beta\). This direction is not universal: differential, nonlinear, outcome-dependent, or correlated error can bias estimates in other directions.
Misclassification
Misclassification occurs when a categorical state is recorded incorrectly. A true positive may be recorded as negative, or a true negative as positive.
| Recorded Positive | Recorded Negative | |
|---|---|---|
| True Positive | True positive | False negative |
| True Negative | False positive | True negative |
Sensitivity and specificity summarize two parts of classification performance:
\[ \text{sensitivity} = P(\text{recorded positive}\mid\text{true positive}), \] \[ \text{specificity} = P(\text{recorded negative}\mid\text{true negative}). \]Differential and Nondifferential Error
Differential measurement error changes according to treatment, exposure, outcome, or comparison group. For example, assessors may search more intensely for adverse events in a new-treatment group.
Nondifferential measurement error follows the same broad measurement process across groups. It can still distort estimates. Although simple nondifferential misclassification often weakens an association, it does not always bias toward no effect, especially with multiple categories, several error-prone variables, or complex models.
The direction of measurement bias should be reasoned from the actual error mechanism rather than from a universal slogan.
Recall Bias
Recall bias occurs when past events or exposures are remembered differently across comparison groups. People who experience an adverse outcome may search their memories more intensely for possible causes than people without the outcome.
Long recall periods, emotionally significant outcomes, vague exposure definitions, and unequal prompting can increase the risk. Contemporaneous records, shorter recall windows, neutral questions, and validation against independent data sources can reduce it.
Interviewer and Observer Bias
An interviewer can influence responses through wording, tone, follow-up prompts, body language, or selective recording. An observer can interpret the same behavior differently depending on expectations.
Protection includes standardized scripts, training, behaviorally anchored rules, blinded assessment, inter-rater checks, recorded interviews, and adjudication of ambiguous cases.
Response and Social-Desirability Bias
Participants may provide answers they believe are acceptable, safe, impressive, or desired by the researcher. Sensitive behaviors may be underreported, while socially approved behaviors may be overstated.
Privacy protections, neutral wording, self-administered instruments, indirect questioning, clearly bounded response categories, and assurance that individual answers will not affect services can improve measurement.
Instrument and Calibration Bias
Instruments can drift, saturate, round, truncate, or respond differently across environmental conditions. Software pipelines can also transform or code data incorrectly.
A measurement plan should specify:
- reference standards and calibration schedules;
- operating ranges and resolution;
- maintenance and quality-control rules;
- device, laboratory, or software version;
- handling of values below or above detection limits;
- unit conversions and rounding; and
- audit trails for manual corrections.
Reliability and Validity
Reliability concerns consistency across repeated measurements, observers, items, or occasions. Validity concerns whether the measurement captures the intended construct accurately enough for its use.
A questionnaire can be highly reliable—producing similar scores repeatedly—while measuring only one narrow part of the intended concept. Conversely, a valid construct definition can be implemented unreliably through inconsistent scoring.
Both matter. Reliability is necessary for stable measurement, but it is not sufficient for valid inference.
Outcome Ascertainment
Ascertainment is the process of finding and confirming outcomes. If one treatment group receives more follow-up, testing, or record review, more outcomes may be detected even when the true rates are the same.
Equal follow-up schedules, common case definitions, blinded adjudication, linked records, and standardized search procedures can reduce differential ascertainment. The study should report whether outcome opportunities were comparable across groups.
Selection and Measurement Can Interact
The two forms of bias often reinforce one another. Suppose participants with severe symptoms are more likely to leave one treatment group, and the remaining participants underreport mild symptoms. The analytic sample is selected and the outcome is measured differently from the truth.
Correcting only one pipeline may leave the other distortion intact. A complete audit follows units from eligibility through analysis and follows variables from definition through final coding.
Preventing Selection Bias
- Define the target population and sampling frame explicitly.
- Use probability sampling when population representation is required.
- Apply eligibility rules consistently before outcomes are known.
- Conceal treatment allocation during enrollment in randomized experiments.
- Use repeated, accessible, multilingual, and privacy-conscious follow-up.
- Track reasons for refusal, exclusion, attrition, and missingness by group.
- Preserve randomized groups in the primary analysis when appropriate.
- Use weighting or other adjustment only with a defensible selection model.
- Perform sensitivity analyses for plausible departures from that model.
Preventing Measurement Bias
- Define every exposure, treatment, outcome, and covariate operationally.
- Use validated instruments suited to the population and setting.
- Calibrate equipment and verify units, ranges, and detection limits.
- Train observers with explicit scoring rules and practice cases.
- Blind participants, providers, assessors, or analysts where feasible.
- Use the same measurement schedule and procedures across groups.
- Collect validation subsamples or duplicate measurements when useful.
- Record instrument versions, code changes, and data corrections.
- Prespecify how ambiguous, missing, censored, or conflicting values will be handled.
Analysis Cannot Replace Prevention
Statistical tools can address some known selection and measurement mechanisms:
- survey and nonresponse weights;
- inverse-probability-of-selection or censoring weights;
- multiple imputation under stated missingness assumptions;
- measurement-error correction using validation data;
- latent-variable or misclassification models;
- calibration estimators; and
- sensitivity analyses for unobserved selection or error.
Each method depends on assumptions and adequate information. If excluded units were never described, or the measurement process was never validated, the data may not identify the needed correction.
Example: A Workplace Satisfaction Survey
A company emails a voluntary satisfaction survey to all current employees. Only \(28\%\) respond, and the company reports an average satisfaction score of \(8.4\) out of \(10\).
What Is the Target?
The intended target may be all current employees—or perhaps all employees during the year, including those who left. The report must specify this.
What Selection Threats Exist?
Current employees exclude former employees. Voluntary respondents may differ from nonrespondents. Employees who distrust confidentiality may be less willing to answer, and satisfaction itself may influence response.
What Would Improve the Study?
Use a probability sample or well-designed census follow-up, provide strong privacy protections, offer multiple response modes, compare respondents and nonrespondents on available workforce variables, and report response patterns by unit and shift.
What Cannot Be Claimed Automatically?
The average among respondents cannot be assumed to equal the average among all employees merely because the respondent count is large.
Example: Two Blood-Pressure Devices
A treatment group is measured with a new device, while the control group is measured with an older device that reads an average of \(5\) mmHg higher than a reference standard.
Device type is perfectly aligned with treatment group. Any observed treatment difference mixes the real treatment effect with device calibration bias.
A stronger design uses the same calibrated devices across groups, randomly distributes device use, or measures a validation subset on both devices so the device difference can be estimated. Device identifiers and calibration records should be preserved in the data.
Report the Flow of Units and Measurements
A transparent study reports how many units were:
- identified;
- assessed for eligibility;
- excluded and why;
- invited;
- enrolled or randomized;
- treated as assigned;
- followed at each measurement time;
- missing each primary variable; and
- included in each analysis.
It should also report measurement instruments, timing, calibration, blinding, coding rules, and deviations from the original measurement plan. These are part of the evidence, not administrative details.
A Bias Audit Checklist
- Define the target. Which population, treatment contrast, and outcome are intended?
- Map selection. How does a unit move from the target population to the analytic sample?
- Check dependence. Could inclusion depend on exposure, outcome, or causes of either?
- Track loss. Who refused, dropped out, survived, or had incomplete data?
- Define variables. What does each recorded value operationally mean?
- Map measurement. Who or what records each variable, when, and with which instrument?
- Check group differences. Could measurement quality or intensity differ across groups?
- Validate. Are calibration, reference, duplicate, or adjudication data available?
- Test assumptions. How do conclusions change under plausible selection and error mechanisms?
Common Errors
Assuming a Large Sample Is Unbiased
Sample size reduces random error. It does not repair undercoverage, self-selection, differential attrition, or systematic measurement error.
Using Response Rate as the Entire Bias Diagnosis
Response rate is informative, but bias depends on how respondents and nonrespondents differ in relation to the target quantity.
Treating Complete Cases as Automatically Representative
Units with complete data may differ systematically from those with missing values. Complete-case analysis requires assumptions about the missingness process.
Equating Precision with Accuracy
Repeated readings can agree closely while all being shifted from the truth. Reliability does not guarantee validity.
Assuming Nondifferential Error Always Biases Toward Zero
That tendency holds only in certain simple settings. Complex misclassification and measurement-error structures can bias estimates in other directions.
Guided Practice
Problem 1
A website posts an open poll asking whether visitors approve of its redesign. One hundred thousand people respond. What major bias remains possible?
Show the solution
Self-selection bias remains. Visitors with strong opinions may be more likely to respond, and website visitors may not represent the intended target population. The large respondent count does not correct the voluntary selection mechanism.
Problem 2
In a treatment study, \(20\%\) of the treatment group and \(4\%\) of the control group are lost to follow-up. Why is this concerning?
Show the solution
Attrition differs substantially by group. If loss is related to side effects, treatment response, or prognosis, the remaining groups may no longer be comparable. The study should report reasons for loss and examine sensitivity to plausible missing outcomes.
Problem 3
A scale gives readings of \(72.1\), \(72.2\), and \(72.1\) kilograms for an object whose reference mass is \(70.0\) kilograms. Is the scale precise, accurate, both, or neither?
Show the solution
The readings are highly precise because they agree closely. They are inaccurate because they are systematically about \(2.1\) kilograms above the reference.
Problem 4
An outcome assessor knows which participants received the new treatment and asks them more follow-up questions about improvement. Which bias is possible?
Show the solution
Differential measurement or ascertainment bias is possible. The new-treatment group has more opportunity to report improvement. Blinded assessment and a standardized interview would reduce the problem.
Problem 5
A diagnostic rule has sensitivity \(0.90\). Among 200 truly positive cases, how many are expected to be recorded positive and false negative?
Show the solution
Expected recorded positives are \(0.90(200)=180\). The remaining \(200-180=20\) are expected false negatives.
A Final Two-Part Check
Before accepting a result, ask:
\[ \boxed{ \begin{aligned} &\text{Who is missing from the evidence, and why?}\\ &\text{How could the recorded values differ from the truth?} \end{aligned} } \]Those two questions expose the selection and measurement assumptions supporting the analysis.
Summary
- Selection bias distorts who enters, remains in, or is included in a study or comparison.
- Undercoverage, self-selection, nonresponse, attrition, survivorship, and conditioning on selected groups are common selection pathways.
- Measurement bias systematically distorts recorded exposures, treatments, outcomes, or covariates.
- Recall, interviewer, observer, social-desirability, calibration, misclassification, and differential ascertainment errors are common measurement pathways.
- Precision and reliability do not guarantee accuracy or validity.
- Large samples reduce random variability but do not automatically remove either selection or measurement bias.
- Prevention, transparent flow reporting, validation data, design-aware analysis, and sensitivity analysis are all required for credible conclusions.
Next Step
Selection, measurement, and confounding operate through causal pathways. The next lesson introduces causal diagrams—a compact visual language for organizing those pathways and deciding which variables should be measured, controlled, or left alone.