The Central Idea

Statistical quality is not described by sample size alone.

When an estimate differs from the population quantity of interest, we should ask at least three questions:

  1. Is the procedure systematically shifted away from the truth?
  2. How much would the estimate change if the study were repeated?
  3. Does the observed data adequately reflect the target population?

These questions correspond to bias, variability, and representativeness.

Bias

Bias is systematic error.

For an estimator \(\hat{\theta}\) of a parameter \(\theta\), statistical bias is defined by

\[ \boxed{ \operatorname{Bias}(\hat{\theta}) = E(\hat{\theta})-\theta }. \]

A positive bias means the estimator tends to be too high.

A negative bias means it tends to be too low.

Unbiasedness

An estimator is unbiased when

\[ E(\hat{\theta})=\theta. \]

Equivalently,

\[ \operatorname{Bias}(\hat{\theta})=0. \]

Unbiasedness is a repeated-sampling property. It does not mean that every observed estimate equals the population parameter.

Variability

Variability describes how much a statistic changes from one sample to another.

For an estimator \(\hat{\theta}\), variability is commonly summarized by its variance

\[ \operatorname{Var}(\hat{\theta}) \]

or its standard error

\[ \operatorname{SE}(\hat{\theta}) = \sqrt{\operatorname{Var}(\hat{\theta})}. \]

Lower variability generally means greater precision.

Representativeness

Representativeness concerns how well the observed data reflect the population relevant to the inferential question.

It is not a single numerical property like variance.

It depends on:

Accuracy and Precision

Accuracy describes closeness to the true value.

Precision describes how tightly repeated estimates cluster together.

A procedure can be:

Precision alone does not guarantee accuracy.

Four Conceptual Cases

Imagine repeatedly estimating the same parameter.

If the estimates are tightly clustered around the truth, the procedure has low bias and low variability.

If the estimates are widely scattered around the truth, it has low bias but high variability.

If the estimates are tightly clustered away from the truth, it has high bias but low variability.

If they are widely scattered and systematically displaced, both problems are present.

Bias Is Not the Same as One Bad Estimate

A single estimate can be far from the truth even when the estimator is unbiased.

Random sampling variability can occasionally produce an unusual sample.

Bias refers to the center of the repeated-sampling distribution, not the error in one isolated realization.

Variability Is Not the Same as Bias

Suppose an estimator has

\[ E(\hat{\theta})=\theta \]

but a large standard error.

It is unbiased but imprecise.

Suppose another estimator has a very small standard error but

\[ E(\hat{\theta})\neq\theta. \]

It is precise but biased.

Sampling Bias

Sampling bias arises when the mechanism that produces the observed sample systematically favors some parts of the target population over others.

Examples include:

Undercoverage

Undercoverage occurs when some members of the target population have no practical chance, or too little chance, of appearing in the sample.

Suppose a city wants information about all residents but samples only from a directory that omits people without listed addresses.

The resulting sample may be random within the directory while still failing to cover the full target population.

Overcoverage

Overcoverage occurs when the sampling frame contains units that do not belong to the target population or contains duplicate representations of some units.

For example, a customer database may contain inactive accounts or duplicate customer records.

If these are not identified, some population units can receive the wrong selection probabilities.

Convenience Sampling

A convenience sample is selected primarily because the observations are easy to obtain.

Surveying only people near a building entrance, only customers who visit during one time period, or only students in one readily available class can create systematic differences between the sample and the target population.

Convenience can reduce cost while increasing inferential risk.

Voluntary-Response Bias

In a voluntary-response sample, people decide for themselves whether to participate.

People with unusually strong opinions, unusually positive experiences, or unusually negative experiences may be more likely to respond.

The resulting bias is often called voluntary-response bias or self-selection bias.

Nonresponse Bias

Nonresponse bias occurs when selected units who respond differ systematically from selected units who do not respond in ways related to the quantity being estimated.

A low response rate does not mathematically guarantee large bias, but it increases concern because more of the selected sample is unobserved.

The critical question is whether response behavior is related to the study variables after accounting for the design.

Example: Nonresponse

Suppose \(2{,}000\) employees are randomly selected for a workplace satisfaction survey, but only \(600\) respond.

If highly dissatisfied employees are less willing to respond because they distrust confidentiality, the responding group may report satisfaction that is systematically too high.

The original random sample does not automatically protect the final respondent set from nonresponse bias.

Measurement Bias

Bias can arise even when sampling is excellent.

Measurement bias occurs when the measurement process systematically pushes recorded values away from the quantity intended to be measured.

Sources include:

Response Bias

Response bias occurs when respondents systematically report values that differ from the underlying quantity of interest.

Questions about income, exercise, voting, spending, or socially sensitive behavior can be affected by memory errors or pressure to provide a socially acceptable response.

Random sampling does not remove this form of bias.

Question Wording Can Create Bias

Survey questions should not pressure respondents toward a particular answer.

Compare:

“Do you support the proposed policy?”

with:

“Do you support the wasteful proposed policy that will increase costs?”

The second version contains evaluative language that can influence responses.

Question Order Can Matter

Earlier questions can affect how later questions are interpreted.

For example, asking respondents to think first about a recent negative experience can change how they answer a later general satisfaction question.

Survey design therefore includes wording, ordering, context, and response options—not only sampling.

Interviewer Effects

Respondents may answer differently depending on who asks the question and how it is asked.

Tone, identity cues, body language, perceived authority, and interviewer expectations can influence responses.

Standardized interviewer procedures can reduce, but not necessarily eliminate, these effects.

Instrument Calibration

Suppose a scale consistently reads \(1.5\) kilograms too high.

Repeating the measurement many times with the same miscalibrated scale can produce extremely precise measurements that are still systematically wrong.

This is a clear example of low variability with high bias.

Random Error in Measurement

Not all measurement problems are systematic.

Some measurement error fluctuates unpredictably around the true value.

Repeated measurements may sometimes be too high and sometimes too low.

Such random measurement error increases variability rather than necessarily shifting the center.

Systematic Error versus Random Error

Systematic error tends to move results in a consistent direction.

Random error creates unpredictable fluctuation.

In simplified terms:

\[ \text{systematic error} \longleftrightarrow \text{bias}, \] \[ \text{random error} \longleftrightarrow \text{variability}. \]

Real studies can contain both at the same time.

Mean Squared Error

Bias and variability can be combined through mean squared error, or MSE.

For an estimator \(\hat{\theta}\),

\[ \boxed{ \operatorname{MSE}(\hat{\theta}) = E\left[(\hat{\theta}-\theta)^2\right] }. \]

MSE measures the average squared distance between the estimator and the true parameter.

Bias-Variance Decomposition

Mean squared error separates into variance and squared bias:

\[ \boxed{ \operatorname{MSE}(\hat{\theta}) = \operatorname{Var}(\hat{\theta}) + \left[\operatorname{Bias}(\hat{\theta})\right]^2 }. \]

This identity formalizes the fact that estimation quality can suffer from either excessive spread or systematic displacement.

Deriving the Bias-Variance Decomposition

Write

\[ \hat{\theta}-\theta = \left(\hat{\theta}-E(\hat{\theta})\right) + \left(E(\hat{\theta})-\theta\right). \]

Squaring and taking expectations gives

\[ E[(\hat{\theta}-\theta)^2] = E[(\hat{\theta}-E\hat{\theta})^2] + (E\hat{\theta}-\theta)^2, \]

because the cross term has expectation zero.

Therefore,

\[ \operatorname{MSE} = \operatorname{Var} + \operatorname{Bias}^2. \]

Example: Comparing Two Estimators

Suppose Estimator A has

\[ \operatorname{Bias}(A)=0,\qquad \operatorname{Var}(A)=25. \]

Then

\[ \operatorname{MSE}(A)=25. \]

Suppose Estimator B has

\[ \operatorname{Bias}(B)=2,\qquad \operatorname{Var}(B)=9. \]

Then

\[ \operatorname{MSE}(B)=9+2^2=13. \]

Estimator B is biased, but it has the lower mean squared error in this comparison.

Unbiased Is Not Always Best

Unbiasedness is desirable, but it is not the only criterion for evaluating an estimator.

A slightly biased estimator with much smaller variance can have lower mean squared error than an unbiased estimator.

Statistical procedures often balance bias and variability rather than optimizing only one dimension.

Increasing Sample Size and Variability

Increasing the sample size often reduces sampling variability.

For a sample mean under independent sampling,

\[ \operatorname{SE}(\bar{X}) = \frac{\sigma}{\sqrt{n}}. \]

Thus a larger \(n\) generally produces a narrower sampling distribution.

Increasing Sample Size Does Not Automatically Remove Bias

Suppose a survey systematically excludes one important population group.

Increasing the sample from \(1{,}000\) to \(100{,}000\) observations from the same incomplete frame can make the estimate more precise relative to that flawed frame while leaving the systematic error intact.

A very large sample can therefore be very precisely wrong.

Representativeness Is Relative to a Target Population

A sample cannot be called representative without specifying representative of what population.

A sample of university students may accurately describe students at one institution but not all adults in a country.

Representativeness is always tied to the inferential target.

Representativeness Is Multidimensional

A sample can match the population on one characteristic and differ greatly on another.

Matching age proportions does not guarantee matching income, geography, education, attitudes, or health behaviors.

Therefore, checking one visible demographic characteristic cannot by itself prove general representativeness.

Probability Sampling Supports Representativeness

Probability sampling gives eligible units known or specified selection probabilities.

This does not guarantee that every realized sample perfectly resembles the population.

It does create a formal selection mechanism that makes chance variation analyzable and reduces reliance on uncontrolled human selection.

Chance Imbalance Is Not Necessarily Bias

Suppose a population is \(50\%\) Group A and \(50\%\) Group B.

A legitimate random sample might contain \(47\%\) Group A and \(53\%\) Group B.

This difference can arise from ordinary sampling variability.

A sample does not become biased merely because its realized composition does not exactly equal the population composition.

Systematic Imbalance Is More Concerning

If the sampling method repeatedly produces too few observations from the same subgroup, the imbalance is no longer merely chance fluctuation.

Examples include:

Selection Probabilities

Representativeness often depends on knowing how units entered the sample.

If unit \(i\) has inclusion probability

\[ \pi_i, \]

then unequal probabilities may require weighting or design-based analysis.

Ignoring unequal selection can distort population estimates.

Sampling Weights

A basic sampling weight is

\[ w_i=\frac{1}{\pi_i}. \]

Units selected with smaller probability receive larger design weights.

Weighting can help reconstruct the population structure implied by the sampling design.

Weights do not automatically correct every form of bias.

Weighting and Variability

Weighting can improve representation while increasing variance.

If some observations receive very large weights, the estimate may become more sensitive to those observations.

This illustrates another practical tradeoff between bias control and precision.

Poststratification

Poststratification adjusts sample weights so that weighted sample totals align with known population totals for selected categories.

For example, weights might be adjusted so that the weighted sample matches known population distributions by age group and region.

This can reduce some forms of imbalance when the adjustment variables are relevant to both selection and the outcome.

Weighting Cannot Recover Missing Information Automatically

Suppose an important subgroup has no observations in the sample.

No finite weight applied to observed units can directly reproduce information that was never collected from that subgroup.

Weighting is therefore not a universal repair for coverage failure.

Selection Bias in Observational Data

Many datasets were not originally created through a probability sample.

Administrative records, website activity, customer transactions, medical records, and platform data can contain enormous numbers of observations but still represent only the units that entered the underlying system.

Statistical size and population representativeness are different concepts.

Survivorship Bias

Survivorship bias occurs when analysis focuses on units that remain observable after a filtering or survival process while ignoring units that disappeared.

Examples include:

The surviving group may differ systematically from the original population.

Attrition

Attrition occurs when observations are lost over time in a longitudinal study.

If dropout is unrelated to the outcomes of interest, the damage may be limited.

If dropout is systematically related to those outcomes, attrition can create bias.

Missing Data Mechanisms Matter

Missing observations are not all equivalent.

A useful conceptual distinction is whether missingness is:

The appropriate response to missing data depends on the mechanism that generated the missingness.

Randomization Controls Some Forms of Bias

Random sampling reduces dependence on investigator choice during selection.

Random assignment reduces systematic treatment-group differences in experiments.

Neither process guarantees perfect balance in one realization.

Their value lies in creating a probability mechanism that makes deviations analyzable.

Stratification Can Improve Representation

If an important subgroup is small, a simple random sample may contain only a few observations from it.

Stratified sampling can guarantee that observations are drawn from every designated stratum.

When combined with appropriate weighting, this can support both subgroup analysis and population estimation.

Oversampling

Oversampling means deliberately selecting some groups at a higher rate than their population share.

This is useful when a small but important subgroup would otherwise produce too little data for reliable analysis.

Population-level estimates must then account for the unequal selection probabilities.

Cluster Sampling and Variability

Cluster sampling can reduce field costs but often increases sampling variability when units within a cluster are similar.

A sample of \(500\) people drawn from only a few highly similar clusters can contain less independent information than a dispersed simple random sample of \(500\) people.

Sample size should therefore be interpreted together with design structure.

Effective Sample Size

Complex designs and unequal weights can make the information content of a sample differ from its raw count.

The term effective sample size refers to the size of a simpler reference sample that would provide approximately comparable precision.

There is no single universal effective-sample-size formula for all designs, but the concept is useful for comparing precision.

Design Effect

A common comparison is

\[ \operatorname{DEFF} = \frac{\operatorname{Var}(\text{estimator under actual design})} {\operatorname{Var}(\text{estimator under SRS})}. \]

If

\[ \operatorname{DEFF}>1, \]

the actual design produces greater variance than the simple-random-sample benchmark for the estimator being compared.

Representative Does Not Mean Identical

A useful sample does not have to reproduce every population characteristic exactly.

Random samples naturally fluctuate.

The practical goal is a defensible relationship between the observed sample, the sampling process, and the target population—not perfect duplication of every population percentage.

Representative Does Not Mean Large

A small probability sample can be more informative about a target population than a much larger self-selected sample.

Sample size primarily addresses random variability.

Selection design addresses which units the sample can legitimately represent.

Representative Does Not Mean Demographically Balanced by Hand

Manually selecting observations to make a sample “look like” the population can introduce judgment into the selection process.

Matching known margins can be useful when performed through a principled design or weighting procedure, but visual resemblance alone does not establish unbiased selection.

External Validity

External validity concerns whether study findings can be generalized beyond the observed study units and conditions.

Representativeness is one important contributor to external validity.

Other contributors include:

Internal Validity and External Validity

Internal validity concerns whether the study supports the conclusion being made for the observed study conditions.

External validity concerns whether that conclusion extends to other populations, places, times, or settings.

A study can have strong internal validity but limited external validity.

Example: Precise but Biased Survey

Suppose an online platform surveys \(100{,}000\) of its active users to estimate an opinion among all adults.

The resulting percentage may have very small sampling variability within the platform's active-user population.

But if active platform users differ systematically from all adults, the estimate may not represent the broader target population.

Precision within the observed frame does not guarantee population accuracy.

Example: Unbiased but Variable Estimator

Suppose an estimator satisfies

\[ E(\hat{\theta})=\theta \]

but has standard error

\[ 20. \]

The estimator is correctly centered but individual estimates may be far from the truth.

Larger samples or a more efficient design may reduce the variability.

Example: Biased but Stable Instrument

A thermometer consistently reads \(2^\circ\) too high with almost no measurement noise.

Its readings are highly precise.

They are also systematically biased upward.

Repeating the measurement more times without recalibration does not remove the calibration bias.

Example: Random Measurement Noise

Suppose a device measures a fixed quantity with independent errors having mean \(0\) and standard deviation \(6\).

Averaging \(n\) repeated measurements reduces the standard deviation of the average to

\[ \frac{6}{\sqrt{n}}. \]

This is a variability problem that repeated independent measurement can help reduce.

Data Quality Has Multiple Dimensions

Statistical data quality can involve:

A strong sampling design cannot repair every weakness elsewhere in the data pipeline.

A Census Can Still Be Biased

Observing every unit on a list eliminates sampling variability relative to that list, but it does not automatically eliminate bias.

A census can still suffer from:

Sampling error is only one component of total survey error.

Total Survey Error

The phrase total survey error emphasizes that survey quality is affected by multiple error sources rather than sampling variation alone.

These can include:

Improving one component does not guarantee that the others are small.

Reducing Bias

Bias reduction depends on its source.

Possible strategies include:

Reducing Variability

Possible strategies for reducing variability include:

These strategies do not automatically remove bias.

Improving Representativeness

Representativeness can be strengthened by:

Tradeoffs in Study Design

Statistical design often involves competing objectives.

A geographically dispersed sample may improve population coverage but increase data-collection cost.

Oversampling small groups may improve subgroup precision but require weighting for population estimates.

More intensive follow-up may reduce nonresponse bias but increase cost and time.

Good design balances accuracy, precision, feasibility, and transparency.

Bias Can Enter Before Sampling Begins

Bias can be created by the way the research question itself is framed.

If the target population is defined too narrowly, or the outcome variable does not match the intended concept, later statistical calculations cannot fully repair the mismatch.

Sound inference begins with a precise question and a defensible target population.

Bias Can Enter After Data Collection

Analysis choices can also introduce systematic distortion.

Examples include:

Diagnostic Comparisons

Researchers often compare the observed sample with known population benchmarks.

For example, they may compare distributions of:

Large discrepancies can reveal possible representation problems, although close agreement does not prove the absence of bias.

Sensitivity Analysis

When bias cannot be eliminated or directly measured, researchers may study how conclusions change under plausible alternative assumptions.

This is called sensitivity analysis.

A conclusion that remains similar across a broad range of assumptions is more robust than one that changes sharply under small adjustments.

Transparency Is Part of Statistical Quality

A responsible analysis should report enough information for readers to evaluate the data-generating process.

Important information can include:

Common Error: Believing a Large Sample Must Be Representative

A large sample can have very low random sampling variability while still being systematically unrepresentative.

Sample size primarily addresses precision.

It does not by itself guarantee coverage, random selection, valid measurement, or unbiased response.

Common Error: Calling Any Imbalance Bias

Random samples naturally differ from the population by chance.

A discrepancy between a sample proportion and population proportion is not automatically evidence of biased sampling.

The key question is whether the selection mechanism systematically creates the discrepancy.

Common Error: Assuming Random Sampling Solves Measurement Problems

Random sampling controls how units enter the sample.

It does not correct a miscalibrated instrument, misleading survey question, inaccurate record, or inconsistent coding system.

Selection quality and measurement quality must be evaluated separately.

Common Error: Treating Low Variance as Proof of Accuracy

A procedure can return nearly the same answer every time and still be wrong every time.

Low variance means high precision.

Accuracy also requires that systematic error be sufficiently small.

Common Error: Assuming Weighting Removes All Bias

Weighting can correct known unequal selection probabilities and sometimes reduce imbalance on observed characteristics.

It cannot automatically repair:

Common Error: Reporting a Margin of Error as Total Error

A reported margin of error commonly reflects sampling variability under a specified probability model.

It usually does not include every possible source of error, such as nonresponse bias, undercoverage, wording effects, or measurement error.

The margin of error should therefore not be interpreted as a complete bound on all possible error in the study.

A Reliable Procedure for Evaluating an Estimate

  1. Identify the population parameter being estimated.
  2. Identify the statistic used as the estimator.
  3. Define the target population.
  4. Examine the sampling frame.
  5. Examine how units were selected.
  6. Examine response and missing-data patterns.
  7. Examine how variables were measured.
  8. Assess likely sources of systematic bias.
  9. Assess sampling variability or standard error.
  10. Check whether weighting or design adjustments are required.
  11. Evaluate whether the sample supports the intended generalization.
  12. Report remaining limitations.

Example: Full Bias and Variability Analysis

A retailer wants to estimate average satisfaction among all customers.

It emails a survey only to loyalty-program members and receives responses from \(40\%\) of those contacted.

Several issues must be separated.

Increasing the number of loyalty members contacted can reduce some variability, but it does not automatically make the frame representative of all customers.

Example: Mean Squared Error Comparison

Suppose Estimator A has

\[ \operatorname{Bias}(A)=1,\qquad \operatorname{Var}(A)=16. \]

Then

\[ \operatorname{MSE}(A) = 16+1^2 = 17. \]

Estimator B has

\[ \operatorname{Bias}(B)=3,\qquad \operatorname{Var}(B)=4. \]

Therefore,

\[ \operatorname{MSE}(B) = 4+3^2 = 13. \]

Estimator B has more bias but lower total squared estimation error under the MSE criterion.

Practice 1: Bias or Variability?

A measurement device gives values that are consistently about \(4\) units too high with very little fluctuation.

Is the primary problem bias, variability, or both?

Practice 2: Precision

Two unbiased estimators estimate the same parameter.

Estimator A has standard error \(5\), while Estimator B has standard error \(2\).

Which is more precise?

Practice 3: Undercoverage

A university wants to survey all students but selects names only from students who live in university housing.

Identify the primary sampling problem.

Practice 4: Voluntary Response

A television program asks viewers to visit a website and vote on a controversial issue.

What form of bias is a major concern?

Practice 5: Mean Squared Error

An estimator has bias

\[ 2 \]

and variance

\[ 9. \]

Find its mean squared error.

Practice 6: Increasing Sample Size

A survey uses a badly incomplete sampling frame.

Will increasing the sample size automatically eliminate the coverage bias?

Practice 7: Random Imbalance

A population is \(50\%\) Group A and \(50\%\) Group B. A legitimate random sample is \(47\%\) Group A.

Does this fact alone prove the sampling method is biased?

Practice 8: Nonresponse

A simple random sample of \(1{,}500\) people is selected, but only \(300\) respond.

Why might the final respondent set fail to represent the target population?

Practice 9: Precise but Wrong

A scale gives readings \(1.2\) kilograms too high every time, with almost no random fluctuation.

Describe its accuracy and precision.

Practice 10: Margin of Error

A survey reports a margin of error of \(\pm 3\) percentage points.

Does that number necessarily include possible undercoverage, nonresponse, and measurement bias?

Practice Answers

Practice 1

The primary problem is bias.

The measurements are systematically too high but have little variability.

Practice 2

Estimator B is more precise because it has the smaller standard error:

\[ \boxed{2<5}. \]

Practice 3

The primary problem is undercoverage.

Students who do not live in university housing cannot enter the sample from that frame.

Practice 4

A major concern is voluntary-response bias, also called self-selection bias.

Practice 5

\[ \operatorname{MSE} = \operatorname{Var} + \operatorname{Bias}^2 = 9+2^2 = \boxed{13}. \]

Practice 6

No.

Increasing sample size can reduce sampling variability within the available frame, but it does not automatically add the population units missing from that frame.

Practice 7

No.

A random sample can differ from the population by ordinary chance variation. The imbalance alone does not prove systematic bias.

Practice 8

If response behavior is related to characteristics relevant to the study, the respondents can differ systematically from the selected nonrespondents.

This creates possible nonresponse bias.

Practice 9

The scale is highly precise but inaccurate because its readings are systematically biased upward.

Practice 10

No.

A conventional margin of error usually quantifies sampling variability under a specified model. It does not automatically include undercoverage, nonresponse bias, measurement bias, or every other source of survey error.

What to Remember

Bias, variability, and representativeness describe different dimensions of statistical quality.

The most important habit is: identify whether a problem concerns systematic bias, random variability, population representation, or some combination before deciding how to fix it.

Where We Go Next

Once we understand how sampling design, bias, and variability affect the quality of a statistic, the next step is to use sample statistics deliberately as estimates of unknown population parameters.

The next lesson, Point Estimation, develops the language and methods of estimators, estimates, standard errors, efficiency, consistency, and the practical use of sample information to estimate unknown population quantities.