Sections 2.1 – 2.8
From Population to Sample: Choosing Data Well
Why Collection Matters · Census and Sample · SRS · Stratified · Cluster · Systematic · Bias · Wording
Section 2.1
Why Data Collection Matters
When students first encounter statistics, they often focus on formulas: how to compute a mean, how to draw a histogram, how to calculate a probability. These skills are real and important. But statistical reasoning begins much earlier — at the moment data are gathered.
Consider a simple example. Suppose you want to know what fraction of students at your school eat breakfast on school days. You ask the first 30 students you see arriving before 7:30 a.m. Many of them probably do eat breakfast, since they arrived early. The fraction you compute might be quite high. But it might not represent students who arrive later, sleep through alarms, skip breakfast to save time, or have no food at home. Your number is calculated correctly, but it describes the wrong group.
The lesson: before trusting any statistical result, ask how the data were collected.
This question matters because most statistical conclusions are generalizations. A study measures some people and draws conclusions about many people. A survey samples some consumers and reports on all consumers. A clinical trial tests a treatment on some patients and recommends it for many patients. In every case, the quality of the generalization depends on the quality of the data collection process.
Poor data collection introduces bias — a systematic tendency for the data to overrepresent or underrepresent certain values. Bias does not mean randomness or noise. Bias means the data consistently point in a wrong direction. Calculations performed on biased data are themselves biased, no matter how carefully the arithmetic is done.
Student Tip
"Garbage in, garbage out" is a principle borrowed from computing that applies equally well to statistics. If the data collection process is flawed, no amount of careful calculation will fix the result.
Section 2.2
Census and Sample
A population is the entire group of individuals about which a statistical question is being asked. A census is an attempt to measure every individual in the population. A sample is a subset of the population that is actually measured. A sample statistic is computed from the sample and used to estimate a population parameter.
Censuses have an obvious appeal: if you measure everyone, you do not have to estimate anything. But censuses are rarely practical. Measuring every individual in a large population is expensive. By the time a complete census is finished, the population may have changed. Some measurements destroy the item being tested — a quality-control engineer cannot test every lightbulb to failure and still have products to sell. And some individuals in a population cannot be reached at all.
For these reasons, statistics usually works with samples drawn from populations. The goal is to choose a sample that is representative — one that reflects the characteristics of the larger population without distortion.
Key Distinctions
Population — the full group being studied
Sample — the group actually measured
Census — measuring the full population
Parameter — a number describing the population (e.g., the true mean μ)
Statistic — a number computed from the sample (e.g., the sample mean x̄)
Example 2.2.1
Identifying population and sample
- ProblemA school district wants to estimate the average number of hours per week that high school students spend on homework. Administrators survey 150 randomly chosen students from the district's four high schools.
- ThinkIdentify who is being studied in full (population) and who was actually measured (sample).
- RepresentPopulation = all high school students in the district. Sample = the 150 students who were surveyed.
- InterpretAny average computed from the 150 students is a sample statistic. It estimates — but does not exactly equal — the average for all students in the district.
Answer: Population = all district high school students. Sample = 150 surveyed students. The computed average is a statistic estimating a parameter.
Section 2.3
Simple Random Samples
The most important principle in sampling is randomness. If individuals are selected randomly, every individual in the population has a chance of being chosen, and no group is systematically favored or excluded. Random sampling does not eliminate all error — a randomly chosen sample will never be a perfect miniature of the population — but it eliminates systematic bias and allows statisticians to use probability to describe how much the sample might differ from the population.
A simple random sample (SRS) is a sample chosen so that every possible group of size n from the population has an equally likely chance of being selected. This is stronger than saying each individual has an equal chance; it means every possible combination of n individuals is equally likely.
How to Choose a Simple Random Sample
1. List every individual in the population and assign each a unique number. This list is called the sampling frame.
2. Use a random number generator to produce n numbers in the appropriate range.
3. Select the individuals whose numbers were chosen.
Example 2.3.1
Choosing a simple random sample
- ProblemA class has 30 students. The teacher wants to select 5 students to present their projects. Describe how to choose a simple random sample.
- ThinkAssign each student a number from 01 to 30. Use a random number generator to select 5 distinct numbers. The students with those numbers present.
- RepresentStudents are numbered 01 through 30. A random number generator produces: 07, 19, 03, 25, 12. These five students are selected.
- CheckEach student had an equal chance of being chosen, and the selection was not influenced by performance, seat location, or any other characteristic.
- InterpretThis is a fair, unbiased selection. No student was systematically favored or excluded.
Answer: Assign numbers 01–30. Generate 5 random numbers. Select corresponding students. This is a valid SRS.
Student Warning
Choosing "whoever is nearest" or "whoever volunteers" is not a simple random sample, even if you intend to be fair. Randomness requires a mechanical, chance-based process — not human judgment.
Section 2.4
Stratified Samples
Sometimes the population has distinct subgroups — called strata — that differ from each other in important ways. If those differences matter to the question being studied, a simple random sample might, by chance, include too many or too few individuals from a particular stratum.
A stratified random sample divides the population into non-overlapping strata and then selects a simple random sample from each stratum. The individual samples are combined to form the full sample. Use stratified sampling when the population has distinct subgroups likely to respond differently, when you need guaranteed representation from each subgroup, or when subgroup comparisons are a key goal.
Example 2.4.1
Stratified sampling design
- ProblemA university wants to survey 200 students about dining hall satisfaction. There are 5,000 undergraduates and 1,000 graduate students. The researcher suspects satisfaction differs between groups. Describe a stratified sampling plan.
- ThinkDivide the population into two strata: undergraduates and graduate students. Sample from each stratum separately.
- RepresentUndergraduates: 5,000 of 6,000 total = 83.3%. Sample ≈ 167 undergraduates. Graduate students: 1,000 of 6,000 = 16.7%. Sample ≈ 33 graduate students. Use SRS within each stratum.
- CheckTotal sample size ≈ 200. Both groups are represented proportionally.
- InterpretThis design ensures both undergraduate and graduate voices are included. Comparing the two strata is also possible, which a simple random sample might not guarantee.
Answer: Stratified sample — 167 undergraduates and 33 graduate students, each selected by SRS within their stratum.
Section 2.5
Cluster Samples
In a cluster sample, the population is divided into groups called clusters. A random sample of entire clusters is selected, and then all individuals within the chosen clusters are measured. Cluster sampling is used when the population is spread across many locations and it is not practical to sample individuals scattered everywhere.
Example 2.5.1
Cluster sampling design
- ProblemA state department of education wants to survey high school students about library use. There are 350 high schools in the state. Describe a cluster sampling plan.
- ThinkEach high school is a cluster. Randomly select a subset of schools and survey all students in those schools.
- RepresentNumber the 350 schools from 001 to 350. Use a random number generator to select 20 schools. Survey all students in those 20 schools.
- InterpretThis is more practical than selecting individual students from all 350 schools. The tradeoff is that if chosen clusters happen to be unusual, the sample may not represent the full population well.
Answer: Cluster sample — randomly select 20 of 350 schools; survey all students within those schools.
| Feature | Stratified Sample | Cluster Sample |
|---|
| Are all groups represented? | Yes — every stratum is sampled | No — only selected clusters |
| Primary goal | Ensure subgroup representation | Practicality and cost savings |
| Within each group | Random sample from each stratum | Measure everyone in chosen clusters |
Section 2.6
Systematic Samples
A systematic sample selects individuals at regular intervals from a list of the population. The procedure: (1) list all individuals; (2) decide the sampling interval k, where k = population size ÷ desired sample size; (3) randomly choose a starting point between 1 and k; (4) select every kth individual after the starting point.
Example 2.6.1
Systematic sample in quality control
- ProblemA factory produces 8,000 items per shift. Quality control wants to inspect 200 items. Describe a systematic sampling plan.
- ThinkSampling interval k = 8,000 ÷ 200 = 40. Randomly choose a starting point between 1 and 40, then inspect every 40th item.
- Represent
Random starting point: 17. Inspector examines items:
17, 57, 97, 137, 177, ... continuing until 200 items are selected
- CheckEvery item had a chance of being selected. The process is mechanical and does not depend on human judgment.
- InterpretThis method is practical for production settings. It is approximately equivalent to an SRS if the list is not organized with a repeating pattern at multiples of 40.
Answer: Systematic sample — k = 40, start at item 17, then every 40th item: 17, 57, 97, 137, …
Student Warning — Periodic Patterns
Systematic sampling can fail if there is a periodic pattern in the population list that aligns with the interval. If a shift schedule repeats every 40 items in a way that places lower-quality items at those positions, every selected item would be a lower-quality one. Always check whether the list has a structure that could interact with the sampling interval.
Section 2.7
Convenience Samples and Voluntary Response Bias
Convenience Samples
A convenience sample consists of individuals who are easy to reach or happen to be nearby when data are collected. The sample is defined not by randomness but by accessibility — a reporter asking opinions from people near a studio entrance, a researcher polling students in the first class that agrees to participate, a doctor gathering data only from her own patients. Convenience samples almost always contain bias because the individuals who are easy to reach are rarely representative of the full population.
Voluntary Response Bias
A voluntary response sample consists of individuals who choose to respond to an open invitation — a call-in poll, an online questionnaire, a mailed survey. Voluntary response samples are biased because people with strong opinions are far more likely to respond than people with mild views or no opinion. This creates a systematic distortion: the sample overrepresents the most motivated voices and underrepresents the quiet majority.
Example 2.7.1
Recognizing voluntary response bias
- ProblemA television news program asks viewers to call in and vote on whether a new city ordinance is a good idea. Of 4,200 callers, 72 percent say it is a bad idea. The anchor reports: "Our viewers overwhelmingly oppose the ordinance."
- ThinkWho called in? Who did not?
- AnalyzePeople with strong negative feelings about the ordinance had the most motivation to call. People who support it or feel neutral may have watched the same program but did not bother to call. The sample represents motivated opponents, not the full viewing audience — and certainly not the full city.
- InterpretThe 72 percent figure reflects voluntary response bias, not a representative opinion of the city's residents. The anchor's conclusion is not supported by this survey design.
Answer: Voluntary response bias — motivated opponents self-selected into the sample. The result does not represent the full population's opinion.
Data Ethics Note
Voluntary response surveys are common in online reviews, political call-in polls, and social media votes. They may produce dramatic-sounding numbers, but those numbers should not be used to make claims about the general population. Responsible statistical communication always acknowledges the limitations of the data source.
Section 2.8
Survey Wording and Response Bias
Even a well-designed random sample can produce biased data if the survey instrument itself is flawed. Two important sources of survey bias are wording bias and response bias.
Wording Bias
Wording bias occurs when the phrasing of a question influences how respondents answer. This can happen deliberately (when a survey is designed to produce a desired result) or accidentally (when a question's language is vague, loaded, or leading). Examples: "Do you support giving hardworking taxpayers a well-deserved tax break?" or "Do you oppose dangerous cuts to essential school funding?" Both use emotionally loaded language that steers the respondent toward a particular answer.
Example 2.8.1
Comparing survey wordings
- Problem
A city is studying public transit. Two versions of the same question are tested:
Version A: "Do you support improving public transportation?"
Version B: "Do you support raising property taxes by $150 per year to fund improvements to public transportation?"
- ThinkBoth questions address the same underlying policy. Do their different framings likely produce different responses?
- AnalyzeVersion A asks about a vague benefit. Most people support "improvement" in principle. Version B attaches a specific cost, which makes the trade-off concrete. Many people who say yes to Version A would say no to Version B.
- InterpretThe version used in a survey dramatically shapes the results. Both must be reported with their exact wording for readers to interpret the data responsibly.
Answer: Both versions address the same policy but likely produce very different results due to wording bias. Neither is inherently "right," but exact wording must always be disclosed.
Response Bias and Undercoverage
Response bias occurs when respondents give inaccurate answers, often because the topic is sensitive, the respondent wants to appear a certain way, or the respondent does not know the true answer but answers anyway. Examples: underreporting alcohol use, overreporting exercise habits, or answering about a political figure one has never heard of rather than saying "I don't know."
Undercoverage occurs when some members of the population are less likely to be included in the sample. A telephone landline survey systematically undercovers younger adults who use only cell phones. An email survey undercovers people without reliable internet access. In both cases, the missing groups may differ in important ways from those who are included.
Student Tip
Read survey questions carefully. Ask: Is the wording neutral? Are important options missing? Could the question be understood in more than one way? Could social pressure cause respondents to answer dishonestly? These questions are as important as the numbers the survey produces.
- Explain the difference between a census and a sample. Give one reason why a census is sometimes impractical.
- What does it mean for a sample to be representative of a population?
- A polling organization wants to survey 400 registered voters from a list of 12,000. Describe how to select a simple random sample using a random number generator.
- A university has four class years. A researcher wants to survey 200 students with 50 from each year. Name the sampling method. Why might this design be preferred over an SRS?
- A school has 50 classrooms. A researcher randomly selects 5 classrooms and surveys every student in those rooms. Name the sampling method and explain the tradeoff compared to an SRS.
- A factory produces 3,000 items per day. Quality control wants to inspect 100 items. They randomly choose a starting point of 14 and inspect every 30th item. Name the method and list the first four items inspected.
- A magazine asks readers to mail in a response card about a proposed policy change. Of 2,000 responses, 68 percent oppose the change. Identify the sampling method and the most serious bias present.
Sections 2.9 – 2.12
Observational Studies and Experiments
Observational Studies · Experiments · Controls and Placebos · Random Assignment and Blinding
Section 2.9
Observational Studies
There are two fundamentally different types of statistical studies, and they support very different kinds of conclusions. An observational study is a study in which the researcher observes and records data on individuals without intervening — without assigning treatments, manipulating variables, or controlling conditions. The researcher watches what happens but does not change what happens.
Observational studies are valuable for describing what is happening in a population, identifying patterns and relationships, and generating hypotheses for further investigation. They are limited because the researcher cannot establish that one variable causes changes in another — confounding variables can explain the observed patterns.
Example 2.9.1
Identifying an observational study
- ProblemResearchers examine medical records from 500 adults aged 40–60. They find that adults who regularly drink green tea have lower rates of cardiovascular disease. The researchers did not tell anyone to drink green tea; they simply recorded what participants already did.
- ThinkDid the researchers assign green tea to some people and not others?
- AnalyzeNo. They observed existing behavior and existing health outcomes. Nothing was manipulated.
- ClassifyThis is an observational study.
- InterpretThe finding shows an association between green tea consumption and lower cardiovascular disease. It does not prove that green tea causes lower rates, because green tea drinkers might differ from non-drinkers in many other ways (diet, income, exercise, stress, genetics) that could explain the health difference.
Answer: Observational study. Finding is an association, not proven causation — confounding variables may explain the relationship.
Section 2.10
Experiments
An experiment is a study in which the researcher actively assigns conditions to individuals in order to observe the effect of those conditions. The researcher does not merely observe what happens — the researcher deliberately creates different conditions and measures the result.
The explanatory variable (also called the independent variable) is the variable whose effect is being studied. The response variable (also called the dependent variable) is the outcome being measured. Experimental units are the individuals to which treatments are applied. Experiments are the gold standard for establishing causation because random assignment distributes other influences evenly across groups, allowing the researcher to isolate the effect of the explanatory variable.
Key Distinction
The fundamental difference between an observational study and an experiment is assignment. In an experiment, the researcher decides who gets which treatment. In an observational study, the researcher measures conditions that already exist.
Example 2.10.1
Identifying an experiment
- ProblemA pharmaceutical company randomly assigns 300 patients with migraines to one of two groups. One group receives a new migraine medication; the other receives a standard medication. After four hours, both groups report their pain levels on a scale from 0 to 10.
- ThinkDid the researchers assign conditions, or did they simply observe existing behavior?
- AnalyzePatients were randomly assigned to groups, and the researcher controlled which medication each patient received. This is active manipulation, not passive observation.
- ClassifyThis is an experiment.
- IdentifyExplanatory variable = type of medication (new vs. standard). Response variable = pain level after four hours. Experimental units = the 300 patients.
Answer: Experiment. Explanatory variable: medication type. Response variable: pain level at 4 hours. Experimental units: 300 patients.
Section 2.11
Treatments, Control Groups, and Placebos
In a well-designed experiment, not all experimental units receive the treatment being studied. Instead, the experiment creates comparison groups so that the researcher can see whether the treatment makes a difference relative to the alternative.
A treatment is a condition deliberately imposed on experimental units. A control group is a group of experimental units that does not receive the active treatment being tested — it provides a baseline against which the treatment group can be compared.
Without a control group, an experimenter cannot distinguish the effect of the treatment from effects that would have occurred anyway. A placebo is an inert treatment — typically a sugar pill or saline injection — that looks identical to the real treatment. Placebos are used because of the placebo effect: the well-documented phenomenon in which people experience real physical improvements simply because they believe they are receiving an effective treatment.
Example 2.11.1
Recognizing the role of a placebo
- ProblemIn a study of a new pain reliever, Group A receives the new medication and Group B receives a sugar pill that looks identical. Neither group knows which they received. After two hours, 82 percent of Group A reports significant pain reduction, compared to 61 percent of Group B.
- ThinkWhat does the difference between 82 percent and 61 percent measure?
- AnalyzeThe 61 percent improvement in Group B shows the size of the placebo effect in this study. The 82 percent in Group A includes both the placebo effect and the real pharmacological effect. The difference — 21 percentage points — estimates the additional effect of the active medication.
- InterpretWithout Group B, the researcher would have seen only 82 percent improvement and might have attributed all of it to the medication. The control group reveals how much of the apparent effect is psychological.
Answer: The placebo accounts for 61% improvement. The active medication adds approximately 21 additional percentage points beyond the placebo effect.
Section 2.12
Random Assignment
Of all the design principles in this chapter, random assignment is among the most powerful. It is the feature that distinguishes a rigorous experiment from a poorly controlled one — and it is what allows researchers to draw conclusions about causation.
Random assignment means that experimental units are assigned to treatment groups by a chance mechanism — a random number generator, a coin flip, a random drawing — not by the researcher's choice, the participant's preference, or any characteristic of the participant.
When people self-select into groups, the groups may differ in important ways before the experiment begins. Volunteers for a new exercise program may already be more motivated to improve their health than non-volunteers. Random assignment spreads these pre-existing differences evenly across groups by chance. With a large enough experiment, the randomly assigned groups will be approximately equivalent at the start, so any systematic difference in outcomes after treatment can be attributed to the treatment itself.
Blinding
Knowledge of treatment assignment can influence behavior. A patient who knows they received a new medication may report better outcomes because of optimism. A researcher who knows which group received the treatment may unconsciously record or interpret results differently. Blinding is the practice of keeping participants and/or researchers unaware of treatment assignments.
Single-blind experiment: Participants do not know which treatment they received, but researchers do.
Double-blind experiment: Neither participants nor the researchers measuring outcomes know which treatment each participant received. A separate administrator manages the assignments. Double-blind experiments are considered the strongest protection against psychological bias.
Example 2.12.1
Designing a double-blind experiment
- ProblemResearchers want to test whether a new sleep supplement improves sleep duration. Describe the key elements of a well-designed experiment.
- ThinkThe study needs a treatment group, a control group, a placebo, random assignment, and blinding.
- Design
Participants: Recruit adults who report trouble sleeping. Assign randomly using a random number generator.
Treatment Group: Takes the sleep supplement each night.
Control Group: Takes an identical-looking placebo capsule each night.
Blinding: Participants do not know which capsule they received. The researchers measuring sleep duration also do not know group membership (double-blind). Assignment records held separately until analysis is complete.
Response Variable: Average nightly sleep duration measured by a wristband tracker over 30 days.
- InterpretAny difference in average sleep duration between the two groups at the end of the study can be attributed to the supplement, because random assignment made the groups equivalent at the start and blinding prevented psychological effects from favoring one group.
Answer: Randomized double-blind placebo-controlled experiment. Random assignment ensures group equivalence; blinding prevents psychological bias; placebo isolates the supplement's real pharmacological effect.
MGU Library Connection
- The logic of double-blind experiments is foundational to clinical trial design and medical research
- Students interested in medicine, public health, psychology, or biology will encounter experimental design in Research Methods courses in the MGU Library
- These principles connect directly to later work in Data Science Foundations and Research Methods
- For each study, identify whether it is an observational study or an experiment. Explain your answer.
(a) Researchers record the eating habits and cholesterol levels of 1,000 adults for five years.
(b) Researchers randomly assign students to two reading programs and compare scores at semester end.
(c) A biologist records the nesting success of birds in different forest habitats.
(d) A company randomly gives some patients a new cholesterol drug and others a placebo, measuring cholesterol after three months.
- A school district wants to know whether a new tutoring program raises test scores. They offer the program to all seventh graders whose parents volunteer them, and compare their year-end scores to seventh graders whose parents did not volunteer. What is the most serious problem with this design?
- Describe the key differences between a single-blind and a double-blind experiment. Why is double-blinding preferred?
- Explain the role of a placebo in an experiment. Why is it not sufficient to simply give the treatment group the medication and give the control group nothing?
- A clinical trial tests a new drug for depression. Group A receives the drug for 12 weeks; Group B receives a placebo. After 12 weeks, 64 percent of Group A and 48 percent of Group B report significant improvement. (a) What is the explanatory variable? (b) What is the response variable? (c) How large is the estimated effect of the drug beyond the placebo effect? (d) Why is it important that patients did not know which group they were in?
Sections 2.13 – 2.15
Confounding, Causation, and Common Mistakes
Confounding Variables · Association vs. Causation · Ten Common Data Collection Errors
Section 2.13
Confounding Variables
Even in a well-conducted observational study, there is a risk that an observed association between two variables is not what it seems. The relationship might not be a direct one — a third variable may be responsible for both. A confounding variable (sometimes called a lurking variable) is a variable that is associated with both the explanatory variable and the response variable, and whose presence can make it appear that the explanatory variable causes changes in the response variable when it may not. Confounding variables do not announce themselves. They hide in the background, influencing both what is being measured and what is being explained.
Example 2.13.1
Identifying a confounding variable — shoe size and reading
- ProblemA study of elementary school children finds that children with larger shoe sizes tend to have higher reading scores. A newspaper headline reads: "Bigger Feet Linked to Better Reading."
- ThinkCould shoe size directly cause reading ability to improve?
- AnalyzeAlmost certainly not. Both shoe size and reading ability are associated with a third variable: age. Older children have larger feet (they've grown more) and score higher on reading tests (they've had more instruction and practice). Age is the confounding variable — it explains the association without requiring any direct connection between feet and literacy.
- InterpretThe association is real in the data, but it is explained by age, not by any causal relationship between shoe size and reading.
Answer: Age is the confounding variable. It causes both larger feet and higher reading scores. The association between shoe size and reading is not causal.
Example 2.13.2
Confounding in a health study — dog ownership and blood pressure
- ProblemAn observational study finds that adults who own dogs have lower average blood pressure than adults who do not. Can we conclude that owning a dog lowers blood pressure?
- ThinkWhat other variables are associated with both dog ownership and blood pressure?
- AnalyzeDog owners may walk more (because dogs require exercise), have stronger social connections (pets reduce loneliness), or differ in age, income, living situation, or stress levels. Any of these variables — not dog ownership itself — might explain lower blood pressure. They are potential confounders.
- InterpretDog ownership is associated with lower blood pressure in this study, but the observational design cannot rule out confounding. A statement of causation is not supported.
Answer: Potential confounders include exercise habits, social connections, age, income, and stress. The observational design cannot establish causation.
Student Warning
Confounding variables are one of the most common reasons why observational studies cannot establish causation. When you read about a study finding that some behavior or exposure is "linked to" an outcome, ask: Is this an experiment or an observational study? If it is observational, what confounding variables might explain the association?
Section 2.14
Association and Causation
The most important and most frequently violated principle in data interpretation is this: association is not causation.
An association (or correlation) between two variables means that knowing the value of one variable gives information about the likely value of the other. Two variables are associated if they tend to move together. Causation means that changing one variable produces a change in the other. Causation is a much stronger claim than association.
| Source of Association | Description | Example |
|---|
| Direct causation | Variable X genuinely produces changes in variable Y | Studying more causes higher quiz scores |
| Reverse causation | Y produces changes in X, not the other way around | Critically ill patients go to hospitals; illness (→ death) causes hospitalization, not the reverse |
| Common cause | A third variable Z causes both X and Y to change | Hot weather causes both more ice cream sales and more drownings; ice cream does not cause drowning |
| Coincidence | Variables move together in a dataset by chance with no meaningful relationship | Per capita cheese consumption correlated with deaths by becoming tangled in bedsheets |
Establishing Causation Requires
Evidence of an association
A plausible mechanism by which X could cause Y
Evidence that the time order is correct (X precedes Y)
Elimination of alternative explanations through experimental control or strong observational evidence from multiple independent studies
Example 2.14.1
Distinguishing association from causation — fast food and obesity
- ProblemA public health researcher reports that adults who eat more fast food have higher rates of obesity. A food industry spokesperson says this proves nothing about fast food's role in obesity. Who is more statistically correct?
- ThinkWhat kind of study design could support a claim of causation? What are the limitations of observational data?
- AnalyzeIf this is an observational study, confounding is possible. People who eat more fast food may also be more sedentary, have lower incomes (with less access to fresh food), sleep less, experience more stress, or differ in other ways that also contribute to obesity. The association is real, but the observational design cannot rule out these alternative explanations.
- InterpretThe association between fast food consumption and obesity is well-established and consistent across many studies, and there are plausible biological mechanisms. Scientists take the evidence seriously. But the spokesperson is technically correct that any single observational study does not prove causation — it shows association consistent with causation, subject to potential confounding. A scientific consensus on causation builds through multiple lines of evidence, including biological research.
Answer: The association is real. The spokesperson is correct that one observational study does not prove causation. Scientific consensus requires multiple independent lines of evidence, including mechanism and experimental data.
Data Ethics Note
The association/causation distinction is frequently misused in advertising, political advocacy, and media reporting. Claims like "studies show that X causes Y" often describe association from observational studies, not established causation from experiments. Reading such claims carefully is an essential part of statistical and scientific literacy.
Section 2.15
Common Data Collection Mistakes
This section summarizes the ten most common errors in data collection. Recognizing these mistakes in real studies is as important as knowing how to design a good study.
Mistake 1 — Using a Convenience Sample
Reaching for whoever is available produces a sample that reflects who was available, not the population of interest. Online polls, hallway interviews, and voluntary surveys all share this flaw.
Mistake 2 — Relying on Voluntary Response
Open invitations produce responses primarily from people with strong opinions. The result overrepresents motivated voices and underrepresents the general population.
Mistake 3 — Biased Survey Wording
Leading questions, emotionally loaded language, and ambiguous phrasing push respondents toward certain answers. Surveys must be carefully reviewed for neutral, clear language.
Mistake 4 — Ignoring Undercoverage
If parts of the population are systematically excluded from the sampling frame — because they lack internet access, do not speak the survey language, or cannot be reached — the results cannot represent the full population.
Mistake 5 — No Control Group
An experiment without a comparison group cannot determine whether the treatment made a difference. Improvement in the treatment group might have occurred without treatment.
Mistake 6 — No Random Assignment
Allowing participants to self-select into treatment groups allows confounding variables to accumulate in one group. The groups are not comparable before treatment begins.
Mistake 7 — No Blinding
When participants know which treatment they received, or when researchers know group membership, psychological expectations and unconscious bias can influence results.
Mistake 8 — Confusing Association with Causation
Observational studies show patterns; they do not establish cause and effect. This is the most consequential interpretive error in popular science reporting.
Mistake 9 — Overgeneralizing from a Nonrepresentative Sample
A study conducted on college students cannot automatically be generalized to adults over 50. A study of patients in one hospital cannot be generalized to patients nationwide. The population to which conclusions apply is limited by who was actually studied.
Mistake 10 — Ignoring Nonresponse
When a significant fraction of the sample does not respond, the remaining responses may not represent the full sample. People who respond may differ systematically from those who do not.
Example 2.15.1
Identifying multiple mistakes — gym satisfaction survey
- ProblemA gym posts a link to an online survey on its social media page and receives 312 responses out of approximately 2,400 members. Of respondents, 91 percent say they are satisfied with the facilities. The marketing team announces: "91 percent of our members are satisfied!"
- Identify
Voluntary response bias: Members who feel strongly — enthusiastic supporters or highly dissatisfied members — are more likely to click the link and complete the survey. Members with neutral feelings may ignore it.
Undercoverage: Not all members follow the gym's social media accounts. Members who are less tech-savvy or who joined recently may not see the survey.
Nonresponse: Only 312 of 2,400 members (about 13 percent) responded. The other 87 percent did not participate.
Overgeneralization: The marketing team applies the 91 percent figure to all members, but only 13 percent of members responded, and those respondents are likely not representative.
- InterpretThe 91 percent figure describes how the 312 self-selected respondents answered, not the satisfaction level of the full membership of 2,400.
Answer: Four mistakes — voluntary response bias, undercoverage, nonresponse, and overgeneralization. The 91% figure cannot responsibly be applied to all members.
- A study finds that cities with more libraries have lower crime rates. Does this prove that libraries reduce crime? Name two confounding variables that might explain the association.
- A study finds that adults who own dogs have lower average blood pressure. Name two confounding variables. What type of study would be needed to establish causation more convincingly?
- Ice cream sales and drowning rates are both higher in summer. What type of association is this? Explain why ice cream does not cause drowning.
- A news article reports: "Countries with more televisions per household have higher life expectancy. Televisions are good for your health." Critique this conclusion. What type of association best describes this relationship?
- A study gives patients a new anti-anxiety medication for three months. Eighty percent report reduced anxiety. The researchers conclude the medication is effective. What critical design element is missing?
- A school principal wants to know how students feel about the new homework policy. She announces that any student who wants to share their opinion can stop by the office during lunch. Of 47 students who stop by, 38 oppose the policy. She reports: "81 percent of students oppose the new policy." Identify all mistakes and explain how each affects the conclusion.
Chapter 2 — Back Matter
Chapter Summary
- Sec 2.1Statistical reasoning begins at the moment of data collection. Bias — a systematic tendency to overrepresent or underrepresent certain values — cannot be corrected by careful arithmetic after the fact.
- Sec 2.2A census measures the entire population; a sample measures a subset. Statistics computed from samples estimate population parameters. A good sample is representative — it reflects the population without distortion.
- Sec 2.3A simple random sample (SRS) gives every possible group of size n an equally likely chance of selection. SRSs are chosen using random number generators, not human judgment. They require a complete sampling frame.
- Sec 2.4Stratified random samples divide the population into non-overlapping strata and select an SRS from each. They guarantee representation of subgroups and allow subgroup comparisons.
- Sec 2.5Cluster samples divide the population into groups (clusters), randomly select some clusters, and measure all individuals in those clusters. They trade precision for practicality.
- Sec 2.6Systematic samples select every kth individual from an ordered list, beginning at a random starting point. They are practical for production and institutional settings but can fail if periodic patterns align with the interval.
- Sec 2.7Convenience samples are biased because accessibility is rarely independent of the characteristics being measured. Voluntary response samples are biased because people with strong opinions are disproportionately likely to respond.
- Sec 2.8Wording bias, response bias, and undercoverage can distort survey results even in randomly chosen samples. Exact survey wording must always be reported so readers can judge the results.
- Sec 2.9Observational studies record data without manipulation. They can identify associations and generate hypotheses but cannot establish causation because confounding variables may explain any observed relationship.
- Sec 2.10Experiments actively assign treatments to experimental units. The explanatory variable is what is manipulated; the response variable is what is measured. Experiments can support causal conclusions when properly designed.
- Sec 2.11Treatments create conditions; control groups provide baselines; placebos isolate the real treatment effect from the placebo effect. Without a control group, apparent improvement cannot be attributed to the treatment.
- Sec 2.12Random assignment distributes confounding variables evenly across groups by chance. Blinding prevents psychological effects from distorting results. Double-blind experiments offer the strongest protection against both sources of bias.
- Sec 2.13Confounding variables are associated with both the explanatory and response variables. They are the primary reason observational studies cannot prove causation. Random assignment in experiments controls for confounding.
- Sec 2.14Association is not causation. Apparent associations can arise from direct causation, reverse causation, a common third cause, or coincidence. Establishing causation requires multiple lines of evidence, including a plausible mechanism, correct time order, and elimination of alternatives.
- Sec 2.15Ten common data collection mistakes: convenience sampling, voluntary response, biased wording, undercoverage, no control group, no random assignment, no blinding, confusing association with causation, overgeneralizing, and ignoring nonresponse.
Key Terms Review
association
A statistical relationship between two variables; knowing one helps predict the other. Association is not causation.
bias
A systematic tendency for data to overestimate or underestimate the true population value. Bias cannot be corrected by arithmetic alone.
blinding
Keeping participants and/or researchers unaware of treatment assignments to prevent psychological effects from influencing results.
census
A study that attempts to measure every individual in the population.
cluster sample
A sample in which the population is divided into clusters, some clusters are randomly selected, and everyone within chosen clusters is measured.
confounding variable
A variable associated with both the explanatory and response variables that can create a false appearance of causation. Also called a lurking variable.
control group
The group in an experiment that does not receive the active treatment; provides a baseline for comparison.
convenience sample
A sample consisting of individuals who are easy to reach; prone to bias because accessibility is rarely independent of the characteristic being studied.
double-blind
An experiment in which neither participants nor the researchers measuring outcomes know which treatment each participant received.
experiment
A study in which the researcher actively assigns conditions (treatments) to experimental units and measures the effect.
experimental unit
The individual (person, animal, plant, or object) to which a treatment is applied.
explanatory variable
The variable whose effect is being studied; the "cause" variable in an experiment. Also called the independent variable.
lurking variable
A confounding variable that is not directly measured or accounted for in a study.
observational study
A study in which the researcher observes and records data without manipulating variables; cannot establish causation.
placebo
An inactive treatment given to the control group to mimic the experience of receiving treatment, controlling for the placebo effect.
placebo effect
The improvement in outcomes experienced by people who believe they are receiving an effective treatment, independent of any pharmacological action.
population
The entire group of individuals about which a statistical question is asked.
random assignment
Using a chance mechanism to assign experimental units to treatment groups, distributing confounding variables evenly across groups.
response bias
Inaccurate answers caused by sensitive topics, social pressure, or respondents not knowing the true answer but answering anyway.
response variable
The outcome being measured in an experiment; the "effect" variable. Also called the dependent variable.
sample
The subset of the population that is actually measured; used to estimate population parameters.
simple random sample
A sample chosen so that every possible group of size n is equally likely to be selected.
stratified sample
A sample in which the population is divided into strata and a random sample is selected from each stratum, guaranteeing subgroup representation.
systematic sample
A sample in which individuals are selected at regular intervals (every kth individual) from an ordered list after a random starting point.
treatment
A condition deliberately imposed on experimental units in order to observe its effect on the response variable.
undercoverage
Bias that results when parts of the population are systematically excluded from the sampling frame.
voluntary response bias
Bias that results when individuals self-select into a sample, overrepresenting those with strong opinions.
wording bias
Bias introduced by the phrasing of survey questions — loaded language, leading questions, or ambiguous wording that steers responses.
Concept Review Questions
- Why is randomness important in choosing a sample?
- Describe one difference between a stratified sample and a cluster sample.
- What is voluntary response bias? Why does it tend to distort results in a predictable direction?
- What is the difference between an observational study and an experiment?
- Why does random assignment help researchers draw conclusions about causation?
- What is a confounding variable? Give an original example not from this chapter.
- Name four sources of apparent association between two variables. Which source is the only one that implies a true causal relationship?
- A study finds that people who wear seatbelts are more likely to survive car accidents. Is this association likely to reflect causation? Explain.
Bias Identification Practice
For each study, identify the type of bias (wording bias, voluntary response bias, convenience sampling, undercoverage, response bias, or nonresponse) and explain how it distorts results.
- A student surveys her 20 friends about their study habits to estimate the typical study time for students at her school. All 20 are in the same after-school drama club that practices until 9:30 p.m.
- A survey asks: "Do you agree that the local government's excessive spending is out of control?" to measure attitudes toward city spending.
- A telephone survey using only landline numbers is used to estimate the political opinions of adults under 35.
- A health survey asks respondents to self-report the number of alcoholic beverages they consume per week.
- An online news site posts a poll asking: "Should the president resign?" One hour after posting, 84 percent of 5,600 respondents say yes. The site reports this as a national opinion poll.
- A mail survey about neighborhood safety is sent to 500 households. Only 120 responses are returned. The researcher analyzes only the returned responses and reports on the full neighborhood.
Real-World Applications
- A news report states: "A study of 1.2 million social media users found that users who posted more frequently had higher rates of anxiety." The study analyzed posts and medical records — no treatments were assigned. (a) Is this an experiment or an observational study? (b) Can the study conclude that social media use causes anxiety? Why or why not? (c) Name two confounding variables. (d) What type of study would establish causation more convincingly?
- A city government wants to estimate the percentage of households that recycle regularly. The city has three district types: downtown apartments, suburban houses, and rural properties. Explain why a stratified random sample might be preferable to a simple random sample.
- A fast-food chain wants feedback on a new menu item. They leave comment cards on tables at five of their busiest urban locations. Describe at least two ways this could produce a biased sample.
- A polling company calls 1,000 randomly selected adults. Of those called, 380 agree to answer the survey. The pollsters report results from the 380 respondents. What concern should a careful reader have about this report?
- A pharmaceutical company publishes a study showing that patients who took their new vitamin supplement had 40 percent fewer sick days than the general population. The patients were not randomly selected — they purchased the supplement on their own. (a) Is this an experiment or an observational study? (b) Name at least two confounding variables. (c) Is the company justified in advertising that the supplement "causes" fewer sick days?
Error Analysis
Each of the following contains a statistical or methodological error. Identify the error and explain how it affects the conclusion.
Problem 1
A school principal surveys 47 volunteers during lunch and reports "81% of students oppose the homework policy."
What sampling method is being used? Name at least two errors. What should the principal report instead?
Problem 2
A researcher assigns the three most motivated students to the new teaching method group. After one month, they score higher. The researcher concludes the new method is superior.
What critical design principle is violated? How does this affect the conclusion?
Problem 3
A study gives patients a new anti-anxiety medication for three months. 80% report reduced anxiety. The researchers conclude the medication is effective.
What essential element is missing from this experiment? Why is the 80% figure alone insufficient to conclude the medication works?
Problem 4
"Nine less than twice a number n" written as 9 − 2n
Error: this belongs in the Algebra I chapter. A statistics version: "Countries with more televisions per household have higher life expectancy, so buying a television improves health." Identify the error in causal reasoning.
Problem 5
A gym survey gets 312 of 2,400 members responding online. The gym reports "91% of members are satisfied."
Identify all sources of bias. Rewrite the claim in a statistically responsible way.
Chapter 2 Checkpoint
Chapter 2 Checkpoint
Complete without assistance — assesses all key skills and concepts from Chapter 2
Section 1 — Vocabulary and Concepts
- Define bias in the context of data collection. Give one example of a biased sampling method and explain why it produces a biased result.
- Explain the difference between a parameter and a statistic.
- What is the fundamental difference between an observational study and an experiment?
Section 2 — Sampling Design
- A researcher wants to estimate the average number of books read per year by adults in a large city. The city has three districts: downtown (population 40,000), midtown (population 80,000), and uptown (population 30,000). (a) What is the population? (b) If the researcher wants a total sample of 300 adults allocated proportionally by district, how many should be sampled from each district? (c) What type of sampling design is this? (d) Why might this design be preferable to a convenience sample taken at a single downtown library?
- Describe how you would use a systematic sample to inspect 150 items from a production run of 6,000. What is the sampling interval? If the random starting point is 22, what are the first four items inspected?
Section 3 — Study Design
- A health organization studies whether daily meditation reduces blood pressure. They recruit 200 adults with elevated blood pressure. Participants are randomly assigned to two groups. Group A attends a daily 20-minute meditation session for 8 weeks. Group B attends a daily 20-minute general wellness lecture for 8 weeks. Neither group is told which intervention is considered the "active treatment." Blood pressure is measured by a nurse who does not know which group each participant is in. (a) Is this an experiment or an observational study? (b) What is the explanatory variable? (c) What is the response variable? (d) Why did the researchers give Group B a wellness lecture rather than nothing? (e) Is this study single-blind or double-blind? Explain. (f) What does random assignment protect against?
Section 4 — Association and Causation
- A study finds that students who participate in extracurricular activities have higher GPAs than students who do not. (a) Is this likely an experiment or an observational study? (b) Name two confounding variables that might explain the association. (c) Write one sentence interpreting the finding responsibly, without overclaiming causation.
- For each of the following, state whether the study is an experiment or observational study, identify the explanatory and response variables, and state whether the study can support a conclusion of causation. Explain. (a) Researchers randomly assign 80 kindergartners to two phonics programs and measure reading fluency at year end. (b) A sociologist analyzes police records and finds neighborhoods with more streetlights have lower car theft rates. (c) Adults who voluntarily began running three times per week are compared to non-exercisers; runners have lower depression rates one year later.
Section 5 — Error Analysis
- A company surveys customers by placing an online feedback form at the bottom of every customer receipt email. Of 12,000 customers who received the email, 890 filled out the survey. Of those, 76 percent rated their experience as "excellent." (a) What sampling method is being used? (b) What is the response rate? (c) Describe two sources of bias. (d) Why should the company be cautious about claiming "76 percent of our customers rate us excellent"?
Bridge Note — Collecting Data and the Road Ahead
Chapters 3–5: Organizing, Displaying, and Summarizing Data
The skills in Chapter 2 are foundational to everything that follows in this book. When you encounter a graph in Chapter 3 or 4, ask: where did these data come from, and how were they collected? When you calculate a mean or standard deviation in Chapter 5, ask: does this sample represent the population? When you study correlation in Chapter 6, ask: could this association be explained by a confounding variable?
Statistical reasoning is not a series of disconnected procedures. It is a chain of judgment calls, beginning with how data are gathered. The choices made at the collection stage echo through every calculation and conclusion that follows. In college statistics courses, sampling distributions and confidence intervals rely explicitly on the assumption that data come from properly randomized samples. In data science and machine learning, questions about training data — how it was collected, who is represented, what biases it contains — are among the most important questions in the entire field.
Learning to collect data carefully, and to recognize when data have been collected carelessly, is one of the most transferable skills in statistics.
MGU Library Connection
- Data Collection Reference — Appendix D of this textbook
- Chapter 1: What Statistics Studies — population, sample, and the nature of statistical questions
- Chapter 12: Sampling Distributions and Estimation — how sampling variation is modeled mathematically
- Chapter 13: Hypothesis Testing — where proper study design determines whether a test is valid
- Research Methods (MGU Library) — experimental design in psychology, biology, public health, and education
- Data Science Foundations, Chapter 3 (MGU Library) — bias in training data and machine learning
- MGU Dictionary entries: bias, confounding variable, random assignment, simple random sample, placebo effect, association, causation