Datasets, data collection, cleaning, visualization, descriptive statistics, correlation and causation, prediction, bias, and responsible interpretation — how data becomes knowledge
Every day, enormous quantities of information are recorded, transmitted, and stored. Hospitals record patient measurements and treatment outcomes. Retailers track purchases and browsing behavior. Weather stations measure temperature, humidity, and wind speed every few minutes. Social platforms log every post, like, and scroll. All of this recorded information is called data — but raw data does not automatically answer questions. Data science is the discipline that turns raw data into useful knowledge.
This chapter introduces data science as the study of how to collect, organize, clean, explore, analyze, visualize, model, and interpret data to answer questions, discover patterns, and support decisions. You will learn what datasets are, how data is collected and cleaned, how visualization reveals patterns, how descriptive statistics summarize data, what correlation and causation mean, how prediction works at an introductory level, how bias enters data and models, and how to interpret data responsibly.
No prior statistics or calculus is required. You should be comfortable with arithmetic, ratios, percentages, tables, and basic graph reading. This chapter prepares you for later courses in databases, data science, machine learning, and artificial intelligence.
A school district wants to understand which students are most at risk of falling behind in reading. They have access to attendance records, past test scores, course enrollment data, and socioeconomic indicators collected from surveys. How would you turn that data into a useful decision? What would you need to clean, measure, visualize, and watch out for? This chapter gives you the vocabulary and habits to answer those questions carefully.
Data science is the study of how to collect, organize, clean, explore, analyze, visualize, model, and interpret data to answer questions, discover patterns, and support decisions. Data scientists work across medicine, business, science, government, education, journalism, sports, environmental monitoring, and many other fields.
Data science is not a single subject. It sits at the intersection of three disciplines. Computing provides the tools for storing, retrieving, processing, and programming with data. Statistics provides methods for measuring, summarizing, testing, and reasoning under uncertainty. Domain knowledge tells a data scientist which questions matter, what the data actually represents, and whether an answer makes sense.
A data scientist studying hospital readmissions needs all three: programming knowledge to work with the database, statistical reasoning about patient populations, and medical knowledge to interpret what the findings mean in practice.
Data science is not the same as machine learning or artificial intelligence, though all three are related. Machine learning (Chapter 26) is one technique data scientists sometimes use. Data science is the broader practice of working with data from beginning to end — collecting it, understanding it, cleaning it, summarizing it, visualizing it, modeling it, and interpreting results carefully. It is also not magic: data can be incomplete, biased, mismeasured, or simply irrelevant to the question at hand.
In computer science, data refers to recorded information — numbers, text, dates, images, audio, sensor readings, or any other representation of something in the world. Before data can be analyzed, it must be organized. The most common way to organize data for analysis is a dataset — a structured collection of related data organized as a table.
A dataset table has rows, also called records or observations — each row represents one thing being studied. It also has columns, also called features, variables, or attributes — each column represents one characteristic that was measured or recorded.
| Student ID | Age | Study Hours | Practice Tests Taken | Score |
|---|---|---|---|---|
| 1001 | 16 | 12 | 3 | 78 |
| 1002 | 17 | 20 | 5 | 91 |
| 1003 | 16 | 8 | 1 | 65 |
| 1004 | 18 | 15 | 4 | 83 |
| 1005 | 17 | 5 | 0 | 54 |
In this dataset, each row represents one student (five observations), each column represents one feature, and the dataset has five features including the ID column. The term observation is useful because it emphasizes that each row is an observation of one real-world thing at one moment in time.
| Feature Type | Description | Example | Arithmetic Meaningful? |
|---|---|---|---|
| Quantitative | Measurable numbers | Study hours, test score, temperature | Yes — averages and differences make sense |
| Categorical | Groups or categories | Country, color, diagnosis (yes/no) | No — adding "sophomore + junior" is meaningless |
| Ordinal | Ordered categories, unequal spacing | Pain scale 1–10, satisfaction rating | Partially — rank is meaningful, spacing is not |
| Temporal | When something happened | Date of visit, timestamp, duration | Special — has direction, periodicity, and gaps |
Recognizing the type of each feature is one of the first steps in any data science project. The type of feature determines which operations make sense, which visualizations are appropriate, and which statistical methods apply. Treating a categorical feature like a number — for example, treating grade levels (9, 10, 11, 12) as if their differences are meaningful quantities — leads to incorrect analysis.
Before a dataset exists, someone must collect data. How data is collected has enormous consequences for what can and cannot be learned from it.
| Method | Description | Example |
|---|---|---|
| Surveys | Asking people questions and recording answers | Annual census, customer satisfaction poll |
| Sensors and instruments | Devices that automatically record physical measurements | Weather stations, fitness trackers, traffic counters |
| Transaction records | Events recorded as they happen | Purchases, hospital visits, web clicks |
| Experiments | Deliberately varying conditions and measuring outcomes | Clinical drug trials, A/B website tests |
| Administrative records | Data from normal institutional operations | Tax records, school enrollment, court records |
Suppose a data scientist wants to understand the reading habits of teenagers in a country of 50 million people. It would be impossible to survey every teenager. Instead, data scientists typically study a sample — a subset of the full population of interest.
Representativeness is the key property a good sample needs. A representative sample accurately reflects the diversity of the population. A sample of teenagers from one wealthy neighborhood is not representative of all teenagers in a country. When a sample is unrepresentative, the conclusions drawn from it will not apply to the broader population — this is called sampling bias, explored further in Section 25.9.
| Approach | Description | Strength | Limitation |
|---|---|---|---|
| Observational study | Observe things as they naturally occur without intervention | Can study situations where experiments are impossible | Cannot easily establish causation — confounders remain |
| Experiment | Deliberately vary one condition; randomly assign subjects to groups | Random assignment distributes confounders equally; causation more justified | May be impossible or unethical in many real situations |
Data collection always involves choices about what to measure and what to ignore. A hospital dataset might record treatments but not whether patients had strong family support at home — which could also affect outcomes. What is not in a dataset is often as important as what is. A data scientist must always ask: what information is missing, and could it change the conclusions?
Real-world data is rarely perfect when collected. Before analysis, data almost always requires data cleaning — the process of finding and correcting problems so the data accurately represents what it is supposed to represent.
Data cleaning is unglamorous but essential. Experienced data scientists estimate that cleaning consumes a majority of time on most data projects. A small number of errors in a dataset can distort the results of an entire analysis.
| Problem | Description | Example | Possible Handling |
|---|---|---|---|
| Missing values | Data was not collected for some observations | Survey respondent skipped a question | Remove row, or impute using average of known values |
| Duplicate records | Same observation appears more than once | Form submitted twice | Remove the duplicate; keep one copy |
| Inconsistent formatting | Same value represented multiple ways | Dates as "2024-03-15" and "March 15, 2024" | Standardize all values to one format |
| Outliers | Value far from the typical range | Weight recorded as 1500 kg instead of 150 kg | Investigate: genuine extreme (keep) or error (correct/remove) |
| Incorrect types | Values stored as wrong type | A number column stored as text | Convert to the correct type |
Customer ID | Delivery Time (min)
------------|--------------------
2001 | 32
2002 | 28
2003 | 9999 ← sensor error
2004 | 31
2005 | ← missing
2006 | 29
2001 | 32 ← duplicate
Customer ID | Delivery Time (min)
------------|--------------------
2002 | 28
2004 | 31
2006 | 29
Numbers alone are difficult to interpret, especially in large quantities. Data visualization is the practice of representing data graphically so that patterns, trends, outliers, and relationships become easier to see. A good visualization communicates information more efficiently than a table of numbers. A poor one obscures or distorts information.
One of the most famous examples in data science, created by statistician Francis Anscombe in 1973, consists of four small datasets with nearly identical descriptive statistics: the same mean, variance, correlation, and regression line. Yet when plotted, the four datasets look completely different — one is linear, one is curved, one has an outlier, and one forms a vertical cluster. This demonstrates that summary statistics alone can miss important features of data. Visualization catches what numbers summarize away.
Line charts connect data points in sequence, usually over time. Use them when a single quantitative feature is measured at multiple ordered points. The line makes seasonal patterns, trends, and sudden changes easy to see.
Temperature (°C) 35 | *** 30 | *** *** 25 | ** ** 20 | *** ** 15 | *** ** 10 | *** *** 5 | **** *** |------------------------------------------- Jan Mar May Jul Sep Nov Dec Note: Pattern of seasonal rise and fall is immediately visible.
Bar charts use rectangular bars to compare quantities across distinct categories. Use them when comparing a quantitative feature across groups.
Score
100 |
90 | █████
80 | █████ █████ █████
70 | █████ █████ █████ █████
60 | █████ █████ █████ █████ █████
| Sec A Sec B Sec C Sec D Sec E
Note: Bars make it easy to compare five groups at a glance.
A histogram shows how often values fall into different ranges (called bins) for a single quantitative feature. It reveals the shape of the data — whether it is symmetric, skewed, has gaps, or has multiple peaks.
Count 12 | █████ 10 | █████ █████ 8 | █████ █████ █████ 6 | █████ █████ █████ █████ 4 | █████ █████ █████ █████ █████ 2 | █████ █████ █████ █████ █████ █████ | 50-59 60-69 70-79 80-89 90-99 Note: Most scores cluster in the 70–89 range. A bell-shaped histogram is called a "normal distribution," though real data does not always take this form.
Scatter plots place dots on a grid where each axis represents one quantitative feature. Each dot is one observation. Use them to explore whether two numerical features are related.
Score
100 | •
90 | • •
80 | • •
70 | • • •
60 | • •
50 | •
|----------------------------
0 5 10 15 20 25
Study Hours
Note: The general upward pattern suggests students who
studied more tended to score higher — but the relationship
is not perfect. Individual variation exists even within a trend.
Pie charts represent parts of a whole using wedge-shaped slices. They are familiar but frequently misleading. Human eyes are poor at accurately judging the sizes of different-shaped wedges, especially when slices are similar. For most purposes, a bar chart communicates the same information more accurately. Pie charts work reasonably well only when there are a small number of categories and the differences are large and obvious.
| Principle | What It Means | Common Violation |
|---|---|---|
| Accuracy | The visual should not distort the data | A y-axis not starting at zero makes small differences look dramatic |
| Clarity | Labels, axes, and titles should be readable without guessing | No axis labels or units; illegible font sizes |
| Honesty | The chart should show what the data actually says | Showing only the time period where a trend looks favorable |
| Appropriate choice | The chart type should match the data and the question | Using a line chart for unordered categories; a pie chart with 15 slices |
Descriptive statistics are numerical summaries that describe the main features of a dataset. Rather than examining every row individually, descriptive statistics compress a dataset's key properties into a small number of meaningful numbers.
A measure of center describes the typical or central value in a dataset.
Mean = (78 + 91 + 65 + 83 + 54) / 5 = 371 / 5 = 74.2
The mean is sensitive to outliers. Adding a score of 0 (absent student):Mean = (78 + 91 + 65 + 83 + 54 + 0) / 6 = 371 / 6 = 61.8
A single extreme value drops the mean by more than 12 points.
Sorted: 54, 65, 78, 83, 91
Middle value (3rd of 5): 78
The median is resistant to outliers. Adding the 0:Sorted: 0, 54, 65, 78, 83, 91
Two middle values: 65 and 78
Median = (65 + 78) / 2 = 71.5
Median shifted only from 78 to 71.5 — a much smaller change than the mean's shift.
| Situation | Better Measure | Reason |
|---|---|---|
| Data roughly symmetric, no extreme outliers | Mean | Uses all values; most mathematically precise |
| Data contains extreme outliers | Median | Resistant to pull from extreme values |
| Income, housing prices, wealth | Median | These distributions are heavily skewed by high earners |
| Categorical data | Mode | Mean and median are not meaningful for categories |
Knowing the center of a dataset is not enough. Two datasets can have the same mean but very different distributions. Dataset A: 50, 50, 50, 50, 50 — mean = 50, no variation. Dataset B: 10, 30, 50, 70, 90 — mean = 50, values spread widely. A measure of spread describes how much variation exists.
| Measure | How to Calculate | Sensitivity to Outliers |
|---|---|---|
| Range | Maximum − Minimum | Very sensitive — one outlier changes it completely |
| Interquartile Range (IQR) | Q3 (75th percentile) − Q1 (25th percentile) | Resistant — ignores the top and bottom 25% |
| Standard Deviation | Average distance of each value from the mean (roughly) | Sensitive to outliers, but captures all values |
The standard deviation is a measure of how far typical values are from the mean. A small standard deviation means values cluster closely around the mean. A large standard deviation means values are spread out widely. Students who continue to statistics or data science courses will learn to compute it precisely. For now: small std dev = tightly clustered; large std dev = widely spread.
| Feature | Count | Min | Max | Mean | Median | Std Dev |
|---|---|---|---|---|---|---|
| Age | 5 | 16 | 18 | 16.8 | 17 | 0.84 |
| Study Hours | 5 | 5 | 20 | 12.0 | 12 | 5.52 |
| Practice Tests | 5 | 0 | 5 | 2.6 | 3 | 1.95 |
| Score | 5 | 54 | 91 | 74.2 | 78 | 13.7 |
For the Score feature: the median (78) is higher than the mean (74.2), suggesting the lower scores pull the average down. The standard deviation of 13.7 tells us that typical scores are about 13–14 points away from the mean.
One of the most important ideas in data science — and one of the most frequently misunderstood — is the relationship between correlation and causation. Two features are correlated when changes in one feature tend to accompany changes in another. Causation means that one thing directly produces a change in another. These are not the same.
Correlation is measured by the correlation coefficient, written as r, which ranges from −1 to +1.
| r value | Interpretation |
|---|---|
+1.0 | Perfect positive correlation — as one increases, the other always increases proportionally |
+0.7 | Strong positive correlation |
+0.3 | Weak positive correlation |
0.0 | No linear correlation |
−0.3 | Weak negative correlation |
−0.7 | Strong negative correlation |
−1.0 | Perfect negative correlation — as one increases, the other always decreases proportionally |
Consider a famous example: cities that sell more ice cream tend to have more drowning deaths. Ice cream sales and drowning deaths are positively correlated. Does eating ice cream cause drowning? Obviously not.
The explanation is a confounding variable: hot weather. When the weather is hot, people buy more ice cream and people swim more, so more drowning deaths occur. Hot weather causes both. Ice cream and drowning are correlated not because one causes the other, but because both are caused by a third variable.
A confounding variable (also called a lurking variable) is a variable related to both the feature being studied and the outcome, creating a spurious appearance of correlation between them.
CASE 1: A causes B
A ──────────► B
(Correlation reflects genuine causation)
CASE 2: B causes A
B ──────────► A
(They're correlated, but causation runs the opposite direction
from what was assumed)
CASE 3: A confounding variable C causes both A and B
C
/ \
▼ ▼
A B
(A and B are correlated but neither causes the other —
a third variable explains both)
CASE 4: Coincidence
(A and B happen to correlate in this dataset but have
no real relationship — common in small datasets or
when many correlations are tested at once)
REMEMBER: A scatter plot or correlation coefficient tells you
whether A and B tend to vary together.
It does NOT tell you WHY they vary together.
The strongest evidence comes from randomized controlled experiments, where researchers randomly assign subjects to a treatment group or a control group. Random assignment distributes confounding variables equally between groups, so differences in outcomes can be attributed to the treatment. Where experiments are impossible or unethical, scientists use careful observational studies, multiple independent datasets, and accumulated evidence over time. A correlation in data is a clue worth investigating — not a conclusion.
One of the most powerful uses of data science is prediction — using patterns found in historical data to estimate unknown or future values. The feature being predicted is called the target or label. The features used to make the prediction are called inputs or predictors.
| Inputs (known features) | Target (to be predicted) |
|---|---|
| Study hours, practice tests taken | Exam score |
| Patient age, weight, symptoms | Diagnosis |
| Historical temperature, humidity | Tomorrow's weather |
| Transaction amount, time, location | Fraud or not fraud |
| House size, location, age | Sale price |
For two quantitative features with a linear relationship, prediction can be done with a trend line (also called a line of best fit or regression line). The trend line passes through the scatter plot in the direction that best represents the overall relationship.
If the trend line for study hours and scores gives an equation like Predicted score ≈ 45 + 2.5 × (Study Hours), then for a student who studied 16 hours: Predicted score ≈ 45 + 2.5 × 16 = 85. This is an estimate — not a guarantee. A student who studies 16 hours might actually score 79 or 92.
| Problem | Description | Effect |
|---|---|---|
| Underfitting | Model is too simple to capture real patterns | Performs poorly on both training data and new data |
| Overfitting | Model learns the training data too precisely, memorizing noise instead of patterns | Performs well on training data but fails on new data |
| Good fit | Model captures real patterns without memorizing random variation | Performs well on both training data and new data |
Even a good prediction model produces estimates, not guarantees. Presenting model outputs as certainties is one of the most common and harmful mistakes in data science communication. A model that predicts a 73% chance of hospital readmission means exactly that — a 73% probability, not a guaranteed outcome.
Data science can produce conclusions that are unfair, inaccurate, or harmful when the data itself — or decisions made about it — introduce bias. Bias in data science refers to systematic errors that skew results in a consistent direction.
| Type of Bias | How It Arises | Example |
|---|---|---|
| Sampling bias | The sample does not represent the intended population | A medication study conducted only on men whose findings do not apply to women |
| Historical bias | Training data accurately reflects the past, but the past was unfair | A hiring algorithm trained on historical decisions learns to reject women in technical roles because fewer were historically hired |
| Measurement bias | The process of collecting data produces systematic errors | A face-detection system trained mostly on lighter-skinned faces performs poorly on darker-skinned faces |
| Label bias | Labels in training data reflect biased human judgments | A recidivism prediction model trained on data where some judges were systematically harsher toward certain groups |
| Confirmation bias | The analyst focuses on patterns that support preexisting beliefs | Reporting only findings that support a preferred teaching method, ignoring contradictory evidence in the same dataset |
Historical bias is particularly hard to address because the data is technically accurate — it really does reflect what happened. The problem is that what happened was unfair. A system that learns from historical patterns and replicates them is not "neutral" just because it is based on real data. Data encodes the choices, inequalities, and judgments of those who collected it.
Recognizing bias requires asking careful questions at every stage of a project. Who collected this data, and who is represented — and who might be missing? Does the training data reflect historical patterns that should not be perpetuated? Could the measurement process systematically favor certain groups? Are the labels produced by reliable, unbiased judgments? Am I looking for evidence that challenges my assumptions, or only evidence that supports them?
No dataset or analysis is perfectly free of bias. Responsible data science involves making bias visible, addressing it where possible, and being honest about its effects on conclusions.
Data science has significant power to influence decisions — in medicine, criminal justice, hiring, education, and finance. With that power comes responsibility.
| Principle | What It Means |
|---|---|
| Report the full picture | Do not select only results that support a preferred conclusion. Report what the data shows overall, including inconvenient findings. |
| Distinguish statistical from practical significance | A 0.2-point average score difference might be statistically detectable in a huge dataset but have no practical meaning for real decisions. |
| State limitations clearly | Every analysis has limitations: unrepresentative samples, missing variables, models that may not generalize. Stating these is not weakness — it is honesty. |
| Separate description from prescription | Describing what data shows is different from recommending what to do. Decisions require values, context, and judgment that data alone cannot supply. |
When data involves people, ethical responsibilities include privacy — data about individuals should be collected and used in ways that protect them. Even data that seems anonymous can sometimes be re-identified when combined with other information. This is called re-identification and is a serious privacy risk.
Consent means people should know what data is being collected about them and have meaningful choices about how it is used. Data minimization — collecting only the data needed for a specific purpose — reduces the risk of misuse or breach.
[1] DEFINE THE QUESTION
What do we want to know?
What decision will the analysis support?
│
▼
[2] CONSIDER ETHICAL OBLIGATIONS
Who is affected? What are our responsibilities to them?
│
▼
[3] COLLECT DATA
Surveys, sensors, records, experiments
Prioritize representativeness and informed consent
│
▼
[4] CLEAN DATA
Handle missing values, duplicates, formatting errors
Document all changes and why they were made
│
▼
[5] EXPLORE AND VISUALIZE
Summary statistics, charts, distributions
Look for patterns, outliers, anomalies, and sources of bias
│
▼
[6] ANALYZE AND MODEL
Correlation, prediction, classification
Evaluate for bias and generalization to new data
│
▼
[7] INTERPRET AND COMMUNICATE
Report findings, limitations, and uncertainty honestly
Distinguish what data shows from what should be done
│
▼
[8] INFORM DECISIONS
Data is one input — not the only one
Values, context, and human judgment also matter
Note: Steps are often revisited. Analysis may reveal the need
to collect more data or reconsider the original question.
Sum = 23+31+28+19+34+27+31+25+42+22 = 282
Count = 10
Mean = 282 / 10 = 28.2 minutes
Sorted: 19, 22, 23, 25, 27, 28, 31, 31, 34, 42
Even count — average the 5th and 6th values:
Median = (27 + 28) / 2 = 27.5 minutes
What type of data problem most likely explains the discrepancy? What should the data scientist have done before computing the mean?
What is wrong with this chart? How does the y-axis choice affect the viewer's perception? What would the chart look like if drawn honestly?
Identify two errors in reasoning in this recommendation. What is the likely confounding variable?
What problem has occurred? What does it mean, and what should the data scientist do?
Identify the flaw in this conclusion. What population does the sample actually represent?
These questions revisit concepts from earlier chapters.
ID | Salary | Department
---|---------|----------
01 | 45000 | Marketing
02 | 9999999 | ← ?
03 | | Finance
01 | 45000 | Marketing
Chapter 25 built the foundation of data thinking: how to collect, clean, visualize, summarize, and interpret data. Chapter 26 extends these ideas into artificial intelligence and machine learning — systems that learn patterns from data to make predictions, classify inputs, generate content, and support complex decisions.
You have already encountered the core ideas that make machine learning possible: datasets of observations and features, the concept of a model trained on examples to predict new ones, the challenge of avoiding both overfitting and underfitting, and the critical importance of bias in training data. Chapter 26 will show how these ideas are assembled into real AI systems — and what it means to evaluate, trust, and responsibly use them.