Purpose and Overview

Learning Objectives

1. 1Explain what files are, why programs need them, and how memory and storage differ.
2. 2Read a file system path and describe what it points to, distinguishing absolute from relative paths.
3. 3Describe plain text files and explain why they are useful across computing.
4. 4Interpret a tabular dataset, identifying rows, columns, fields, headers, and records.
5. 5Read and interpret a CSV file and explain how the format handles special cases.
6. 6Trace the read-process-write pattern in a data program.
7. 7Identify common data quality problems and describe how cleaning addresses them.
8. 8Trace filtering, sorting, and summarizing operations on small datasets.
9. 9Discuss the ethical principles that govern responsible data use and explain why they matter.

Sections at a Glance

• 14.1Why Programs Use FilesMemory vs. storage; persistence; the four uses of files; why files are necessary for real software
• 14.2Files, Folders, and PathsFile systems; directories; root; absolute and relative paths; file extensions; path notation
• 14.3Plain Text FilesCharacter-only storage; UTF-8; log files; config files; reading and writing in Python style
• 14.4Tables, Rows, Columns, and FieldsTabular structure; records; fields; headers; reading table data
• 14.5CSV Data at an Introductory LevelCSV format; header row; commas; special cases; reading CSV in Python style
• 14.6Reading and Writing DataRead-process-write pattern; line-by-line reading; writing to files; a complete example
• 14.7Cleaning DataMissing values; inconsistent formats; duplicates; out-of-range values; cleaning in Python style
• 14.8Filtering and Sorting DataFiltering conditions; sorting ascending and descending; multi-level sorting; combining operations
• 14.9Summarizing DataCounting; sum and average; min and max; grouping; a complete summary pipeline
• 14.10Responsible Use of DataSensitive data; PII; re-identification; six principles; data and power; a developer checklist

Why Programs Use Files

Memory vs. Storage

The distinction between memory and storage is worth making precise before going further.

Files, Folders, and Paths

/ (root)
├── home/
│   └── student/
│       ├── documents/
│       │   ├── essay.txt
│       │   └── notes.md
│       ├── data/
│       │   ├── students.csv
│       │   └── scores.csv
│       └── programs/
│           └── grade_calculator.py
└── etc/
    └── config.txt

File Paths

Absolute (Linux / macOS):   /home/student/data/students.csv
Absolute (Windows):         C:\Users\student\data\students.csv

Relative (from student/):   data/students.csv
Relative (from programs/):  ../data/students.csv   ← go up, then into data/

File Extensions

Most file names include an extension — a short suffix after a period that indicates the file type by convention.

Plain Text Files

What Plain Text Looks Like

2024-09-01 08:14:22 User login: asmith
2024-09-01 08:15:03 File opened: report.csv
2024-09-01 08:16:45 File saved: report.csv
2024-09-01 08:22:11 User logout: asmith

username = student1
language = English
theme = dark
font_size = 14

Reading and Writing Plain Text in Python Style

# Reading: basic pattern
file = open("notes.txt", "r")   # "r" means open for reading
contents = file.read()           # read everything into a string
file.close()                     # close the file when done
print(contents)

# Safer pattern using 'with' (auto-closes on exit)
with open("notes.txt", "r") as file:
    contents = file.read()
    print(contents)

# Writing plain text
with open("output.txt", "w") as file:   # "w" means open for writing
    file.write("Hello, file system.\n")
    file.write("This is a second line.\n")

Tables, Rows, Columns, and Fields

CSV Data at an Introductory Level

student_id,name,grade,score       ← header row (column names)
1001,Amara Osei,10,87
1002,Jin Park,11,93
1003,Fatima Nkrumah,10,78
1004,Liam Torres,12,95
1005,Sofia Reyes,11,82

Complications in CSV

Real CSV files are rarely as clean as the example above. Three common complications arise.

Values containing commas → wrap the field in quotation marks:
1006,"Torres, Maria",12,90

Values containing quotation marks → double the quotes inside:
1007,Kai ""The Coder"" Kim,11,88

Missing values → comma is present but nothing appears between:
1008,,10,75    ← the name field is empty

Reading CSV in Python Style

Python has a built-in csv module that handles these complications automatically. DictReader reads each row as a dictionary, where keys are column names from the header row.

import csv

with open("students.csv", "r") as file:
    reader = csv.DictReader(file)    # reads header row automatically
    for row in reader:
        print(row["name"], row["score"])

Reading and Writing Data

┌───────────────┐     ┌───────────────┐     ┌───────────────┐
│               │     │               │     │               │
│   SOURCE      │────▶│   PROCESS     │────▶│  DESTINATION  │
│               │     │               │     │               │
│ students.csv  │     │ clean         │     │ report.csv    │
│ database      │     │ filter        │     │ screen output │
│ network API   │     │ sort          │     │ database      │
│ user input    │     │ summarize     │     │ email / log   │
│               │     │ transform     │     │               │
└───────────────┘     └───────────────┘     └───────────────┘

Reading Line by Line

For large files, reading the entire file at once can waste memory. Reading line by line processes one row at a time.

with open("students.csv", "r") as file:
    for line in file:
        print(line.strip())    # strip() removes the trailing newline character

A Complete Read-Process-Write Example

This program reads student scores, adds 5 bonus points to each, and writes the updated records to a new file.

import csv

# Step 1: Read
students = []
with open("students.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        students.append(row)

# Step 2: Process
for student in students:
    original = int(student["score"])
    student["score"] = original + 5

# Step 3: Write
with open("students_updated.csv", "w", newline="") as file:
    writer = csv.DictWriter(file, fieldnames=["student_id", "name", "grade", "score"])
    writer.writeheader()
    writer.writerows(students)

print("Done. Updated scores written to students_updated.csv")

Cleaning Data

Missing Values

Inconsistent Formats

Cleaning in Python Style

import csv

cleaned = []
with open("students_raw.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        # Skip rows with missing names or scores
        if not row["name"] or not row["score"]:
            continue

        # Convert score to integer; skip if it fails
        try:
            score = int(row["score"])
        except ValueError:
            continue

        # Skip out-of-range scores
        if score < 0 or score > 100:
            continue

        # Standardize name capitalization
        row["name"] = row["name"].strip().title()

        cleaned.append(row)

print(f"Cleaned: {len(cleaned)} valid records")

Filtering and Sorting Data

Filtering

Filtering means selecting records that satisfy a condition and discarding the rest. Filters can test a single field or combine multiple conditions.

# Filter: grade 10 students only
grade_10 = []
for student in students:
    if int(student["grade"]) == 10:
        grade_10.append(student)

# Filter: combined condition — grade 10 AND score above 80
for student in students:
    if int(student["grade"]) == 10 and int(student["score"]) > 80:
        print(student["name"])

Sorting

Sorting arranges records in ascending (low to high) or descending (high to low) order based on one or more fields. Python's sorted() function accepts a key argument that specifies how to compare records.

# Sort by score, highest first
sorted_students = sorted(students, key=lambda s: int(s["score"]), reverse=True)

for student in sorted_students:
    print(student["name"], student["score"])

# Multi-level sort: grade ascending, then score descending within each grade
sorted_students = sorted(
    students,
    key=lambda s: (int(s["grade"]), -int(s["score"]))
)

Filtering and Sorting Together

Real programs usually combine filtering and sorting in sequence.

# Grade 11 students, sorted by score descending
grade_11_top = sorted(
    [s for s in students if int(s["grade"]) == 11],
    key=lambda s: int(s["score"]),
    reverse=True
)

Summarizing Data

Counting, Sum, Average, Min, and Max

# Counting
total = len(students)
grade_10_count = sum(1 for s in students if int(s["grade"]) == 10)

# Sum and average
scores = [int(s["score"]) for s in students]
average_score = sum(scores) / len(scores)
print(f"Average: {average_score:.1f}")    # .1f = one decimal place

# Min and max
print(f"Lowest: {min(scores)}")
print(f"Highest: {max(scores)}")

Grouping

Grouping means computing summaries for each category separately — for example, the average score per grade level.

from collections import defaultdict

grade_scores = defaultdict(list)   # each grade maps to a list of scores

for student in students:
    grade = student["grade"]
    score = int(student["score"])
    grade_scores[grade].append(score)

for grade in sorted(grade_scores.keys()):
    scores = grade_scores[grade]
    avg = sum(scores) / len(scores)
    print(f"Grade {grade}: {len(scores)} students, average {avg:.1f}")

Responsible Use of Data

What Makes Data Sensitive

Some data is more sensitive than other data. Categories that require particular care include:

Six Principles of Responsible Data Use

Data and Power

A Developer Checklist

Before building a program that collects or processes data about people, thoughtful developers ask:

□  What data do we actually need?
□  Where will it be stored, and who can access it?
□  How long will we keep it, and when will we delete it?
□  Are we collecting this data with users' knowledge and consent?
□  What could go wrong if this data were accessed by the wrong person?
□  Who is accountable if something goes wrong?

Chapter Summary

• Sec 14.1Files are named collections of data stored on persistent devices. They allow programs to save, load, share, and archive information. Memory is fast and temporary; storage is slower and permanent. Programs copy data between them when reading and writing files.
• Sec 14.2File systems organize files into hierarchical trees of folders. The root directory is the starting point. Absolute paths begin at the root; relative paths begin at the current working directory. The .. symbol moves up one level. File extensions indicate type by convention.
• Sec 14.3Plain text files store only characters using a standard encoding. They have no special formatting and can be read by any program that handles characters. They are used for source code, configuration files, logs, and data files. Opening with "w" erases existing content; use "a" to append.
• Sec 14.4Tabular data organizes information into rows (records) and columns (fields). Headers name the columns. Individual cells at row-column intersections are field values. This structure appears in spreadsheets, databases, CSV files, and data science datasets.
• Sec 14.5CSV files represent tables as plain text with commas separating fields and newlines separating rows. The first line is the header row. Special cases — values with commas, embedded quotes, and missing fields — are handled by quoting conventions. Python's csv.DictReader automates parsing.
• Sec 14.6The read-process-write pattern is the standard structure for data programs: read from a source, transform or analyze the data, write results to a destination. This pattern scales from small scripts to industrial pipelines.
• Sec 14.7Data cleaning identifies and fixes quality problems before analysis. Common problems include missing values, inconsistent formats, duplicate records, and out-of-range values. Incorrect data produces incorrect conclusions regardless of how sophisticated the analysis is.
• Sec 14.8Filtering selects records that satisfy conditions; sorting arranges them in order. Both operations appear constantly in real programs. Multi-level sorting uses a tuple as the sort key. Programs commonly filter first, then sort the filtered results.
• Sec 14.9Summarizing produces aggregate statistics: counts, sums, averages, minimums, maximums, and group-by breakdowns. Grouping computes summaries for each category separately. These summaries appear in reports, analytics dashboards, and scientific studies.
• Sec 14.10Data about people carries ethical weight. Responsible data use means collecting only what is necessary, storing securely, using data only for stated purposes, respecting consent, deleting data when no longer needed, and being honest about limitations. Re-identification shows that removing names does not make data automatically anonymous.

Key Terms Review

Diagram Review

/ (root)
├── home/
│   └── student/
│       ├── data/
│       │   ├── students.csv    ←  /home/student/data/students.csv  (absolute)
│       │   └── scores.csv
│       └── programs/
│           └── process.py      ←  /home/student/programs/process.py (absolute)
└── etc/
    └── config.txt

From programs/, the relative path to students.csv is:   ../data/students.csv
  ( .. moves up to student/ → then data/ → then students.csv )

student_id , name              , grade , score    ← header row (column names)
-----------   -----------------   -----   -----
1001       , Amara Osei        , 10    , 87       ← row 1 (record 1)
1002       , Jin Park          , 11    , 93       ← row 2 (record 2)
1003       , Fatima Nkrumah    , 10    , 78       ← row 3 (record 3)
  ↑                ↑               ↑       ↑
field 1          field 2         field 3  field 4

Each cell (row ∩ column) is one field value.
The field at row 1002, column "score" = 93.

Code Review

students = [
    {"name": "Mia", "grade": "10", "score": "72"},
    {"name": "Leo", "grade": "11", "score": "88"},
    {"name": "Zara", "grade": "10", "score": "91"},
]
for s in students:
    if int(s["grade"]) == 10:
        print(s["name"], int(s["score"]) + 5)

sorted_students = sorted(students, key=lambda s: int(s["score"]))

for s in sorted_students:
    print(s["name"], s["score"])

with open("notes.txt", "r") as file:
    for line in file:
        print(line.strip())

name,score
Alice,85
Bob,92
Carol,78

import csv

scores = []
with open("scores.csv", "r") as file:
    reader = csv.DictReader(file)
    for row in reader:
        scores.append(int(row["score"]))

print(min(scores))
print(max(scores))
print(sum(scores) / len(scores))

Worked Example Review

Concept Check

1. What is the difference between a file and a variable? Why are files necessary?
2. What does .. mean in a relative file path? Give an example.
3. Why does CSV wrap some field values in quotation marks?
4. What is the difference between filtering and sorting?
5. Name three common data quality problems that data cleaning addresses.
6. What does int(row["score"]) do, and why might it raise an error?
7. What is re-identification, and why does it matter for privacy?
8. In the read-process-write pattern, what happens during the process step?

Practice Questions

Identify each as an absolute path or a relative path. Explain how you can tell.

1. /home/student/data/grades.csv
2. data/grades.csv
3. ../reports/summary.txt
4. C:\Users\admin\desktop\notes.txt

Read this CSV data and answer the questions.

employee_id,department,salary,years
E001,Engineering,72000,3
E002,Marketing,58000,7
E003,Engineering,81000,5
E004,HR,54000,2
E005,Marketing,61000,4
E006,Engineering,94000,9

1. How many rows of data are in this table, not counting the header?
2. How many columns does it have? Name them.
3. What is the salary of employee E003?
4. Which department has the most employees in this dataset?
5. What is the average salary across all six employees?

Describe in plain English what each condition selects from a student dataset.

1. int(s["score"]) >= 90
2. s["grade"] == "12"
3. int(s["score"]) > 70 and int(s["score"]) < 85
4. s["name"].startswith("A")

Answer both parts without using a computer.

1. Put these names in alphabetical (ascending) order: Torres, Osei, Nkrumah, Park, Reyes
2. Put these scores in descending order (highest first): 78, 95, 87, 82, 93

Use these scores. Show your arithmetic.

Scores: 91, 76, 84, 63, 88, 72, 95, 80, 68, 77

1. Count: how many scores?
2. Sum: what is the total?
3. Average: sum ÷ count?
4. Minimum: lowest score?
5. Maximum: highest score?
6. Filtered count: how many are above 80?

Identify every data quality problem in this CSV. For each, name the type and describe the specific issue.

student_id,name,grade,score
1001,Marcus Chen,10,87
1002,,11,93
1003,Layla Hassan,10,78
1003,Layla Hassan,10,78
1004,Raj Patel,12,115
1005,amara osei,11,82
1006,Sofia Torres,9/1/2008,74

1. List every problem you can find. For each: name the type (missing value, duplicate, format inconsistency, or out-of-range) and describe what is wrong.

A school stores a CSV file containing student names, home addresses, parent contact information, health conditions, and academic records.

1. Which fields would you classify as sensitive personal data? Why?
2. Who should be allowed to access this file? Who should not?
3. What risks would arise if this file were accidentally made publicly accessible?
4. When might it be appropriate to delete records from this file?
5. If a researcher wanted to study the relationship between health conditions and academic performance, what ethical questions would need to be addressed before sharing the data?

Debugging and Error Analysis

Design and Application Questions

Design a CSV table to store information about library books. The table must be able to answer these questions: Who is the author? What is the title? Is the book currently checked out? When is it due back? How many times has it been borrowed this year?

1. Write out the column names you would use.
2. For each column, state what type of data it holds (text, number, date, or yes/no).
3. Write two sample rows of data that would fit your design.

A program will receive a CSV file of product inventory data from an external supplier. The file contains product names, prices, quantities, and category codes.

1. Write a cleaning checklist of at least five specific checks the program should perform before using this data. For each check, explain what problem it catches and what the program should do when it finds one.

Describe in plain English — not code — a complete read-process-write pipeline that reads a CSV file of employee records (name, department, salary, years of service), removes records with missing values, filters for employees with more than 3 years of service, calculates the average salary for each department, and writes a summary CSV with one row per department.

1. Write out each stage (read, process steps, write) in your own words.
2. What columns would the output CSV file contain?
3. How many rows would the output have, given the input described?

Real-World Applications

Web servers generate log files recording every request they receive. A typical log line:

2024-09-15 14:22:03 192.168.1.45 GET /index.html 200 1543

This records: date and time · visitor IP address · request type · page requested · status code (200 = success) · bytes transferred.

1. What questions could a developer answer by analyzing a log file?
2. What data cleaning steps might be needed before analyzing logs at scale?
3. A website gets 50,000 log entries per day. What summary statistics would be most useful?
4. What privacy concerns arise from storing IP addresses in log files?

1. Why would a clinic export data to CSV rather than always analyzing directly in the database?
2. Which fields in medical records would you classify as highly sensitive?
3. If a researcher wants to study which medications are most commonly prescribed, how could the data be anonymized before sharing?
4. If an error in the data showed a patient had been prescribed the wrong medication, what should happen — technically and ethically?

1. What data would a search engine need to store for each indexed web page?
2. How is filtering relevant when a user submits a search query?
3. How is sorting relevant when displaying search results?
4. What data quality problems might arise when indexing pages from across the entire web?

Mixed Review

1. From Chapters 3 and 5. Describe in pseudocode an algorithm that reads a list of numbers and prints only those greater than a specified threshold. Include inputs, outputs, and the main steps.
2. From Chapter 8. A loop reads records from a file one at a time. What type of loop would you use — a for loop or a while loop? Explain your choice.
3. From Chapter 4. A CSV file uses UTF-8 encoding. What does that mean? Why does encoding matter when opening a file that may contain names from many languages?
4. From Chapter 12. A CSV file has 1,000,000 records. A program needs to find all records where score > 90. Would this be faster if the records were pre-sorted by score? Explain using the concept of binary search.
5. From Chapter 13. A data scientist builds a model using historical student data and warns the results might not generalize to students in other schools. How does the concept of model assumptions and limitations from Chapter 13 apply here?

Chapter 14 Checkpoint