MGU Computer Science Series — Programming with Python

Part V: Files, Data, and Practical Automation

Scripts · Batch Processing · Command-Line Arguments · Logging · Safe Automation

Chapter 25 of 30

MGU Computer Science Series · Programming with Python · Library Textbook Edition

Chapter 25:
Automating Everyday
Tasks

Scripts as tools — batch processing, command-line arguments, report generation, configuration, logging, and the discipline of safe, testable automation

■ Part V of VII ■

■ 10 Sections • Chapter 25 of 30 ■

Prerequisite: Chapters 21–24 — files, CSV data, exceptions, and standard library tools

Chapter 25 — Front Matter

Chapter Introduction and Learning Objectives

A program that runs once is a solution. A program that runs a thousand times without breaking is a tool.

Every chapter in this book has taught you to build programs that solve one problem at a time. You have learned to read files, process CSV data, handle exceptions, generate random numbers, and design functions that do their job cleanly. You now have all the ingredients for something more powerful: automation.

Automation means writing programs that do repetitive, rule-following work on your behalf — without you having to sit at the keyboard each time. Scripts that process many files. Programs that generate formatted reports. Tools that apply the same transformation to dozens of records. Utilities that run on a schedule or in response to a trigger.

This chapter shows you how to think about automation, how to design scripts that work reliably on real data, and how to write automation that is safe, testable, and responsible. You will also learn two new practical tools: command-line arguments, which let you pass information to a script when you run it, and introductory logging, which lets a script leave a record of what it did.

By the end of this chapter, you will be able to write Python scripts that automate genuine everyday work — and more importantly, you will know how to test and safeguard them so they do not cause damage when something unexpected happens.

10Sections

12Objectives

14Key Terms

30Total Chapters

Learning Objectives

By the end of this chapter, you should be able to:

1Explain what automation means and identify tasks that are good candidates for it.
2Distinguish between interactive programs and scripts, and describe the structure of a well-designed script.
3Write batch-processing loops using os.listdir(), os.path.join(), and endswith().
4Accept command-line arguments in a Python script using sys.argv and handle missing arguments safely.
5Process many files safely: check before writing, handle per-file errors, and write to separate output folders.
6Implement dry-run mode to preview script behavior before committing to a run.
7Generate formatted text reports by combining data processing with file output.
8Manage configuration values using top-of-file constants, command-line arguments, and plain-text configuration files.
9Add basic logging to a script using both manual log files and Python's built-in logging module.
10Test automation scripts before running them widely, using test folders, check functions, and hand-traced examples.
11Apply safe automation practices: backing up data, writing to output files rather than overwriting originals, and requiring confirmation before irreversible actions.
12Design and build a complete batch-processing script that combines the major techniques of this chapter.

Chapter 25 — Contents

Sections at a Glance

25.1Why Automation MattersWhat automation is; tasks well-suited for automation; why automation amplifies both good and bad code; the importance of testing
25.2Scripts as ToolsScripts versus interactive programs; a minimal script; well-structured script design; separating functions from the main flow
25.3Repetitive Tasks and Batch ProcessingBatch processing concept; looping over a list of items; looping over files with os.listdir(), endswith(), and os.path.join(); tracing a batch loop
25.4Command-Line Arguments at an Introductory Levelsys.argv; reading filename and value arguments; checking argument count; multiple arguments; when to use command-line arguments
25.5Processing Many Files SafelyChecking before writing with os.path.exists(); per-file error handling with try/except; output folders with os.makedirs(); dry-run mode
25.6Generating ReportsPlain text report pattern; report from CSV data; building report output incrementally
25.7Configuration ValuesConstants at the top of a script; command-line arguments for changing values; plain-text configuration files; choosing between options
25.8Logging at an Introductory LevelWhy logging matters; manual log file with timestamps; Python's built-in logging module; log levels; what to log
25.9Testing Automation Before Running It WidelySingle-file testing; test folder strategy; compare before and after; check functions; hand-tracing a loop
25.10Safe and Responsible AutomationNever delete without backups; write output files not input files; reversible automation; confirmation prompts; documenting scripts

Section 25.1

Why Automation Matters

Computers are exceptionally good at doing the same thing many times without getting tired, distracted, or bored. Humans are not. When a task is repetitive, rule-following, and predictable, it is a strong candidate for automation.

Consider a few scenarios:

A teacher receives a folder of student response files each week and needs to count how many are missing.
A small business keeps daily sales records as CSV files and wants a weekly summary.
A researcher has three hundred text files and needs to find every file that contains a specific phrase.
A student wants to rename fifty downloaded files from messy names like img_4892.jpg to organized names like photo_001.jpg, photo_002.jpg, and so on.

In each case, doing the work by hand would take a long time, involve clicking through many files or rows, and risk human error. A well-written Python script can do the same work in seconds, apply rules perfectly consistently, and run again next week with no extra effort.

Automation also makes programs more useful. Instead of writing a program that processes one file whose name you type inside the code, you can write a script that processes any file whose name you pass in from the command line. That single change turns a personal tool into something you can share and reuse.

Important Caution

Automation is not magic. A script that has a bug in its logic will apply that bug to every file you give it. Automation amplifies both good code and bad code. This is why testing, careful design, and safe defaults matter even more in automation than in interactive programs — and why this chapter pays so much attention to testing and safety.

Section 25.2

Scripts as Tools

An interactive program waits for the user to type something at every step. A script is different. A script is a program that runs from start to finish, taking in information at the beginning (through files, command-line arguments, or configuration) and producing output at the end (in files, reports, or printed summaries). The user does not need to sit at the keyboard while it runs.

Think of a script as a tool, the way a hammer is a tool. You pick it up, use it, and put it down. You do not hold it and watch it while it rests on the shelf. A good script is the same: you run it when you need it, it does its job, and it stops.

A Minimal Script

Here is a simple script that counts the lines in a text file:

Python — count_lines.py

# count_lines.py
# Counts the number of lines in a given text file.

def count_lines(filename):
    """Return the number of lines in a file."""
    with open(filename, "r") as f:
        lines = f.readlines()
    return len(lines)

filename = "notes.txt"
total = count_lines(filename)
print(f"{filename} contains {total} lines.")

This script does one thing cleanly. It has a function that does the real work. It uses a with statement so the file is always closed properly. But notice the limitation: the filename is hardcoded as "notes.txt". If you want to count lines in a different file, you have to open the script and change the code. Section 25.4 will fix this using command-line arguments.

Structure of a Well-Designed Script

Good scripts tend to follow a consistent pattern:

Python — clean script structure pattern

# A pattern for clean script structure.

import os

def process_file(filename):
    """Do the real work here."""
    # ... logic goes here
    pass

def main():
    """Coordinate the overall script."""
    filename = "data.txt"
    process_file(filename)
    print("Done.")

main()

Design Habit

Putting the main logic inside a main() function and calling it at the bottom keeps the script organized and makes testing easier. The functions can be tested independently, and the script's behavior is easy to read from the bottom up.

Section 25.3

Repetitive Tasks and Batch Processing

The power of automation shows most clearly when you need to do the same thing to many items. This is called batch processing: applying one operation to a whole batch of inputs.

Processing a List of Items

Python — generate_greetings.py

def make_greeting(name):
    """Return a personalized greeting string."""
    return f"Hello, {name}! Welcome to the course."

students = ["Alice", "Bob", "Carmen", "David", "Elena"]

for name in students:
    greeting = make_greeting(name)
    print(greeting)

Output

Hello, Alice! Welcome to the course. Hello, Bob! Welcome to the course. Hello, Carmen! Welcome to the course. Hello, David! Welcome to the course. Hello, Elena! Welcome to the course.

The loop does the work. The function keeps the logic clean. If you want to change the greeting, you change one function — not five separate lines.

Processing Many Files

Python — batch file summary

import os

def summarize_file(filepath):
    """Print a simple summary of a text file."""
    with open(filepath, "r") as f:
        lines = f.readlines()
    word_count = sum(len(line.split()) for line in lines)
    print(f"{filepath}: {len(lines)} lines, {word_count} words")

folder = "reports"

for filename in os.listdir(folder):
    if filename.endswith(".txt"):
        filepath = os.path.join(folder, filename)
        summarize_file(filepath)

os.listdir(folder)

List folder contents

Returns the names of all items in a folder — files and subfolders alike. Does not include path prefix.

endswith(".txt")

Filter by extension

Filters so you only process files of the right type, skipping folders, images, or other formats.

os.path.join(folder, name)

Build full path safely

Constructs a correct path for any operating system. Safer than folder + "/" + name because Windows uses backslashes.

Tracing the Loop

Suppose the reports folder contains these files: january.txt, february.txt, notes.docx, march.txt

Iteration 1january.txt ends with .txt → path built → file summarized.
Iteration 2february.txt ends with .txt → path built → file summarized.
Iteration 3notes.docx does not end with .txt → skipped entirely.
Iteration 4march.txt ends with .txt → path built → file summarized.
PatternVisit everything, filter what you need, process what passes the filter. This is the core batch-processing pattern.

Section 25.4

Command-Line Arguments at an Introductory Level

When you run a Python script from the terminal, you can pass information to it directly on the same line. These pieces of information are called command-line arguments. Instead of hardcoding a filename inside the script, the script receives it as an argument when it starts — making the script reusable with any file without changing any code.

How sys.argv Works

Python stores command-line arguments in a list called sys.argv, available in the sys module. sys.argv[0] is always the script name. Arguments you provide start at sys.argv[1]. All values are strings — convert with int() or float() if you need numbers.

Python — show_args.py

import sys

print("Script name:", sys.argv[0])
print("All arguments:", sys.argv)
print("Number of arguments:", len(sys.argv))

Terminal command

python show_args.py hello world 42

Output

Script name: show_args.py All arguments: ['show_args.py', 'hello', 'world', '42'] Number of arguments: 4

Using a Command-Line Argument for a Filename

Python — count_lines.py (improved with sys.argv)

import sys

def count_lines(filename):
    """Return the number of lines in a file."""
    with open(filename, "r") as f:
        lines = f.readlines()
    return len(lines)

if len(sys.argv) < 2:
    print("Usage: python count_lines.py <filename>")
else:
    filename = sys.argv[1]
    total = count_lines(filename)
    print(f"{filename} contains {total} lines.")

Always Check First

The if len(sys.argv) < 2 check is essential. If the user runs the script without a filename, accessing sys.argv[1] causes an IndexError. Checking first and printing a usage message is far friendlier — and it is standard practice in script design.

Multiple Arguments

Python — search_file.py (two arguments)

import sys

def count_word(filename, word):
    """Count how many lines contain the given word."""
    count = 0
    with open(filename, "r") as f:
        for line in f:
            if word.lower() in line.lower():
                count += 1
    return count

if len(sys.argv) < 3:
    print("Usage: python search_file.py <filename> <word>")
else:
    filename = sys.argv[1]
    word = sys.argv[2]
    result = count_word(filename, word)
    print(f'Found "{word}" in {result} lines of {filename}.')

Situation	Best approach
Filename to process	`sys.argv[1]`
Value changes every run	Command-line argument
Flag like `--dry-run`	`"--dry-run" in sys.argv`
Many options with help text	Python's `argparse` module (beyond this chapter)

Section 25.5

Processing Many Files Safely

Batch processing that touches many files needs careful design. A careless script can overwrite files, skip files silently, fail partway through, or produce incorrect output with no warning. A careful script checks before it acts, handles errors gracefully, and reports what it did.

Check Before Writing

Python — check before overwrite

import os

output_path = "output/report.txt"

if os.path.exists(output_path):
    print(f"Warning: {output_path} already exists. Skipping.")
else:
    with open(output_path, "w") as f:
        f.write("Report content here.\n")
    print(f"Wrote {output_path}.")

Handle Errors Per File, Not Per Script

When processing many files, a single broken file should not stop the entire batch. Wrap per-file work in a try/except block and collect errors for review at the end.

Python — per-file error handling

import os

def process_file(filepath):
    """Process one file and return a summary string."""
    with open(filepath, "r") as f:
        content = f.read()
    word_count = len(content.split())
    return f"{filepath}: {word_count} words"

folder = "documents"
results = []
errors = []

for filename in os.listdir(folder):
    filepath = os.path.join(folder, filename)
    try:
        summary = process_file(filepath)
        results.append(summary)
    except Exception as e:
        errors.append(f"Could not process {filepath}: {e}")

print("=== Results ===")
for r in results:
    print(r)

if errors:
    print("\n=== Errors ===")
    for e in errors:
        print(e)

Build Output Paths Carefully

Python — separate input and output folders

import os

input_folder = "raw_data"
output_folder = "processed_data"

os.makedirs(output_folder, exist_ok=True)  # Create folder if it doesn't exist.

for filename in os.listdir(input_folder):
    if filename.endswith(".txt"):
        input_path = os.path.join(input_folder, filename)
        output_path = os.path.join(output_folder, filename)
        # Process input_path and write to output_path.

os.makedirs with exist_ok

os.makedirs(output_folder, exist_ok=True) creates the output folder if it does not exist. The exist_ok=True argument prevents an error if the folder already exists. Without it, the script crashes on a second run.

Dry-Run Mode

Python — dry-run pattern

DRY_RUN = True  # Set to False to actually process files.

def process_file(filepath, dry_run=False):
    if dry_run:
        print(f"[DRY RUN] Would process: {filepath}")
    else:
        print(f"Processing: {filepath}")
        # ... real processing here

folder = "data"
for filename in os.listdir(folder):
    if filename.endswith(".csv"):
        process_file(os.path.join(folder, filename), dry_run=DRY_RUN)

When you are confident the script works correctly, change DRY_RUN = True to DRY_RUN = False and run it for real. This pattern prevents expensive mistakes.

Section 25.6

Generating Reports

One of the most common and useful automation tasks is generating reports: taking raw data and producing a formatted, readable summary. Combining file I/O, CSV processing, and formatted strings lets you produce reports that would take a long time to create by hand.

A Simple Text Report

Python — score report generator

def compute_stats(scores):
    """Return a dictionary of statistics for a list of scores."""
    if not scores:
        return {"count": 0, "total": 0, "average": 0, "highest": 0, "lowest": 0}
    return {
        "count": len(scores),
        "total": sum(scores),
        "average": sum(scores) / len(scores),
        "highest": max(scores),
        "lowest": min(scores),
    }

def write_report(stats, output_path):
    """Write a formatted report to a text file."""
    with open(output_path, "w") as f:
        f.write("=== Score Report ===\n\n")
        f.write(f"Number of scores:  {stats['count']}\n")
        f.write(f"Total:             {stats['total']}\n")
        f.write(f"Average:           {stats['average']:.2f}\n")
        f.write(f"Highest score:     {stats['highest']}\n")
        f.write(f"Lowest score:      {stats['lowest']}\n")

scores = [88, 92, 75, 61, 94, 83, 77, 90, 68, 85]
stats = compute_stats(scores)
write_report(stats, "score_report.txt")
print("Report written to score_report.txt.")

Output file — score_report.txt

=== Score Report === Number of scores: 10 Total: 813 Average: 81.30 Highest score: 94 Lowest score: 61

Generating a Report from CSV Data

Python — weekly sales report

import csv

def load_sales(filepath):
    """Load sales records from a CSV file."""
    records = []
    with open(filepath, "r", newline="") as f:
        reader = csv.DictReader(f)
        for row in reader:
            records.append({
                "product": row["product"],
                "quantity": int(row["quantity"]),
                "price": float(row["price"]),
            })
    return records

def total_revenue(records):
    """Return the total revenue across all records."""
    return sum(r["quantity"] * r["price"] for r in records)

def top_product(records):
    """Return the product name with the highest total revenue."""
    revenues = {}
    for r in records:
        name = r["product"]
        revenue = r["quantity"] * r["price"]
        revenues[name] = revenues.get(name, 0) + revenue
    return max(revenues, key=revenues.get)

def write_sales_report(records, output_path):
    """Write a sales summary report."""
    revenue = total_revenue(records)
    best = top_product(records)
    with open(output_path, "w") as f:
        f.write("=== Weekly Sales Report ===\n\n")
        f.write(f"Total transactions: {len(records)}\n")
        f.write(f"Total revenue:      ${revenue:.2f}\n")
        f.write(f"Top product:        {best}\n")

records = load_sales("sales.csv")
write_sales_report(records, "sales_report.txt")
print("Sales report written.")

Building Report Output Incrementally

For longer reports, collect lines in a list first and write them all at once. Build the report in memory, then call f.writelines(lines). This is clean, efficient, and easy to read.

Section 25.7

Configuration Values

Scripts often need values that might change from one use to the next: a folder path, a threshold, a date, a maximum number of rows. There are three common patterns for managing these.

Option 1: Constants at the Top of the Script

The simplest approach: put adjustable values in named constants near the top of the file, in ALL_CAPS by convention.

Python — top-of-file configuration constants

# config values — adjust these before running
INPUT_FOLDER = "raw_data"
OUTPUT_FOLDER = "processed_data"
MIN_SCORE = 60
REPORT_TITLE = "Quarterly Results"

Option 2: Command-Line Arguments

For values that change every time you run the script, command-line arguments are more flexible than editing the file each time.

Python — folder paths from command line

import sys

if len(sys.argv) < 3:
    print("Usage: python process.py <input_folder> <output_folder>")
else:
    input_folder = sys.argv[1]
    output_folder = sys.argv[2]

Option 3: A Plain-Text Configuration File

For larger scripts, store configuration in a separate file that the script reads at startup. Someone who is not a programmer can edit the config file without touching Python code.

config.txt

# Settings input_folder = raw_data output_folder = processed min_score = 60

Python — load_config function

def load_config(filepath):
    """Load key-value configuration from a text file."""
    config = {}
    with open(filepath, "r") as f:
        for line in f:
            line = line.strip()
            if line and not line.startswith("#"):
                key, value = line.split("=", 1)
                config[key.strip()] = value.strip()
    return config

config = load_config("config.txt")
input_folder = config.get("input_folder", "data")
min_score = int(config.get("min_score", 0))

Situation	Best approach
Simple personal script	Constants at the top
Value changes every run	Command-line argument
Many settings, shared with others	Configuration file

Section 25.8

Logging at an Introductory Level

When a script runs automatically — perhaps overnight, on a schedule, or on a server — you may not be watching the output. Logging solves this: writing a record of events to a file as the program runs, so you can review it afterward.

Simple Manual Logging

Python — manual log with timestamps

import os
from datetime import datetime

def write_log(log_path, message):
    """Append a timestamped message to the log file."""
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    with open(log_path, "a") as f:
        f.write(f"[{timestamp}] {message}\n")

LOG_FILE = "process.log"
input_folder = "data"

write_log(LOG_FILE, "Script started.")

for filename in os.listdir(input_folder):
    if filename.endswith(".txt"):
        filepath = os.path.join(input_folder, filename)
        try:
            # Process the file here.
            write_log(LOG_FILE, f"Processed: {filepath}")
        except Exception as e:
            write_log(LOG_FILE, f"ERROR processing {filepath}: {e}")

write_log(LOG_FILE, "Script finished.")

Resulting log file — process.log

[2024-10-14 09:15:02] Script started. [2024-10-14 09:15:02] Processed: data/report_a.txt [2024-10-14 09:15:03] Processed: data/report_b.txt [2024-10-14 09:15:03] ERROR processing data/broken.txt: 'utf-8' codec can't decode byte... [2024-10-14 09:15:03] Processed: data/report_c.txt [2024-10-14 09:15:03] Script finished.

Python's Built-In `logging` Module

Python — logging module (introductory)

import logging

# Configure logging to write to a file.
logging.basicConfig(
    filename="script.log",
    level=logging.INFO,
    format="%(asctime)s %(levelname)s: %(message)s"
)

logging.info("Script started.")
logging.info("Processing file: data.txt")
logging.warning("File was empty.")
logging.error("Could not open config.txt")
logging.info("Script finished.")

Resulting log file — script.log

2024-10-14 09:15:02,341 INFO: Script started. 2024-10-14 09:15:02,342 INFO: Processing file: data.txt 2024-10-14 09:15:02,343 WARNING: File was empty. 2024-10-14 09:15:02,344 ERROR: Could not open config.txt 2024-10-14 09:15:02,345 INFO: Script finished.

DEBUG

Detailed diagnostic info

Lowest level. Used during development to trace every step. Suppressed in production.

INFO

Confirmation of normal events

Records that things are working as expected. Script started, file processed, count completed.

WARNING

Something unexpected, but recoverable

A file was empty, a value was out of the expected range. Processing continued.

ERROR

A serious problem occurred

A file could not be opened. A record was malformed. Processing for this item stopped.

What to Log

Log when the script started and finished. Log which files or records were processed. Log any errors or warnings with enough detail to diagnose the problem. Log counts and summaries. Avoid logging so much that the file becomes too large to read.

Section 25.9

Testing Automation Before Running It Widely

Before you let a script loose on a hundred files or an important dataset, you need to be confident it works correctly. Automation that has a bug in it will apply that bug to every single input.

Testing Strategy

Five habits for testing automation scripts before committing to a full run

One file firstStart by running the script on a single, small, known input. Check the output manually. Confirm it is exactly what you expected. Only then test on a larger set.
Test folderKeep a small folder of test files with known properties: a normal file, an empty file, a file with unusual formatting, and a file that is intentionally broken.
Compare before and afterSave copies of original files first — or use a separate output folder. Compare the output to the originals to confirm the transformation is correct.
Check functionWrite a function that validates the output: expected line count, expected keys in a dictionary, expected file size range. Checking your own output is a sign of careful programming.
Hand-trace one iterationBefore running any batch script on real data, trace through one pass of your main loop. Ask: what does this do to this specific input? What if the file is empty? What if the file is missing?

Python — check function example

def check_output(output_path, expected_line_count):
    """Verify that the output file has the expected number of lines."""
    with open(output_path, "r") as f:
        lines = f.readlines()
    if len(lines) != expected_line_count:
        print(f"WARNING: Expected {expected_line_count} lines, got {len(lines)}.")
        return False
    return True

Section 25.10

Safe and Responsible Automation

Automation is powerful. That power requires responsibility. Before writing a script that modifies, moves, renames, or deletes files — or sends messages, submits forms, or changes records — think carefully about what could go wrong.

Never Delete Files Automatically Without Backups

A script that deletes files can destroy data permanently. Keep backups before running any deletion script. Use dry-run mode first. Consider moving files to a "trash" folder rather than deleting immediately — you can empty it later once you are sure. Always print what will be deleted and ask the user to confirm before proceeding.

Python — confirmation before irreversible action

files_to_delete = ["old_backup_1.txt", "old_backup_2.txt"]

print("The following files will be deleted:")
for f in files_to_delete:
    print(f"  {f}")

confirm = input("Type YES to proceed: ")
if confirm == "YES":
    for f in files_to_delete:
        os.remove(f)
        print(f"Deleted: {f}")
else:
    print("Cancelled. No files were deleted.")

Principle 1

Write output files, not input files

Write results to a new output file rather than modifying the original. This preserves the original data and lets you compare results at any time.

Principle 2

Test on small, fake data first

Before running on real data, create a small set of test data that mimics the real data. Test every case, including errors. Only then run on real data.

Principle 3

Write reversible automation

Design scripts so effects can be undone. Writing to a new output file is reversible; overwriting the original is not. Moving to a backup folder is reversible; deleting is not.

Principle 4

Document what the script does

Always include a comment block at the top explaining purpose, inputs, outputs, and usage. This takes thirty seconds and saves future readers — including yourself — a great deal of confusion.

Python — script documentation block

# batch_summarize.py
#
# PURPOSE: Reads all .txt files in the input folder and writes
#          a one-line summary of each to summaries.txt.
#
# INPUT:   A folder named "reports/" containing .txt files.
# OUTPUT:  A file named "summaries.txt" in the current directory.
#
# USAGE:   python batch_summarize.py
#
# NOTES:   Does not modify input files.
#          Overwrites summaries.txt if it already exists.

Chapter 25 — Worked Example

A Batch Word-Count Report Script

This example builds a complete script that reads all .txt files in a given folder, counts the words in each file, and writes a formatted report. It incorporates command-line arguments, safe file handling, per-file error logging, and a dry-run option. It is designed to be read carefully from top to bottom.

Step 1

Plan the script before writing any code

InputsInput folder path and output report path, accepted as command-line arguments. Optional --dry-run flag.
ProcessingList all .txt files. For each: count words, record result, log success or failure.
OutputFormatted report file with per-file counts, totals, and error list. Log file with all events timestamped.

Step 2

Write the helper functions

Python — word_count_report.py (helper functions)

import os
import sys
from datetime import datetime

def count_words_in_file(filepath):
    """Return the number of words in a text file."""
    with open(filepath, "r", encoding="utf-8") as f:
        content = f.read()
    return len(content.split())

def collect_txt_files(folder):
    """Return a sorted list of .txt file paths in a folder."""
    files = []
    for filename in sorted(os.listdir(folder)):
        if filename.endswith(".txt"):
            files.append(os.path.join(folder, filename))
    return files

def write_log(log_lines, message):
    """Append a timestamped message to the log list."""
    timestamp = datetime.now().strftime("%H:%M:%S")
    log_lines.append(f"[{timestamp}] {message}")

Step 3

Write the main processing logic

Python — process_files function

def process_files(file_list, dry_run=False):
    """
    Process each file in file_list.
    Returns (results, log_lines) where results is a list of (filepath, word_count)
    tuples and log_lines is a list of log messages.
    """
    results = []
    log_lines = []

    write_log(log_lines, f"Starting. {len(file_list)} files to process.")

    for filepath in file_list:
        if dry_run:
            write_log(log_lines, f"[DRY RUN] Would process: {filepath}")
            results.append((filepath, None))
        else:
            try:
                count = count_words_in_file(filepath)
                results.append((filepath, count))
                write_log(log_lines, f"OK: {filepath} ({count} words)")
            except Exception as e:
                results.append((filepath, None))
                write_log(log_lines, f"ERROR: {filepath} — {e}")

    write_log(log_lines, "Processing complete.")
    return results, log_lines

Step 4

Write the report and log functions

Python — write_report and write_log_file functions

def write_report(results, report_path, dry_run=False):
    """Write a formatted word-count report."""
    if dry_run:
        print(f"[DRY RUN] Would write report to: {report_path}")
        return

    successful = [(path, count) for path, count in results if count is not None]
    failed = [(path, count) for path, count in results if count is None]
    total_words = sum(count for _, count in successful)

    with open(report_path, "w") as f:
        f.write("=== Word Count Report ===\n")
        f.write(f"Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}\n\n")
        f.write(f"Files processed:    {len(successful)}\n")
        f.write(f"Files with errors:  {len(failed)}\n")
        f.write(f"Total words:        {total_words}\n\n")
        f.write("--- File Details ---\n")
        for path, count in successful:
            f.write(f"  {os.path.basename(path):30s}  {count:>6} words\n")
        if failed:
            f.write("\n--- Files with Errors ---\n")
            for path, _ in failed:
                f.write(f"  {os.path.basename(path)}\n")

    print(f"Report written to: {report_path}")

def write_log_file(log_lines, log_path):
    """Write log messages to a log file."""
    with open(log_path, "a") as f:
        for line in log_lines:
            f.write(line + "\n")

Step 5

Write the main function

Python — main function

def main():
    dry_run = "--dry-run" in sys.argv

    if len(sys.argv) < 3:
        print("Usage: python word_count_report.py <input_folder> <output_report> [--dry-run]")
        return

    input_folder = sys.argv[1]
    output_report = sys.argv[2]

    if not os.path.isdir(input_folder):
        print(f"Error: '{input_folder}' is not a valid folder.")
        return

    files = collect_txt_files(input_folder)

    if not files:
        print(f"No .txt files found in '{input_folder}'.")
        return

    print(f"Found {len(files)} .txt file(s) in '{input_folder}'.")
    if dry_run:
        print("DRY RUN: No files will be written.")

    results, log_lines = process_files(files, dry_run=dry_run)
    write_report(results, output_report, dry_run=dry_run)
    write_log_file(log_lines, "word_count.log")

    print(f"Log written to: word_count.log")

main()

Step 6

Sample run and output

Terminal command

python word_count_report.py reports/ word_count_report.txt

Terminal output

Found 4 .txt file(s) in 'reports/'. Report written to: word_count_report.txt Log written to: word_count.log

Resulting report file — word_count_report.txt

=== Word Count Report === Generated: 2024-10-14 09:15:03 Files processed: 4 Files with errors: 0 Total words: 2847 --- File Details --- january.txt 743 words february.txt 812 words march.txt 698 words notes.txt 594 words

What Makes This Script Well Designed

It separates concerns into focused functions. It handles errors per file rather than stopping on the first error. It supports dry-run mode. It logs what it did. It validates its inputs before processing. It writes to an output file rather than modifying the originals. And it is documented at the top.

No single technique here is complicated. The quality comes from applying many good habits together.

Chapter 25 — Back Matter

Chapter Summary

Section 25.1Automation means writing programs that perform repetitive, rule-following work reliably without human intervention for each run. Automation amplifies both good code and bad code, which is why testing and safe design matter especially here.
Section 25.2A script runs from start to finish without interactive prompting. Well-designed scripts separate work into focused functions, accept input from outside the code, and are organized so the main flow is readable from the bottom up.
Section 25.3Batch processing applies one operation to many items using a loop. The core pattern for files is: visit everything with os.listdir(), filter by extension with endswith(), and build paths safely with os.path.join().
Section 25.4Command-line arguments are stored in sys.argv, a list of strings where index 0 is the script name and arguments start at index 1. Always check len(sys.argv) before accessing an index, and print a usage message if required arguments are missing.
Section 25.5Safe batch file processing checks whether output files already exist, handles errors per file rather than crashing the whole script, writes to separate output folders, and uses dry-run mode to preview actions before committing.
Section 25.6Reports turn raw data into formatted, readable summaries written to text files. Combining data processing, formatted strings, and file output produces reports that would take much longer to create by hand.
Section 25.7Configuration values separate a script's adjustable settings from its logic. Simple settings belong in named constants at the top of the script. Frequently-changing values belong in command-line arguments or configuration files.
Section 25.8Logging records what a script did so you can review it later. Python's built-in logging module provides log levels (DEBUG, INFO, WARNING, ERROR) that control which messages are recorded. Open log files in append mode ("a") so each run adds to the history.
Section 25.9Test automation scripts on a single file first, then a test folder with known cases. Write check functions that validate output. Hand-trace one loop iteration before running at scale.
Section 25.10Safe automation is reversible automation. Write output files rather than modifying originals. Back up data before scripts that delete or rename. Ask for confirmation before irreversible actions. Document every script at the top.

Key Terms

automation

Using a program to perform repetitive, rule-following work without manual intervention for each item processed.

script

A program that runs from start to finish, accepting input at the start and producing output at the end, without interactive prompting during execution.

batch processing

Applying the same operation to a collection of inputs — typically using a loop — rather than handling each item manually.

command-line argument

A value passed to a script on the same terminal line as the run command, available inside the script through sys.argv.

sys.argv

A list in the sys module containing the script name at index 0 and any command-line arguments at index 1 onward. All values are strings.

configuration value

An adjustable setting that controls a script's behavior without changing its core logic — stored as a constant, argument, or config file entry.

logging

Writing a timestamped record of events to a file during program execution so the record can be reviewed afterward.

log level

A category indicating a log message's severity. Python's levels in order: DEBUG, INFO, WARNING, ERROR, CRITICAL.

dry-run mode

A script execution mode that prints what would happen without making any real changes — used as a safety check before committing to a run.

os.listdir(folder)

Returns a list of filenames (not full paths) for all items in the given folder.

os.path.join(folder, filename)

Builds a file path correctly for the current operating system, handling slash differences between Windows and Unix systems.

os.path.exists(path)

Returns True if a file or folder exists at the given path; False otherwise.

os.makedirs(path, exist_ok=True)

Creates a folder — and any intermediate folders — at the given path. Does not raise an error if the folder already exists when exist_ok=True.

reversible automation

Automation designed so its effects can be undone — typically by writing to output files rather than modifying originals, and backing up data before destructive operations.

PracticeCode Reading

Read each snippet carefully and answer the questions. Do not run the code — reason through it.

25-R1

Python

import sys
import os

def count_lines(filepath):
    with open(filepath, "r") as f:
        return len(f.readlines())

if len(sys.argv) < 2:
    print("Usage: python script.py <folder>")
else:
    folder = sys.argv[1]
    for name in sorted(os.listdir(folder)):
        if name.endswith(".txt"):
            path = os.path.join(folder, name)
            lines = count_lines(path)
            print(f"{name}: {lines} lines")

What does this script expect to receive as a command-line argument?
What does sorted() do to the output of os.listdir(folder)?
What does os.path.join(folder, name) do that folder + name would not do correctly?
What happens if the folder contains a subfolder named notes/? Does count_lines work on folders?

25-R2

Python

def write_log(log_path, message):
    from datetime import datetime
    timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    with open(log_path, "a") as f:
        f.write(f"[{timestamp}] {message}\n")

What file mode is used? What does "a" do differently than "w"?
What does strftime("%Y-%m-%d %H:%M:%S") produce? Give a concrete example.
What would happen to the log file across multiple runs if "w" were used instead of "a"?

25-R3

Python

DRY_RUN = True

for filename in ["a.txt", "b.txt", "c.txt"]:
    if DRY_RUN:
        print(f"[DRY RUN] Would delete: {filename}")
    else:
        os.remove(filename)
        print(f"Deleted: {filename}")

What output does this produce when DRY_RUN = True?
What would happen if you changed DRY_RUN to False?
Why is dry-run mode a useful safety feature specifically for loops?

PracticeCode Tracing

25-T1

Trace this program. What does it print?

Python

import sys

sys.argv = ["report.py", "data", "output.txt"]

if len(sys.argv) < 3:
    print("Not enough arguments.")
else:
    folder = sys.argv[1]
    output = sys.argv[2]
    print(f"Folder: {folder}")
    print(f"Output: {output}")
    print(f"Argument count: {len(sys.argv)}")

25-T2

Trace this batch loop. Assume the data folder contains: report_a.txt, notes.docx, report_b.txt, image.png, report_c.txt. List the output lines in order.

Python

import os

folder = "data"
count = 0

for filename in sorted(os.listdir(folder)):
    if filename.endswith(".txt"):
        count += 1
        print(f"Processing file {count}: {filename}")

print(f"Total .txt files: {count}")

25-T3

Trace this function. What does result contain after the call?

Python

def load_config(lines):
    config = {}
    for line in lines:
        line = line.strip()
        if line and not line.startswith("#"):
            key, value = line.split("=", 1)
            config[key.strip()] = value.strip()
    return config

sample = [
    "# Settings\n",
    "folder = reports\n",
    "  max_files = 50  \n",
    "\n",
    "# End of config\n",
    "output = results.txt\n",
]

result = load_config(sample)
print(result)

PracticePredict the Output

25-P1

Python

import sys
sys.argv = ["check.py", "data.txt", "summary.txt", "--dry-run"]

print(len(sys.argv))
print(sys.argv[1])
print("--dry-run" in sys.argv)

25-P2

Python

files = ["a.txt", "b.csv", "c.txt", "d.py", "e.txt"]
txt_files = [f for f in files if f.endswith(".txt")]
print(len(txt_files))
print(txt_files[0])
print(txt_files[-1])

25-P3

Python

log = []

def add_log(messages, text):
    messages.append(f"LOG: {text}")

add_log(log, "Started.")
add_log(log, "Processed file 1.")
add_log(log, "Processed file 2.")
add_log(log, "Finished.")

for entry in log:
    print(entry)
print(f"Total log entries: {len(log)}")

PracticeDebugging and Error Analysis

Each script contains one or more errors. Identify each error and write the corrected version.

25-D1

Python — buggy version

import sys

if len(sys.argv) < 2:
    print("Usage: python script.py <filename>")

filename = sys.argv[1]
print(f"Processing: {filename}")

What error occurs when the user runs python script.py with no arguments? Why does the check not prevent it?
Fix the script so it exits safely when the argument is missing.

25-D2

Python — buggy version

import os

folder = "reports"
for filename in os.listdir(folder):
    if filename.endswith(".txt"):
        with open(filename, "r") as f:
            print(f.read())

This script fails to open most files even though they exist. What is the problem?
Fix the script.

25-D3

Python — works but has a reliability problem

def write_report(data, output_path):
    f = open(output_path, "w")
    for item in data:
        f.write(item + "\n")

write_report(["line one", "line two", "line three"], "report.txt")

What reliability problem does this code have even though it runs?
Rewrite the function using a with statement.

25-D4

Python — buggy version

import sys

if len(sys.argv) == 3:
    folder = sys.argv[1]
    count = sys.argv[2]
    for i in range(count):
        print(f"Processing item {i + 1} in {folder}")

There are two distinct problems. Identify both.
Fix the script.

PracticeShort Programming Tasks

25-S1. Write a script called line_counter.py that accepts a filename as a command-line argument and prints the number of lines in that file. If no argument is given, print a usage message.
25-S2. Write a function list_txt_files(folder) that returns a sorted list of all .txt file paths in the given folder. Use os.listdir(), os.path.join(), and endswith().
25-S3. Write a function write_summary_report(stats, output_path) where stats is a dictionary with keys "count", "total", and "average". Write a formatted report to output_path using a with statement.
25-S4. Write a function load_config(filepath) that reads a plain-text file where each non-blank, non-comment line has the format key = value. Return a dictionary of the key-value pairs. Lines starting with # should be ignored.
25-S5. Write a function process_with_log(file_list, dry_run=False) that loops through file_list. For each filename, print either [DRY RUN] Would process: <name> or Processing: <name> depending on the flag. Return the count of files actually processed.
25-S6. Write a script that accepts a folder path as a command-line argument and prints a count of .txt, .csv, and .py files. Output format: .txt files: 4, .csv files: 2, .py files: 7.

Automation Safety Checklist

Complete before running any script that modifies, moves, renames, deletes, or overwrites files

Before Running

I have read the entire script and understand what it does.
I have run the script in dry-run mode and reviewed the output.
I have tested the script on a small folder of test data.
I have verified the output is correct for the test data.

Data Safety

I have backed up any files that will be modified or deleted.
The script writes to a separate output folder, not the input folder.
I can undo the changes if something goes wrong.

Script Design

The script handles errors per file, not per entire run.
The script logs what it does.
The script is documented at the top with purpose, inputs, and outputs.

Mini-ProjectReport Generator or File Organizer

Choose one of the two options below. Both require applying the major techniques of this chapter.

Option A

Batch Grade Report Generator

OverviewWrite a script called grade_report.py that reads a CSV file of student scores and writes a formatted grade report.
InputCSV file path as a command-line argument. Columns: name, score1, score2, score3.
ProcessingCompute each student's average. Assign grades: A (90+), B (80–89), C (70–79), D (60–69), F (below 60).
OutputA formatted report to grade_report.txt listing name, average, and grade for each student. Class average and grade distribution at the end. Log events and any rows that cannot be processed.
RequirementSupport --dry-run which prints the report to the terminal instead of writing a file.

Option B

File Organizer by Extension

OverviewWrite a script called organize_files.py that organizes files in a folder into subfolders by extension.
InputInput folder path as a command-line argument. Optional --dry-run flag.
ProcessingScan all files. Organize into subfolders: text_files/ for .txt, data_files/ for .csv, scripts/ for .py, other/ for everything else.
OutputDry run: print what would happen for each file. Normal mode: copy each file to the appropriate subfolder using shutil.copy2(src, dst). Print a summary count per folder. Log every operation.
RequirementsMust not crash if a file cannot be copied. Must not overwrite existing files — log a warning and skip. Must create output folders only when needed.

ReviewCumulative Review

These questions draw on skills from earlier chapters.

(Chapter 23) Write a function safe_read_csv(filepath) that tries to open and read a CSV file using csv.DictReader. If the file is not found, return an empty list. If any other exception occurs, print an error message and return an empty list. If successful, return the list of row dictionaries.
(Chapter 21) A log file contains lines in the format [2024-10-14 09:15:03] Processed: report_a.txt. Write a function count_errors(log_path) that reads the log file and returns the count of lines containing the word ERROR.
(Chapter 18) A script records results as a list of dictionaries, each with keys "filename" and "word_count". Write a function most_words(results) that returns the filename of the entry with the highest word count.
(Chapter 14) A script has three functions: load_data(), process_data(), and write_report(). Explain in your own words why this decomposition is good design. What would the disadvantages be if all three functions' code were written directly in the main loop?
(Chapter 9) Write a for loop that iterates over a list of filenames and prints only the filenames longer than 15 characters (including the extension).

ReviewReview Questions

Recall and Vocabulary

What is automation, and what kinds of tasks are good candidates for it?
What is the difference between a script and an interactive program?
Where are command-line arguments stored in Python? What is always in position 0?
Why must you check len(sys.argv) before accessing sys.argv[1]?
What does os.path.join(folder, filename) do? Why is it safer than string concatenation?
What does os.path.exists(path) return?
What does os.makedirs(path, exist_ok=True) do?
What is dry-run mode? Why is it a useful practice for automation scripts?

Understanding and Explanation

What is logging? Why is logging more useful than print statements alone when a script runs automatically?
What is the difference between opening a log file in "w" mode and "a" mode? Which is correct for a running log, and why?
Name three things a well-designed automation script should do to handle file errors safely.
Why is it safer for an automation script to write results to a new output folder rather than modifying the input files directly?

Application and Analysis

A script processes student score files in a folder and has a bug: it divides by zero when a student has no scores. In what way is automation's relationship with bugs different from a bug in a single interactive program? What would happen here?
You have a script that renames files in a folder. Before running it on 500 real files, describe the full testing strategy you would use, drawing on Section 25.9.
Compare these two designs for passing the output path to a script: (a) hardcode output_path = "report.txt" at the top; (b) use sys.argv[1]. When is each design appropriate?
A classmate says "Logging is just debugging. I'll delete the logging code once the script works." Write a response explaining why logging in automation scripts serves a different purpose from debugging print statements.

Preview — Chapter 26

Classes and Objects

Chapter 25 brings Part V — Files, Data, and Practical Automation to a close. You have learned to read and write files, process CSV data, handle exceptions, use randomness and dates, and build scripts that automate real work.

Part VI turns to a different kind of program organization: object-oriented programming. Chapter 26 introduces classes and objects — Python's way of combining data and the functions that work with it into a single, named unit. You have already used objects throughout this book. Strings, lists, and dictionaries are all objects. Every time you call my_list.append(item) or my_string.lower(), you are calling a method on an object. Chapter 26 shows you how to design your own types: how to define a class, create instances, give them attributes, and write methods that operate on their data. It starts gently, with small concrete examples, and builds your understanding step by step.

MGU Library Connection

Programming with Python — Appendix B: Command Line and File Paths Reference
Programming with Python — Appendix I: File and CSV Reference
Programming with Python — Appendix J: Standard Library Beginner Map (os, sys, logging)
Companion resource: Debugging Checklist
Companion resource: Automation Safety Checklist
Companion resource: Python Project Checklist
Next chapter: Chapter 26 — Classes and Objects (Part VI)

Chapter 25:Automating EverydayTasks

Chapter Introduction and Learning Objectives

Learning Objectives

Sections at a Glance

Why Automation Matters

Scripts as Tools

A Minimal Script

Structure of a Well-Designed Script

Repetitive Tasks and Batch Processing

Processing a List of Items

Processing Many Files

Command-Line Arguments at an Introductory Level

Using a Command-Line Argument for a Filename

Multiple Arguments

Processing Many Files Safely

Check Before Writing

Handle Errors Per File, Not Per Script

Build Output Paths Carefully

Dry-Run Mode

Generating Reports

A Simple Text Report

Generating a Report from CSV Data

Configuration Values

Option 1: Constants at the Top of the Script

Option 2: Command-Line Arguments

Option 3: A Plain-Text Configuration File

Logging at an Introductory Level

Simple Manual Logging

Python's Built-In logging Module

Testing Automation Before Running It Widely

Safe and Responsible Automation

A Batch Word-Count Report Script

Chapter Summary

Key Terms

25-R1

25-R2

25-R3

25-T1

25-T2

25-T3

25-P1

25-P2

25-P3

25-D1

25-D2

25-D3

25-D4

Recall and Vocabulary

Understanding and Explanation

Application and Analysis

Chapter 25:
Automating Everyday
Tasks

Python's Built-In `logging` Module