Chapter 14 — Front Matter
Purpose and Overview
Statistics is the grammar of science.
— Karl Pearson
Every chapter of this book has built one part of a larger structure. In Chapter 1, probability was introduced as the mathematics of uncertainty. Chapters 2 through 4 developed probability rules, random variables, and the distributions that describe how uncertainty behaves. Chapters 5 and 6 extended those ideas to multiple variables and transformations. Chapter 7 established the limit theorems — the law of large numbers and the central limit theorem — that make statistical inference mathematically justified rather than merely plausible.
With that foundation in place, Chapters 8 through 10 introduced the core procedures of statistical inference: confidence intervals, hypothesis testing, likelihood estimation, and Bayesian reasoning. Chapters 11 and 12 developed regression and ANOVA as tools for modeling relationships and comparing groups. Chapter 13 showed how uncertainty unfolds over time through stochastic processes.
Chapter 14 does not introduce a new technique and then apply it. Instead, it brings everything together. It shows how to move from a real question to a probability model, from a model to an inference procedure, from a procedure to a numerical result, and from a numerical result to a responsible, clearly communicated conclusion. It addresses the decisions a statistician or data scientist must make at every step: which model fits? which method is appropriate? which assumptions must be checked? how should uncertainty be reported? how should results be communicated honestly?
By the end of this chapter, you should be able to look at an unfamiliar data problem, identify what kind of model and method it requires, execute the analysis carefully, check the assumptions behind it, quantify the uncertainty in your conclusions, and communicate your findings in a way that is accurate, honest, and useful.
15Sections
9Objectives
30+Key Terms
14Worked Examples
Learning Objectives
- 1Describe and apply the six-stage statistical modeling cycle from question formulation through communication.
- 2Choose an appropriate probability model for a given dataset using subject-matter reasoning and graphical checks.
- 3Select the correct inference procedure for a given question, data type, and sample size.
- 4Check the assumptions of statistical procedures using graphical and formal methods.
- 5Compare competing models using AIC, BIC, and likelihood ratio tests.
- 6Quantify and communicate uncertainty through standard errors, confidence intervals, and prediction intervals.
- 7Apply basic statistical decision theory and expected utility to decisions under uncertainty.
- 8Identify and avoid common statistical errors including p-hacking, multiple comparisons, and conflation of association with causation.
- 9Conduct and communicate a complete capstone data analysis integrating all stages of the modeling cycle.
Sections 14.1 – 14.5
The Full Modeling Cycle: From Question to Model Comparison
Process · Model Choice · Inference Selection · Assumption Checking · Model Comparison
Section 14.1
The Probability and Statistics Modeling Process
Statistical analysis is not a collection of separate procedures. It is a process — a disciplined cycle of question, model, data, inference, checking, and communication. Every procedure in this book is one step inside that cycle.
Before running any calculation, a statistician must ask: What question am I trying to answer? What data are available, and how were they collected? What model describes the process that generated the data? What method answers the question under that model? What assumptions does the method require, and are those assumptions plausible here? How should I report what I find?
The Six-Stage Modeling Cycle
Stage 1 — Formulate the question. Name the population, the variable(s), and the type of claim.
Stage 2 — Collect and describe the data. Summarize centers, spreads, shapes, outliers, and missing values.
Stage 3 — Choose a probability model. Match the data-generating process to an appropriate distribution or regression.
Stage 4 — Estimate, test, or predict. Extract information from data using point estimates, intervals, or tests.
Stage 5 — Check assumptions. Use residual plots, Q-Q plots, and formal tests to verify model conditions.
Stage 6 — Communicate conclusions. Report findings honestly with uncertainty, context, and appropriate hedging.
The Cycle Is Not Linear
Checking assumptions in Stage 5 sometimes reveals that the model chosen in Stage 3 must be revised. Examining data in Stage 2 often reshapes the question from Stage 1. Real statistical work moves back and forth among stages. Recognizing this is part of statistical maturity.
Example 14.1.1
The modeling cycle in practice — disease spread analysis
- Stage 1A public health analyst asks: what is the rate at which new infections arrive per day, and does it differ across neighborhoods?
- Stage 2She collects daily case counts from hospital records, discovers a two-day reporting lag in some neighborhoods, and adjusts accordingly.
- Stage 3She considers Poisson models for count data but notes that variance exceeds the mean in several neighborhoods, suggesting a negative binomial model may fit better.
- Stage 4She fits both models, estimates rate parameters, and constructs confidence intervals for each neighborhood.
- Stage 5She checks residuals, compares models using AIC, and discovers that one neighborhood's data was miscoded — returning to Stage 2.
- Stage 6She reports estimated rates with confidence intervals, notes the assumptions and their limitations, and avoids claiming which neighborhood is riskier.
This is real statistical practice: iterative, careful, honest.
Section 14.2
Choosing a Probability Model
A probability model is a mathematical description of the process that generated the data. Models are not found in nature — they are chosen by the analyst. The goal is not to find the true model (which is usually unknowable) but to find a model that is useful: one that captures the essential features of the data-generating process, is simple enough to work with, and is accurate enough that conclusions based on it are trustworthy.
George Box's observation that "all models are wrong, but some are useful" applies throughout statistics. The question is never whether a model is perfectly correct. The question is whether the model's approximations are close enough to matter.
| Data Type | Structure | Suggested Model | Key Parameters |
|---|
| Discrete | Successes in fixed trials, constant probability | Binomial | n, p |
| Discrete | Trials until first success | Geometric | p |
| Discrete | Arrivals at constant rate, independent intervals | Poisson | λ |
| Discrete | Successes drawn without replacement | Hypergeometric | N, K, n |
| Continuous | Bell-shaped, symmetric | Normal | μ, σ |
| Continuous | Non-negative, waiting time or lifetime | Exponential / Gamma | λ or α, β |
| Continuous | Constrained to [0, 1], proportion | Beta | α, β |
| Continuous | Uniform over an interval | Uniform | a, b |
Model Fit Must Be Checked
Choosing a candidate model based on subject-matter reasoning is the start, not the end. Use a histogram or density estimate compared against the proposed distribution, and a Q-Q plot comparing empirical to theoretical quantiles. A straight-line Q-Q plot supports the model; systematic curvature signals a mismatch.
Example 14.2.1
Checking a Poisson model for call center data
- SetupA data scientist models calls per hour at a call center over 200 hours. Sample mean = 4.7; sample variance = 4.9.
- CheckFor a Poisson distribution, mean and variance are equal. The near-equality of 4.7 and 4.9 supports a Poisson model with λ̂ ≈ 4.7.
- VerifyA histogram of counts is plotted alongside the Poisson(4.7) PMF — close agreement is observed. A chi-squared goodness-of-fit test returns p = 0.38.
No evidence against the Poisson model at any standard significance level. Proceed with Poisson(λ̂ = 4.7).
Example 14.2.2
Comparing Exponential and Gamma fits for component lifetimes
- SetupAn engineer records lifetimes of 150 electronic components. The histogram is right-skewed.
- CheckA Q-Q plot against the Exponential distribution shows systematic deviation in the right tail — the data are heavier-tailed than Exponential.
- ActionShe fits both Gamma and Weibull models, then uses AIC to compare (see Section 14.5).
One graphical check can send the analysis in a meaningfully different direction. Always look before you compute.
Section 14.3
Choosing an Inference Method
Estimation is appropriate when the question asks: what is the value of an unknown parameter? Use estimation when the goal is to measure — a rate, a proportion, a mean, a regression coefficient — and report a point estimate with a confidence interval.
Hypothesis testing is appropriate when the question asks: is a specific claim consistent with the data? Testing produces a test statistic and a p-value. In most serious analyses, both estimation and testing are reported together: the interval tells you where the parameter plausibly lies; the test tells you whether it excludes a specific null value.
| Setting | Procedure | Key Condition |
|---|
| One mean, σ known | z-procedure | Normal or n ≥ 30 |
| One mean, σ unknown | t-procedure (df = n − 1) | Normal or n large |
| One proportion | Large-sample z-procedure | np₀ ≥ 10 and n(1−p₀) ≥ 10 |
| Two independent means | Two-sample t-test | Independence, approx. Normality |
| Paired data | Paired t-test | Differences approx. Normal |
| Two proportions | Two-proportion z-test | Large samples in each group |
| More than two means | One-way ANOVA + F-test | Independence, equal variance, Normality |
| One continuous predictor | Simple linear regression | Linearity, independence, constant variance, Normality of errors |
| Multiple predictors | Multiple linear regression | Same as simple, plus no multicollinearity |
| Any parametric model | Maximum likelihood estimation | Model correctly specified |
| Prior information available | Bayesian inference | Prior + likelihood → posterior |
Example 14.3.1
Selecting the correct procedure for a clinical comparison
- QuestionDoes a new treatment reduce mean recovery time compared to standard care?
- DesignTwo independent groups, continuous outcome, approximate Normality confirmed by Q-Q plots.
- MethodTwo-sample t-test for the hypothesis test; 95% confidence interval for the difference in means to report effect size in clinically interpretable units.
Report both the test result and the confidence interval. A p-value alone does not communicate the size of the effect.
Section 14.4
Checking Assumptions
Every inferential procedure rests on assumptions. Confidence intervals for a mean assume either Normality or a large sample. Two-sample t-tests assume independence and approximate Normality. Linear regression assumes linearity, independence of errors, constant error variance, and approximate Normality of errors. When assumptions hold, inference is valid. When assumptions fail badly, nominal confidence levels and p-values may be wrong — sometimes dramatically so.
Graphical Methods
Normal Q-Q plots. Plot sample quantiles against theoretical Normal quantiles. A straight-line pattern supports Normality. Curvature suggests skewness; S-shapes suggest heavy or light tails. Residual plots. In regression, plot residuals against fitted values. A fan shape signals heteroscedasticity; a curved pattern signals non-linearity. Time-order plots. Residuals plotted in time order check independence — trends or oscillations suggest correlated observations.
Formal Tests
| Test | What It Checks | Caution |
|---|
| Shapiro-Wilk | Consistency with Normality | Non-significant ≠ proves Normal; just fails to reject |
| Levene's test | Equal variances across groups | More robust to non-Normality than Bartlett's |
| Bartlett's test | Equal variances across groups | Sensitive to non-Normality; use Levene's if data skewed |
| Chi-squared goodness-of-fit | Fit of a discrete model to observed frequencies | Requires adequate expected counts (≥ 5) per cell |
When Assumptions Fail
Possible responses include: (1) Transformations — the log, square root, or reciprocal of a skewed response can restore approximate Normality; (2) Nonparametric methods — rank-based procedures (Wilcoxon, Kruskal-Wallis) avoid distributional assumptions; (3) Robust methods — designed to maintain validity under departures from assumptions; (4) Model revision — a curved residual plot signals the wrong functional form, not just a variance problem.
Example 14.4.1
Addressing heteroscedasticity with a log transformation
- ProblemRegression of systolic blood pressure on body weight. Residual plot shows a fan shape — variance increases with fitted values.
- ActionApply log transformation to the response variable. Refit the regression.
- CheckResidual plot for the transformed model shows an approximately random scatter — no fan pattern remains.
Proceed with the log-transformed model and interpret all coefficients accordingly.
Section 14.5
Comparing Models
When more than one model is plausible, a principled criterion is needed to choose among them. Model comparison balances two competing demands: fit (how well the model describes the data) and parsimony (how simple the model is). Adding parameters always improves fit, but adding irrelevant parameters introduces noise and reduces generalizability.
Information Criteria
AIC = −2 ln L(θ̂) + 2k
where k is the number of parameters. Lower AIC is better. AIC is preferred when the goal is prediction or approximation.
BIC = −2 ln L(θ̂) + k ln n
BIC penalizes additional parameters more heavily for larger n. BIC is preferred when the goal is selecting the true model from a candidate set, favoring simpler models.
Likelihood Ratio Test — For Nested Models
When one model is a special case of another (nested), the likelihood ratio statistic Λ = −2[ln L(θ̂₀) − ln L(θ̂₁)] has an approximate chi-squared distribution with r degrees of freedom under H₀, where r is the number of parameters fixed in the simpler model. A large Λ favors the more complex model.
Example 14.5.1
AIC and likelihood ratio test for regression model selection
- SetupTwo regression models (n = 100): Model A has 2 parameters, log-likelihood = −152.3. Model B has 3 parameters, log-likelihood = −150.1.
- AIC
AIC_A = −2(−152.3) + 2(2) = 308.6
AIC_B = −2(−150.1) + 2(3) = 306.2Model B has the lower AIC.
- LR TestΛ = 2(150.1 − 152.3) = 4.4, df = 1. The p-value ≈ 0.036. Both criteria support adding the second predictor.
Model B is preferred by both AIC and the likelihood ratio test at α = 0.05.
Sections 14.6 – 14.8
Uncertainty, Decisions, and Expected Utility
Standard Errors · Confidence Intervals · Decision Theory · Utility Functions
Section 14.6
Quantifying Uncertainty
A statistical result without a measure of uncertainty is incomplete. Every quantity estimated from data is subject to sampling variability, and that variability must be reported. The standard error of an estimator is the standard deviation of its sampling distribution — it measures how much the estimator would vary across repeated samples of the same size from the same population.
| Quantity | Standard Error Formula | Notes |
|---|
| Sample mean X̄ | SE = σ / √n | Use s when σ unknown |
| Sample proportion p̂ | SE = √(p(1−p)/n) | Estimate with p̂ in practice |
| Regression slope β̂₁ | From regression output | Depends on residual variance and variability of X |
| Difference of means | √(s₁²/n₁ + s₂²/n₂) | Assumes independence between groups |
Confidence Interval Interpretation
A 95% confidence interval does not mean there is a 95% probability that the true parameter lies in the computed interval. The parameter is fixed — not random in the frequentist framework. The correct interpretation: if this procedure were applied to many independent samples, 95% of the resulting intervals would contain the true value. The Bayesian credible interval provides the direct probability statement — P(a ≤ θ ≤ b | x) = 0.95.
Example 14.6.1
Confidence vs. prediction intervals in regression
- SetupRegression: Ŷ = 45 + 8x (exam score on study hours). SE(β̂₁) = 1.4. For a student studying 6 hours, predicted score = 93.
- CI for mean95% confidence interval for the mean score of all students who study 6 hours: (90.1, 95.9). Narrower — only captures uncertainty about the mean.
- PI for individual95% prediction interval for a specific student studying 6 hours: (82.3, 103.7). Wider — also captures natural variability among individual students.
Prediction intervals are always wider than confidence intervals for the mean. Both must be reported in applications where individual predictions matter.
Section 14.7
Decision-Making Under Uncertainty
Statistical inference answers questions about the world. But many real situations require not just answers but decisions — actions that must be chosen under uncertainty, with consequences that depend on which possible state of the world is true. Statistical decision theory extends inference into a formal framework that accounts for these consequences.
The Three Components of a Decision Problem
States of nature θ — the unknown true state. In statistics, usually a parameter.
Actions a — the possible decisions the analyst can make.
Loss function L(θ, a) — the cost of taking action a when the true state is θ.
The goal is to choose the action that minimizes expected loss. The Bayes risk of an action is the expected loss under the posterior: r(a | x) = E[L(θ, a) | x].
| Loss Function | Optimal Bayesian Action |
|---|
| Squared error: L(θ, a) = (θ − a)² | Posterior mean E[θ | x] |
| Absolute error: L(θ, a) = |θ − a| | Posterior median |
| 0-1 loss (discrete θ) | Posterior mode |
Example 14.7.1
Drug approval as a decision problem
- SetupA company must decide: seek approval for a new drug (a₁) or abandon it (a₂). The unknown state is the true effect size δ.
- Loss structureDrug works, approval sought: low loss. Drug works, abandoned: high loss (missed opportunity, patient harm). Drug fails, approval sought: high loss (cost, patient harm from ineffective treatment). Drug fails, abandoned: low loss.
- DecisionCompute the Bayes risk of each action using the posterior distribution of δ from the clinical trial. Choose the action with lower expected loss.
This is more complete than simply reporting a p-value and applying a significance threshold. The loss function explicitly accounts for asymmetric consequences.
Section 14.8
Risk, Loss, and Expected Utility as an Introduction
Loss functions measure costs. But real decisions involve both costs and benefits, and a decision-maker's preferences over outcomes may be non-linear. Expected utility theory provides a framework for rational decision-making under uncertainty that accounts for these complexities.
A utility function U(c) assigns a numerical value to each consequence c. Expected utility is EU(a) = E[U(c(a, θ))]. The optimal action maximizes expected utility. The shape of the utility function determines the decision-maker's attitude toward risk.
| Risk Attitude | Utility Shape | Condition | Preference |
|---|
| Risk-averse | Concave | U″(c) < 0 | Certain outcome over equally valuable gamble |
| Risk-neutral | Linear | U″(c) = 0 | Indifferent between certain outcome and gamble |
| Risk-seeking | Convex | U″(c) > 0 | Gamble over equally valuable certain outcome |
Jensen's Inequality and Risk Aversion
For a concave utility function U: U(E[c]) ≥ E[U(c)]. The utility of the expected outcome exceeds the expected utility of the random outcome. A risk-averse decision-maker prefers a certain outcome over a gamble with the same expected value — a fact that underlies insurance markets, option pricing, and much of finance.
Sections 14.9 – 14.11
Reproducibility, Ethics, and Communication
Statistical Practice · Ethics · Tables and Graphs · Honest Reporting
Section 14.9
Reproducibility and Statistical Practice
A result is reproducible if independent researchers following the same procedure on the same data reach the same conclusion. A result is replicable if independent researchers applying the same procedure to a new independent sample reach consistent conclusions. Across fields including psychology, medicine, economics, and biology, many published findings have failed to replicate.
This reproducibility crisis has multiple statistical causes: small sample sizes producing low statistical power; selective reporting of results that survive a significance threshold (publication bias); multiple comparisons made without correction; flexible data analysis decisions that inflate the false positive rate; and the conflation of statistical significance with practical importance.
p-Hacking — Definition
Running 20 analyses until one yields p < 0.05, then reporting only that result
Because 5% of true null hypothesis tests will yield p < 0.05 by chance, this practice reliably produces false positives. The Bonferroni correction (use α/m as the threshold when testing m hypotheses) is one remedy. The Benjamini-Hochberg procedure controls the false discovery rate more efficiently.
Practices That Support Reproducibility
Pre-registration — specify hypotheses, data collection, and analysis plan before the data are collected. Separates confirmatory from exploratory analysis.
Open data and code — make data and analysis scripts publicly available wherever privacy allows. Enables verification and error correction.
Report all tests — when multiple comparisons are made, say so and apply corrections. Label exploratory findings as hypothesis-generating, not confirmatory.
Meta-analysis — synthesize evidence across multiple independent studies. A replicated effect across studies provides much stronger evidence than any single analysis.
Section 14.10
Statistical Ethics
Statistics is not ethically neutral. Statistical analyses influence decisions that affect people's lives: medical treatments are approved or rejected, policies are adopted or abandoned, loans are granted or denied. The statistician who designs the analysis and reports the results bears responsibility for the quality and honesty of that work.
Selective Reporting and Cherry-Picking
Reporting only the significant results, or only results that support a preferred hypothesis
A result's significance depends on the analysis chosen, and the analysis chosen should be determined by the question — not by the desired answer. This applies equally to presenting a chosen statistic without disclosing other relevant findings.
Confusing Association and Causation
"X is correlated with Y, therefore X causes Y"
Regression and correlation measure association. Causal claims require randomized experimentation or explicit causal modeling with identified assumptions. The language used matters: "X is associated with higher Y" is a statistical claim. "X causes Y" is a causal claim requiring additional justification.
Algorithmic Bias
Using a model trained on historically biased data without auditing for differential error rates
Models used in consequential decisions — credit scoring, hiring, medical diagnosis — can encode and amplify biases present in training data. The ethical question is not only whether a model is accurate overall but whether its errors are distributed fairly across different groups.
Section 14.11
Communicating Results with Tables and Graphs
A technically excellent analysis that is communicated poorly is worth little. Statistical results must be translated into language and visuals that make the findings clear, honest, and actionable for the intended audience.
Principles of Effective Statistical Communication
Report estimates with uncertainty — every point estimate should be accompanied by a standard error or confidence interval.
Use appropriate precision — report enough decimal places to convey meaningful information, but not so many as to suggest false precision.
Distinguish statistical and practical significance — a p-value says whether the effect is detectable, not whether it matters. Always report effect sizes.
Describe the data — sample size, collection method, key characteristics, exclusions. A reader who doesn't know how the data were obtained cannot assess the conclusions.
Be clear about what was tested — when multiple tests were performed, say so, and report any multiplicity correction applied.
Model Table: Comparing Two Treatments
| Group | n | Mean SBP (mmHg) | SD | 95% CI for Mean |
|---|
| Treatment A | 45 | 128.4 | 11.2 | (124.1, 132.7) |
| Treatment B | 48 | 134.7 | 12.6 | (131.0, 138.4) |
| Difference (A − B) | — | −6.3 | — | (−10.9, −1.7) |
SBP = systolic blood pressure. Two-sample t-test: t(91) = −2.77, p = 0.007. Equal variances not assumed.
Sections 14.14 – 14.15
Common Mistakes and the Road Ahead
Ten Critical Errors · Advanced Statistics · Machine Learning · Research Methods
Section 14.14
Common Modeling and Interpretation Mistakes
Mistake 1 — Treating the Sample as the Population
Sample mean = $54,200, therefore population mean = $54,200
Correct: The sample mean estimates the population mean with uncertainty. Always report a confidence interval alongside the point estimate.
Mistake 2 — Misinterpreting the Confidence Level
"There is a 95% probability that the true mean is in (51.8, 56.6)"
Correct: The confidence level refers to the procedure, not the specific interval. The population mean is fixed. 95% of intervals produced by this procedure contain the true mean — we do not know whether this one does.
Mistake 3 — Misinterpreting the p-Value
"p = 0.03 means there is a 3% probability the null hypothesis is true"
Correct: The p-value is the probability of observing a test statistic at least as extreme as the one computed, assuming H₀ is true. It is not the probability that H₀ is true — that belongs to Bayesian inference.
Mistake 4 — Confusing Statistical and Practical Significance
With n = 10,000, a 0.2-point wellness score difference (p = 0.001) is "highly effective"
Correct: Large samples detect small differences with high statistical significance. A 0.2-point difference on a 100-point scale may be statistically detectable but practically meaningless. Always report and discuss effect sizes.
Mistake 5 — Ignoring Assumptions
Applying a two-sample t-test with n = 8 per group and strongly skewed data without checking
Correct: With small samples and non-Normal data, the t-test may not be reliable. Check assumptions with Q-Q plots and consider the Wilcoxon rank-sum test as a nonparametric alternative.
Mistake 6 — Confusing Correlation and Causation
Cities with more libraries have higher median incomes → build libraries to increase income
Correct: Regression measures association, not causation. The library-income association could reflect a common cause (wealthier cities fund both) or reverse causation. Causal claims require randomization or explicit causal modeling.
Mistake 7 — Multiple Comparisons Without Correction
Testing 30 hypotheses at α = 0.05 and reporting the 3 significant results as confirmed findings
Correct: With 30 tests and no true effects, P(at least one false positive) ≈ 1 − (0.95)³⁰ ≈ 0.79. Apply Bonferroni correction (α/m) or Benjamini-Hochberg, and label exploratory findings as hypothesis-generating.
Mistake 8 — Data Dredging
Running many models on the same data until finding a significant result, then reporting only that result
Correct: Pre-register the analysis plan before examining the data, or report all models considered alongside the final result. Exploratory flexibility inflates the Type I error rate.
Mistake 9 — Overgeneralizing from the Sample
Undergraduate study finds strategy X improves decisions → strategy X works for everyone
Correct: Results apply directly only to the population from which the sample was drawn. Generalizing requires arguing representativeness or plausibility of similar effects in other populations.
Mistake 10 — Ignoring Missing Data
Excluding all observations with any missing value and proceeding as if the remaining data are a random sample
Correct: If data are missing not at random (e.g., sicker patients more likely to miss follow-up), the complete-case analysis is biased. Multiple imputation and explicit missing-data models should be considered.
Section 14.15
Readiness for Advanced Statistics, Machine Learning, and Research Methods
You have completed a calculus-based introduction to probability and statistics. You have built probability from axioms through distributions, limit theorems, inference, regression, and stochastic processes. You have seen how mathematical structure — integration, differentiation, linear algebra, and analysis — enters statistical reasoning. This is a substantial foundation. The following describes where the major threads of this book lead.
Pathways Forward
- Mathematical Statistics — Cramér-Rao bound, sufficient statistics, Neyman-Pearson lemma, optimal tests. Prerequisites: this book, advanced calculus.
- Bayesian Statistics and Computation — MCMC algorithms, probabilistic programming (Stan, PyMC), hierarchical models. Prerequisites: this book, programming experience.
- Machine Learning and Statistical Learning — Regularization (lasso, ridge), trees, neural networks, unsupervised methods. Prerequisites: this book, linear algebra, Python or R.
- Stochastic Processes and Time Series — Continuous Markov chains, Brownian motion, ARIMA models, spectral analysis. Prerequisites: this book, differential equations.
- Econometrics and Causal Inference — Instrumental variables, difference-in-differences, regression discontinuity, potential outcomes. Prerequisites: this book, multiple regression (Ch. 12).
- Design of Experiments — Randomized trials, factorial designs, power analysis. Prerequisites: this book.
A Final Note
The mathematician John Tukey wrote: "The best thing about being a statistician is that you get to play in everyone's backyard." Every field that collects data — and today that is every field — uses the ideas in this book. The modeling cycle is never finished. Every answer raises new questions. Every model reveals its own gaps. This is not a limitation of statistics — it is what makes it a living discipline.
Chapter 14 — Back Matter
Chapter Summary
- Sec 14.1Statistical analysis follows a six-stage cycle: question, data description, model choice, inference, assumption checking, and communication. The cycle is iterative — later stages may require returning to earlier ones.
- Sec 14.2Models are chosen, not discovered. Discrete data suggest Binomial, Poisson, Geometric, or Hypergeometric distributions; continuous data suggest Normal, Exponential, Gamma, Beta, or Uniform depending on shape and support. Fit is checked graphically (histograms, Q-Q plots) and formally (goodness-of-fit tests).
- Sec 14.3Estimation is appropriate when the question asks for a parameter's value; hypothesis testing when evaluating a specific claim. The choice among procedures (z, t, F, chi-squared) follows from data type, sample size, and assumptions. In most analyses, both estimation and testing are reported.
- Sec 14.4Every procedure rests on assumptions that must be checked. Graphical tools (Q-Q plots, residual plots, time-order plots) and formal tests (Shapiro-Wilk, Levene's, chi-squared goodness-of-fit) are the primary methods. When assumptions fail, transformations, nonparametric methods, or model revision may be needed.
- Sec 14.5AIC = −2 ln L(θ̂) + 2k penalizes log-likelihood by the number of parameters; lower is better. BIC uses k ln n and penalizes complexity more heavily for large n. Likelihood ratio tests compare nested models. Cross-validation directly estimates predictive accuracy.
- Sec 14.6Standard errors measure estimator precision. Confidence intervals quantify uncertainty about parameters. Prediction intervals quantify uncertainty about future observations and are always wider than confidence intervals for the mean. Bayesian credible intervals provide direct probability statements about parameters conditional on data and prior.
- Sec 14.7Statistical decision theory formalizes decisions as choices among actions under uncertainty, using a loss function to quantify consequences. The optimal Bayesian action minimizes expected posterior loss. Different loss functions (squared error, absolute error, 0-1) lead to different optimal actions.
- Sec 14.8Utility functions assign numerical values to consequences. Expected utility theory accounts for non-linear preferences. Risk-averse decision-makers (concave utility) prefer certain outcomes over equally valued gambles. Risk-seeking decision-makers prefer gambles. Jensen's inequality formalizes this for concave utilities.
- Sec 14.9P-hacking, publication bias, multiple comparisons without correction, and lack of pre-specification inflate false positive rates. Bonferroni correction and Benjamini-Hochberg control multiplicity. Pre-registration, open data, and open code support reproducibility. Meta-analysis synthesizes evidence across studies.
- Sec 14.10Statistical ethics requires: avoiding selective reporting, distinguishing association from causation, disclosing uncertainty, checking assumptions honestly, and attending to fairness in consequential algorithmic applications. The statistician's responsibility extends to honest communication of limitations.
- Sec 14.11Good communication pairs estimates with uncertainty, distinguishes statistical and practical significance, describes the data and its collection, and uses honest, well-labeled tables and graphs. Axis scales should reveal rather than obscure variation; error bars should always accompany bar charts.
- Sec 14.12The capstone project integrates all modeling cycle stages: question formulation, EDA, model selection, inference, assumption checking, uncertainty quantification, and written conclusions readable by a non-statistician.
- Sec 14.13Probability and statistics underlie data science (likelihood, regression, Bayesian methods), finance (stochastic processes, portfolio theory, risk quantiles), medicine (clinical trials, diagnostic testing, survival analysis), and engineering (reliability, quality control, signal detection).
- Sec 14.14The ten most important errors are conceptual, not arithmetic: misinterpreting p-values and confidence intervals, confusing significance with importance, ignoring assumptions, overgeneralizing, conflating association with causation, multiple comparisons without correction, data dredging, and ignoring missing data.
- Sec 14.15This course opens pathways to mathematical statistics, Bayesian computation, machine learning, stochastic processes, time series, econometrics, and experimental design. The probabilistic foundation developed here accompanies every subsequent course in quantitative work.
Key Terms Review
AIC
Akaike Information Criterion: −2 ln L(θ̂) + 2k. Balances fit and parsimony; lower is better; preferred for prediction.
BIC
Bayesian Information Criterion: −2 ln L(θ̂) + k ln n. Stronger parsimony penalty than AIC for n > 7; preferred for model identification.
Bayes risk
The expected loss of an action under the posterior distribution of the parameter. The optimal Bayesian action minimizes Bayes risk.
credible interval
A Bayesian interval containing the true parameter with specified posterior probability: P(a ≤ θ ≤ b | x) = 0.95 for a 95% credible interval.
cross-validation
Resampling method for estimating predictive accuracy by fitting on training data and evaluating on held-out test data. K-fold CV averages over K splits.
effect size
A measure of the magnitude of an effect, independent of sample size. Essential for distinguishing statistical from practical significance.
expected utility
The expected value of a utility function over uncertain consequences: E[U(c(a, θ))]. The optimal action maximizes expected utility.
false discovery rate
The expected proportion of rejected null hypotheses that are false rejections. Controlled by the Benjamini-Hochberg procedure in multiple testing.
goodness-of-fit
A hypothesis test for whether observed data are consistent with a proposed probability model. The chi-squared goodness-of-fit test is the most common version.
heteroscedasticity
Non-constant variance of errors in a regression model. Detected by a fan-shaped residual plot; often addressed by a log transformation of the response.
likelihood ratio test
A hypothesis test comparing nested models using Λ = −2[ln L(θ̂₀) − ln L(θ̂₁)], which has an approximate chi-squared distribution under H₀.
loss function
A function L(θ, a) measuring the cost of taking action a when the true parameter is θ. Different loss functions yield different optimal Bayesian actions.
p-hacking
Running many analyses until one yields a significant result, then reporting only that result. Inflates the Type I error rate; countered by pre-registration.
parsimony
The principle that simpler models are preferred when they explain the data equally well. AIC and BIC operationalize parsimony through complexity penalties.
prediction interval
An interval for a single future observation accounting for uncertainty about the mean and natural individual variability. Always wider than a confidence interval for the mean.
pre-registration
Specifying hypotheses, data collection, and analysis plan before data are collected. Separates confirmatory from exploratory analysis and counters p-hacking.
Q-Q plot
Quantile-quantile plot comparing sample quantiles to theoretical quantiles of a proposed distribution. A straight line supports the distributional assumption.
replicability
The ability of independent researchers applying the same procedure to a new sample to reach consistent results. Distinguished from reproducibility (same data).
reproducibility
The ability of independent researchers following the same procedure on the same data to reach the same result. Supported by open data and open code.
risk aversion
The preference for a certain outcome over a gamble with the same expected value. Corresponds to a concave utility function; U(E[c]) ≥ E[U(c)] by Jensen's inequality.
standard error
The standard deviation of an estimator's sampling distribution. Measures precision. Decreases as sample size increases: SE(X̄) = σ/√n.
utility function
A function U(c) assigning a numerical value to each consequence, representing the decision-maker's preferences. Shape determines risk attitude.
Model Selection Practice
- A dataset records defects per production unit over 500 runs. Sample mean = 2.3; sample variance = 2.4. (a) Does a Poisson model seem plausible? (b) Describe two graphical methods for checking the Poisson assumption. (c) A chi-squared test with 6 categories gives a statistic of 8.2. At α = 0.05, is there evidence against the Poisson model? (d) The Poisson model has log-likelihood −812.1 (1 parameter); a negative binomial has −811.7 (2 parameters). Which model does AIC prefer?
- A researcher measures resting heart rate for 80 adults. (a) Why is the Normal distribution a natural candidate? (b) How would a Q-Q plot be used to check this? (c) A Shapiro-Wilk test yields p = 0.13. What does this tell you, and what does it not tell you? (d) The histogram shows mild right skew. Should the researcher proceed with a Normal model? Discuss.
- A statistician considers two regression models for housing prices (n = 250): Model 1 has 2 predictors (log-likelihood = −1204.3); Model 2 has 5 predictors (log-likelihood = −1199.8). (a) Compute AIC for each. (b) Compute BIC for each. (c) Do they agree? (d) Conduct a likelihood ratio test. What are the degrees of freedom, test statistic, and approximate p-value?
- A data scientist fits a Gamma model to 200 insurance claims (log-likelihood = −1403.7) and an Exponential model (log-likelihood = −1411.2). (a) Are these models nested? (b) Compute the likelihood ratio statistic and degrees of freedom. (c) At α = 0.01, does the Gamma fit significantly better? (d) Compare AIC values.
Inference Method Review
- For each scenario, identify the most appropriate inference procedure and justify your choice. (a) Estimating mean daily sodium intake from a random sample of 60 adults. (b) Testing whether a new battery lasts more than 20 hours on average; n = 15, approximately Normal. (c) Comparing the proportion of voters supporting a measure in two counties; 200 surveyed per county. (d) Measuring blood glucose in 25 mice before and after a dietary intervention. (e) Estimating the mean defect rate from counts over 100 production hours.
- A statistician tests whether three group means differ. (a) Why is running three separate two-sample t-tests inappropriate? (b) What procedure is correct? (c) If the overall test is significant, what additional analysis is needed?
- A study randomly assigns 50 students to a new teaching method and 50 to standard instruction. x̄₁ = 82.4, x̄₂ = 78.9, s₁ = 10.2, s₂ = 11.8. (a) State the hypotheses. (b) Compute the two-sample t-statistic. (c) Find the approximate p-value (df ≈ 97). (d) Construct a 95% confidence interval for the difference in means. (e) Interpret the result in plain language.
- A Bayesian analysis uses a Beta(2, 20) prior on defect proportion p. A sample of 100 items has 8 defectives. (a) Find the posterior distribution. (b) Compute the posterior mean. (c) Express a 95% credible interval. (d) Compare the posterior mean to the MLE 0.08 and explain the difference.
Assumption Checking
- A linear regression of systolic blood pressure on age yields a fan-shaped residual plot — residuals grow with fitted values. (a) What assumption does this violate? (b) What transformation might address it? (c) How should the researcher verify the fix?
- A one-sample t-test is applied to 12 observations. A Q-Q plot shows substantial curvature in both tails. (a) What does this suggest about the distribution? (b) Should the t-test be trusted? (c) What alternative procedure is appropriate?
- In a two-way ANOVA with three groups (n = 20 per group), residuals in Group 3 are notably smaller than in Groups 1 and 2. (a) What assumption may be violated? (b) Name a formal test for this assumption. (c) What modifications could be considered?
- A Poisson regression dispersion parameter estimate is 3.7 (substantially greater than 1). (a) What does this suggest? (b) What alternative model family is appropriate? (c) How would you decide between models?
Capstone Project Guide
Step 2: Exploratory Data Analysis Checklist
- Report the sample size.
- For each quantitative variable: mean, median, SD, min, max, histogram or boxplot.
- For each categorical variable: frequency table, bar chart.
- For pairs of variables: scatter plots (two quantitative) or side-by-side boxplots (one categorical, one quantitative).
- Note outliers, missing values, or unusual patterns.
- Write a paragraph describing what you observe.
Step 4: Inference Checklist
- Compute point estimates.
- Compute standard errors.
- Construct confidence intervals (at least 95%).
- If testing: state H₀ and Hₐ, compute the test statistic, find the p-value, state the conclusion.
- Present results in a well-labeled table.
Step 5: Assumption Checking Checklist
- Produce at least one graphical assumption check (Q-Q plot, residual plot, or similar).
- Apply at least one formal assumption check where appropriate.
- State whether each assumption appears to be satisfied.
- Describe what was done if any assumption appears to fail.
- If comparing models: show AIC or BIC for each candidate.
Step 7: Conclusion Checklist
- Answer the original question directly.
- State the effect size and whether it is practically meaningful.
- Acknowledge the assumptions and their plausibility.
- State at least one limitation of the analysis.
- Propose at least one direction for further investigation.
- Write the conclusion in language readable by a non-statistician.
Capstone Evaluation Criteria
| Component | Points |
|---|
| Question formulation and data description | 10 |
| Exploratory data analysis | 15 |
| Model and method selection with justification | 15 |
| Inference (correctness of computations) | 20 |
| Assumption checking | 15 |
| Uncertainty quantification | 10 |
| Communication (clarity, honesty, completeness) | 15 |
| Total | 100 |
Communication Practice
- A clinical trial finds a mean reduction in systolic blood pressure of 5.8 mmHg (treatment) vs. 1.2 mmHg (placebo). The 95% CI for the difference is (2.9, 7.3) mmHg; p = 0.0003. Write a two-paragraph summary for a general audience addressing: what was studied and found; what the confidence interval means in plain language; how to distinguish statistical from clinical significance; and one question the results do not answer.
- A researcher reports: "The correlation between study hours and exam scores was r = 0.34, which was statistically significant (p < 0.05). This proves that studying more causes higher exam scores." Identify at least three errors in this statement and write a corrected version.
- Design a table presenting regression results predicting salary from years of experience, education level (high school, bachelor's, graduate), and industry (5 categories). Include intercept and all coefficients with estimates, standard errors, t-statistics, p-values, and 95% CIs. (a) Sketch the table structure. (b) What footnote information is needed? (c) What graph would accompany it best?
- A newspaper headline reads: "New Study Finds No Link Between Coffee and Heart Disease." The study had 120 participants; r = −0.11; p = 0.23. (a) Explain why "no link" may be an overstatement. (b) What would a confidence interval for the true correlation tell you beyond the p-value? (c) How does sample size affect interpretation?
Error Analysis
Each problem contains a statistical error. Identify it and explain the correction.
Problem 1
From 500 participants, men earn $4,200 more than women → "The gender pay gap is exactly $4,200"
The sample statistic is an estimate with sampling variability. State the estimate, its standard error, and a confidence interval. The population gap might differ substantially.
Problem 2
p = 0.04 → "There is a 4% probability that the null hypothesis is true"
The p-value is the probability of results at least as extreme as observed, assuming H₀ is true — not the probability H₀ is true. That statement requires Bayesian reasoning with a prior on H₀.
Problem 3
Testing 40 predictors at α = 0.05, reporting the 3 significant ones as "confirmed risk factors"
P(at least one false positive among 40 tests) is very high. Apply Bonferroni or Benjamini-Hochberg correction and label findings as exploratory, requiring independent replication.
Problem 4
Residual plot shows a clear U-shape → "Some minor random variation; the linear model is adequate"
A U-shaped residual pattern is not random — it signals that the linear model misses a curvature in the data. A quadratic term or other functional form should be considered.
Problem 5
CI for regression slope is (0.23, 0.87) → "95% probability the true slope is in this interval"
The frequentist CI refers to the long-run procedure, not the specific interval. This specific interval either contains the true slope or it does not. The Bayesian credible interval provides the direct probability statement.
Problem 6
Selecting the model with highest R² among 10 models fit to the same training data, reporting R² = 0.94 as the model's performance
R² on training data always increases with complexity. Performance must be evaluated on held-out data (cross-validation or a test set). Report cross-validated predictive accuracy instead.
Chapter 14 Checkpoint
Chapter 14 Checkpoint
Complete without assistance — assesses all key skills and concepts from Chapter 14
Problem 1 — Model Selection
- A call center records customer complaints per day over 180 days. Sample mean = 3.1; sample variance = 8.4. (a) Does the Poisson model seem appropriate? Why or why not? (b) Write the AIC expressions for a Poisson model (log-likelihood = −412.3, 1 parameter) and a negative binomial model (log-likelihood = −406.7, 2 parameters). (c) Which model does AIC prefer? (d) A likelihood ratio test gives Λ = 11.2 with 1 degree of freedom. At α = 0.01, is there significant evidence the negative binomial fits better?
Problem 2 — Inference
- A sample of 36 adult males has mean resting heart rate x̄ = 72.4 bpm, SD s = 9.8 bpm. (a) Construct a 95% confidence interval for the population mean. (b) Test H₀: μ = 70 vs. Hₐ: μ ≠ 70 at α = 0.05. State the test statistic, p-value, and conclusion. (c) Interpret the confidence interval in plain language. (d) How would the interval change with n = 144?
Problem 3 — Assumption Checking
- A simple linear regression of salary on years of experience (n = 50) gives Ŷ = 35.2 + 3.1X, R² = 0.61. The residual plot shows a fan shape. (a) List four assumptions of simple linear regression. (b) What does the fan shape signal, and what should be done? (c) The Q-Q plot of residuals is approximately linear. What does this suggest? (d) One observation has a very large positive residual. How should this be handled?
Problem 4 — Decision-Making with Bayes' Theorem
- A diagnostic test has sensitivity 0.90 and specificity 0.95. Disease prevalence is 2%. (a) Using Bayes' theorem, find the positive predictive value (P(disease | positive test)). (b) Find the negative predictive value. (c) If a positive test triggers expensive treatment, what additional information is needed to decide whether to treat everyone who tests positive? (d) Recompute the positive predictive value if prevalence increases to 10%. How does this change the decision context?
Problem 5 — Communication
- A multiple regression of home prices yields these results (n = 180, R² = 0.71):
Square footage: estimate = 88.4, SE = 12.3, 95% CI = (64.2, 112.6)
Bedrooms: estimate = −3,200, SE = 4,100, 95% CI = (−11,200, 4,800)
Distance to city center (mi): estimate = −1,850, SE = 310, 95% CI = (−2,460, −1,240)
(a) Interpret the coefficient on square footage in plain language. (b) Why might the bedrooms coefficient be negative in a multiple regression? (c) Interpret the distance coefficient. (d) What does it mean that the bedrooms CI includes zero? (e) Write a two-sentence summary of the model's predictive performance.
Problem 6 — Ethics and Reproducibility
- A researcher tests 12 predictors for association with depression scores (n = 80) and finds 2 significant at α = 0.05, reporting them as "significant predictors of depression" without mentioning the other 10 tests. (a) What problem does this create? (b) With 12 true null hypotheses, how many significant results are expected? (c) Describe two practices the researcher could have used. (d) Write a two-sentence honest version of the findings.
End of Textbook
You Have Completed Probability and Statistics — MGU Mathematics Series
You have built probability from axioms through distributions, limit theorems, inference, regression, and stochastic processes. You have seen how integration, differentiation, and linear algebra enter statistical reasoning, and how statistical reasoning answers questions in science, medicine, engineering, finance, and data science. The path forward branches in many directions — mathematical statistics, Bayesian computation, machine learning, stochastic processes, econometrics, experimental design — but the foundation you carry into each of them is the same: the discipline to reason carefully under uncertainty, to quantify what you know and what you do not know, and to communicate both honestly.
MGU Library Connection
- Appendix A: Calculus Readiness Review — integration, differentiation, and series for probability
- Appendix O: Likelihood and Bayesian Reference — connects to Sections 14.2, 14.3, 14.5, and 14.7
- Appendix S: Capstone Project Rubric — full scoring guide for Section 14.12
- Appendix T: Advanced Statistics and Data Science Readiness Check — connects to Section 14.15
- Linear Algebra, Chapters 3–4: Matrix Methods — used in multiple regression and multivariate inference
- Differential Equations, Chapter 8: Stochastic DEs — next step beyond Chapter 13 random walks