The Central Idea
Surveys, experiments, and forecasts answer different kinds of questions.
- Surveys: what does a population look like?
- Experiments: what happens when we deliberately change something?
- Forecasts: what is likely to happen next?
The common mathematical structure is:
\[ \boxed{ \text{data} + \text{design} + \text{probability model} \longrightarrow \text{inference or prediction} } \]Population and Sample
A population is the full set of units we want to understand.
A sample is the subset actually observed.
The central survey problem is whether the sample gives trustworthy information about the population.
Parameters and Statistics
A parameter is a numerical characteristic of a population.
A statistic is a numerical characteristic computed from a sample.
For example, a population proportion is
\[ p, \]while a sample proportion is
\[ \boxed{\hat p=\frac{x}{n}}. \]A First Survey Example
Suppose \(600\) people are surveyed and \(372\) support a proposal.
The sample proportion is
\[ \hat p=\frac{372}{600}=0.62. \]Therefore,
\[ \boxed{\hat p=62\%}. \]Random Sampling
Random sampling uses a probability mechanism to choose units from a target population.
Randomization helps make the sampling process less dependent on human choice and provides a basis for probability-based inference.
Simple Random Samples
In a simple random sample of size \(n\), each possible sample of that size has the same probability of selection under the idealized design.
This is a clean mathematical model, though real surveys often use more complex designs.
Sampling Without Replacement
If a population contains \(N\) units and a sample of size \(n\) is selected without replacement, the number of possible unordered samples is
\[ \boxed{\binom{N}{n}}. \]This connects survey design directly to combinatorics.
Stratified Sampling
Stratified sampling divides the population into groups and samples within each group.
Stratification can improve precision or ensure representation of important subgroups.
Common strata include:
- region;
- age group;
- organization size;
- grade level;
- customer category.
Cluster Sampling
Cluster sampling samples naturally occurring groups, then observes some or all units within selected clusters.
Examples of clusters include:
- schools;
- neighborhoods;
- worksites;
- households.
Cluster designs can reduce collection cost but may increase dependence among observations.
Sampling Bias
A large sample can still be misleading if the sampling process systematically favors some parts of the population.
Bias can arise from:
- undercoverage;
- voluntary response;
- nonresponse;
- poor sampling frames;
- selection procedures tied to the outcome.
Nonresponse Bias
Nonresponse becomes a problem when the people who do not respond differ in relevant ways from those who do.
A high response rate can help, but response rate alone does not guarantee absence of bias.
Measurement Bias
Even a representative sample can produce poor information if the measurement process is flawed.
Sources include:
- leading questions;
- ambiguous wording;
- social-desirability effects;
- instrument error;
- interviewer effects;
- recall error.
Sampling Variability
Different random samples from the same population produce different statistics.
This is sampling variability.
It is not necessarily a mistake; it is a natural consequence of sampling.
Standard Error of a Sample Proportion
Under a simple random-sampling model,
\[ \boxed{ \operatorname{SE}(\hat p) = \sqrt{\frac{p(1-p)}{n}} }. \]In practice, \(p\) is often replaced by \(\hat p\) when estimating the standard error.
Sample Size and Precision
Standard error decreases roughly like
\[ \boxed{\frac{1}{\sqrt n}}. \]Therefore, quadrupling sample size roughly halves standard error under the basic model.
Confidence Intervals
A confidence interval gives a range of plausible values for a population parameter under a specified inferential procedure.
A common large-sample form is
\[ \boxed{ \text{estimate} \pm \text{critical value}\times\text{standard error} }. \]Margin of Error
The margin of error is the amount added to and subtracted from the point estimate in a symmetric confidence interval.
In a simple proportion interval,
\[ \boxed{ ME=z^\ast \sqrt{\frac{\hat p(1-\hat p)}{n}} }. \]Survey Confidence Example
Suppose
\[ \hat p=0.60, \qquad n=400. \]Approximate standard error is
\[ \sqrt{\frac{0.60(0.40)}{400}} = \sqrt{0.0006} \approx0.0245. \]Using \(z^\ast\approx1.96\), the approximate margin of error is
\[ 1.96(0.0245)\approx\boxed{0.048}. \]The interval is approximately
\[ 0.60\pm0.048. \]Random Sampling and Random Assignment Are Different
Random sampling concerns how units enter a study.
Random assignment concerns how study units are assigned to treatments.
Random sampling supports population generalization.
Random assignment supports causal comparison.
Experiments
An experiment deliberately assigns one or more conditions to study units and measures outcomes.
A well-designed experiment tries to isolate the effect of the treatment from competing explanations.
Treatments
A treatment is a condition or intervention deliberately assigned in an experiment.
Examples include:
- different teaching methods;
- different software interfaces;
- different process settings;
- different marketing messages;
- different medical interventions.
Control Groups
A control group provides a comparison condition.
Depending on context, it may receive:
- no intervention;
- standard practice;
- a placebo;
- an existing treatment.
Random Assignment
Random assignment uses chance to allocate study units to treatment groups.
Its purpose is to reduce systematic differences between groups before treatment.
Random assignment does not guarantee perfectly identical groups, but it makes systematic allocation bias less likely.
Replication
Replication means applying treatments to enough independent experimental units to estimate variability and distinguish treatment effects from noise.
One observation per treatment rarely provides enough information for reliable inference.
Blocking
Blocking groups similar units together before treatment assignment.
Treatments are then compared within blocks.
Blocking can reduce unwanted variability from known sources.
Blinding
Blinding limits knowledge of treatment assignment when that knowledge could influence measurement, behavior, or evaluation.
Who can be blinded depends on the experimental setting.
Confounding
A confounder is a factor whose effect cannot be separated cleanly from the treatment effect in the observed comparison.
Randomization, blocking, and design controls help reduce confounding.
Experimental Units
The experimental unit is the smallest unit independently assigned to a treatment.
Identifying the correct unit matters for determining sample size and independence.
Pseudoreplication
Pseudoreplication occurs when multiple measurements from the same experimental unit are incorrectly treated as independent replications.
Repeated measurements can provide useful information, but they do not create new independent treatment assignments.
Difference in Means
For two treatment groups, a simple effect estimate is
\[ \boxed{ \bar X_T-\bar X_C }. \]Here \(T\) denotes treatment and \(C\) denotes control.
Difference in Proportions
For binary outcomes, a common effect estimate is
\[ \boxed{ \hat p_T-\hat p_C }. \]This is an absolute treatment-effect estimate.
Observational Studies
In an observational study, researchers observe exposures and outcomes without randomly assigning the exposure.
Such studies can reveal associations but generally require stronger assumptions for causal interpretation.
Forecasting
A forecast is an estimate of a future quantity.
Forecast targets may include:
- sales;
- demand;
- population;
- traffic;
- inventory usage;
- economic indicators;
- weather-related quantities.
Point Forecasts
A point forecast gives one predicted value.
For example,
\[ \boxed{\hat y_{t+1}=1250}. \]A point forecast is incomplete without some understanding of uncertainty.
Prediction Intervals
A prediction interval gives a range intended to capture a future observation with a specified long-run probability under the model.
Prediction intervals are generally wider than confidence intervals for a mean response because they include future random variation.
Forecast Error
If the actual value at time \(t\) is \(y_t\) and the forecast is \(\hat y_t\), define forecast error as
\[ \boxed{ e_t=y_t-\hat y_t }. \]A positive error means the forecast was too low under this sign convention.
Absolute Error
Absolute forecast error is
\[ \boxed{|e_t|}. \]It measures error magnitude without regard to direction.
Squared Error
Squared error is
\[ \boxed{e_t^2}. \]Squaring gives greater weight to large errors.
Mean Absolute Error
For \(n\) forecasts, mean absolute error is
\[ \boxed{ \operatorname{MAE} = \frac{1}{n} \sum_{t=1}^{n}|e_t| }. \]Mean Squared Error
Mean squared error is
\[ \boxed{ \operatorname{MSE} = \frac{1}{n} \sum_{t=1}^{n}e_t^2 }. \]Root Mean Squared Error
Root mean squared error is
\[ \boxed{ \operatorname{RMSE} = \sqrt{\operatorname{MSE}} }. \]RMSE has the same units as the forecasted quantity.
Naive Forecasts
A simple benchmark forecast is
\[ \boxed{ \hat y_{t+1}=y_t }. \]This says the next value will equal the most recent observed value.
Simple benchmarks are important because a complicated forecasting model should generally outperform an appropriate naive alternative to justify its complexity.
Moving Averages
A moving-average forecast smooths recent observations.
A \(k\)-period moving average is
\[ \boxed{ \hat y_{t+1} = \frac{y_t+y_{t-1}+\cdots+y_{t-k+1}}{k} }. \]Moving-Average Example
Suppose the last three observations are
\[ 100,\qquad110,\qquad120. \]The three-period moving-average forecast is
\[ \frac{100+110+120}{3} = \boxed{110}. \]Exponential Smoothing
Simple exponential smoothing updates a forecast using
\[ \boxed{ \hat y_{t+1} = \alpha y_t +(1-\alpha)\hat y_t }. \]Here
\[ 0\le\alpha\le1. \]Larger \(\alpha\) places more weight on the most recent observation.
Exponential-Smoothing Example
Suppose
\[ \alpha=0.4, \qquad y_t=120, \qquad \hat y_t=110. \]Then
\[ \hat y_{t+1} = 0.4(120)+0.6(110). \]Therefore,
\[ \hat y_{t+1} = 48+66 = \boxed{114}. \]Trend
A trend is a persistent long-term increase or decrease.
A simple linear trend model is
\[ \boxed{ y_t=a+bt+\varepsilon_t }. \]The coefficient \(b\) represents average change per time period under the model.
Seasonality
Seasonality is a repeating pattern tied to a calendar or recurring cycle.
Examples include:
- day-of-week patterns;
- monthly retail cycles;
- annual energy demand;
- school-year effects.
Cycles and Structural Change
Not every repeating pattern is strict seasonality.
Longer irregular cycles and sudden structural changes can also affect a time series.
Forecasting models can fail when historical relationships stop holding.
Autocorrelation
Time-series observations are often dependent across time.
Autocorrelation measures association between a series and lagged versions of itself.
This dependence means time-series observations cannot always be treated as independent samples.
Forecast Horizon
Forecast uncertainty generally grows as the forecast horizon increases.
Predicting tomorrow is usually easier than predicting years ahead because more unobserved changes can accumulate over longer horizons.
Calibration
A probabilistic forecast is calibrated when events assigned probability \(p\) occur about proportion \(p\) of the time across many comparable forecasts.
For example, among many events assigned probability \(0.70\), roughly \(70\%\) should occur if the forecasts are well calibrated.
Sharpness
Calibration alone does not reward informative forecasts.
A forecaster that always gives broad or moderate probabilities may be calibrated but uninformative.
Sharpness refers to concentration or decisiveness of predictive distributions, subject to maintaining calibration.
Scoring Probabilistic Forecasts
Proper scoring rules reward accurate probability forecasts.
For a binary event with forecast probability \(p\) and outcome
\[ y\in\{0,1\}, \]the Brier score is
\[ \boxed{ (p-y)^2 }. \]Brier-Score Example
Suppose a forecaster assigns probability
\[ p=0.8 \]and the event occurs, so
\[ y=1. \]The Brier score is
\[ (0.8-1)^2 = (-0.2)^2 = \boxed{0.04}. \]Training and Testing Data
A forecasting model should be evaluated on data not used to fit it.
Otherwise, the model may appear accurate simply because it adapted too closely to the historical sample.
Overfitting
Overfitting occurs when a model captures random noise in the training data rather than stable predictive structure.
Overfit models can perform extremely well in-sample and poorly out-of-sample.
Validation
Validation uses held-out or future-like data to compare forecasting models.
In time series, validation should respect time order rather than randomly mixing future observations into the training period.
Rolling Forecast Evaluation
A rolling evaluation repeatedly:
- fits or updates a model using data available up to a given time;
- forecasts the next period or horizon;
- records the error;
- moves forward through time.
This mimics real forecasting more closely than evaluating on data the model has already seen.
Common Error: Treating a Convenience Sample as Representative
Easy-to-reach respondents may differ systematically from the target population.
Sample size does not repair a biased selection mechanism.
Common Error: Confusing Random Sampling and Random Assignment
Random sampling addresses generalization.
Random assignment addresses causal treatment comparison.
A study may have one, both, or neither.
Common Error: Treating Association as Experimental Evidence
Observational association can arise from confounding or selection effects.
Causal conclusions require design or assumptions beyond correlation alone.
Common Error: Ignoring Experimental Unit
Repeated measurements on one unit are not equivalent to independent treatment replications.
The unit of random assignment determines the effective replication level.
Common Error: Reporting a Point Forecast Without Uncertainty
A forecast of
\[ 1000 \]units does not indicate whether plausible outcomes range from \(990\) to \(1010\) or from \(500\) to \(1500\).
Prediction uncertainty matters for decisions.
Common Error: Comparing Forecast Models In-Sample Only
A flexible model can fit historical data closely while forecasting poorly.
Model comparison should use out-of-sample or future-like evaluation.
Common Error: Ignoring Time Order
Forecasting models must not use future information that would not have been available at the time of prediction.
This mistake is often called leakage.
Common Error: Assuming the Past Must Continue
Forecasts based on historical relationships can fail after:
- policy changes;
- technology changes;
- market shifts;
- behavioral changes;
- rare disruptions.
Structural change is a major source of forecast risk.
A Reliable Procedure for Survey Problems
- Define the target population.
- Define the sampling frame.
- Choose the sampling design.
- Identify possible coverage and nonresponse problems.
- Measure the variable consistently.
- Calculate the sample statistic.
- Estimate sampling uncertainty.
- Apply weights or design corrections when required.
- State the population to which conclusions apply.
A Reliable Procedure for Experiments
- Define the experimental units.
- Define treatments and outcomes.
- Identify potential confounders.
- Randomize treatment assignment when appropriate.
- Block or stratify important known sources of variation when useful.
- Use adequate replication.
- Blind participants or evaluators when feasible and relevant.
- Compare treatment outcomes with uncertainty measures.
- Interpret causal conclusions within the design's scope.
A Reliable Procedure for Forecasting
- Define the forecast target.
- Define the horizon.
- Inspect trend, seasonality, and structural changes.
- Choose a simple benchmark.
- Fit one or more models using only available historical information.
- Generate point forecasts and uncertainty intervals.
- Evaluate on future-like data.
- Compare errors using appropriate scoring measures.
- Check calibration for probabilistic forecasts.
- Update the model as new data arrive.
Example: Full Survey Workflow
Suppose \(800\) randomly selected respondents are surveyed and \(456\) support an initiative.
Sample proportion:
\[ \hat p=\frac{456}{800}=0.57. \]Approximate standard error:
\[ \sqrt{\frac{0.57(0.43)}{800}} \approx0.0175. \]Approximate \(95\%\) margin of error:
\[ 1.96(0.0175)\approx0.0343. \]Thus an approximate interval is
\[ 0.57\pm0.034. \]Example: Full Experimental Workflow
Suppose \(200\) participants are randomly assigned:
- \(100\) to treatment;
- \(100\) to control.
Suppose \(68\) treatment participants and \(54\) control participants reach the desired outcome.
Treatment proportion:
\[ \hat p_T=\frac{68}{100}=0.68. \]Control proportion:
\[ \hat p_C=\frac{54}{100}=0.54. \]Estimated treatment difference:
\[ 0.68-0.54=\boxed{0.14}. \]Example: Full Forecast Workflow
Suppose the last four monthly demand values are
\[ 120,\ 130,\ 125,\ 135. \]A four-period moving-average forecast is
\[ \frac{120+130+125+135}{4} = \frac{510}{4} = \boxed{127.5}. \]If actual next-month demand is \(132\), forecast error under
\[ e=y-\hat y \]is
\[ 132-127.5=\boxed{4.5}. \]Practice 1: Sample Proportion
In a sample of \(500\), \(315\) respondents choose option A.
Find \(\hat p\).
Practice 2: Number of Samples
How many unordered samples of size \(3\) can be selected from \(10\) units?
Practice 3: Standard Error
Using
\[ \hat p=0.50, \qquad n=400, \]estimate the standard error of the sample proportion.
Practice 4: Difference in Proportions
A treatment group has success proportion \(0.72\) and control has \(0.60\).
Find the difference in proportions.
Practice 5: Forecast Error
Actual demand is \(210\) and forecast demand is \(195\).
Using
\[ e=y-\hat y, \]find the forecast error.
Practice 6: Absolute Error
If forecast error is
\[ -12, \]find the absolute error.
Practice 7: Three-Period Moving Average
The last three observations are
\[ 80,\ 90,\ 100. \]Find the next moving-average forecast.
Practice 8: Exponential Smoothing
Suppose
\[ \alpha=0.5,\qquad y_t=140,\qquad\hat y_t=120. \]Find \(\hat y_{t+1}\).
Practice 9: Brier Score
A forecaster assigns probability \(0.6\) and the event occurs.
Find the Brier score.
Practice 10: Randomization Purpose
Which supports causal comparison more directly: random sampling or random assignment?
Practice Answers
Practice 1
\[ \hat p = \frac{315}{500} = \boxed{0.63}. \]Practice 2
\[ \binom{10}{3} = \frac{10\cdot9\cdot8}{3\cdot2\cdot1} = \boxed{120}. \]Practice 3
\[ \operatorname{SE}(\hat p) \approx \sqrt{\frac{0.5(0.5)}{400}} = \sqrt{0.000625} = \boxed{0.025}. \]Practice 4
\[ 0.72-0.60=\boxed{0.12}. \]Practice 5
\[ e=210-195=\boxed{15}. \]Practice 6
\[ |-12|=\boxed{12}. \]Practice 7
\[ \frac{80+90+100}{3} = \frac{270}{3} = \boxed{90}. \]Practice 8
\[ \hat y_{t+1} = 0.5(140)+0.5(120) = 70+60 = \boxed{130}. \]Practice 9
Since the event occurs,
\[ y=1. \]Therefore,
\[ (0.6-1)^2 = (-0.4)^2 = \boxed{0.16}. \]Practice 10
The answer is
\[ \boxed{\text{random assignment}}. \]What to Remember
Surveys, experiments, and forecasting all depend on disciplined data collection and uncertainty modeling.
- surveys estimate population characteristics from samples;
- random sampling supports probability-based generalization;
- sampling variability decreases as sample size increases;
- bias is not repaired simply by increasing sample size;
- random assignment supports causal treatment comparison;
- blocking and replication improve experimental design;
- forecast error is the difference between actual and predicted values;
- MAE, MSE, and RMSE summarize forecast accuracy differently;
- moving averages and exponential smoothing are basic forecasting tools;
- trend, seasonality, and autocorrelation matter in time series;
- forecast models should be evaluated out of sample;
- probabilistic forecasts should be judged for calibration as well as sharpness.
The central habit is: separate the data-collection design from the statistical model, and never report an estimate, treatment effect, or forecast without asking what uncertainty and bias remain.
Where We Go Next
Surveys, experiments, and forecasts turn observed data into estimates, causal comparisons, and predictions. The final applied problem family combines these ideas with algorithmic experimentation, allowing entire stochastic systems and decision rules to be tested computationally.
The next lesson, Simulation and Data-Driven Decision-Making, develops Monte Carlo simulation, scenario generation, model validation, sensitivity analysis, optimization under uncertainty, and evidence-based decision workflows.