Training and Validation Data | Malone Math

The Central Idea

Model development and model evaluation must use information in different roles:

\[ \boxed{ \text{learn} \quad\longrightarrow\quad \text{choose} \quad\longrightarrow\quad \text{audit} }. \]

If the same outcomes repeatedly influence both learning and auditing, the reported performance becomes optimistic.

The Three Data Roles

Role	Primary use	May guide model choices?
Training	Estimate coefficients and learned preprocessing	Yes, within the development procedure
Validation	Compare candidates and tune hyperparameters	Yes
Test	Evaluate the finalized workflow	No

Training Data

Training data are used to estimate model coefficients, split points, neighbor structures, feature representations, and other learned quantities. Any statistic used to transform the predictors—such as an imputation mean or scaling standard deviation—is also a fitted quantity.

Training performance describes fit to known cases and is generally optimistic for future performance.

Validation Data

Validation data guide choices not determined by ordinary coefficient fitting. Examples include

polynomial degree;
regularization strength;
tree depth or number of neighbors;
feature sets and transformations;
classification thresholds and calibration methods.

Because validation outcomes guide decisions, validation performance is no longer a fully untouched final audit.

Test Data

Test data are held aside until the complete modeling procedure is locked. The final test estimates performance on comparable unseen observations.

\[ \boxed{ \text{one final evaluation} \ne \text{an iterative tuning dashboard} }. \]

A test result that triggers another model revision has become validation information.

A Clean Development Flow

Training and validation support development. The untouched test set audits the selected workflow.

A Holdout Split

A holdout design assigns each eligible observation to one role. A common starting allocation is \(60\%\) training, \(20\%\) validation, and \(20\%\) test, but no ratio is universally optimal.

Large datasets can support smaller percentages with large absolute evaluation counts. Small datasets often benefit from cross-validation within a protected development set.

A Worked Holdout Example

With \(N=1{,}000\) eligible observations and a \(60/20/20\) allocation,

\[ n_{\mathrm{train}}=0.60(1000)=600, \] \[ n_{\mathrm{validation}}=0.20(1000)=200, \] \[ n_{\mathrm{test}}=0.20(1000)=200. \]

The split provides \(800\) development observations and \(200\) untouched test observations.

Allocation Is a Resource Decision

More training data can improve the fitted model. More validation data can distinguish candidate models more reliably. More test data can estimate final performance more precisely.

Choose allocation from the total sample size, model complexity, number of tuning decisions, outcome rarity, and precision required for final metrics.

Random Assignment to Data Roles

For independent and similarly distributed observations, random assignment helps create comparable subsets. Set and record a random seed so the split can be reproduced.

Randomness does not repair an unrepresentative source sample or make dependent rows independent.

Stratified Splitting

Stratification approximately preserves selected category proportions across splits. In binary classification, stratifying by the outcome can prevent a rare positive class from concentrating in one subset.

If the \(1{,}000\)-case example contains \(120\) positives, an exact \(60/20/20\) stratified allocation would place \(72\), \(24\), and \(24\) positives in training, validation, and test data.

Stratification Has Limits

Stratifying on many variables can create tiny or impossible strata. Preserve the variables most important to evaluation while retaining enough observations in every subset.

Stratification balances observed categories; it does not guarantee balance in every predictor or unmeasured characteristic.

The Independent-and-Identically-Distributed Ideal

Ordinary random splitting is most defensible when observations are approximately independent and the training and future cases share a common distribution:

\[ (X_i,Y_i) \overset{\mathrm{iid}}{\sim} P. \]

Many practical datasets violate this ideal through time, groups, geography, repeated measurements, or selection.

Split to Match Deployment

\[ \boxed{ \text{validation boundary} \approx \text{deployment boundary} }. \]

If the model will predict future months, validate on later months. If it will predict new customers, keep each customer's records in one subset. If it will transfer to new sites, hold out sites.

Time-Based Splits

For future prediction, train on earlier observations and validate on later observations:

\[ \text{train time} < \text{validation time} < \text{test time}. \]

Randomly mixing future records into training can create temporal leakage and hide drift, seasonality, or policy changes.

Grouped Splits

When several rows belong to the same person, firm, machine, household, or experiment, place the entire group in one role. Otherwise the model may encounter nearly identical group-specific information in both training and validation data.

The unit of splitting should match the unit of future generalization.

Spatial Splits

Nearby observations can share environmental, economic, or measurement conditions. A random row split may therefore exaggerate geographic transfer. Holding out regions or spatial blocks provides a more demanding and relevant evaluation when predictions will be made in new locations.

Duplicate and Near-Duplicate Records

Exact duplicates, repeated images, overlapping text, or measurements of the same event can cross a random split. The validation case then provides little evidence about performance on genuinely new information.

Deduplicate or group related records before splitting.

Cross-Validation

\(K\)-fold cross-validation partitions development data into \(K\) folds. For each fold:

fit the complete workflow on the other \(K-1\) folds;
predict the held-out fold;
calculate the selected loss on those held-out cases.

Every observation receives an out-of-fold prediction from a model that did not train on that observation.

Cross-Validated Loss

If fold \(k\) contains \(n_k\) observations and has average loss \(L_k\), the size-weighted cross-validated loss is

\[ \boxed{ L_{\mathrm{CV}} = \frac{\sum_{k=1}^{K}n_kL_k} {\sum_{k=1}^{K}n_k} }. \]

With equal fold sizes, this reduces to the ordinary mean \(\frac{1}{K}\sum_{k=1}^{K}L_k\).

A Worked Five-Fold Example

Suppose five equal folds produce these held-out RMSE values:

Fold	Held-out RMSE
1	\(4.8\)
2	\(5.1\)
3	\(4.6\)
4	\(5.0\)
5	\(5.5\)

\[ \overline{\operatorname{RMSE}} = \frac{4.8+5.1+4.6+5.0+5.5}{5} = 5.0. \]

The fold values range from \(4.6\) to \(5.5\), revealing meaningful split-to-split variation.

Average RMSE versus Pooled RMSE

Averaging fold RMSE values gives each fold a direct contribution. Another approach pools all out-of-fold squared errors and then takes one square root:

\[ \operatorname{RMSE}_{\mathrm{OOF}} = \sqrt{ \frac{1}{N} \sum_{i=1}^{N} (y_i-\hat{y}_{i,\mathrm{OOF}})^2 }. \]

These summaries can differ because the square-root operation is nonlinear. State which convention is used.

Out-of-Fold Predictions

Out-of-fold predictions support residual plots, calibration checks, and threshold selection without using each observation's outcome to fit its own prediction model.

They are development estimates, not a substitute for a final independent test when extensive choices were guided by the same cross-validation process.

Choosing \(K\)

Five- and ten-fold cross-validation are common. Larger \(K\) trains each fitted model on more data but requires more computation and can change the variability of the estimate.

The appropriate choice depends on sample size, outcome rarity, computation, grouping, and the stability of the learning procedure.

Leave-One-Out Cross-Validation

Leave-one-out cross-validation uses \(K=N\): each observation is held out once. Every fitted model uses \(N-1\) observations.

It can be computationally costly and its error estimate may have high variability because the fitted training sets are almost identical. It is not automatically superior to five- or ten-fold validation.

Repeated Cross-Validation

Repeating \(K\)-fold cross-validation with several random partitions measures sensitivity to a particular fold assignment. Summarize the distribution of performance and record every seed.

Repetition does not create new independent observations; it reuses the same cases in different partitions.

Fold Results Are Dependent

Training sets overlap across folds, so fold losses are not independent replicates. A simple standard deviation across \(K\) folds is a useful descriptive stability measure but does not automatically yield an exact standard error for generalization performance.

Nested Cross-Validation

Nested cross-validation separates tuning from evaluation:

the inner loop selects hyperparameters;
the outer loop evaluates the selected procedure.

Each outer assessment fold must remain invisible to all inner-loop preprocessing and model choices.

Rolling-Origin Validation

Time-series validation can use expanding or rolling training windows:

\[ \text{train through }t \quad\longrightarrow\quad \text{validate at }t+h. \]

Repeating this origin across time estimates performance for the intended forecast horizon \(h\) without training on the future.

Blocked and Gap Validation

Adjacent time or spatial observations can be strongly related. A gap between training and validation periods, or blocked spatial folds, can reduce information spillover.

The gap should reflect how quickly dependence decays and how deployment separates past information from future outcomes.

Stratified Cross-Validation

In classification, stratified folds approximately preserve class proportions. This can prevent individual folds from containing too few positive cases for stable metric calculation.

When data are grouped, stratification must operate at the group level or through a procedure that preserves both constraints.

Rare Outcomes

If there are fewer positive cases than folds, some validation folds must contain no positives. Sensitivity, precision, and related measures may be undefined or extremely unstable.

Use fewer folds, repeated carefully designed splits, suitable probability metrics, or additional outcome collection.

The Entire Pipeline Must Be Fitted Inside Training

\[ \boxed{ \text{split first} \quad\longrightarrow\quad \text{fit preprocessing inside each training partition} }. \]

The pipeline includes imputation, scaling, encoding, transformation, feature selection, resampling, model fitting, and any learned calibration.

Imputation Leakage

Filling missing values using the full-dataset mean allows validation and test predictors to influence the training transformation. Instead, estimate the mean or other imputation model on training data and apply it unchanged to held-out data.

Scaling Leakage

Standardization parameters must be learned from training data:

\[ z = \frac{x-\bar{x}_{\mathrm{train}}} {s_{\mathrm{train}}}. \]

The stored training mean and standard deviation are then applied to validation, test, and deployment observations.

Encoding and Vocabulary Leakage

Category levels, text vocabularies, rare-level groupings, and learned embeddings can reveal held-out information. Fit these representations on the training partition and define how genuinely unseen values will be handled.

Feature-Selection Leakage

Selecting predictors with the full set of outcomes before cross-validation makes every fold indirectly visible. Feature screening and dimensionality reduction must be repeated inside each training fold.

The evaluation target is the full selection procedure, not a feature set chosen with privileged information.

Resampling Leakage

Oversampling a rare class before splitting can place copies or synthetic descendants of the same observation in training and validation data. Perform over- or undersampling inside each training partition only.

Evaluate on held-out data with the natural class distribution relevant to deployment.

Target Leakage

A predictor leaks the target when it contains the outcome itself or information created after the outcome became known. Examples include a discharge code used to predict admission risk or a collection notice used to predict an earlier default.

Define the prediction timestamp and audit when every feature becomes available.

Group Leakage

Records from one entity can share identifiers, habits, background, or measurement artifacts. If related records cross folds, a model may recognize the entity rather than learn a transferable relationship.

Group-aware splitting is necessary even when direct identifiers have been removed.

Hyperparameter Tuning

For each candidate hyperparameter setting, fit the pipeline on training data and evaluate it on validation data or through cross-validation. Choose the setting using a prespecified metric and tie-breaking rule.

A large search can overfit the validation process, which strengthens the case for nested evaluation or a protected test set.

Classification Thresholds

Probability thresholds are operational hyperparameters. Select a threshold on validation data using documented error costs, capacity constraints, or required sensitivity and specificity.

Report final confusion-matrix metrics on test data using the locked threshold.

Probability Calibration

Calibration procedures such as logistic recalibration or isotonic regression are learned transformations. Fit them on data not used to fit the underlying score model, often with a dedicated calibration split or cross-validated predictions.

Evaluate the complete calibrated pipeline on untouched data.

Choose Metrics before Viewing Results

The evaluation metric should represent the prediction target and decision cost. Examples include

MAE or RMSE for quantitative predictions;
log loss or Brier score for probability estimates;
sensitivity, specificity, or precision for threshold decisions;
interval coverage and width for prediction intervals.

Changing the metric after seeing results creates another selection step.

Baseline Models

Compare candidate models with a simple baseline: the training mean for regression, class prevalence for probability prediction, or a seasonal naive forecast for time series.

A complex model that cannot outperform an appropriate baseline on held-out data has not demonstrated operational value.

Uncertainty in Test Accuracy

Suppose a locked classifier correctly predicts \(170\) of \(200\) independent test cases:

\[ \hat{p} = \frac{170}{200} = 0.85. \]

A rough binomial standard error is

\[ \sqrt{ \frac{0.85(0.15)}{200} } \approx 0.0252. \]

Thus, \(85\%\) is an estimate, not an exact property of the deployed model. Formal intervals should use an appropriate binomial method and account for dependence or weighting when present.

Test-Set Size Controls Precision

Rare events and subgroup analyses require adequate absolute counts. A test set with thousands of negatives but only ten positives cannot estimate sensitivity precisely.

Plan the evaluation sample around the least common outcome and the smallest decision-relevant group, not only the total number of rows.

External Validation

External validation evaluates a locked model in a different site, time period, organization, or population. It provides stronger evidence about transportability than a random split from one source.

Differences in measurement, eligibility, prevalence, and policy should be documented rather than treated as nuisance details.

Recalibration Is a New Development Step

If external data are used to update an intercept, slope, threshold, or feature representation, those data are no longer pure external test data. Evaluate the updated model on another untouched sample or through a valid resampling design.

Refitting after Selection

After hyperparameters are locked, a final model may be refitted on all development data and evaluated once on the test set. This uses more information for coefficient estimation while preserving the test audit.

If the model is later refitted using test outcomes, the prior test score remains historical evidence for the earlier workflow; no untouched test set remains for the expanded fit.

Learning Curves

A learning curve evaluates training and validation loss at increasing training sample sizes. It can indicate whether more representative data may reduce a generalization gap or whether both errors have plateaued at a level suggesting underfitting.

Each point must preserve the same leakage-free preprocessing and deployment-relevant split design.

Bootstrap and Validation Answer Different Questions

Bootstrap resampling can estimate variability, optimism, or interval uncertainty. It does not automatically create an external test population, and ordinary bootstrap samples do not respect time or group dependence unless the resampling unit is redesigned.

Cross-Fitting

Cross-fitting constructs nuisance estimates or predictions for each observation using other folds. It is used in stacking, calibration, causal estimation, and other multistage procedures to reduce self-influence.

Cross-fitting is a design principle, not permission to use a final test set during development.

Dataset Shift

A valid split estimates performance only for the relationship between its development and evaluation distributions. Deployment can still differ through changed predictor distributions, outcome prevalence, measurement, or the conditional relationship itself.

Stress tests and external validation extend evidence but cannot guarantee future stability.

Postdeployment Monitoring

Once outcomes arrive, compare live performance with validation and test expectations. Monitor

input quality and missingness;
predictor and score distributions;
residuals, calibration, and error rates;
subgroup performance and coverage;
operational overrides and downstream effects.

Version the Data and Split

Record eligibility rules, extraction dates, data versions, group identifiers, split assignments, random seeds, and excluded observations. A reproducible split is part of the model specification.

Silent changes to the source data can otherwise move observations across roles and invalidate comparisons.

Test-Set Governance

Define who may access test outcomes, when evaluation occurs, what criteria authorize launch, and what happens after failure. A documented gate reduces the temptation to revise repeatedly against the same test set.

For long-running programs, maintain future evaluation cohorts rather than treating one historical test set as permanently authoritative.

A Practical Data-Splitting Workflow

Define the prediction timestamp, unit, population, and deployment boundary.
Audit duplicates, groups, time order, geography, and outcome rarity.
Reserve a test set using a deployment-relevant split.
Choose holdout or cross-validation within the remaining development data.
Place every learned preprocessing step inside the training pipeline.
Select metrics, candidates, and tuning rules before comparing results.
Tune models and thresholds using validation evidence only.
Lock the complete workflow and refit on development data if planned.
Evaluate once on the untouched test set with uncertainty.
Version the result and monitor new deployment cohorts.

Reporting Checklist

What is the unit of observation and prediction?
How were groups, time, location, and duplicates handled?
What are the training, validation, and test sample sizes?
Which transformations were learned inside training?
What metric and baseline guided selection?
How extensive was the hyperparameter and feature search?
Was the test set accessed before the workflow was locked?
What uncertainty and subgroup results accompany final performance?

Common Error: Splitting Rows before Groups

Random row splitting can place one person's, firm's, or device's records in multiple subsets. Split unique groups first when the target is generalization to new groups.

Common Error: Preprocessing before Splitting

Full-dataset imputation, scaling, selection, or encoding lets held-out data influence training. Split first and fit the entire preprocessing pipeline within each training partition.

Common Error: Calling Validation Data Test Data

A dataset repeatedly used to choose models is validation data regardless of its filename. Data roles are determined by use, not labels.

Common Error: Random Splitting across Time

Mixing future observations into training can make historical prediction appear easier than real forecasting. Preserve chronological order when the deployment target is the future.

Common Error: Reporting One Convenient Split

A favorable random split may overstate performance. Record the split in advance, use repeated or cross-validated development evidence where appropriate, and retain a protected final test.

Practice 1: Allocate a Holdout Split

Divide \(2{,}500\) observations using a \(70/15/15\) training, validation, and test allocation.

Practice 2: Identify the Data Role

A team uses a dataset to choose between ridge penalties \(\lambda=0.1,1,10\). What role is that dataset serving?

Practice 3: Calculate Cross-Validated Loss

Three equal folds have losses \(6\), \(9\), and \(12\). Find the mean cross-validated loss.

Practice 4: Diagnose Scaling Leakage

An analyst calculates predictor means and standard deviations from all observations before cross-validation. What should change?

Practice 5: Choose a Grouped Split

A dataset contains ten visits for each patient, and deployment will predict outcomes for new patients. At what level should folds be assigned?

Practice 6: Choose a Time Split

A model will forecast next quarter's demand. Why is a random split across five years of monthly observations questionable?

Practice 7: Protect the Test Set

A model fails the test criterion, so the team changes features and evaluates on the same test set again. What has happened to the data role?

Practice 8: Calculate Test Accuracy

A locked classifier correctly predicts \(360\) of \(450\) test cases. Find the test accuracy.

Practice 9: Handle a Rare Class

A binary dataset has only four positive observations. Why is ordinary ten-fold stratified cross-validation infeasible for class-specific metrics?

Practice 10: Diagnose Target Leakage

A hospital model predicts whether a patient will be admitted using a room assignment recorded after the admission decision. What is wrong?

Practice Answers

Practice 1

\[ n_{\mathrm{train}} = 0.70(2500) = 1750, \] \[ n_{\mathrm{validation}} = n_{\mathrm{test}} = 0.15(2500) = 375. \]

Practice 2

It is serving as validation data because its outcomes guide the hyperparameter choice.

Practice 3

\[ \frac{6+9+12}{3} = 9. \]

Practice 4

Fit the means and standard deviations separately within each training fold, then apply those stored values to that fold's held-out observations.

Practice 5

Assign folds at the patient level so all ten visits for a patient remain in one fold.

Practice 6

The random split can train on months that occur after validation months and hide temporal change. Train on earlier periods and validate on later periods at the intended forecast horizon.

Practice 7

The test outcomes influenced a model revision, so the dataset has become validation data. A new untouched evaluation sample is needed for an honest final audit.

Practice 8

\[ \frac{360}{450} = 0.80. \]

The test accuracy is \(80\%\).

Practice 9

Ten folds cannot each contain a positive case when only four positives exist. Several folds would have no positives, making measures such as sensitivity undefined or highly unstable.

Practice 10

The room assignment is unavailable at the prediction time and encodes the completed decision. It is target and temporal leakage.

What to Remember

Training data fit; validation data choose; test data audit.
Data roles are determined by use, not by filenames.
The validation boundary should reproduce the deployment boundary.
Groups, time, geography, and duplicates often require structured splits.
Every learned preprocessing step belongs inside training.
Cross-validation gives each development case an out-of-fold assessment.
Nested validation separates hyperparameter search from performance estimation.
Repeated test access converts test data into validation data.
Final metrics are estimates and should include uncertainty.
Deployment monitoring is the next evaluation cohort.

Where We Go Next

Honest data separation can show that a relationship predicts well in new cases. Predictive stability, however, does not establish that changing a predictor would change the response.

The next lesson, Association versus Causation, separates predictive relationships from causal effects and examines the additional design and assumptions required for causal conclusions.