Data Science and Machine Learning

The Central Idea

A data-driven modeling workflow can be summarized as

The model is only one part of the pipeline. Poor data, leakage, inappropriate objectives, or weak validation can invalidate an otherwise sophisticated algorithm.

Observations and Features

Suppose a dataset contains \(n\) observations and \(p\) measured features. A feature vector for observation \(i\) may be written

The outcome or target is often denoted \(y_i\).

The Design Matrix

Collecting feature vectors into rows gives a design matrix

Many machine-learning methods can be expressed as operations on \(X\), a target vector \(\mathbf{y}\), and model parameters.

Supervised Learning

In supervised learning, observations include both inputs and target values:

The objective is to learn a function

that predicts targets for new inputs.

Regression

Regression predicts a quantitative target. A linear regression model is

The coefficients describe how the fitted prediction changes as features change, holding the other modeled features fixed.

Residuals

A residual is the difference between observed and predicted outcome:

Residual patterns can reveal model misspecification, nonlinear structure, changing variance, outliers, or omitted variables.

Least Squares

Ordinary least squares chooses parameters to minimize the sum of squared residuals:

Squaring emphasizes larger errors and creates a smooth optimization problem.

Matrix Form of Linear Regression

A linear model may be written

When \(X^TX\) is invertible, the least-squares solution is

In numerical practice, more stable matrix factorizations are generally preferred to forming the inverse explicitly.

Nonlinear Regression

Regression models need not be linear in the inputs. A model may include powers, interactions, splines, kernels, trees, neural networks, or other nonlinear structures.

Greater flexibility can improve fit but also increases the risk of overfitting.

Classification

Classification predicts a discrete category. In binary classification,

A model may output either a class label or an estimated probability

Logistic Regression

Logistic regression maps a linear score to a probability:

The logistic function constrains the model output to the interval \((0,1)\).

Odds and Log-Odds

If \(p\) is an event probability, the odds are

The log-odds are

Logistic regression models the log-odds as a linear function of the features.

Decision Thresholds

A probability model becomes a class decision only after a threshold is chosen:

Confusion Matrix

Binary classification outcomes can be organized into

• true positives;
• false positives;
• true negatives;
• false negatives.

Different performance metrics summarize different combinations of these counts.

Accuracy

Accuracy is

Accuracy can be misleading when classes are highly imbalanced.

Precision and Recall

Precision is

Recall is

Precision emphasizes reliability of positive predictions, while recall emphasizes finding positive cases.

F1 Score

The F1 score is the harmonic mean of precision and recall:

It is useful when both false positives and false negatives matter.

Probability Calibration

A classifier can rank cases well while producing poor probability estimates. A calibrated model should satisfy, approximately,

Calibration is especially important when probabilities support decisions.

Unsupervised Learning

Unsupervised learning uses data without target labels and seeks structure in the feature space.

Common tasks include clustering, dimensional reduction, density estimation, and anomaly detection.

Clustering

Clustering groups observations according to a similarity or distance rule.

A clustering result is not automatically a discovery of natural or causal categories. It depends on representation, scaling, distance, and algorithm.

K-Means Clustering

K-means assigns observations to \(K\) cluster centers and minimizes within-cluster squared distance:

Here \(c_i\) is the cluster assigned to observation \(i\).

Distance Matters

Euclidean distance between vectors is

Features measured on larger numerical scales can dominate this distance unless the representation is standardized appropriately.

Feature Scaling

A standardized feature is

Standardization gives the feature mean \(0\) and standard deviation \(1\) on the dataset used to estimate \(\mu\) and \(\sigma\).

Training-Time Transformations

Scaling, imputation, feature selection, and other learned preprocessing steps must be estimated using training data only.

Applying information from validation or test data during preprocessing creates leakage.

Dimensional Reduction

High-dimensional data can contain redundant or correlated directions. Dimensional-reduction methods seek lower-dimensional representations that preserve selected structure.

Principal Component Analysis

Principal component analysis, or PCA, finds orthogonal directions of maximum variance in centered data.

If \(\Sigma\) is a covariance matrix, principal directions satisfy

Eigenvectors with larger eigenvalues capture more variance.

Explained Variance

If PCA eigenvalues are

the fraction of total variance explained by component \(j\) is

Train, Validation, and Test Sets

A common workflow separates data into

• training data for fitting parameters;
• validation data for model and hyperparameter choices;
• test data for final evaluation.

The test set should remain untouched until model development is complete.

Generalization

Generalization is the ability to perform well on new data drawn from the intended deployment environment.

Low training error alone does not demonstrate generalization.

Overfitting

Overfitting occurs when a model learns noise or idiosyncratic structure in the training data that does not transfer to new data.

A common symptom is very low training error combined with substantially worse validation error.

Underfitting

Underfitting occurs when a model is too simple or insufficiently trained to capture important structure.

Both training and validation error may remain high.

Bias-Variance Tradeoff

Flexible models can reduce approximation bias but may become more sensitive to sampling variation. Simpler models may be more stable but less expressive.

Model complexity should be matched to data quantity, noise, and the prediction task.

Cross-Validation

In \(K\)-fold cross-validation, data are divided into \(K\) parts. Each part is used once for validation while the remaining \(K-1\) parts are used for training.

The average validation score estimates out-of-sample performance more efficiently than a single split in many settings.

Time-Series Validation

Random train-test splitting is inappropriate when future observations must be predicted from past information.

Time-series validation preserves temporal order:

Data Leakage

Leakage occurs when information unavailable at prediction time enters model training or feature construction.

Leakage can produce unrealistically strong validation results that collapse in deployment.

Loss Functions

Training requires an objective. A generic empirical risk is

The loss function determines which kinds of errors the training process penalizes.

Mean Squared Error

For regression,

Mean Absolute Error

Another regression loss is

MAE gives large residuals less leverage than squared error.

Cross-Entropy Loss

For binary probabilistic classification,

Confident incorrect probabilities are penalized strongly.

Regularization

Regularization penalizes model complexity. A general objective is

The parameter \(\lambda\) controls the tradeoff between fit and penalty.

L2 Regularization

Ridge-type regularization uses

It discourages large coefficients and can stabilize models with correlated features.

L1 Regularization

Lasso-type regularization uses

L1 penalties can drive some coefficients exactly to zero and therefore perform a form of feature selection.

Optimization

Training many models requires numerical optimization:

For differentiable objectives, gradients provide local information about how the loss changes with parameters.

Gradient Descent

Gradient descent updates parameters by

The learning rate \(\eta\) controls step size.

Learning Rate

If the learning rate is too small, training may be unnecessarily slow. If it is too large, the algorithm may oscillate or diverge.

Learning-rate selection is therefore part of implementation, not merely a software setting.

Stochastic Gradient Methods

Large datasets often estimate the gradient from a subset, or mini-batch, of the training data.

This reduces computation per update and introduces gradient noise that can be useful in large-scale optimization.

Decision Trees

Decision trees recursively partition feature space into regions and assign a prediction within each region.

Trees are flexible and interpretable locally but can overfit when grown without constraint.

Ensemble Models

Ensembles combine predictions from multiple models:

Averaging can reduce variance, while sequential ensembles can focus later models on errors made by earlier ones.

Random Forests

Random forests combine many decision trees trained on resampled data with randomized feature selection.

Their diversity reduces the variance associated with a single tree.

Boosting

Boosting builds models sequentially so later learners focus on errors not yet captured adequately.

The resulting model can be highly predictive but requires careful regularization and validation.

Nearest-Neighbor Methods

A nearest-neighbor model predicts using observations located near a query point in feature space.

Performance depends heavily on the distance metric, feature scaling, and local data density.

Kernel Methods

Kernel methods compare observations through a similarity function

They can represent nonlinear relationships without constructing every nonlinear feature explicitly.

Neural Networks

A neural network composes linear transformations with nonlinear activation functions. A single layer may be written

Multiple layers create flexible function approximators.

Parameters in Neural Networks

Weights and biases are estimated by minimizing a training loss, usually with gradient-based optimization.

Large neural networks can contain millions or billions of parameters, making data quality, regularization, evaluation, and computational design especially important.

Representation Learning

Traditional models often depend on manually designed features. Deep learning can learn intermediate representations jointly with the prediction task.

This is especially useful for images, audio, text, and other high-dimensional structured data.

Classification versus Ranking

Some applications care less about the final class threshold and more about ordering observations by likelihood or priority.

Ranking quality and probability calibration are distinct performance goals.

ROC Curves

A receiver operating characteristic curve compares true-positive and false-positive rates across classification thresholds.

It measures ranking behavior across thresholds rather than performance at one operational cutoff.

Precision-Recall Curves

Precision-recall curves are often more informative than ROC curves when positive cases are rare and the operational focus is on positive predictions.

Imbalanced Classes

If only \(1\%\) of cases are positive, a model that always predicts the negative class achieves \(99\%\) accuracy while being useless for detecting positives.

Metrics must match the actual task.

Missing Data

Missing values can arise from sensor failure, survey nonresponse, incomplete records, or data-integration problems.

The consequences depend on why the data are missing, not merely on the fraction missing.

Imputation

Imputation replaces missing values using a rule or predictive model.

Imputation must be fitted using training information only, and uncertainty introduced by missingness should not be ignored when it materially affects the decision.

Outliers

Extreme observations may represent errors, rare valid cases, regime changes, or the most important events in the dataset.

Removing them automatically can destroy information about tail behavior.

Feature Engineering

Feature engineering constructs informative model inputs from raw data.

Examples include

• ratios and rates;
• lags and rolling averages;
• interaction terms;
• log transformations;
• cyclical encodings;
• domain-specific physical or operational quantities.

Feature Selection

Feature selection removes inputs that add noise, instability, cost, or complexity without sufficient predictive value.

Selection should occur within the training and validation process to avoid optimistic evaluation.

Correlation Does Not Establish Causation

Predictive models learn statistical relationships. A feature can be highly predictive because it is a cause, an effect, a proxy, or simply correlated with an unobserved common factor.

Causal interpretation requires additional design or assumptions.

Causal Modeling

Causal analysis asks how outcomes would change under interventions.

The relevant quantity may be a contrast such as

This is conceptually different from predicting \(Y\) from observed features.

Distribution Shift

A model may fail when deployment data differ from training data.

Changes can occur in

• feature distributions;
• outcome prevalence;
• relationships between features and targets;
• measurement systems;
• behavior or policy.

Concept Drift

Concept drift occurs when the relationship between inputs and outcomes changes over time.

Models used in dynamic environments may require monitoring, retraining, or explicit time-varying structure.

Monitoring

Production models should be monitored for

• input drift;
• missingness changes;
• prediction drift;
• performance decay;
• calibration changes;
• latency or system failures.

Uncertainty in Prediction

A point prediction gives one number. Decision-making often requires a range or probability distribution.

A prediction interval may be written

Its interpretation depends on the statistical method used to construct it.

Aleatoric and Epistemic Uncertainty

Aleatoric uncertainty refers to irreducible variability in the data-generating process. Epistemic uncertainty refers to uncertainty in the model or parameters that may shrink with better information.

The distinction is useful when deciding whether more data can materially improve performance.

Probabilistic Models

Probabilistic models represent a conditional distribution

This framework supports uncertainty quantification, likelihood-based training, and principled comparison with observations.

Maximum Likelihood

Maximum-likelihood estimation chooses parameters that maximize the probability of the observed data:

Equivalently, one often minimizes negative log-likelihood.

Bayesian Learning

Bayesian inference combines a prior distribution with the likelihood:

Predictions then average over parameter uncertainty rather than relying only on one fitted parameter value.

Mechanistic and Data-Driven Models

A mechanistic model encodes known relationships such as conservation, geometry, or physical law. A data-driven model learns relationships primarily from observations.

Neither approach is universally superior.

Hybrid Models

Hybrid models combine mechanistic structure with learned components:

This can improve data efficiency and preserve constraints while still capturing unresolved behavior.

Physics-Informed Learning

Physical equations can enter training through architecture, constraints, or penalty terms.

A generic loss might be

The physical term penalizes violation of known relationships.

Interpretability

Interpretability concerns how readily model structure or predictions can be understood.

Linear coefficients, small trees, feature effects, local explanations, and sensitivity measures provide different kinds of interpretability.

Interpretability Is Model-Dependent

A coefficient in a linear regression, a split in a decision tree, and an attribution score from a complex model do not carry the same meaning.

Explanations should be interpreted according to the underlying model and assumptions.

Fairness and Group Performance

Aggregate accuracy can hide large performance differences across subgroups.

When decisions affect people, evaluation may need to compare error rates, calibration, coverage, or other metrics across relevant groups.

Objective Functions Encode Priorities

The loss function, class weights, threshold, and optimization target determine which errors are treated as costly.

These choices are part of model governance because they translate operational priorities into mathematics.

Data Quality

Model performance is constrained by the quality of the underlying data. Important questions include

• Are measurements accurate?
• Are labels reliable?
• Is sampling representative?
• Are timestamps aligned?
• Are units consistent?
• Are duplicated or impossible records present?

Sampling Bias

If the training sample differs systematically from the target population, model performance estimates can be misleading.

More data do not automatically correct biased sampling.

Label Noise

Classification labels may contain mistakes, ambiguity, or inconsistent human judgment.

Label quality can become the limiting factor even when feature data are abundant.

Reproducibility

A reproducible machine-learning workflow records

• data version;
• feature transformations;
• training split;
• random seeds where relevant;
• model configuration;
• software environment;
• evaluation metrics;
• final artifacts.

Computational Cost

Model quality is not the only deployment criterion. Training time, inference latency, memory, energy use, and hardware requirements may matter.

A slightly less accurate model may be operationally superior if it is far cheaper, faster, or easier to maintain.

Model Compression

Large models can sometimes be approximated by smaller models through pruning, quantization, distillation, or low-rank approximation.

Compression introduces a tradeoff among size, speed, accuracy, and robustness.

Example: Linear Prediction

Suppose

For

the prediction is

Example: Logistic Probability

Suppose a logistic model produces score

Then

Example: Mean Squared Error

Suppose observed values are

and predictions are

The residuals are

Therefore,

Example: Standardization

Suppose a feature value is \(x=70\), the training mean is \(\mu=50\), and the training standard deviation is \(\sigma=10\). Then

Example: Precision and Recall

Suppose

Precision is

Recall is

Verification

Verification asks whether the data pipeline, transformations, mathematics, and software implement the intended model correctly.

Useful checks include

• shape and dimension checks;
• unit and scaling checks;
• reproducing small analytical examples;
• confirming split logic;
• testing metric calculations;
• checking gradient or optimization behavior;
• verifying preprocessing is applied consistently.

Validation

Validation asks whether the model performs adequately on data representative of its intended deployment setting.

A model can be implemented perfectly and still fail because the data are unrepresentative, the target is poorly defined, the environment has shifted, or the evaluation metric does not match the operational goal.

Common Error: Training on the Test Set

Repeatedly checking the test set and adjusting the model in response turns the test set into a validation set.

Final performance will then be optimistically biased.

Common Error: Leakage through Preprocessing

Computing means, standard deviations, feature selections, or imputations using the full dataset leaks information from validation or test data into training.

Common Error: Optimizing One Metric Blindly

A high accuracy score can hide poor calibration, weak minority-class recall, unacceptable false positives, or poor tail performance.

Metrics should reflect the actual operating decision.

Common Error: Assuming More Complexity Is Better

A more complex model can fit more patterns but can also become less stable, harder to validate, more expensive to operate, and more vulnerable to overfitting.

Common Error: Treating Correlation as Causation

Predictive association does not establish how an intervention would change an outcome.

Causal claims require additional assumptions or study design.

Common Error: Ignoring Distribution Shift

Excellent historical performance does not guarantee future performance if the deployment population, measurement process, policy environment, or behavior changes.

Common Error: Ignoring Uncertainty

Point predictions can be misleading when several actions have similar expected outcomes but very different uncertainty or downside profiles.

Decision systems should use calibrated probabilities or intervals when uncertainty materially affects the choice.

A Reliable Modeling Procedure

1. Define the prediction, inference, or decision objective.
2. Identify the target population, target variable, and available features.
3. Audit data quality, sampling, missingness, units, and label reliability.
4. Separate training, validation, and test information before learned preprocessing.
5. Choose a baseline model and evaluation metric matched to the decision.
6. Train models using an appropriate loss function and regularization strategy.
7. Evaluate generalization with cross-validation or time-aware validation as appropriate.
8. Check calibration, subgroup performance, uncertainty, and failure cases.
9. Compare against simpler and mechanistic alternatives where available.
10. Monitor deployment data for drift, performance decay, and changing assumptions.

Practice 1: Linear Prediction

A model is

Find \(\hat y\) when \(x_1=5\) and \(x_2=2\).

Practice 2: Residual

The observed value is \(12\) and the model predicts \(9.5\). Find the residual.

Practice 3: Mean Squared Error

Observed values are

and predictions are

Find the mean squared error.

Practice 4: Logistic Probability

A logistic model produces \(z=0\). Find the predicted probability.

Practice 5: Accuracy

A classifier has

Find accuracy.

Practice 6: Precision and Recall

For the same classifier,

Find precision and recall.

Practice 7: Standardization

A feature value is \(85\), the training mean is \(70\), and the training standard deviation is \(5\). Find the standardized value.

Practice 8: PCA Explained Variance

Three PCA eigenvalues are

What fraction of total variance is explained by the first component?

Practice 9: Detect Leakage

A team standardizes every feature using the mean and standard deviation computed from the entire dataset before splitting into training and test sets. Explain the problem.

Practice 10: Choose the Better Validation Design

A model predicts next month's demand using several years of historical data. Explain why a random train-test split may be misleading and identify the better validation direction.

Practice Answers

Practice 1

Practice 2

Practice 3

The residuals are

Therefore,

Practice 4

Practice 5

Practice 6

Practice 7

Practice 8

Total variance is

Therefore, the first component explains

That is \(60\%\) of total variance.

Practice 9

The preprocessing uses information from the future test set to transform the training data. This is data leakage. The mean and standard deviation should be estimated from the training data and then applied unchanged to validation and test observations.

Practice 10

A random split allows later observations to influence training while earlier observations appear in testing, which does not match the real forecasting task. A time-ordered validation or rolling-origin design should train on earlier periods and evaluate on later periods.

What to Remember

• data science models depend on both mathematical algorithms and the quality of the surrounding data pipeline;
• regression predicts quantitative outcomes, while classification predicts categories or probabilities;
• clustering and dimensional reduction seek structure without target labels;
• loss functions and regularization define what the training process rewards and penalizes;
• training performance does not establish generalization;
• validation design must match the deployment setting, especially for time-dependent data;
• data leakage can make weak models appear excellent;
• prediction, causal inference, calibration, ranking, and decision-making are different objectives;
• distribution shift and concept drift can invalidate models after deployment;
• verification, validation, uncertainty analysis, monitoring, and reproducibility are essential parts of a production-quality modeling system.

Where We Go Next

Data science and machine learning show how mathematical models can infer patterns directly from observations and improve through optimization. In many engineered systems, however, prediction is only one part of the problem. The model must also regulate a changing physical process through feedback.

The next lesson, Control Systems and Robotics, develops models of state, feedback, stability, sensors, actuators, transfer functions, state-space systems, trajectory tracking, estimation, and automated control.