The Central Idea

Reproducibility connects a reported result to the full chain of choices that produced it.

The basic structure is

\[ \boxed{ \text{model} + \text{data} + \text{parameters} + \text{software} + \text{workflow} \longrightarrow \text{reproducible result} }. \]

If any essential part of that chain is missing, the result may be difficult or impossible to reconstruct.

What Is Reproducibility?

In computational modeling, reproducibility means that the same computational procedure can be repeated using the documented inputs and settings to obtain the same or appropriately equivalent result.

Reproducibility depends on preserving:

Repeatability, Reproducibility, and Replication

These terms are sometimes used differently across fields, but it is helpful to distinguish three ideas.

All three strengthen confidence in computational work.

Why Reproducibility Matters

Reproducibility supports:

Without reproducibility, even a correct result can become difficult to trust later.

A Result Needs Context

A number such as

\[ 0.734 \]

has little computational meaning by itself.

To interpret it, we may need to know:

Model Versioning

Computational models evolve.

Equations may change, bugs may be corrected, parameters may be renamed, and algorithms may be improved.

Every reported result should therefore be associated with a specific model or code version.

A version identifier might take a form such as

\[ \texttt{v1.4.2}. \]

Why Version Numbers Matter

Suppose version \(1.3\) used one boundary rule while version \(1.4\) corrected that rule.

If an old result does not record its model version, it may be impossible to know which implementation generated it.

Versioning makes computational history traceable.

Source-Control Ideas

Source-control systems help preserve the history of changes to computational code.

A useful record can show:

Reproducibility improves when model results can be linked to a specific revision.

Input Data Provenance

Data provenance records where input data came from and how they were transformed before entering the model.

Useful provenance information includes:

Raw Data Versus Processed Data

It is useful to distinguish original data from transformed data.

A reproducible workflow might follow

\[ \boxed{ \text{raw data} \rightarrow \text{cleaning} \rightarrow \text{transformation} \rightarrow \text{model input} }. \]

Each transformation should be documented or scripted so that the final inputs can be regenerated.

Parameter Records

Every simulation should preserve the parameter values used.

If the parameter vector is

\[ \boldsymbol{\theta} = (\theta_1,\theta_2,\ldots,\theta_k), \]

then the complete vector should be associated with the run.

Default values should not be assumed silently.

Initial Conditions

Dynamic models may depend strongly on their initial state.

If

\[ \mathbf{x}(0)=\mathbf{x}_0, \]

then \(\mathbf{x}_0\) is part of the computational experiment.

Reproducibility requires recording the initial conditions just as carefully as the model parameters.

Boundary Conditions

Boundary conditions can materially alter numerical solutions.

A record should specify whether a boundary is:

Boundary details should never be left to memory.

Numerical Settings

Two runs of the same mathematical model can differ numerically if they use different computational settings.

Important settings may include:

Solver Tolerances

Iterative algorithms often stop when an error or residual becomes sufficiently small.

A stopping rule may be

\[ \|r_k\|<\varepsilon, \]

where \(\varepsilon\) is a numerical tolerance.

Different tolerances can produce different runtime and accuracy, so the tolerance should be recorded.

Random Seeds

Stochastic simulations depend on pseudorandom-number sequences.

A random seed initializes that sequence.

If the same code, generator, seed, and execution conditions are used, the same pseudorandom sequence can often be reproduced.

Seed values should therefore be stored with stochastic runs.

Seed Policies

A reproducible stochastic experiment should define how seeds are assigned.

For example,

\[ s_r=s_0+r, \]

where \(r\) is the run number.

Another strategy is to store an explicit seed list.

The important principle is that seed assignment should be deliberate and recorded.

Random Streams

Complex simulations may use separate random streams for different mechanisms.

For example:

Separating streams can make experiments easier to debug and compare.

Software Environment

Reproducibility may depend on the software environment in which the model runs.

Important information can include:

Dependency Versions

A library update can change numerical behavior.

For example, version \(A\) of a solver may use a different algorithm or default tolerance from version \(B\).

Recording dependency versions reduces ambiguity when results are reproduced later.

Environment Locking

A computational environment can be described explicitly with a list of exact software dependencies.

Conceptually,

\[ \text{environment} = \{ \text{language version}, \text{libraries}, \text{solver versions}, \ldots \}. \]

Recreating the same environment reduces differences caused by software drift.

Run Identifiers

Each simulation should have a unique identifier.

A run ID allows output files, logs, parameter settings, and error messages to be connected to one computational experiment.

A run table might contain:

Run Metadata

Metadata are data about the computational run itself.

Useful metadata include:

Logs

A log records what happened during execution.

Logs may capture:

Logs are especially valuable when a result cannot be reproduced immediately.

Deterministic Workflows

A reproducible pipeline should minimize undocumented manual choices.

Instead of:

“open the file, edit several values, click run, copy the output,”

prefer a scripted sequence such as

\[ \boxed{ \text{load} \rightarrow \text{transform} \rightarrow \text{simulate} \rightarrow \text{analyze} \rightarrow \text{report} }. \]

Configuration Files

Model settings can be separated from program logic.

A configuration file may store:

This makes experiments easier to reproduce without editing source code.

Single Source of Truth

Reproducibility improves when each important setting is defined in one clear place.

If the same parameter appears in several scripts, spreadsheets, and manual notes, inconsistencies can arise.

A single authoritative configuration reduces this risk.

Automated Pipelines

An automated pipeline executes the same sequence of steps every time.

This reduces variation caused by manual intervention.

A computational pipeline may automatically:

  1. load data;
  2. validate inputs;
  3. run simulations;
  4. store outputs;
  5. generate summary statistics;
  6. create figures;
  7. record metadata.

Checksums and File Integrity

Large datasets or important input files may be identified by a checksum or hash.

If the file changes, the checksum changes.

This allows a reproducibility record to confirm that the exact intended input file was used.

Data Snapshots

External datasets may change over time.

If a model uses a live or regularly updated data source, reproducing an old run may require preserving the specific data snapshot used at that time.

Recording only the data source name may not be enough.

Reproducible Parameter Sweeps

Parameter sweeps generate many runs, so reproducibility becomes especially important.

Each run should remain traceable to:

Reproducible Monte Carlo Simulation

A Monte Carlo result depends on both the probability model and the random-number sequence.

A reproducible report should record:

Parallel Computation and Reproducibility

Parallel execution can complicate exact reproducibility.

The order in which computations finish may vary between runs.

Floating-point addition is not perfectly associative:

\[ (a+b)+c \]

may differ slightly from

\[ a+(b+c). \]

Therefore, parallel reductions can produce tiny numerical differences.

Exact Versus Numerical Reproducibility

Reproducibility does not always require bit-for-bit identical output.

In some numerical workflows, a reproduced result may be considered equivalent if differences remain within an accepted tolerance:

\[ |y_{\text{new}}-y_{\text{reference}}| \le\varepsilon. \]

The tolerance should be defined before comparison.

Floating-Point Effects

Different processors, compilers, libraries, or operation orders can produce small differences in floating-point results.

These differences may be harmless, but they should be distinguished from larger discrepancies that indicate a real change in the model or implementation.

Checkpoints

Long-running simulations may periodically save their internal state.

A checkpoint can include:

Checkpoints support restart and help reproduce failures that occur late in a run.

Audit Trails

An audit trail preserves the history of how a computational result was created.

A strong audit trail can answer:

Provenance Graph

A complex workflow can be viewed as a dependency graph.

For example,

\[ \text{raw data} \rightarrow \text{processed data} \rightarrow \text{simulation} \rightarrow \text{summary table} \rightarrow \text{figure}. \]

Reproducibility improves when each output retains a clear link to its upstream inputs and transformations.

Documentation

Good computational documentation should explain:

Documentation should be written for someone who did not create the model.

Reproducible Figures

A figure should ideally be generated directly from stored data using a documented plotting procedure.

Manual adjustments that are not recorded can break reproducibility.

The figure-generation workflow should preserve:

Reproducible Tables

Summary tables should also be generated from defined computational steps rather than manually copied values.

If a reported mean is

\[ \bar{x} = \frac{1}{N} \sum_{i=1}^{N}x_i, \]

the exact dataset and filtering rule used to compute that mean should be known.

Regression Tests

Reproducibility supports regression testing.

A trusted model version can store reference outputs.

After code changes, the same test cases are rerun and compared with the reference values.

Unexpected differences reveal that behavior has changed.

Reference Outputs

A reference output should be associated with:

Otherwise, the reference itself may be ambiguous.

Reproducing a Failure

Reproducibility is valuable not only for successful runs.

If a model fails under one configuration, preserving the run state allows the failure to be recreated and studied.

Important information may include:

Archiving

Long-term reproducibility requires preserving important project artifacts.

An archive may contain:

Common Error: Recording Only Final Outputs

Saving a plot or final number without the configuration that produced it is not a reproducible workflow.

Results should remain linked to their model, data, parameters, and numerical settings.

Common Error: Relying on Default Settings

Software defaults can change between versions.

Important settings should be specified explicitly rather than relying on whatever default happens to be active.

Common Error: Overwriting Data or Results

If each new run replaces the previous output file, the history of the experiment is lost.

Unique run identifiers or versioned output directories help preserve traceability.

Common Error: Using Unrecorded Manual Steps

A workflow that depends on undocumented clicking, copying, renaming, or editing is difficult to reproduce.

Important transformations should be scripted or documented explicitly.

Common Error: Forgetting the Random Seed

A stochastic simulation may be impossible to reproduce exactly if the seed or random-number state is not recorded.

Seed information should be treated as part of the run configuration.

Common Error: Ignoring Software Versions

The same source code can behave differently under different library or compiler versions.

Environment information is therefore part of the computational record.

Common Error: Confusing Reproducibility with Validity

A model can be perfectly reproducible and still be wrong.

Reproducibility shows that the computational procedure can be repeated.

It does not establish that the model is scientifically valid or appropriate for its intended purpose.

A Reliable Reproducibility Workflow

  1. Assign a model or code version.
  2. Preserve the exact input data or a traceable data snapshot.
  3. Record all parameter values.
  4. Record initial and boundary conditions.
  5. Record numerical settings and solver tolerances.
  6. Define and store the random-seed policy.
  7. Record software and dependency versions.
  8. Assign a unique run identifier.
  9. Store logs and run metadata.
  10. Automate data preparation, simulation, analysis, and reporting where practical.
  11. Generate tables and figures from reproducible scripts or configurations.
  12. Preserve reference outputs for regression testing.
  13. Archive the materials needed to rerun the experiment.

Practice 1: Identify Missing Information

A report states that a simulation produced

\[ Y=42.7 \]

but does not record the parameter values.

What reproducibility problem does this create?

Practice 2: Random Seed

A stochastic model is rerun with the same code and parameters but a different seed.

Should the exact same random trajectory necessarily be expected?

Practice 3: Numerical Settings

Two simulations use the same equations and parameters but different time steps.

Are they computationally identical experiments?

Practice 4: Data Provenance

A model input file was cleaned manually, but the cleaning steps were never recorded.

What reproducibility weakness does this introduce?

Practice 5: Run Count Metadata

A Monte Carlo study reports a mean but does not state how many trials were used.

Why is this information important?

Practice 6: Software Environment

A model depends on a numerical library whose algorithm changed between versions.

What should be recorded?

Practice 7: Exact Versus Tolerance-Based Reproduction

Two parallel runs differ only by

\[ 10^{-12} \]

because of floating-point operation order.

Must this automatically be treated as a failed reproduction?

Practice 8: Reproducibility Versus Validation

A simulation can be reproduced exactly by another researcher.

Does that prove the model accurately represents the real system?

Practice Answers

Practice 1

The result cannot be connected reliably to the model configuration that produced it. Without the parameters, the experiment cannot be reconstructed fully.

Practice 2

No. A different seed generally produces a different pseudorandom sequence and therefore may produce a different trajectory.

Practice 3

No. The time step is part of the numerical configuration, so the two runs are not identical computational experiments.

Practice 4

The final model input may not be regenerable from the raw data. The manual cleaning process should be documented or automated.

Practice 5

The number of trials affects Monte Carlo precision and sampling uncertainty. Without it, the reported mean cannot be evaluated properly.

Practice 6

Record the exact numerical-library version, along with other relevant environment information.

Practice 7

No. If the accepted reproducibility criterion allows differences within a defined numerical tolerance, such a tiny discrepancy may be acceptable.

Practice 8

No. Reproducibility shows that the computation can be repeated. Validation is still required to assess whether the model adequately represents the intended system.

What to Remember

Reproducibility turns a computational result into a reconstructable experiment.

The central principle is: preserve enough information that the path from inputs to reported output can be reconstructed, rerun, and audited.

Where We Go Next

Reproducibility ensures that computational results can be regenerated and checked. The final challenge is communicating those results clearly without hiding uncertainty, exaggerating precision, or confusing visual appearance with evidence.

The next lesson, Visualization and Interpretation, develops principles for turning model outputs into informative tables, plots, fields, trajectories, distributions, and comparative displays while preserving the mathematical meaning of the simulation.