Reproducibility is non-negotiable

You generate results 5–10× faster. If those results are not reproducible, you generate garbage 5–10× faster. Internalize this chapter before the others.

The four-tuple

Every result in the repo must trace back to exactly four pieces of state:

If any of the four is missing, the result is provisional at best, undefendable at worst.

Determinism in practice

Set seeds across every RNG that touches your run:

import os, random, numpy as np, torch

def seed_everything(seed: int) -> None:
    os.environ["PYTHONHASHSEED"] = str(seed)
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

Then call this once, at the top of every entry point. Not in a library function, not conditionally. The entry point.

For JAX:

key = jax.random.PRNGKey(seed)
# pass key explicitly, never call np.random or random in JAX code

GPU non-determinism is real even with the above (cuDNN convolution algorithms, floating point reduction order). Document the residual non-determinism and budget for it: if a metric fluctuates by ±0.2 between identical runs, your threshold for "improvement" must exceed 0.2.

Pin the environment

A single source of truth for dependencies:

• Python: uv lock or pip-compile → committed requirements.lock. Never rely on pip install resolving consistently.
• System libraries: Dockerfile or Nix file. CUDA version, cuDNN version, glibc.
• Hardware: logged. nvidia-smi -L output saved per run if GPU matters.

Anti-pattern: a requirements.txt with transformers and no version. The agent will install whatever the latest is, and your "same" experiment will produce different numbers next month.

The data is part of the experiment

Code reproducibility is meaningless if you cannot reconstruct the dataset.

Three levels, in order of preference:

1. Immutable, versioned data store. HF Hub revision pin, S3 with versioning, DVC pointers. The bytes are recoverable forever.
2. Manifest with hashes. A JSON or parquet listing every file's path + sha256. You cannot recover the bytes if the source disappears, but you can detect silent drift.
3. Description only. Anti-pattern. Don't.

For derived datasets (filtered, deduped, augmented): commit the derivation script and a hash of the output. The derivation script + base data + script version reconstruct the artifact.

The "rerun in 6 months" test

A practical test for reproducibility: can a colleague (or future-you) run

git checkout <sha>
make reproduce-run RUN=2026-05-04-charngram

and get the same metric within noise tolerance? If no, the result is not reproducible, no matter how thorough the writeup looks.

This is the one test that matters. Run it occasionally on old experiments. The results will surprise you.

Reproducibility for AI agents specifically

AI-generated code has a few extra failure modes:

• Library version drift. The agent imports the latest API; your lock file is stale. Pin and audit.
• Hidden state in ~/.cache. The agent downloads a tokenizer or weights from HF. That cache is not in your repo. Either commit a small fixture or pin the HF revision and document the download.
• pip install mid-run. The agent will sometimes add a dependency on the fly. Forbid this in CLAUDE.md. Dependencies go through the lock file.
• Off-the-shelf eval scripts. "I used the standard eval harness" is not reproducible if you do not pin the harness version. Pin it.

Cost vs. value

Reproducibility costs ~10–20% wall-clock per experiment (logging, manifests, env capture). It saves the catastrophic case:

• The result reported six months ago cannot be reproduced.
• A reviewer asks "how did you get this number?" and you cannot answer.
• A regression happens and you cannot bisect.

Pay the 20% tax. Skipping is false economy.