ML Handbook: ML Engineering with AI
Recipes/Chapter 33 of 35

Recipe: running a benchmark

5 min readEdit on GitHub

A benchmark is a runnable measurement script that produces a number you can publish. This recipe walks through writing and running one for inference performance — the most common case.

What you need before starting

  • A trained model in a known location (runs/<id>/checkpoint.pt).
  • A representative input distribution (real samples, not random tensors).
  • Hardware you can describe specifically (GPU model, CPU model, RAM).

If any are missing, fix that first.

The script template

# benchmarks/run_inference.py
import json, time, statistics, argparse, platform
from pathlib import Path
from datetime import datetime, timezone

import torch

from my_pkg.model import load_model
from my_pkg.data import load_inference_samples


def benchmark(
    model,
    inputs: list,
    n_warmup: int = 3,
    n_trials: int = 7,
) -> dict:
    """Best-of-N latency. Warm up before timing."""
    model.eval()
    with torch.inference_mode():
        # warmup
        for _ in range(n_warmup):
            _ = model(inputs[0])

        # measure
        timings = []
        for x in inputs[:n_trials]:
            torch.cuda.synchronize() if torch.cuda.is_available() else None
            t0 = time.perf_counter()
            _ = model(x)
            torch.cuda.synchronize() if torch.cuda.is_available() else None
            timings.append(time.perf_counter() - t0)

    return {
        "n_warmup": n_warmup,
        "n_trials": n_trials,
        "median_ms": 1000 * statistics.median(timings),
        "min_ms": 1000 * min(timings),
        "max_ms": 1000 * max(timings),
        "stddev_ms": 1000 * statistics.stdev(timings) if len(timings) > 1 else 0.0,
        "raw_ms": [1000 * t for t in timings],
    }


def gather_environment() -> dict:
    env = {
        "platform": platform.platform(),
        "python": platform.python_version(),
        "torch": torch.__version__,
        "cuda_available": torch.cuda.is_available(),
        "git_rev": _git_rev(),
        "timestamp_utc": datetime.now(timezone.utc).isoformat(),
    }
    if torch.cuda.is_available():
        env["gpu"] = torch.cuda.get_device_name(0)
        env["cuda"] = torch.version.cuda
    return env


def _git_rev() -> str:
    import subprocess
    try:
        return subprocess.check_output(
            ["git", "rev-parse", "HEAD"], text=True
        ).strip()
    except Exception:
        return "unknown"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--checkpoint", required=True)
    ap.add_argument("--out", required=True)
    ap.add_argument("--n-warmup", type=int, default=3)
    ap.add_argument("--n-trials", type=int, default=7)
    args = ap.parse_args()

    model = load_model(args.checkpoint)
    samples = load_inference_samples(n=max(args.n_warmup, args.n_trials))

    result = {
        "checkpoint": args.checkpoint,
        "environment": gather_environment(),
        "latency": benchmark(model, samples, args.n_warmup, args.n_trials),
    }

    out = Path(args.out)
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_text(json.dumps(result, indent=2))
    print(f"wrote {out}")
    print(f"median: {result['latency']['median_ms']:.2f} ms")


if __name__ == "__main__":
    main()

Running it

uv run python benchmarks/run_inference.py \
    --checkpoint runs/2026-05-04-final/checkpoint.pt \
    --out benchmarks/results/2026-05-04-final-l4.json

Output:

{
  "checkpoint": "runs/2026-05-04-final/checkpoint.pt",
  "environment": {
    "platform": "Linux-6.5.0-...",
    "torch": "2.4.0",
    "cuda_available": true,
    "git_rev": "a1b2c3d",
    "timestamp_utc": "2026-05-04T08:11:43+00:00",
    "gpu": "NVIDIA L4",
    "cuda": "12.4"
  },
  "latency": {
    "n_warmup": 3,
    "n_trials": 7,
    "median_ms": 22.8,
    "min_ms": 21.4,
    "max_ms": 25.1,
    "stddev_ms": 1.3,
    "raw_ms": [22.8, 21.4, 23.5, 22.4, 25.1, 22.7, 22.9]
  }
}

Commit benchmarks/results/2026-05-04-final-l4.json. This file is now the source of truth for that number.

Citing the result in docs

| Metric                  | Value   | Source                                                                                                                    |
| ----------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------- |
| Inference latency (p50) | 22.8 ms | [`benchmarks/results/2026-05-04-final-l4.json`](../benchmarks/results/2026-05-04-final-l4.json), NVIDIA L4, batch=1, fp16 |

The link makes the claim verifiable. The source line states the conditions. The reader can re-run if they want.

Comparison benchmarks

When comparing against another tool/library:

| Tool         | Version | Latency (p50) | Source                                             |
| ------------ | ------- | ------------- | -------------------------------------------------- |
| ours         | 0.4.1   | 22.8 ms       | benchmarks/results/2026-05-04-final-l4.json        |
| competitor-X | 9.4.0   | 31.2 ms       | benchmarks/results/2026-05-04-competitor-x-l4.json |

Both rows must come from runnable scripts in your repo. Pin the competitor's version. Re-run when you bump it. Note the bench date so the reader knows the snapshot is from a specific moment, not "current."

Throughput vs. latency

Latency is per-call time; throughput is calls per unit time. They are different benchmarks:

  • Latency: measure single calls with n_warmup ≥ 3, n_trials ≥ 7, report median.
  • Throughput: measure batch of N calls back-to-back over a longer window (e.g., 60 seconds), report calls/second.

Throughput benefits from batching, async, and pipelining; latency does not. Don't conflate them.

Quality benchmarks

The same template applies, with quality metrics replacing latency. Add:

  • Confidence interval (CV folds, bootstrap, or seed reruns).
  • Per-class breakdown if applicable.
  • A regression set with hand-curated examples (catches "improved benchmark, regressed on the use case").

When the number drifts

You re-run the benchmark and get a different number. Possible causes:

  • Hardware difference. Same GPU type can perform differently across spot instances. Pin instance type if you can; report the variation.
  • Driver / CUDA upgrade. The cuDNN algorithm chosen may have changed.
  • Library upgrade. torch / transformers / etc. version bump shifted the perf curve.
  • Warm cache. Earlier runs warmed kernels that the new run did not. Increase n_warmup.
  • Real regression. The model changed. Or the input distribution did.

Drift > 5% deserves an investigation, not a silent doc update. Find the cause, document it, then update the number with the explanation.

CLAUDE.md rules to enforce

## Benchmarks (hard rules)

- All benchmarks live in benchmarks/. All results in benchmarks/results/.
- Every benchmark warms up ≥3 calls before timing.
- Every benchmark reports best-of-N (N≥3), with N stated in the result.
- Every result includes git_rev, timestamp, hardware identifier.
- Comparison benchmarks pin the competitor version and bench date.
- Numbers in docs link to a result file in benchmarks/results/.