Experiment tracking

Pick one. Use it on every run. The worst tracker used consistently beats the best tracker used sometimes.

The shortlist

Tool	Strengths	Weaknesses	Pick when
MLflow	Self-host trivially. Pure Python. No vendor lock-in.	UI is functional, not pretty.	You want self-hosted, simple, durable.
Weights & Biases	Best UI for sweeps and comparisons. Strong community.	SaaS cost at scale. Privacy review needed.	You're a team, you want polish, the bill is fine.
Aim	Fast UI, OSS, runs locally.	Smaller ecosystem.	You like Python-only stacks and self-hosting.
TensorBoard	Universal, zero config.	Just metrics — no run registry, no system logs.	Quick prototypes, single-user.
Plain JSONL + a script	Zero deps. Reproducible. Diff-friendly.	You write your own UI.	Tiny repos, auto-research harnesses (see Designing research loops).

For most ML teams in 2026, the answer is MLflow (self-hosted) or W&B (if budget and policy allow). Aim is the dark horse if you want OSS without MLflow's quirks.

For an auto-research harness or a minimal single-purpose repo, plain JSONL in a structured run dir is often better than any of them — the agent can read and reason about the format directly.

What to log on every run

Independent of tool:

# config (snapshot of all hyperparameters)
seed: 1337
model:
  n_layer: 12
  n_embd: 768
data:
  manifest: data/manifests/train_v3.json
  manifest_sha256: 8e2c...
optim:
  lr: 3.0e-4
  batch_size: 32
  max_iters: 5000

# environment
git_rev: a1b2c3d
git_dirty: false
python_version: 3.12.4
cuda_version: 12.4
gpu: NVIDIA L4
hostname: gpu-box-3
started_at: 2026-05-04T09:23:11+07:00

# metrics (over time)
- {iter: 100, loss: 4.21, val_loss: 4.18}
- {iter: 200, loss: 3.87, val_loss: 3.91}
...

# artifacts
- runs/<id>/checkpoint-best.pt # path; not in tracker, in storage
- runs/<id>/eval_predictions.parquet

That set covers 95% of the questions you will want to answer later: what config produced this, can I reproduce it, how did training behave, where are the artifacts.

Anti-patterns

Logging to two systems. "We log to W&B and TensorBoard" means you log inconsistently to both. Pick one.
Logging only the final number. Loss curves catch divergent runs that ended at a flatteringly average value.
Run IDs that are not stable. run-1, run-2, run-final-2 — useless. Use <date>-<slug> or a UUID. Sortable and unambiguous.
No run registry in code. If "which run produced production model v1.4?" cannot be answered by a single git grep, you have a registry problem.

The agent's role

A useful pattern: have the agent (via a slash command or subagent) enforce the logging schema, not implement it from scratch each time.

#.claude/commands/log-check.md
Verify the most recent run logged everything required:

- config.yaml present and complete
- env.lock present
- metrics.jsonl has at least 10 entries
- final metric matches what BENCHMARK.md claims
  Stop and report any missing piece. Do not edit; just report.

Run this before promoting any result. Two minutes of audit saves a week of "we can't reproduce this number."

Storage and retention

Every run dir grows. After a year, you have hundreds. A few rules:

Keep run metadata forever. config.yaml, metrics.jsonl, env.lock — these are tiny. Never delete.
Tier checkpoints. Last N + tagged "shipped" + "best per quarter." Delete the rest after 90 days.
Summarize quarterly. A runs/QUARTERLY/2026-Q2.md with the 5 most important runs and what they taught you. The agent can draft this from metrics + git log.