Guardrails checklist

The consolidated list. Use it before merging any AI-generated PR that touches data, training, eval, or anything customer-facing. Two minutes. Saves hours.

Pre-merge checklist

Code

• No absolute paths in src/ (config or CLI args only).
• No magic numbers in code (typed config object).
• No try/except: pass — every exception is either re-raised, logged, or asserted on.
• No silent fallbacks (return 0.0 in metrics, return None in loaders).
• No bare return to skip — use pytest.skip(reason="...") instead.
• No mocked tests claiming to test the real model/system.
• No new dependencies without lock-file update + smoke test.

Data

• Splits are deterministic (seed pinned).
• Group-leakage check passes (assertion in code).
• Time-leakage check passes (if time-aware task).
• Preprocessing fits on training data only (assertion or pipeline).
• Provenance column present on assembled datasets.
• Dataset references use pinned revisions, not bare repo IDs.

Training

• Seed set across NumPy, Python random, framework, CUDA.
• Config snapshot saved to run dir.
• Environment snapshot (pip freeze, git rev) saved to run dir.
• Smoke run (10 steps, tiny data) completes and loss decreases.
• Checkpoints written to a new run dir (no overwrites).

Eval

• Eval loads from a saved checkpoint, not the in-memory model.
• Test set is held out from any tuning.
• Comparison vs. baseline included.
• Confidence interval / variance reported (CV folds or seed reruns).
• Hand-inspected at least 20 predictions, including failures.

Documentation

• Every reported number has a script reference (see benchmarks/run.py).
• Every comparison number has a competitor version + bench date.
• Every model card cites a real author + contact email in the author block.
• Updates in same commit as the code/data change that produced them.
• No hardcoded relative paths in doc snippets.

Run-time guardrails

These should fail the build, not be checklist items:

# tests/integration/test_no_leakage.py
def test_train_test_disjoint():...

# tests/integration/test_no_silent_fallback.py
def test_metric_raises_on_empty():...

# scripts/check_pinned_revisions.py
def main():... # CI fails if any HF dataset/model load is unpinned

If a guardrail can be enforced in code, enforce it in code. Checklists discipline humans. CI disciplines all of us.

Slash commands that automate the checklist

#.claude/commands/preflight.md
Before I merge, walk the guardrails checklist (docs/05-failure-modes/guardrails.md)
against the current diff.

For each item, report PASS / FAIL / N/A with a one-line reason.
Do not edit anything. Just report.

Five minutes, every PR. Catches most regressions at the latest moment they're cheap to fix.

Calibrating strictness

Not every project needs every guardrail. A throwaway notebook does not need the model card check. A research prototype does not need the leakage assertion.

The rule of escalation:

Match the rigor to the audience. Over-applying guardrails to throwaway code slows iteration. Under-applying to released code creates the public mistakes that take quarters to recover from.

When a guardrail catches something

Treat it as a small win. Document the near-miss:

# CLAUDE.md → Things that have bitten us

- (2026-05-04) Group leakage check caught a 12-point inflated F1 in
  PR #142. Root cause: switched from GroupKFold to KFold during a
  "simplification" refactor. Reverted; added test that asserts the
  splitter type explicitly.

The list grows, the agent reads it, the same mistake stops happening.