Agent context files

The highest-leverage category of files in any AI-assisted ML repo is the set agents read to figure out how to behave. CLAUDE.md is the most visible. It is one of several. They form a layered system; teams that treat them as a system out-ship teams that treat them as a single file.

The file family

You do not need all of these. You do need to pick which ones you use, write them deliberately, and keep them honest. The biggest mistake is letting them drift so different agents read different versions of "the rules."

Single source of truth

Pick one file as canonical and have the others link to it. Common pattern:

The alternative — duplicating rules across four files — guarantees they go out of sync within a month. When CLAUDE.md says "use pathlib" and .cursorrules does not mention it, two engineers using two tools generate diverging code.

If you only run Claude Code, collapse this: CLAUDE.md is canonical, skip AGENTS.md. The pattern matters; the file count does not.

Scope: repo, user, organization

Three scopes, three locations:

• Repo (./CLAUDE.md) — anything specific to this codebase. Forbidden imports, dataset conventions, deployment quirks. Checked in. Reviewed in PRs.
• User (~/.claude/CLAUDE.md) — your personal preferences. "Always show diffs in unified format." Not checked in.
• Organization — shared rules across repos. Two options: a separate repo that every project clones into .shared/, or a templating tool (cookiecutter) that stamps the same CLAUDE.md header into new projects.

Do not mix scopes. Personal preferences in a checked-in file confuse new contributors. Repo rules in a personal file disappear for everyone except you.

Initialize on day one

Before the first feature, before the first model — drop the canonical file at the repo root. A starter template (works for CLAUDE.md, AGENTS.md, or both):

# <project name> — agent rules

## Project context (1 paragraph)

What this repo does, who uses it, what it must never break.

## Hard rules (non-negotiable)

- All experiments are reproducible: seed pinned, deps pinned, command logged.
- No metric is reported without a runnable script in `benchmarks/`.
- No dataset paths are hardcoded; load via `config/data.yaml`.
- All training runs log to `runs/<date>-<slug>/` with `config.yaml` + `metrics.json`.
- All public APIs are typed; CI fails on `mypy --strict` errors in `src/`.
- Tests in `tests/unit/` must complete in < 30 seconds. Slow tests live in `tests/slow/`.
- Never commit data, model weights, or `.env`. Use DVC / HF Hub / S3 references.

## Conventions

- Package layout: `src/<pkg>/`, tests in `tests/`, configs in `config/`, runs in `runs/` (gitignored).
- Naming: `train_<model>_<task>.py`, `eval_<model>_<task>.py`. No `final_v2_real_FINAL.py`.
- Commits: imperative mood, one logical change per commit.

## Definition of done for a new model

- [ ] Reproducible training script
- [ ] Eval script with at least one baseline comparison
- [ ] Numbers committed in `BENCHMARK.md` (with measurement script reference)
- [ ] Model card in `docs/models/<name>.md`
- [ ] Smoke test in `tests/unit/test_<model>.py`

## Things that have bitten us (so don't do them again)

<!-- grow this section over time -->

- (yyyy-mm-dd) <one-line lesson + link to commit/issue>

Fits on one screen. Answers 90% of "should I do X?" the agent would otherwise guess.

Hard rules vs. soft preferences

Agents treat everything in these files as load-bearing. Be deliberate about what goes in.

The rule of thumb: if violating it would be a bug, it goes in the rules file. If violating it would be a different opinion, it goes elsewhere.

Slash commands and subagents — the action layer

Rules say what to do. Slash commands and subagents say how to do specific recurring jobs. They are the action layer of the same context system.

.claude/
├── commands/
│ ├── bench.md # /bench → run, capture, commit numbers
│ ├── release.md # /release → version bump, tag, model-card refresh
│ ├── review.md # /review → strict diff review
│ └── new-experiment.md # /new-experiment <slug> → scaffold runs/<date>-<slug>/
└── agents/
 ├── leakage-checker.md # subagent that audits a PR for data leakage
 ├── benchmark-auditor.md # subagent that verifies any number in a doc
 └── model-card-writer.md # subagent that drafts cards from training logs

A slash command for "run a benchmark and commit the result" turns a 12-step ritual into one line. More importantly, the command removes the chance the agent skips step 7 (e.g., "verify the comparison target version") — the markdown spells it out every time.

Update the rules as you learn

The most common failure: writing the rules file once and never touching it. It should grow when:

• The agent makes the same mistake twice. Add a rule that prevents it.
• A subtle convention saves you from a bug. Encode it before you forget why.
• Tribal knowledge ("our test set is in JST, training data in UTC") almost burns someone. Write it down.
• A pattern is forbidden for this repo (e.g., "no pickle for model weights"). Make it explicit.

Cycle time matters. A rule added a week after the bug is half as useful as a rule added the same day.

Things to almost-always include for ML repos

A non-exhaustive list of rules that pay for themselves in any ML codebase:

• Determinism. Every training script accepts --seed and uses it consistently across NumPy, Python, PyTorch/JAX/TF, and CUDA where applicable.
• Config over flags. Long argparse chains are a smell. Use Hydra / OmegaConf / Pydantic-settings. Pin the config in the run directory.
• Eval is a separate script. Never compute the "production" metric inside the training loop. Always re-load the checkpoint and re-evaluate from disk.
• No silent fallbacks. If a config key is missing, crash. Do not default to "sensible" values that mask bugs.
• Data manifest, not data path. Code references data/manifests/train_v3.parquet, not /home/user/Desktop/dataset/.
• Forbidden imports. If your serving layer must not depend on training-only packages, declare it: "no torch.utils.data imports under src/serving/."
• Output schema. Every artifact has a known shape (a metrics.json with documented keys, a model card with documented sections). The agent will fill unknown shapes with garbage; known shapes with truth.

Quarterly audit

Every quarter, re-read the agent context files together (CLAUDE.md, AGENTS.md, slash commands, subagents) and ask:

1. Which rules have been violated and not caught? Either enforce them or delete them.
2. Which rules are stale (the underlying tool changed)? Update or delete.
3. Which rules are aspirational ("we should have 80% coverage") rather than binding? Move to a GOALS.md.
4. Which slash commands are unused? Delete.
5. Are the files still consistent with each other? Resolve drift.

A rules file that has not changed in six months is either perfect (rare) or ignored (common).