ML Handbook: ML Engineering with AI
Tooling Stack/Chapter 12 of 35

Claude Code as the engine

5 min readEdit on GitHub

I default to Claude Code as the primary implementation agent. The reasoning is in Foundations; this chapter is about configuration that actually pays off.

Why this default (and when to deviate)

PropertyWhy it matters for ML work
CLI binary, not IDE pluginRuns anywhere — local laptop, remote GPU box, CI
Markdown is the configCLAUDE.md, slash commands, subagents — all editable text. No GUI lock-in.
Tool use is first-classBash, file edits, web fetch — all logged, all reviewable
Repo-aware out of the boxReads your code on demand; does not need an indexer to be useful
Subagent + skill ergonomicsCompose specialized roles (reviewer, planner, bench-runner) without a framework

Deviate when:

  • You need a tightly-coupled IDE flow on a teammate's machine — Cursor / Windsurf may be smoother.
  • You need a CLI optimized for very long autonomous runs on a laptop — Aider's edit/diff model has its fans.
  • You're testing a frontier model not yet in Claude Code — use the vendor's own CLI, then port the workflow back.

The patterns transfer. The choice of agent is reversible; the choice of workflow (small repos, hard rules, reproducible runs) is what matters.

Minimum viable setup for a new ML repo

# 1. CLAUDE.md at repo root (see Section 1)
# 2. Slash commands directory
mkdir -p.claude/commands
# 3. Subagent directory (optional but high-leverage)
mkdir -p.claude/agents
# 4. Personal global rules
mkdir -p ~/.claude
touch ~/.claude/CLAUDE.md

That is the whole filesystem footprint. No SaaS, no backend, no auth flow.

Slash commands worth writing on day one

These four cover ~80% of recurring agent tasks in an ML repo.

/bench

#.claude/commands/bench.md
Run the benchmark for the most recent model checkpoint.

Steps:

1. Find the latest run dir under `runs/` by mtime.
2. Source `.venv/bin/activate`.
3. Execute `python benchmarks/run.py --run-dir <latest>`.
4. Read the resulting `benchmarks/results/<run-id>.json`.
5. Update `BENCHMARK.md` with a new row: date, model, metric, run-id.
6. Show me the diff. Do not commit until I approve.

Hard rules:

- Never overwrite an existing row in BENCHMARK.md.
- If the metric script does not exist, stop and ask. Do not invent numbers.

/new-experiment

#.claude/commands/new-experiment.md
Scaffold a new experiment directory.

Args: <slug>

Steps:

1. Create runs/$(date +%Y-%m-%d)-<slug>/.
2. Copy the template hypothesis.md, config.yaml, command.sh from templates/.
3. Pre-fill code revision (git rev-parse HEAD) and timestamp.
4. Open hypothesis.md and ask me to fill in: Hypothesis, Falsifier, Decision.
5. Do not run anything until I approve the hypothesis.

/release

#.claude/commands/release.md
Cut a release.

Steps:

1. Verify all benchmarks in BENCHMARK.md have matching run dirs and scripts.
2. Bump version in pyproject.toml.
3. Update CHANGELOG.md with merged-since-last-tag commits.
4. Re-render any model cards that reference the new version.
5. Show me everything as a single diff.
6. Wait for explicit "go" before tagging or publishing.

/review

#.claude/commands/review.md
Strict review of the current diff.

Voice: paranoid staff engineer.
Check, in order:

1. Hardcoded paths, magic constants, mutable defaults.
2. Data leakage: any test/eval data flowing through training paths.
3. Metric scripts: any silent fallback to 0, NaN handling.
4. Determinism: seeds, sort stability, set ordering.
5. Documentation: every new metric must update the corresponding doc.
6. Tests: every new code path needs at least one test.
   Be specific. Cite file:line for every issue.

Subagents worth having

Subagents are specialized personas with their own tool subset. Three that earn their keep in ML repos:

  • leakage-checker — given a diff, looks for any path where test data could flow into training. Read-only tools (no edits).
  • benchmark-auditor — given a number cited in a doc, finds the script that produced it, verifies it runs, re-runs it, and compares. Read-only + bash.
  • model-card-writer — given a runs/<id>/ directory, drafts a model card by reading config, metrics, and dataset manifest. Edit-only on docs/models/.

The point of subagents is isolation: they cannot accidentally modify anything outside their scope, and their behavior is encoded once.

CLAUDE.md sections that pay off most

the high-leverage sections are:

  1. "Things that have bitten us." A growing list. Each entry prevents one recurrence. The agent reads these and avoids the same mistake.
  2. "Forbidden patterns." Pickle for weights, hardcoded paths, pip install mid-run, etc. Hard rules.
  3. "Definition of done for X." Checklists for "new model," "new metric," "new dataset version." Removes ambiguity from "is this ready?"

The "nice to haves" that I've found less useful than expected: detailed philosophy, long examples, exhortations to "be careful." The agent treats these as ambient text. Concrete rules and concrete checklists move behavior; prose does not.

Cost discipline

Agent runs cost money. Two settings that matter:

  • Use the smaller model for boilerplate (test stubs, refactors, doc rewording). Reserve the larger model for code that touches the loss function or the metric.
  • Keep CLAUDE.md short. It is read on every turn. A 5,000-line manual burns tokens and dilutes signal. Aim for one screen of hard rules + links to deeper docs.

The cost per task with a well-tuned setup is small. The cost per task with sprawling context is not.