ML Handbook: ML Engineering with AI
Recipes/Chapter 35 of 35

Recipe: debugging with Claude

6 min readEdit on GitHub

Patterns that consistently produce a fix in under an hour.

Mindset

The agent is not a debugger. It is a fast hypothesis generator and code executor. Your job: feed it the right context, constrain its search, verify each "fix" before moving on.

Bad pattern: dump the error, ask "fix this," accept the first patch. That patch usually addresses a symptom, creates two new bugs, and looks like progress.

Good pattern: the steps below.

Step 1 — capture the failure

Before talking to the agent, capture:

  • The exact command that fails.
  • The full stack trace (not just the last line).
  • The relevant config / data manifest.
  • What you expected to happen instead.

Write this in a debug.md scratch file. Sounds bureaucratic; takes 2 minutes; prevents 30 minutes of "let me try this."

# bug-2026-05-04

## Command

make train RUN=2026-05-04-charngram

## What I expected

Training to start, loss to print every 100 steps.

## What happened

RuntimeError: stack expects each tensor to be equal size, but got [128, 768]
and [127, 768] at index 1
[full stack trace]

## What I've already tried

Nothing yet.

Step 2 — hand the agent the focused context

I'm debugging this failure: <paste debug.md>

Your job:
1. Read the relevant source files (point to which: src/data.py, src/train.py).
2. Form 2-3 hypotheses for the root cause. Rank by likelihood.
3. For the top hypothesis, propose a *minimal* check that would confirm
 or deny it. Do not change any code yet.

Three behaviors to demand explicitly:

  • Hypotheses, not fixes. The first patch is rarely the right one.
  • Minimal checks. Add print or assertions, run, observe.
  • No code changes yet. Think first.

Step 3 — confirm the hypothesis before fixing

Run the check the agent proposed. Read the output yourself.

Common pitfall: the agent says "I confirmed the hypothesis" without you having seen the evidence. Look at the evidence. A "confirmed" diagnosis based on a misread log is worse than no diagnosis.

Step 4 — minimal fix

Once the cause is confirmed:

The hypothesis is confirmed: the data loader's last batch is dropping the last
sample because of an off-by-one in the slicing.

Fix:
- Change only the slicing in src/data.py:DataLoader.__iter__.
- Add a unit test in tests/unit/test_data.py that asserts the last sample is included.
- Do not refactor anything else.

The "do not refactor anything else" is important. The agent will, given freedom, also "improve" related code. That improvement is uncontrolled collateral risk during debugging.

Step 5 — verify the fix and the test

Run the failing command. Run the new test. Both should pass.

If they don't, you do not have the fix. Go back to step 2 with new information.

Step 6 — write the lesson

In CLAUDE.md → "Things that have bitten us":

- (2026-05-04) Off-by-one in DataLoader dropped the last sample of every
  epoch. Caught by a stack-trace error, not by a test. Added test_last_sample_included.
  Lesson: every loader needs a length-preservation test.

The next time a similar bug starts to form, the agent reads this entry and avoids it.

Failure-class playbooks

Specific patterns that come up often.

"Loss is NaN"

Hypotheses, in order of frequency:

  1. Learning rate too high → halve and re-check.
  2. Bad input (NaN in data, divide by zero in preprocessing) → log per-batch stats; first NaN appears here.
  3. Numerical instability in loss (log(0), exp overflow) → wrap with stable-version (log_softmax not log(softmax(...))).
  4. Mixed-precision overflow → drop to fp32 to confirm; if so, add gradient scaling.
Loss is NaN at step 200. Add logging that prints per-batch:
- input min/max/mean/has-nan
- output min/max/mean/has-nan
- loss components (if multi-term)
Run for 250 steps. Do not change the model. Show me the log.

"Eval metric dropped after refactor"

Hypotheses:

  1. Refactor changed preprocessing → run old vs. new preprocessing on the same example, diff outputs.
  2. Refactor changed split → check that test set IDs are unchanged.
  3. Refactor changed eval metric implementation → unit-test the metric on a tiny known input.
  4. Random seed changed → grep for seed; verify it's still set.

"Model trains but inference is broken"

Hypotheses:

  1. Tokenizer mismatch (trained with one, serving with another) → assert tokenizer hash matches.
  2. Eval mode forgotten (.train() left on at inference) → assert model.training is False.
  3. Different normalization (different mean/std at inference vs. training) → factor preprocessing into a shared function.
  4. Model loaded with wrong weights (revision mismatch) → log the loaded checkpoint path and hash.

"Tests pass locally, fail in CI"

Hypotheses:

  1. CI has different deps (lock file not used, fresh install picks up newer versions) → pin versions explicitly; use pip install --no-deps.
  2. CI has no GPU and tests silently skip → check for skipped tests.
  3. CI has different filesystem layout (case-sensitive vs. -insensitive) → check for case-mismatched paths.
  4. Race condition exposed by CI's slower / faster execution → look for timing-dependent assertions.

What the agent is bad at debugging

  • Heisenbugs (race conditions, GPU non-determinism). Hard for any tool; agents add hallucinated "fixes." Reproduce minimally first; then debug.
  • Cross-process bugs. Distributed training failures, multi-worker dataloader hangs. The agent's view is one process; you have to provide the cross-process picture.
  • Data quality bugs. "These predictions look weird." The agent will propose model changes; the bug is in the data. Look at the data first.

What to do when an hour passes with no fix

Stop. Reset.

  • Re-read debug.md. Has the symptom evolved? Update.
  • Drop into a Python REPL yourself. Reproduce the bug in 10 lines of code.
  • If you can't reproduce in 10 lines, you don't yet understand the bug. Stop "fixing" until you do.

The agent is fastest when you've narrowed the problem. It is slowest when you ask it to find a needle in a 5,000-line haystack.

Logging the win

When the bug is fixed:

  • Commit the fix and the test in one commit.
  • Update CLAUDE.md if the lesson is general.
  • Close the loop in your team chat / standup. The next person who hits the same shape of bug should find your fix.

This is the compounding part: every debug session leaves a marker the next session can use.