ML Handbook: ML Engineering with AI
Documenting with AI/Chapter 26 of 35

Doc-driven development

4 min readEdit on GitHub

The shortest path to maintainable AI-augmented code: write the doc first, generate the code from the doc, keep them in sync.

The flow

Loading diagram…

The discipline: the doc is the contract. When code and doc disagree, you do not silently update the doc. You decide which is right and update both explicitly.

What "the doc" is

For a new ML feature, the doc has four sections at minimum:

# Feature: <name>

## What it does

One paragraph. Plain English. Audience is "next engineer in 6 months."

## Inputs and outputs

Schemas. Types. Example payloads. Edge cases.

## How to use it

A runnable code block. Copy-pasteable into a Python REPL or notebook.

## Performance characteristics

Throughput, latency, accuracy if relevant. Each number cites the script.

This fits on one screen. The agent generates code from it; reviewers compare implementation to spec.

Why it works with agents

The agent is good at:

  • Generating code from a clear spec.
  • Generating tests that match spec.
  • Detecting drift when you ask "does the code still match the doc?"

The agent is bad at:

  • Inferring intent when no spec exists.
  • Maintaining unwritten conventions.
  • Catching scope creep ("the function now does five things instead of one").

Doc-driven development plays to the strengths and avoids the weaknesses.

A worked example

Bad: "Write a function to dedupe the dataset."

Good:

# data.dedupe

## What it does

Removes near-duplicates from a corpus using MinHash + LSH.

## Input

- `df: pd.DataFrame` with at least column `text: str`
- `threshold: float = 0.85` — Jaccard similarity above which rows are duplicates
- `num_perm: int = 128` — MinHash permutations

## Output

- `pd.DataFrame` with same schema, near-duplicates removed
- Preserves row order: when collapsing a cluster, keeps the first occurrence

## Edge cases

- Empty DataFrame → returns empty
- DataFrame with one row → returns it unchanged
- Identical text → kept first occurrence

## Performance

Median throughput on a 100k-row corpus: <measured value goes here once we run benchmarks/dedupe.py>
Memory: O(n) MinHash signatures.

Now the agent writes the function, you write the test from the spec, the docstring on the function points back to this doc. Everyone knows what "works" means.

Keeping it in sync

The biggest failure mode: doc says X, code does Y, nobody notices for a quarter. Two practices that prevent this:

1. Doctest the code blocks

If the doc has a "How to use it" snippet, that snippet runs in CI. If it breaks, the build fails.

# tests/docs/test_doc_examples.py
import doctest, my_pkg.data
def test_data_doc_examples():
    results = doctest.testmod(my_pkg.data, verbose=False)
    assert results.failed == 0

2. The same-commit rule

Code and its doc move together. The next chapter goes deeper.

The trap to avoid

Don't write so much doc up front that the doc itself becomes a project. The target is one screen of spec for a one-day feature. Bigger features get broken into smaller docs.

The wrong instinct: "I'll write a 5,000-word design doc, then implement." That mode is for genuinely novel architecture. For day-to-day work, a one-page spec is the floor and the ceiling.

When this mode is wrong

  • Pure exploration. You don't know what you're building yet. Write code, get signal, then write the doc.
  • One-off scripts. A throwaway analysis script does not need a spec.
  • Rapid prototyping during a hackathon-style sprint.

But once exploration concludes and the code starts being depended on, write the doc. Code that nobody can describe is code that can't be maintained.