Numbers must be measured

The core rule, restated: every number in user-facing material comes from a measurement script in the repo, or a cited public source with a working URL. Nothing else.

Why this needs its own chapter

The single highest-impact discipline. A team that publishes ten correct numbers a quarter beats a team that publishes a hundred plausible-looking ones, because the second team's reputation collapses the first time someone spot-checks.

The agent fabricates if you let it. It does not flag the fabrication. Only you can.

The taxonomy of "where numbers come from"

Source	Acceptable?	What to write
Script committed to the repo	Yes	"X measured by `bench/run.py`, see `bench/results/2026-05-04.json`"
Public benchmark with cited URL	Yes	"Y reported in Paper Title"
Internal estimate from agent / engineer	No	Leave blank, write "TBD"
"Standard for the industry" / common knowledge	No unless cited	Cite or omit
Number from a previous version of the doc	No unless re-measured	Re-measure or remove

Two rules collapse the table:

Cite the source. Commit it or link it.
If you cannot, leave the cell empty.

What "measured" actually requires

A script existing is necessary. The protocol must also be sound. From the benchmark methodology, specifically:

Throughput / latency: warmup ≥3 calls; report best-of-N (N≥3); state the protocol in the result. Cold-start artifacts inflate ratios.
Comparison targets: pin and report the third-party tool version (e.g. underthesea==9.4.0) and the bench date. Re-run when you bump versions.
Quality metrics: state the corpus size, license, and registers covered. A 50-sentence test set is not a 5,000-sentence test set.
Single-run results don't count. They're noisy. Best-of-N or median.

A "measured" number that violates these is not measured well enough to publish.

Empty is honest. Fake is not.

The instinct to fill every cell of every table is strong. Resist it.

A model card that says "latency: TBD" is a working in progress. A model card that says "latency: ~50ms" with no script is a public claim you cannot defend. Pick the first.

A blog post that compares your model to three others, and only has measured numbers for two, should:

Drop the third column, or
Mark it "not measured," or
Take a day to actually measure it.

Anything else is fiction.

How AI fabrication slips through review

The fabricated number looks like a measured one. The reviewer sees:

"Inference latency: 23ms (NVIDIA L4)"

…and assumes someone measured it. Nothing visually distinguishes a real number from a hallucinated one. The only defense is forcing the citation right next to the number:

"Inference latency: 23ms — median over 7 trials after 3 warmups, see benchmarks/results/2026-05-04.json in the repo, NVIDIA L4, batch=1, fp16."

A reader who follows the link can verify. A reviewer can spot the absent link. The agent reading this template knows the format that is expected.

Convention for numeric claims

In any new file, follow this format:

| Metric            | Value       | Source                                                             |
| ----------------- | ----------- | ------------------------------------------------------------------ |
| F1 (test)         | 0.834       | `eval/run.py` → `runs/2026-04-28-final/metrics.json`               |
| Inference latency | 23ms median | `benchmarks/run_latency.py` → `benchmarks/results/2026-05-04.json` |
| Training time     | 4.2 hours   | `runs/2026-04-28-final/log.txt` line 1247                          |
| Cost              | $1.81       | L4 spot @ $0.43/hr × 4.2 hours                                     |

Every row has a verifiable source. Empty source = remove the row.

Auditing existing docs

Run the audit periodically:

#.claude/agents/number-auditor.md
For each numeric claim in the file:

1. Find the source citation (script path, URL, log file).
2. If no citation, flag it as UNCITED.
3. If a script citation, run the script and compare. Flag drift > 5%.
4. If a URL citation, fetch and confirm the number still appears.
   Report a table. Do not edit the file.

Run on every model card before release. Run on the README quarterly.

A near-miss anecdote

A "~800ms time-to-first-token on Jetson Orin Nano" line appeared in marketing copy for one of my projects. It was an estimate. The hardware had not been tested. The number was removed before publication.

This is what happens when the rule is enforced. Better to remove an unverified claim than ship one.

Summary card

Print this. Stick it next to the model-card template:

Every number ships with a citation. Empty cells are fine. Fake numbers are not.

Three lines. Save you years of recovery from a bad headline.