Hypothesis → experiment → measure

The loop survives 100× speedup only if each step is structured. Here is the structure I use.

Hypothesis: write it before you code

A good hypothesis is a sentence with three parts:

If <change>, then <measurable outcome>, because <mechanism>.

Examples:

• If we add character n-gram features, then word-level F1 improves by ≥1 point on the test set, because current errors are dominated by OOV tokens.
• If we replace ReLU with GELU in the FFN, then validation perplexity drops by ≥0.05, because recent papers show smoother activations help in this size regime.
• If I use the agent's plan-then-act loop, then multi-step task accuracy rises by ≥5 points, because the model fails on tool-call ordering, not capability.

Bad hypotheses (do not start running the agent):

• "Try a bigger model." (No mechanism. No falsifiable outcome.)
• "Improve accuracy." (Not a hypothesis, a wish.)
• "Refactor the trainer." (Not a hypothesis, a chore.)

Experiment: design the smallest falsifying run

The cheapest experiment that can kill the idea wins. Design it before you ask the agent to implement.

The 30-minute falsifier is the unit of work. Anything longer and you are confirming, not exploring.

Measure: pick the metric before the run

The most common failure: defining the metric after seeing the result. This guarantees you will pick the metric that flatters the result.

Three rules:

1. Pin the metric in the hypothesis. "F1 on test set v3, computed by eval/eval_v3.py."
2. Pin the comparison. "Beats baseline B (logged at runs/2026-04-12-baseline) by ≥1 absolute point."
3. Pin the decision. "If yes, promote to full data. If no, drop the line of work."

Without (3), you generate experiments forever and ship none.

A worked example (good)

# Experiment: 2026-05-04-charngram

## Hypothesis

If we add 3–5 char n-gram features to the LR baseline, then macro-F1 on
test_v3 improves by ≥1 point, because error analysis shows 38% of errors
are on OOV tokens.

## Cheapest falsifier

Train on 5% sample, eval on full test_v3. ~6 minutes.
Threshold: improvement < 0.3 points → kill the idea (5% is noisy but a
real effect should still show up).

## Comparison

Baseline: runs/2026-04-12-baseline (macro-F1 0.612).

## Decision

- If sample F1 ≥ 0.617 → run full training (~2 hours).
- If sample F1 < 0.617 → drop, document negative result.

## What I'm willing to be wrong about

The mechanism. Maybe OOV is a symptom, not the cause. If char-ngrams help
on tokens that are not OOV, that is more interesting than the headline number.

A worked example (bad)

# Try char ngrams

Adding char ngram features. Will report results.

This is what the agent will produce if you do not give it the structure. It will run, report a number, and you will be unable to tell whether the idea worked or not — because there was nothing to falsify.

Decisions before data

The whole point is that you make the decision before you see the number. This is the only protection against motivated reasoning.

If you find yourself thinking "well, 0.4 is close enough to my 0.5 threshold, let me run the full version" — you are p-hacking. Either lower the threshold honestly in advance, or drop the experiment.

Logging the loop

Every experiment leaves a paper trail:

runs/2026-05-04-charngram/
├── hypothesis.md # the markdown above
├── config.yaml # exact hyperparameters
├── env.lock # pip freeze + git rev
├── command.sh # exact command, copy-pasteable
├── metrics.json # the number(s)
├── log.txt # stdout/stderr
└── decision.md # what we did with the result

The agent can produce all of this if you ask. The discipline is asking on every run, not just the ones you remember to write up.