Classic vs AI-augmented loop

The loop has the same shape. The economics are different.

The shape (unchanged)

Every textbook draws this. The shape has not changed in 30 years.

The economics (radically different)

Per-lap cost:

The interesting number is not the multiplier on each step. It is the change in which step is the bottleneck.

In the classic regime, "implement" dominated. You had limited shots; you chose carefully. In the augmented regime, idea quality and run cost dominate. You can try more ideas — but shallow ideas waste compute and confuse you.

What this means in practice

Spend more time on hypothesis quality

The agent will not push back on a vague idea. "Try a bigger model" produces a bigger model. "Try a bigger model — current bottleneck is likely compute, not capacity; this rules it out either way" makes the result actionable in both directions.

A useful template before any agent run:

## Hypothesis

<one sentence — falsifiable>

## How would we know we were wrong?

<the result that would kill the idea>

## Cheapest experiment

<smallest change that produces the falsification signal>

Cap compute per idea

Without a cap, the agent will keep training. Set a budget upfront:

• Wall-clock: e.g., 2 GPU-hours per experiment.
• Data: a 1% sample for early ideas, full data only for finalists.
• Sweep size: 8 trials, not 64. Bayesian optimization can wait until the search space is narrow.

Ratchet up the bar to "publish"

If you run 10× more experiments, you need a higher bar to escalate any one of them to "ship." Otherwise your model card fills with noise.

A simple rule: a result is shippable only if a separate, blinded re-run reproduces it within 0.5σ. This costs a 2× compute tax on finalists but kills 99% of false positives from the noisy regime.

What does not change

• Textbook traps: leakage, lookahead bias, multiple-comparison inflation.
• The need for a sane baseline. If your agentic LLM does not beat TF-IDF + logistic regression, you have not learned what you think you learned.
• The need to look at your data. The agent will not catch that 30% of your labels are mojibake until you look.

The loop is the same. Running it badly used to take a week; now it takes a day. Discipline is the only thing between you and a year of fast garbage.