Reward & stop conditions

The two parts of an auto-research loop where teams quietly fool themselves. Both warrant their own discussion.

Reward: what the loop optimizes

Whatever you measure becomes the goal. The agent will find ways to maximize your reward that you did not anticipate. Some of those ways are useful. Some are degenerate.

Common degenerate optima

The fix is not "find the perfect metric." There isn't one. The fix is:

1. Use multiple, opposing metrics. Quality up, length down, calibration stable. A win must improve at least one without regressing others.
2. Hold out a clean eval the loop never sees. Optimize on a dev signal; periodically check the held-out signal. If they diverge, the loop is gaming the dev signal.
3. Spot-check predictions by hand. Every K trials, look at the actual outputs. The metric will lie before your eyes will.

Reward shaping is a tax, not a gift

Designing a reward signal takes more thought than designing the model. A sloppy reward will dominate any modeling cleverness. Spend the time.

A test for whether your reward is well-formed: describe a "perfect" trial output. Would a human rate it as a great result? If your perfect output is weird (200 hedge words, exact target length, no real content), your reward is wrong.

Stop conditions: when to quit

Without a stop condition, you spend money. With a bad one, you stop too early or too late.

The four useful conditions

Budget

The hard cap. Trials, dollars, wall-clock. Always present. Never optional.

Saturation

You hit the target. Stop and ship. Tempting to keep tuning for the last 0.5 points; usually a waste because you're now in noise.

Plateau

No improvement over the best in the last N trials. N depends on the search:

• Random search: N = 20–30 (lots of room to get lucky).
• Bayesian: N = 8–10 (already exploiting; flat means converged).
• Grid: N = remaining trials in the grid.

Pathology

Last K trials crashed or returned nonsense. The loop is broken or the search space is poisoned. Stop and debug; do not let it burn the budget.

What about "the LLM thinks we're done"?

LLM-driven stop is fine as a tiebreaker. It is a bad primary signal because:

• The LLM is not calibrated; "this looks promising" can mean anything.
• It is gameable from history (a string of wins makes the LLM optimistic beyond what the metric supports).
• It is expensive (one extra call per trial).

Use it for:

• Detecting "we are not really exploring; the proposer is stuck on similar configs" — pattern recognition the LLM is good at.
• Suggesting the next direction once the current one stops; not the same as a stop signal.

Robust stopping rules in practice

A composite rule that works for most search problems:

def should_stop(history: list[TrialResult], target: float, plateau_n: int = 10) -> tuple[bool, str]:
    if not history:
        return False, ""

    # Pathology
    last_k = history[-5:]
    if len(last_k) >= 5 and all(r.crashed for r in last_k):
        return True, "5 consecutive crashes"

    # Saturation
    best = max(r.metrics.get("primary", -float("inf")) for r in history if not r.crashed)
    if best >= target:
        return True, f"target {target} reached: {best}"

    # Plateau
    if len(history) >= plateau_n + 5:
        recent_best = max(r.metrics.get("primary", -float("inf")) for r in history[-plateau_n:])
        prior_best = max(r.metrics.get("primary", -float("inf")) for r in history[:-plateau_n])
        if recent_best <= prior_best:
            return True, f"no improvement in last {plateau_n} trials"

    return False, ""

Simple. Mechanical. Auditable. Add the LLM judgment as a separate, advisory signal if you want.

Reward and stop together: the kill switch test

Before launching any loop, ask: if this loop runs in a degenerate way for 12 hours, what will I see at hour 12?

• Will I be able to tell from tail -f that something is wrong?
• Does the budget cap actually kick in?
• Do the artifacts have enough metadata to debug post-hoc?
• Is there a pkill command I can run from another terminal?

If any answer is "no," do not start the loop. Fix the harness first.

Reporting

When the loop stops, the report should include:

• Why it stopped (which condition fired).
• Best trial config + metrics.
• Per-dimension sensitivity (which knobs mattered most).
• Recommended next experiment (what to try next, given what we learned).
• Audit table: claimed best matches the source-of-truth file.

Without these, the next campaign does not learn from this one.