Designing research loops

Once the iteration loop is reproducible, the next step is letting an agent drive it. This is the engineering of an auto-research harness: a bounded action space, machine-readable results, and a hard budget.

What "auto-research" needs

Three things, all in the harness, none in the agent:

1. A bounded action space. The agent picks from a small set of moves (next config, next prompt, next data subset). Open-ended "do anything" loops fail.
2. A machine-readable result. After every run, the harness writes a single JSON the agent can read. No log scraping.
3. A budget. Wall-clock, dollars, trials. Hard cap, enforced in code.

Without these three, the loop diverges or burns money silently.

Reference loop

# auto_research/loop.py
from dataclasses import dataclass
from pathlib import Path
import json

@dataclass
class TrialResult:
    config: dict
    metrics: dict  # {"f1": 0.83, "loss": 0.12, ...}
    artifacts: dict  # {"checkpoint": "runs/<id>/best.pt"}
    cost_usd: float
    wall_s: float
    crashed: bool
    reason: str  # populated when crashed or stopped

@dataclass
class Budget:
    max_trials: int
    max_cost_usd: float
    max_wall_s: float

def auto_research(
    question: str,
    propose_next: callable,  # (history) -> next config
    run_trial: callable,     # (config) -> TrialResult
    should_stop: callable,   # (history) -> (bool, reason)
    budget: Budget,
) -> list[TrialResult]:
    history: list[TrialResult] = []
    spent_cost = 0.0
    spent_wall = 0.0

    for trial_idx in range(budget.max_trials):
        if spent_cost >= budget.max_cost_usd:
            break
        if spent_wall >= budget.max_wall_s:
            break
        stop, reason = should_stop(history)
        if stop:
            break

        config = propose_next(history)
        result = run_trial(config)
        history.append(result)
        spent_cost += result.cost_usd
        spent_wall += result.wall_s

        # checkpoint after every trial — losing 10 trials of state is unacceptable
        Path(f"auto_research/state/{trial_idx}.json").write_text(
            json.dumps([h.__dict__ for h in history])
        )

    return history

That is the whole loop. Around 30 lines. Everything interesting is in the three callables.

The three callables

propose_next

Three flavors:

• Bandit / random. Cheap. Good baseline. Sometimes the right answer forever.
• Bayesian optimizer (Optuna, Ax). Smart use of trials. Add when search space has 5+ dimensions and trials are expensive.
• LLM-driven. Read the history, propose the next config in natural language, parse to typed config. Strongest when the search space is structured (e.g., "try a different prompt template," "swap the loss function").

Start with random. Add the others when the cost of a trial justifies smarter selection.

run_trial

Wraps your existing training/eval pipeline. Two non-negotiables:

• Catches its own crashes. Returns TrialResult(crashed=True, reason=...) instead of throwing into the loop. One bad config should not abort the whole campaign.
• Reports cost. Either measured (cloud bills) or estimated (GPU-hours × spot price). Without this, the budget cap is theater.

def run_trial(config: dict) -> TrialResult:
    t0 = time.time()
    try:
        run_dir = run_training(config)  # your existing function
        metrics = run_eval(run_dir)     # your existing function
        return TrialResult(
            config=config,
            metrics=metrics,
            artifacts={"run_dir": str(run_dir)},
            cost_usd=estimate_cost(time.time() - t0),
            wall_s=time.time() - t0,
            crashed=False,
            reason="",
        )
    except Exception as e:
        return TrialResult(
            config=config, metrics={}, artifacts={},
            cost_usd=estimate_cost(time.time() - t0),
            wall_s=time.time() - t0,
            crashed=True, reason=repr(e),
        )

should_stop

The "knows when to quit" function. Common rules:

• Plateau. Best metric hasn't improved over the last N trials.
• Convergence. Variance in last K trials is below a threshold.
• Saturation. Best metric exceeds the target.
• Pathology. Last K trials all crashed.

The agent itself can be the stopper, given history:

def should_stop(history: list[TrialResult]) -> tuple[bool, str]:
    if len(history) < 5:
        return False, ""
    prompt = f"""Given these trial results, should we stop the campaign?
{summarize(history)}

Reply with a single JSON object: {{"stop": true|false, "reason": "..."}}"""
    response = llm(prompt, response_model=StopDecision)
    return response.stop, response.reason

Use sparingly. Default to mechanical rules; reserve LLM judgment for cases that need it.

Anti-patterns

• No budget. "I'll watch it." You won't. Set the cap.
• State only in memory. A crash wipes the campaign. Checkpoint after every trial.
• Mutating the question. question should be immutable across the loop. If the agent rewrites it, you are no longer answering the original.
• Letting LLM judge override mechanical signals. If the metric is up and the LLM says "looks bad, stop," prefer the metric. LLM judgment is the tiebreaker, not the override.
• Running on production GPUs without a kill switch. A loop on real hardware needs pkill access. Document the kill command in the campaign log.

Observability

A campaign produces:

auto_research/<campaign-id>/
├── question.md
├── budget.json
├── trials/
│   ├── 0/
│   │   ├── config.yaml
│   │   ├── result.json
│   │   └── log.txt
│   ├── 1/
│   └── ...
├── state/        # checkpoints after each trial
└── report.md     # final writeup

The report.md is generated at the end (and incrementally if you want). Include: question, budget, trials run, best result, full leaderboard, recommended next experiment. The agent drafts; you verify.

Cost discipline

A campaign of 20 trials on cloud GPUs at $1/hr × 2 hours each = $40. That is the threshold where you absolutely need the budget cap. Above $200, treat the campaign like a small project: written plan, hard cap, daily check-in.

The teams that get bitten are the ones who let "auto-research" spiral into "I forgot to log out of Modal for three days."

When to use this

Today, useful for:

• Hyperparameter sweeps that are too irregular for a clean grid (e.g., conditional choices, branching configs).
• Prompt template ablations with a programmatic eval.
• Decoding strategy tuning for generation models.
• Data mixture experiments where you can vary the corpus composition.

Not yet useful for:

• Picking the next research direction. Models cannot reliably reason about what is interesting. They reason about what is measurable.
• Evaluating where the eval itself is fuzzy (open-ended generation, art, taste).

Match the tool to the problem. Auto-research is excellent at well-posed search; it does not replace research taste.