Case studies

Three concrete shapes of auto-research that work today, and one that does not. Use them as templates, not gospel — adapt to your own constraints.

Case 1: hyperparameter sweep with early stop (works well)

Goal: find the best lr, weight_decay, dropout combination for a fine-tuning job on a 50k-example dataset.

Search space: 3 dimensions, ~24 reasonable combinations.

Setup:

• Random search seeded for reproducibility.
• Each trial: 1 epoch, 1k validation examples, ~12 minutes on an L4.
• Eval metric: validation F1.
• Budget: 20 trials, $25 cap (≈4 GPU-hours at spot price).
• Stop: plateau N=8.

Outcome (typical for this shape):

• Loop runs 18/20 trials before plateau triggers.
• Best config beats grid-baseline midpoint by a small but real margin.
• Total wall clock: ~3.5 hours, mostly in run_trial.
• Cost: $22.

Why it works:

• Search space is bounded.
• Metric is fast and reliable.
• Cost per wrong trial is small.
• The agent's role is simple: pick the next random config.

What goes wrong:

• If you skip the plateau check, the loop runs all 20 trials even when the last 5 added no information.
• If the val set is too small (fewer than 200), noise dominates and you "find" winners that don't replicate.

Case 2: prompt template ablation with LLM judge (works, with caveats)

Goal: find the best prompt template for an extraction task across 12 template variants, judged by an LLM scorer.

Search space: 12 templates × 3 few-shot example sets = 36 combinations.

Setup:

• Exhaustive search (small enough).
• Each trial: 100 eval examples, ~30 seconds each.
• Eval metric: a stronger LLM rates extraction quality 1–5.
• Budget: $15 cap.
• Stop: exhaustive (no early stop).

Outcome:

• All 36 evaluated.
• Top template clearly dominates by ≥0.4 average score.
• Recommended for production with 50-example human verification.

Why it works:

• The LLM judge has ground-truth structured outputs to compare against, which makes it less gameable than free-form quality judging.
• Cost is low; exhaustive is feasible.

The caveat:

• The LLM judge introduces correlation: a judge that prefers verbose outputs will reward verbose templates. Always sample 50 winning outputs and inspect by hand before promoting.

Case 3: data mixture optimization (works, expensive)

Goal: find the best mixture ratio of three pretraining corpora for domain adaptation.

Search space: continuous (3 ratios summing to 1.0).

Setup:

• Bayesian optimizer (Optuna with TPE).
• Each trial: 500-step pretraining + downstream eval. ~45 minutes on a single H100.
• Eval metric: downstream task average across 3 benchmarks.
• Budget: 30 trials, $400 cap.
• Stop: plateau N=10 + saturation threshold.

Outcome:

• 18 trials before plateau.
• Best mixture clearly beats uniform baseline.
• $235 spent.

Why it works:

• BO efficiently navigates a continuous space.
• Downstream eval is a strong, mechanical signal.
• The team had a real budget and understood that some trials would be lost.

What to watch:

• Reward hacking via mixture extremes (model overfits to one corpus). Mitigate by holding out a "this corpus only" eval as a regression check.
• Compute drift between trials (one trial gets a faster H100 from the spot pool, others get a slower one). Pin instance type or use wall-step rather than wall-clock.

Case 4: agent-driven hypothesis generation (does not work yet — 2026)

Goal: "find the most interesting research direction for our model."

Setup: Give an LLM the codebase, recent results, and prompt it to "propose the next experiment that would be most informative."

Outcome: Reliably underwhelming. Common failures:

• Proposes the safe-and-obvious thing (more data, larger model).
• Proposes the impossible thing (a new architecture sketched in 3 lines).
• Loses the constraint context (forgets you're CPU-bound, not memory-bound).
• Hallucinates citations.

Why it doesn't work yet:

• The model has weak calibration for what is novel and tractable in your specific context.
• It has no way to check whether its proposal is buildable in your stack.
• It cannot reason about budget vs. signal trade-offs.

What to do instead: use the agent for filtering, not generation.

• You propose 5 directions.
• The agent steel-mans each, lists likely outcomes, estimates effort.
• You pick.

This works much better. The model is good at structured analysis when given a small set; it is bad at unbounded creativity.

What separates "works" from "doesn't"

Let the agent search; you set the question. Reverse the roles and the loop falls apart.

Setting up your first auto-research project

A 90-minute checklist:

1. Define the question in one sentence. Not vague.
2. Pick a metric the harness can compute mechanically. Pin its protocol.
3. Define the search space as a dataclass or YAML schema. Bounded.
4. Wrap your existing train + eval as run_trial. Catch its own crashes. Report cost.
5. Pick a propose_next. Random first; upgrade later if cost justifies.
6. Set a budget. Trials, dollars, wall-clock. Hard cap.
7. Run a 3-trial dry run to verify wiring. Inspect every output.
8. Run for real. Watch the first hour. Then check periodically.

The first time, expect the dry run to find at least one bug. Auto-research amplifies bugs that single runs hide.