Recipes/Chapter 32 of 35
Recipe: adding a baseline
5 min readEdit on GitHub
The first model in any new project is the baseline. Not the headline model. The baseline. Without it, no future result has a number to beat.
Why baseline first
Loading diagram…
A "great" F1 of 0.78 is meaningless if the baseline scores 0.77. It is a significant win if the baseline scores 0.62. You cannot tell which without running both.
What counts as a baseline
The simplest model that produces a number on your task. Examples:
- Tabular classification: logistic regression with default hyperparameters.
- Tabular regression: mean prediction, then ridge regression.
- Text classification: TF-IDF + linear classifier.
- Image classification: linear probe on pretrained features.
- Sequence labeling: majority class per position; CRF baseline.
- Generation: copy-input or rule-based extractor.
The baseline is allowed to be "stupid." That is the point. If your fancy model can't beat stupid, that itself is the result.
The 90-minute baseline workflow
Total wall time, with an agent: ~90 minutes for tabular, longer for DL.
Step 1 — define the task (10 min)
Before any code:
# docs/decisions/0001-task-definition.md
## Task
Predict <target> given <inputs>.
## Audience
Who consumes the predictions and how.
## Eval
- Metric: <metric>, computed by <script>.
- Split: <how>, with code reference.
- Test set: <size>, held out from any tuning.
- Baseline target: beat <reasonable floor> by ≥<X>.
## Decision criteria
- If we beat baseline by ≥X, we proceed.
- If not, we revisit assumptions before iterating on the model.
The agent can draft this if you brief it. You verify and commit.
Step 2 — load data + verify split (15 min)
Ask the agent to write src/my_pkg/data.py:
Implement data loading per docs/decisions/0001-task-definition.md.
Requirements:
- load_train(), load_val(), load_test() return typed pandas DataFrames.
- Splits use the methodology defined in the task doc.
- Group/time leakage assertions inside the loader.
- Loader reads paths from config/data.yaml.
After implementing, write tests/unit/test_data.py that asserts:
- The three splits are disjoint.
- Group leakage check passes.
- Each loader returns at least one row.
Run the tests. Show me the output.
If a leakage assertion fires, fix the split before going further.
Step 3 — implement the baseline (30 min)
Implement src/my_pkg/baselines/logreg.py:
- Function train_baseline(train_df) -> sklearn.Pipeline
- Function eval_baseline(model, test_df) -> dict[str, float]
- Pipeline: ColumnTransformer (categorical → OneHotEncoder, numeric → StandardScaler)
+ LogisticRegression with default hyperparameters and seed=1337.
- Save to runs/<date>-baseline-logreg/ with config.yaml + metrics.json + model.joblib.
Then write tests/unit/test_baseline.py:
- A 100-row toy dataset trains and evals end-to-end in <5 seconds.
- The pipeline serialises and deserialises identically.
Run the tests. Show me the output.
Step 4 — run on real data (15 min, mostly compute)
make run-baseline
# or
uv run python -m my_pkg.baselines.logreg --config config/baseline.yaml
Output goes to runs/<date>-baseline-logreg/.
Step 5 — record the number (10 min)
Ask the agent:
Append a row to BENCHMARK.md for the baseline run we just produced.
Format:
| Date | Model | Metric | Value | Run |
|... | logreg-baseline | macro-F1 | 0.612 | runs/2026-05-04-baseline-logreg |
Pull the metric from runs/<id>/metrics.json. Do not invent. Show me the diff.
Verify the number matches the file. Commit.
Step 6 — write a 1-page baseline note (10 min)
docs/models/baseline-logreg.md:
# Baseline: logistic regression
## Result
| Metric | Value | Source |
| --------------- | ---------------- | -------------------------------------------- |
| Macro-F1 (test) | 0.612 | runs/2026-05-04-baseline-logreg/metrics.json |
| Per-class F1 | see metrics.json | same |
## What it is
TF-IDF + logistic regression, sklearn defaults, seed=1337.
## What it is not
Tuned. No feature engineering. No regularization sweep. This is the floor.
## How to reproduce
```bash
git checkout <sha>
make data
make run-baseline
```
Anything beating this needs to clear
- Macro-F1 ≥ 0.62 (≥1 absolute point over baseline).
- No regression on per-class F1 for any class.
Commit. Push.
## What you've earned
After 90 minutes:
- A defensible number on your task.
- Reproducibility for that number.
- A clear "what beats this" criterion.
- A test suite that catches data regressions.
- A `BENCHMARK.md` that future experiments append to, not start from scratch.
This is the foundation. Every subsequent experiment compares to it.
## Common mistakes
- **Skipping the baseline because "we know we need a transformer."** Maybe.
Often the transformer wins by less than you expect, and the baseline
exposes data quality issues you'd otherwise blame on the model.
- **Spending a week tuning the baseline.** A baseline is not "the best
simple model." It is "a simple model that runs." Move on.
- **Calling the baseline "the baseline" forever.** When you ship a better
model, the new floor is the new model. Update BENCHMARK.md to make this
clear.
## When to update the baseline
When the new floor model is in production for >1 month, or when the data
generation process changes substantially. Otherwise, the original baseline
remains the historical anchor. Keep it; don't delete it.