Recipe: spinning up a new project

mkdir my-ml-project && cd my-ml-project
git init -b main
echo "# my-ml-project" > README.md
git add. && git commit -m "init"

uv init --python 3.12
uv add --dev ruff mypy pytest pre-commit
echo "venv = \".venv\"" >>.python-version

mkdir -p src/my_pkg tests/unit tests/integration \
    benchmarks docs/{models,decisions} \
    config data/manifests scripts \
    .claude/{commands,agents}

touch src/my_pkg/__init__.py tests/unit/__init__.py

# my-ml-project — agent rules

## Project context

What this repo does, who uses it, what it must never break. (One paragraph.)

## Hard rules (non-negotiable)

- All experiments are reproducible: seed pinned, deps pinned, command logged.
- No metric is reported without a runnable script in `benchmarks/`.
- No dataset paths are hardcoded; load via `config/data.yaml`.
- All training runs log to `runs/<date>-<slug>/` with `config.yaml` + `metrics.json`.
- All public APIs are typed; `mypy --strict src/` must pass in CI.
- Tests in `tests/unit/` complete in < 30 seconds; slow tests live in `tests/slow/`.
- Never commit data, model weights, or `.env`.
- HF Hub revisions are pinned; no bare repo IDs in production code.
- Pickle is forbidden for model weights.

## Conventions

- Package layout: `src/my_pkg/`, tests in `tests/`, configs in `config/`, runs in `runs/` (gitignored).
- Naming: `train_<model>_<task>.py`, `eval_<model>_<task>.py`.
- Commits: imperative mood, one logical change per commit.

## Definition of done for a new model

- [ ] Reproducible training script
- [ ] Eval script with at least one baseline comparison
- [ ] Numbers committed in `BENCHMARK.md` (with measurement script reference)
- [ ] Model card in `docs/models/<name>.md`
- [ ] Smoke test in `tests/unit/test_<model>.py`

## Things that have bitten us

<!-- grow this section over time -->

cat >.claude/commands/preflight.md <<'EOF'
Walk the guardrails checklist (see Section 5) against the current diff.
For each item, report PASS / FAIL / N/A with a one-line reason.
Do not edit anything. Just report.
EOF

cat >.claude/commands/new-experiment.md <<'EOF'
Scaffold a new experiment.

Args: <slug>

Steps:
1. Create runs/$(date +%Y-%m-%d)-<slug>/.
2. Copy templates/hypothesis.md, config.yaml, command.sh into it.
3. Pre-fill code revision (git rev-parse HEAD) and timestamp.
4. Open hypothesis.md and ask me to fill in: Hypothesis, Falsifier, Decision.
5. Do not run anything until I approve.
EOF

cat >.claude/commands/bench.md <<'EOF'
Run benchmarks/run.py against the most recent run dir.
Update BENCHMARK.md with a new row. Show me the diff.
Do not commit until I approve.
Hard rule: never overwrite an existing row.
EOF

[project]
name = "my-pkg"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = []

[tool.ruff]
line-length = 100

[tool.ruff.lint]
select = ["E", "F", "I", "B", "UP", "N", "RUF"]

[tool.mypy]
strict = true
files = ["src"]

[tool.pytest.ini_options]
testpaths = ["tests/unit"]
addopts = "-q --strict-markers"

.PHONY: setup test fmt lint bench

setup:
	uv sync

test:
	uv run pytest tests/unit -q

fmt:
	uv run ruff format src tests

lint:
	uv run ruff check src tests && uv run mypy --strict src

bench:
	@test -n "$(RUN)" || (echo "RUN=<run-id> required"; exit 1)
	uv run python benchmarks/run.py --run-dir runs/$(RUN)

cat >.gitignore <<'EOF'
__pycache__/
*.pyc
.venv/
.mypy_cache/
.ruff_cache/

data/raw/
runs/
*.pt
*.bin
*.parquet

.env
*.key
EOF

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.5.0
    hooks:
      - id: ruff
      - id: ruff-format
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.10.0
    hooks:
      - id: mypy
        files: ^src/
        args: [--strict]

uv run pre-commit install

git add -A
git commit -m "scaffold project structure and agent contract"