Deep learning with AI

PyTorch, JAX, fine-tuning, distributed training. The economics are different from traditional ML: each experiment costs real money, runs take hours not seconds, and a wrong configuration can burn a day of GPU time.

The agent's role shifts accordingly: less "run more experiments," more "build a harness so the experiments you run are reliable."

What changes vs. traditional ML

The compounding rule: never start a multi-hour run on code the agent modified five minutes ago and you have not read. Hours of L4/H100 cost real money to be wrong about.

The harness is the thing

Spend the first ~20% of any DL project on the harness, not the model. A good harness:

• Single training entry point with a flat config.
• Single eval entry point that re-loads the checkpoint from disk.
• Streaming JSONL metrics + tensorboard / W&B / etc.
• Resumable from checkpoint without subtle behavior change.
• Unit tests for the model forward pass on tiny shapes (catches bugs in 30s).
• Smoke training run on tiny data to verify loss decreases.

The agent will build this if you scaffold the contract:

Scaffold a training harness with these requirements:
- src/train.py: single entry; reads config.yaml; supports --resume.
- src/eval.py: loads checkpoint, runs eval, writes metrics.json.
- src/model.py: model definition, no factories or registries.
- src/data.py: dataset and dataloader, accepts manifest path.
- tests/unit/test_smoke.py: trains 10 steps on 8 examples, asserts loss decreases.
- Config is a Pydantic model in src/config.py.
- Determinism: seed_everything called in main of train.py and eval.py.

Do not start training. We'll review the harness first.

Read the harness diff carefully before any real run.

Mixed precision, multi-GPU, and the bugs they hide

Sources of "the same code gives different numbers" in DL:

• cuDNN nondeterminism. Some convolutions choose different algorithms per run. Set torch.backends.cudnn.deterministic = True and accept the throughput hit.
• Mixed precision. AMP and bf16 produce slightly different gradients per hardware. Two A100s can disagree at 5 decimal places.
• DDP with uneven batches. The last batch on each rank may differ. Pad, drop, or use join context manager.
• Order-dependent reductions. Summing 1M floats in different orders gives different sums.

You will not eliminate these. You will budget for them: if your run-to-run noise is ±0.2 perplexity, your "improvement" threshold is ≥0.4.

Before any conclusion, rerun once. The cheapest sanity check is a second run with a different seed. If the second run agrees with the first within noise, you have a real signal. If not, you have a noisy estimate, not a result.

Fine-tuning specifically

The 2026 fine-tuning landscape:

• LoRA / QLoRA for parameter-efficient fine-tuning of LLMs and vision models. The default for adapting open-weights models on modest hardware.
• Full fine-tuning when you have the budget and need maximum quality.
• DPO / KTO / ORPO for preference fine-tuning without separate reward models. Simpler than RLHF, often sufficient.
• Distillation to compress a big model into a small one for serving.

Common agent-aided workflow for fine-tuning:

The smoke run catches >80% of "wasted overnight runs." Always do it.

Eval is hard, do it anyway

DL eval is where people fool themselves most often. Three rules:

1. Hold out a set the model has not seen during any iteration. Validation set seen during HP tuning is contaminated. You need a clean test set.
2. Have a regression set. A small (~100 example) hand-curated set you run on every checkpoint. Catches "improved benchmark, regressed on the use case."
3. Look at predictions by hand. At least 50, sampled across success and failure cases. The agent will never replace this.

For LLM fine-tuning, add: a "did it forget?" eval. Standard benchmarks (MMLU, etc.) on the base task. A fine-tune that wins on your task and loses 10 points on MMLU is not a free lunch.

The agent's failure modes here

• Confidently wrong about new APIs. PyTorch 2.x compile, FSDP2, AMP context managers — the agent's training data lags the reality. Audit imports against the installed version.
• Silent CPU fallback. Model accidentally on CPU because .to(device) was missed. Run is 100× slower; agent reports "training in progress."
• Tokenizer mismatch. Loading model with one tokenizer, training with another. Loss is fine, generations are gibberish. Caught only at eval time.
• Wrong eval metric. Computes BLEU when you wanted ROUGE. Reports a number anyway.

Each of these is preventable with a slash-command-driven smoke test that asserts the specific invariant. Build them once.

When deep learning is overkill

Same answer as the previous chapter: when a smaller, simpler model would do. Deep learning is the right hammer when:

• The data is unstructured (images, audio, long text).
• The patterns are non-linear and high-dimensional.
• You have enough data (rough rule: 10× the model parameter count, or strong pretrained init you're fine-tuning).
• The win over a simpler baseline is large enough to justify ongoing GPU costs.

If three of those four are not true, run the simpler thing first.