ML Handbook: ML Engineering with AI
Three Modalities/Chapter 17 of 35

Traditional ML with AI

4 min readEdit on GitHub

Tabular, sklearn, XGBoost, LightGBM, classical NLP. This is where AI-augmented workflow shines brightest, because:

  • The libraries are stable. The agent rarely hallucinates sklearn APIs.
  • Iterations are cheap (seconds to minutes). The 3–10× experiment multiplier is fully realized.
  • The failure modes are well-known and easy to encode as guardrails.
  • Domain expertise (which features matter, which leakage to watch for) is what separates competent work from great work — and that comes from you, not the agent.

The high-leverage workflow

Loading diagram…

Each step is one agent task. Total wall-clock for a clean tabular project: 1–2 days end-to-end, where without an agent it would be 1–2 weeks.

Where the agent helps most

  • Feature engineering boilerplate. Lag features, rolling stats, target encoding, group-aware splits — the agent writes these correctly the first time.
  • Cross-validation scaffolding. Stratified, group-aware, time-series CV. Tedious to write, easy to get subtly wrong.
  • Hyperparameter sweeps. Optuna scripts with proper pruning, early stopping, and study persistence.
  • Error analysis. "Slice the predictions by category and tell me where errors concentrate." Two minutes vs. an hour of pivot tables.
  • Ablation tables. "Re-run with each feature group dropped, give me a table." Fully automatable.

Where you must lead

  • Choosing the metric. Accuracy on imbalanced data is a trap. Macro-F1, ROC-AUC, calibration, business-loss-weighted — only you know which.
  • Choosing the split. Random vs. stratified vs. group-aware vs. time-based. The wrong choice silently leaks. The agent will not notice unless told.
  • Reading the data. The agent will not flag that 5% of your timestamps are in 1970, or that one category dominates the train but not the test.
  • Calling the experiment over. When to stop polishing and ship.

A worked starter prompt

For a new tabular project, this prompt produces a useful first pass:

Build a baseline for predicting <target> on data/manifests/train_v1.parquet.

Constraints:
- Use sklearn + XGBoost only.
- Train/test split: stratified k-fold by <group>, k=5.
- Metric: macro-F1, with per-class breakdown.
- Save to runs/<date>-baseline-xgb/ with config.yaml, metrics.json,
 feature_importance.csv, predictions.parquet.
- Do NOT do feature engineering beyond categorical encoding.
- Do NOT tune hyperparameters yet.

Goal: a defensible baseline number we can beat. Stop after the run completes.

The constraints are the whole point. Without them, the agent will introduce clever tricks that confuse the baseline interpretation.

Common failure modes (tabular specific)

  • Leakage via target encoding. Target encoding done before splitting bleeds the target into features. Always fit encoders inside the CV fold.
  • Group leakage. Random split when the unit of analysis is "user" puts the same user in train and test. Use GroupKFold.
  • Time leakage. Shuffling time-series. Use TimeSeriesSplit or a walk-forward harness.
  • Categorical drift. Categories present in test but not train. Decide upfront how to handle: drop, "other" bucket, or model-specific (LGBM handles this natively).
  • NaN handling differences. Train uses median imputation; production uses zero. Production silently degrades. Pin the preprocessing; ideally inside the model pipeline.

Encode these as /review checks. The agent is good at finding these patterns when prompted. It is not good at finding them unprompted.

When traditional ML is the right answer

Always start here. A tabular dataset with fewer than 1M rows and clear features almost certainly does not need a transformer.

The honest sequence:

  1. Logistic regression / linear model. (Floor.)
  2. Gradient-boosted trees (XGBoost / LightGBM). (Strong default for tabular.)
  3. Deep model. (Only if 1 and 2 saturate and you have data and compute.)

Skipping 1 and 2 to "look modern" wastes everyone's time and produces less explainable models. The agent will follow your lead — set the right precedent.

Documentation deliverable

For every shipped tabular model, the model card includes:

  • Target definition (precise, with example rows).
  • Train/test split methodology with code reference.
  • Metric on test, with confidence interval from k-fold.
  • Comparison to dumbest baseline.
  • Feature importances (top 20).
  • Known limitations (slices where it underperforms).
  • Reproducibility command.

The agent can draft this from runs/<id>/. You verify every claim against the underlying file before it goes anywhere external.