ML Handbook: ML Engineering with AI
Failure Modes/Chapter 24 of 35

Hallucinated APIs & versions

4 min readEdit on GitHub

The agent's training data is a snapshot. The libraries you use have moved since. Result: confident code calling functions that no longer exist, with arguments that were renamed, against APIs that were deprecated two majors ago.

What it looks like

# Looks reasonable. Will fail at runtime.
from torch.utils.data import DataLoader
loader = DataLoader(ds, batch_size=32, shuffle=True, num_worker=4)
# ^ typo? no — old name

Or:

# Was valid in transformers 4.30. Renamed in 4.35.
model = AutoModel.from_pretrained(name, torch_dtype=torch.float16, load_in_8bit=True)
# ^ moved into BitsAndBytesConfig

Or worse:

# Doesn't exist. Never did. Sounds plausible.
from sklearn.metrics import classification_report_extended

Why it happens

  • Knowledge cutoff. Models know the API as of their training date. They do not know what changed last quarter.
  • Confabulation under pressure. When you ask for "the function that does X," the model would rather invent a plausible name than admit it does not remember.
  • Training-data bias toward older code. The biggest pile of code on the internet is the historical pile, not the current pile.

Defenses

1. Pin your dependencies and let imports fail loudly

A locked requirements.lock, uv.lock, or poetry.lock. The hallucinated import either exists or it doesn't; a missing import crashes immediately instead of silently degrading.

2. Run the smoke test on a clean install before reviewing

make clean-env && make setup && make test

The agent's "I tested this" claim is checked against a fresh environment, not the agent's session state.

3. Use a type-checker

mypy --strict src/

Many hallucinated APIs surface as "module has no attribute X" or "unexpected keyword argument." Cheap to catch, cheap to run.

4. Cross-check against the installed package

For a suspicious import:

python -c "import torch.utils.data as m; print(dir(m))"
python -c "from transformers import AutoModel; help(AutoModel.from_pretrained)"

The agent can do this autonomously if you ask. A useful slash command:

#.claude/commands/api-check.md
For each external API call in the recent diff:

1. Confirm the function/class exists in the installed version.
2. Confirm each kwarg name and type matches the installed signature.
3. Flag anything you cannot verify against the installed package.
   Never assume. Always import and inspect.

5. Force the agent to check docs for major libraries

For a library where the API churns (transformers, torch, langchain, openai-python):

## CLAUDE.md rule

When generating code that calls transformers, openai, anthropic, or torch APIs:

- Verify the API exists in the installed version (import + dir() / help()).
- If unsure, fetch the relevant page from the official docs and cite the URL.
- Never invent function names. If the function you "remember" doesn't exist, ask.

Version pinning, again

The agent will sometimes:

  • Suggest upgrading a package to "fix" a missing API ("transformers 4.40 has this").
  • Add the new version to the lock file.
  • Move on without testing.

This is how silent breakage spreads. Pin upgrades to a separate PR, run the full test suite, eyeball the changelog. Never let the agent bump a major version in the same PR as a feature.

Specific high-churn libraries (2026)

These are worth extra scrutiny:

  • transformers — config schemas, quantization, generation API
  • torch — torch.compile, FSDP2, DataLoader internals
  • langchain / langgraph — entire surface area changes regularly
  • openai-python / anthropic — request/response schemas
  • pydantic — v1 vs v2 differences are still a tripwire
  • scikit-learn — small changes per minor; mostly safe but watch deprecations

For these, default to fetch-and-check instead of "the agent will know."

When you find one

In code:

  1. Replace with the correct API for the installed version.
  2. Add a smoke test that exercises it.
  3. Add a CLAUDE.md note: "for <library>, always check the installed version's docs."

In a doc:

  1. Treat the doc as fabricated until verified — the same code might be in your example snippet.
  2. Run any code in the doc on the locked environment.
  3. Update.

The deeper lesson

The agent is a probabilistic recall engine, not an authoritative reference. Treat its API knowledge as a starting hypothesis to verify, not a conclusion. This sounds obvious; it is forgotten roughly twice a week in any active repo.