Agentic AI with AI

Building agent systems with the help of agents. The framing is recursive but the practice is concrete: you are designing software whose primary primitive is "LLM call + tool use," and your IDE is also an agent.

Mental model

An agent system is a control loop wrapping an LLM call. The LLM proposes actions; your code validates, executes, and feeds back results. Everything interesting is in the loop, not the model.

Loading diagram…

Two failure attractors at opposite extremes:

Too thin a loop. "ReAct + retry" with no validation. Agent loops on the same wrong action; cost spirals; user waits.
Too thick a loop. Every possible failure handled; LLM is one node in a graph of 40 nodes. Hard to reason about, slow to iterate.

The right loop is thinnest that still enforces your hard invariants.

Patterns that work in 2026

1. Plan-then-act for multi-step tasks

A separate "planning" call produces a structured plan; an "execution" loop walks it. More predictable than pure ReAct on long horizons. Discussed in the a recent on-device voice agent project of mine.

2. Tool use with strict schemas

Pydantic / JSON-schema-defined tools. The LLM cannot call a tool with an invalid argument shape — your runtime rejects before execution. Catches 90% of "tool call hallucinations."

3. Structured output everywhere

Use instructor / outlines / native structured outputs. Free-text from LLMs into your codebase is asking for parse errors. Structured outputs into typed objects compose cleanly.

4. State as a single typed object

@dataclass
class AgentState:
    task: str
    history: list[Action]
    tool_results: dict[str, Any]
    budget_used: int
    budget_max: int

One object passed around. Easy to log, easy to reason about, easy for the agent to mutate explicitly.

5. Hard budgets

if state.budget_used >= state.budget_max:
 return Result(status="budget_exceeded", state=state)

Tokens, tool calls, wall-clock. Always have at least one budget. Without one, runaway loops are a question of when, not if.

Patterns to avoid

"Let the agent decide everything." It will pick wrong tools, retry too many times, and hallucinate workflows. Constrain.
Untyped tool returns. "Tool returned a string" → next LLM call has to re-parse. Return typed objects.
Multi-agent systems before you've solved single-agent. Multi-agent amplifies failures of single-agent design.
Frameworks before you've built it once. Build with primitives first; feel where the pain is; pick the framework that solves that pain.

Building agent systems with Claude Code

Recursive but practical. The same patterns from this handbook apply:

Small, legible code. A 200-line agent.py is debuggable. A 2000-line framework wrapper is not.
Reproducible runs. Capture LLM call inputs, outputs, tool sequences. Replay them when debugging.
Eval as code. A tests/agent/ directory with end-to-end task scenarios. Run on every change.
Hard rules in CLAUDE.md. "All tool returns are typed." "All loops have budgets." "Every LLM call is logged with model + version."

A useful slash command for agent dev:

#.claude/commands/agent-trace.md
Replay the most recent agent run from logs/agent/<latest>.jsonl.

Show:

1. Task input
2. Each LLM call's prompt summary (first 100 chars) + tool chosen
3. Each tool result summary
4. Final state
5. Total tokens, total wall-clock

Flag any:

- Repeated tool calls with identical args (loop)
- Tool errors that were retried more than 2x
- Budget warnings

Eval for agents

The hardest piece. A few useful primitives:

Eval type	What it tests	Cost
Trajectory match	Did the agent take the same steps as a known-good trace?	Low; brittle to acceptable variation
End-state assertions	Did the world end up in the desired state?	Medium; requires programmatic checks
LLM-as-judge	Did an LLM rate the run as successful?	Cheap, gameable, useful as a smoke screen
Human review	Did a person say it was OK?	Expensive, the gold standard

Combine: end-state assertions for the bulk of CI, LLM-as-judge for triage, human review for the final say on releases.

Tracing and observability

Pick one. The shortlist:

Phoenix (Arize). OSS, local-friendly. Strong span model.
Langfuse. OSS or SaaS. Good UI for prompt-version tracking.
LangSmith. Tight LangChain integration; SaaS.
Plain JSONL traces. Often enough for small projects.

Every LLM call should produce one span: model, version, prompt hash, output, latency, tokens. Without this, debugging an agent failure is guessing.

Cost discipline

Agent systems make cost easy to underestimate. Rules:

Per-task budget cap, enforced in code. Not "guidance"; an exception.
Cheaper models for routine sub-calls. Plan with the big model; act with the small one.
Cache aggressively. Identical prompts → cached responses. Anthropic and OpenAI both support prompt caching natively in 2026; use it.
Log spend per task. A weekly grep over your traces tells you which tasks are expensive.

The team that does not measure agent cost is the team that gets the surprise bill.

When (not) to build agents

Use them when:

The task involves multi-step interaction with tools.
The structure of the steps is not knowable in advance.
The cost of a wrong action is bounded (rollback, retry, human review).

Skip them when:

A single LLM call with structured output would do. (Frequently true.)
The "agent" is really just a script with one branch.
The reliability requirement exceeds what current models can deliver. (Be honest. Most production "must succeed" workflows are not agentic — they are deterministic with LLM-assisted slots.)