Session 09: AI for PMs — Understand AI to Make Better Product Decisions

Part 1: What an LLM Is and Why It Differs From Ordinary Software

You already understand the web, databases, APIs, architecture, deployment, security, performance, and how to work with an engineering team. This session adds one final layer: AI — the technology reshaping how every product is built and run.

An LLM is not a searchable database

An LLM (Large Language Model) is an AI model trained on huge amounts of text to predict the most likely next token in a given context. ChatGPT, Claude, and Gemini are all LLMs.

An LLM does not look up facts the way Google or a database does. It generates text that sounds plausible based on patterns from its training data. That's why AI can confidently produce wrong information.

Token — AI's unit of billing and its limit

A token is the LLM's unit of processing. Not a word, not a character — a token is a chunk of text, usually 3-4 English characters. The word "transformation" is one token; the sentence "Tôi muốn mua áo" might be 6-8 tokens depending on the model.

Tokens matter for two reasons:

First: token = money. Most AI APIs charge by the number of input + output tokens. GPT-4o costs about $2.50 per 1 million input tokens. An AI feature that summarizes email for 10,000 users a day, 500 tokens per email — that's 5 million tokens/day, roughly $12.50/day for input alone. Multiply by 30 days.

Second: token = limit. Every model has a context window — the maximum number of tokens the model can "see" at once (both input and output). When a document is longer than the context window, you have to chunk it — and you lose the AI's ability to understand the full context.

PMs should ask: "How many times a day does this feature call the AI? How many tokens per call? Do we have an estimate of the monthly cost?"

Hallucination — a problem with no patch

Hallucination is when an LLM produces wrong information but presents it confidently, as if it were true. It's not a bug — it's a fundamental property of how LLMs work.

Real example: Ask an AI to summarize a 50-page PDF contract, and it may miss or invent a clause that doesn't exist. If a lawyer trusts that summary without rereading the original — that's a real legal risk.

The level of hallucination depends on the task type (summarizing text hallucinates less than answering specific facts), context (information already in the prompt hallucinates less), and model size (larger models usually hallucinate less, but never fully eliminate it).

PMs must decide: Can this feature tolerate hallucination? A chatbot writing poems for users: acceptable. A system summarizing medical records: not acceptable.

Part 2: Build vs Buy and AI Architecture in a Product

Three paths to AI integration

When a team wants to add AI to a product, there are three options with different trade-offs. Most teams start with an API to validate quickly, then migrate to fine-tune or self-host once volume is large enough to save on cost.

The key question a PM should raise from the start: if the vendor raises prices or drops support for the model, what's plan B? Vendor lock-in in AI is a real risk, not a theoretical one.

RAG — the most common architecture you'll encounter

RAG (Retrieval-Augmented Generation) is a pattern that lets an LLM answer questions based on your data — internal docs, a knowledge base, a product database — without retraining the model.

PMs need to understand RAG because it explains why a company's AI chatbot can answer questions about a specific product — and why the answer quality depends on the quality of the documents you feed in, not just the model.

PMs should ask when reviewing a RAG feature:

• "When a document is updated, how long until the chatbot reflects the new information?"
• "When it can't find a relevant document, how does the chatbot fall back — stay silent or make something up?"
• "How do we measure the accuracy of the answers?"

AI latency — completely different from a normal API

A typical REST API returns in 50-200ms. LLM APIs are different: Time to First Token (TTFT) is often 300ms-2 seconds before the first character appears, and total latency depends on output length — 500 tokens can take 5-10 seconds.

Most LLM APIs support streaming — returning each token as it's generated instead of waiting for the whole response. For UX, this is a huge difference: the user sees text appear gradually within 0.3 seconds instead of staring at a blank screen for 10 seconds.

PMs should ask: "Are we using streaming? What does the loading state look like while waiting for the AI? Is there a timeout if the AI takes too long?"

When NOT to use AI

AI isn't the answer to every problem. Avoid AI when the output needs to be 100% accurate with no human review, when the feature is simpler with rule-based logic, when the cost per query is higher than the value it delivers, or when latency doesn't fit the UX requirement.

Signs of "adding AI for the sake of it": proposing AI without a concrete user problem, an impressive demo with no plan to measure accuracy in production, or building an AI feature with no fallback when the AI returns a bad result.

Part 3: Privacy, Data, and the Questions a PM Must Ask

Where user data goes when you call an AI API

This is the most important question PMs tend to forget to ask.

When your app calls the OpenAI or Anthropic API, the entire prompt content — including the user data you embed in it — is sent to their servers. This has consequences:

GDPR/PDPA compliance: If your users are in the EU or Vietnam, you need to ensure the data transfer to a third party has a legal basis and is documented in your Privacy Policy.

Data retention: OpenAI and Anthropic have policies on how long they keep data and whether they use it for training — read the Business/Enterprise plan vs. the free tier carefully. The two differ significantly.

Sensitive data: PII (name, email, national ID), health data, financial data — don't put these raw into a prompt unless necessary. Anonymize before sending when you can.

Part 4: Prompt Engineering for Everyday PM Work

The structure of a good prompt

PMs use AI daily for: writing PRDs, summarizing meeting notes, analyzing user feedback, drafting email. A good prompt follows the structure: [Role/Persona] + [Task] + [Context] + [Constraints] + [Output format].

Common prompting techniques PMs should know

Chain of Thought: Ask the AI to "think step by step" before reaching a conclusion. Reduces hallucination on complex reasoning tasks.

Before answering, analyze each factor one by one, then give your final conclusion.

Few-shot prompting: Give the AI 2-3 examples of the output you want before asking it to do the real task.

Grounding with context: Paste real content (transcript, data, documents) into the prompt instead of letting the AI reason on its own. This is the most effective way to reduce hallucination in practice.

5 prompt templates PMs can use right away

1. Summarize meeting notes:

Summarize the following meeting into: (1) Decisions made, (2) Action items with owner
and deadline, (3) Open questions. Use only the information in the transcript,
don't infer anything extra.

Transcript: [paste here]

2. Analyze user feedback:

Analyze the following 50 pieces of user feedback. Group them into key themes, count how
many times each theme appears, and quote a representative example for each theme. Sort
by descending frequency.

Feedback: [paste here]

3. Critique a PRD:

You are a demanding senior engineer who's good at spotting gaps in a spec.
Review the following PRD and point out: missing edge cases, unclear assumptions,
and the technical questions you'd ask the PM before estimating.

PRD: [paste here]

4. Estimate complexity:

You are a tech lead with 10 years of experience. Based on the feature description below,
list what needs to be done technically, the unclear risks,
and the questions you'd clarify before estimating.

Feature: [description]

5. Compare options:

Compare the following three options on: development time, maintenance cost,
user experience, and technical risk. End with a recommendation and your reasoning.

Option A: [description]
Option B: [description]
Option C: [description]

Part 5: Measuring AI Features in Production

Rolling out an AI feature safely

An AI feature needs a more cautious rollout than a normal feature because its behavior is hard to predict:

Shadow mode is worth asking for: the AI runs and logs its results but the user doesn't see the output yet. You get real data to evaluate before exposing it to users.