02.06 · Concept
Why the Same Prompt Answers Differently Twice
Explain the sources of non-determinism in a generated answer - sampling, personalisation, freshness, fan-out variance - and say what that does to any measurement built on prompts.
Identical prompts can produce different answers because generation samples from probabilities, products personalise context, retrieval changes, and hidden tool chains vary between runs. Prompt-based measurement therefore behaves like sampling a moving system, not asserting a fixed output, so evaluations need repeated runs, recorded settings, and statistical interpretation.
What this lesson answers
- why does the same AI prompt change answers
- how reliable are prompt based AI search measurements
- what causes non deterministic AI generated responses
Notes
A generated answer is not usually a fixed lookup result. The model produces a distribution of likely next tokens, then a decoding process chooses among them. If sampling is enabled, two runs can choose different valid continuations even for the same prompt. Small early differences compound, so the final answer may differ in wording, examples, ordering, or even which borderline facts are included.
Modern AI products add more sources of variation than the model itself. They may personalize based on account, location, memory, history, device, or experiment bucket.
Common questions
- Why can the same prompt produce different AI answers?
- A generated response is selected from likely continuations, not retrieved as a fixed record. If the system samples during decoding, equally valid paths can be chosen on different runs. Once an early word or phrase differs, later context shifts too, changing structure, emphasis, examples, and sometimes marginal facts.
- Is answer variation only caused by the model?
- No. AI products often add personalisation, live retrieval, experiments, memory, location signals, safety filters, rerankers, and tool calls around the model. Any of those layers can change the context the model sees. Two requests that look identical to the user may not be identical inside the product.
- How should prompt-based visibility or SEO tests handle variability?
- Treat prompt results as observations from a variable system. Run repeated trials, preserve prompts and outputs, record settings where possible, and score answers with a rubric instead of exact string matching. Separate harmless wording changes from real differences in brand mention, factual content, recommendation order, or source selection.