From question to answer: how a chatbot works
A chatbot consists of a large language model (LLM) and an app around it. The model does one thing over and over: it looks at the text so far and calculates how likely each possible next word is. Everything else, memory, tools, skills and the chat itself, is the app that assembles the text the model reads. The example used throughout the page is the question “What's the weather in Oslo tomorrow?”.
All numbers and probabilities are illustrative, made to show the mechanisms, not measured on a specific model.
1. From question to answer
Click a step to see what happens, or press Play. The loop at the bottom runs once for every token in the answer; the loop at the top runs when the model asks for a tool.
Step 1: Question
frame 1 of 25
The user types a message in the chat. To the model this is just text. It has no understanding of who is asking or why.
“What's the weather in Oslo tomorrow?”
2. Inside the model: the transformer
The model does not read letters or words, but tokens. Common words are often one token; rare and long words are split. Spaces usually belong to the following word (shown as `·`). Try typing your own text, the other tabs use it too.
Hover a token to see its ID. Simplified tokenizer for illustration; real tokenizers (BPE) learn the pieces from data.
3. Temperature and sampling: why the answer varies
The model produces a probability distribution over the next token, but something has to decide which token is actually chosen. These settings control that, and they explain why the same question can yield different answers.
“The weather in Oslo tomorrow will be ___”
Final probability (after temperature and filtering)
Model original (T = 1)
Filtered out
rainy
z=3.2
33.6%
cum. 34%
cold
z=2.6
18.4%
cum. 52%
cloudy
z=2.4
15.1%
cum. 67%
grey
z=2.0
10.1%
cum. 77%
nice
z=1.6
6.8%
cum. 84%
wet
z=1.3
5.0%
cum. 89%
sunny
z=1.0
3.7%
cum. 93%
changeable
z=0.8
3.0%
cum. 96%
snowy
z=0.3
1.8%
cum. 98%
unpredictable
z=-0.2
1.1%
cum. 99%
fantastic
z=-0.5
0.8%
cum. 100%
magical
z=-1.2
0.4%
cum. 100%
banana
z=-3.0
0.1%
cum. 100%
`z` is the model’s raw score (logit). Temperature: `p_i = e^(z_i / T) / Σ e^(z_j / T)`. Cumulative probability (“cum.”) is the sum of all candidates that are at least as likely, and that is what top-p looks at. Illustrative numbers.
Try it: generate answers
Press Generate to draw words with the settings on the right. If you change a setting, the draws reset.
Temperature T
1.00
Top-k (keep the k best)
off
Top-p (nucleus)
off
Presets:
Around 1 the model’s own probabilities are used almost unchanged. Common words dominate, but less likely ones still show up now and then.
| Setting | What it does | Effect on the answer |
|---|---|---|
| Greedy | Always picks the token with the highest probability | Deterministic: same input gives the same output. Can become repetitive. |
| Temperature T | Divides all logits by T before softmax. T < 1 makes the distribution sharper, T > 1 flatter | Low: precise and predictable. High: varied and creative, but more risk of nonsense. |
| Top-k | Keeps only the k most likely tokens and redistributes the probability | Removes the long tail of unlikely words, no matter how flat the distribution is. |
| Top-p (nucleus) | Keeps the best tokens until together they make up p of the probability | Adapts: when the model is sure, few words remain; when it is unsure, many remain. |
4. Memory, tools and skills
The language model itself can only do one thing: read text and suggest the next token. Memory, tools and skills are features of the app around the model. They all work the same way: they put more text into the context. Switch them on and off and see how the flow and the answer change.
Sequence diagram: time runs downward. Solid arrows are requests, dashed arrows are replies. Click or hover an arrow for an explanation.
The user sends the message to the app (the chat interface and the code around the model). The model itself receives nothing yet.
The answer the user gets
Temperature: 3 °C
Precipitation: rain, wind 6 m/s
Tip: Bring rain gear for the bike ride to work.
The tool provides real data, so the model doesn’t have to guess. Memory makes the answer personal. The skill decides the format.
What the model actually reads: the context window
1,439 tokens of 128,000 possible
Show against the full 128,000-token window
System prompt
300
Skills
450
Tool descriptions
200
Memories
40
Earlier messages
380
Question
9
Tool result
60
Your question is only 9 tokens. The rest is text the app adds. Everything costs space and compute time, so good apps fetch only the memories and skills that are relevant. Turn the features above on and off to see the context shrink.
5. Same question, different wording
To us the phrasings mean almost the same thing. To the model they are different tokens, which give different vectors, different attention and therefore different probabilities already for the first word of the answer. Since every chosen word affects all the next ones, small differences grow into completely different answers.
Meaning space (2D projection of prompt embeddings). Click a point to compare it with A. Points close together mean roughly the same, but none sit on top of each other. Illustrative positions.
A: “What's the weather in Oslo tomorrow?”
8 tokens
Outlined tokens also appear in the other prompt.
Probability of the first token in the answer
Tomorrow
58%
It
21%
The
12%
Oslo
5%
Tomorrow it will be 3 °C and rain in Oslo. Bring an umbrella!
Starting point: an ordinary, clear question produces an ordinary, clear answer.
B: “weather oslo tmorrow”
4 tokens
Outlined tokens also appear in the other prompt.
Probability of the first token in the answer
Oslo
44%
Tomorrow
24%
Weather
16%
3
8%
Oslo tomorrow: 3 °C, rain.
Style leaks. Short, informal questions resemble training text that was followed by short answers. The typo “tmorrow” also becomes different tokens than “tomorrow”.
Why the answer changes
1. Different words give different tokens, and therefore different numbers through the entire model.
2. Style, length and language carry over from the question to the answer, because the model has learned what tends to follow what.
3. Assumptions and leading wording pull the answer toward a particular conclusion, the model continues the premise instead of checking it.
How to get more stable answers
Be specific about what you want, and say which format and length you would like.
Give context and facts instead of assumptions, and avoid leading questions when you want an honest assessment.
Use a low temperature when you need predictability, and show an example of the answer you expect.