Inference EngineeringInteractive lab

Attention heatmap

Type a sentence, see which words attend to which

A full single-head self-attention pass computed in the page: scores, the scale factor, the causal mask and the softmax, drawn as a heatmap you can read row by row.

What this teaches

Scaled dot-product attention
Causal masking
Softmax temperature

The input

11 tokens · d = 32 · √d = 5.657

Sentencemax 12 tokens

Head dimension1/√d = 0.1768

Softmax temperatureT = 1.00Neutral

ModifiersCausal mask (no peeking ahead)Add sinusoidal position

Attention weights

softmax( qᵢ·kⱼ / (√d × T) ) - every row sums to 1

0.001.00Colour intensity is √weight, so a small weight is still visible. The number in each cell is the weight itself.

Attention weights. Each row is one query token and each column one key token; the cell is how much of that query's attention goes to that key. Masked positions are shown as a dash. Choose a row header to see the arithmetic behind it.
0the	1cat	2sat	3on	4the	5mat	6because	7the	8cat	9was	10tired	Σ
1.00	masked	masked	masked	masked	masked	masked	masked	masked	masked	masked	1.00
0.00	1.00	masked	masked	masked	masked	masked	masked	masked	masked	masked	1.00
0.00	0.06	0.94	masked	masked	masked	masked	masked	masked	masked	masked	1.00
0.01	0.00	0.00	0.98	masked	masked	masked	masked	masked	masked	masked	1.00
0.50	0.00	0.00	0.01	0.50	masked	masked	masked	masked	masked	masked	1.00
0.00	0.11	0.04	0.00	0.00	0.85	masked	masked	masked	masked	masked	1.00
0.01	0.00	0.00	0.00	0.01	0.01	0.98	masked	masked	masked	masked	1.00
0.33	0.00	0.00	0.00	0.33	0.00	0.00	0.33	masked	masked	masked	1.00
0.00	0.46	0.03	0.00	0.00	0.06	0.00	0.00	0.46	masked	masked	1.00
0.02	0.01	0.00	0.01	0.02	0.01	0.00	0.02	0.01	0.91	masked	1.00
0.01	0.00	0.01	0.00	0.01	0.00	0.03	0.01	0.00	0.00	0.93	1.00

Inside row 8: “cat”

the most divided row - pick another above

0.456

Weight on “cat”

5.66

Scale divisor √d

Visible positions

2.19

Softmax denominator

The five steps that turn two token vectors into one attention weight, for every key against the selected query.
Key	q·k	÷ √d	÷ T	exp (shifted)	weight
0 the	-4.57	-0.808	-0.808	0.0016	0.0007
1 cat	32.00	5.657	5.657	1.0000	0.4559
2 sat	16.13	2.851	2.851	0.0604	0.0276
3 on	0.00	0.000	0.000	0.0035	0.0016
4 the	-4.57	-0.808	-0.808	0.0016	0.0007
5 mat	20.16	3.563	3.563	0.1233	0.0562
6 because	-5.28	-0.933	-0.933	0.0014	0.0006
7 the	-4.57	-0.808	-0.808	0.0016	0.0007
8 cat	32.00	5.657	5.657	1.0000	0.4559
9 was	4.57	0.808	masked	0	0.0000
10 tired	-4.03	-0.713	masked	0	0.0000

The exponent column is exp(logit − 5.657). Subtracting the row maximum before exponentiating is what keeps a softmax from overflowing on large logits; it cancels in the division, so the weights are unchanged.

What is real here, and what is standing in

Real: the scores, the 1/√d scaling, the causal mask, the temperature and the softmax - all computed in this tab, on your sentence.

Standing in: the token vectors. A trained model has learned embeddings; this page hashes character 2- and 3-grams into 32 dimensions instead, so the vectors encode spelling rather than meaning. That is why “runs” and “running” attend to each other and “cat” and “kitten” do not. Q, K and V are those vectors unprojected, so a score here is a plain similarity; a real head multiplies by three learned matrices first, which is what lets one head track syntax and another track coreference.

Everything on this page runs in your browser. Nothing you type is sent anywhere, there is no account, and it keeps working offline.

Every lab