Are logits the same as probabilities?

No. Logits are arbitrary real-valued scores before normalisation. Softmax converts them into probabilities by exponentiating each score and dividing by the total across the vocabulary. This means the gap between logits matters, not their absolute scale. A token with the largest logit is the greedy choice, but sampling can still choose another token.

Why do logits come from only the last position during generation?

In autoregressive generation, the next-token distribution is determined by the hidden state at the current final position. The model may compute hidden states for prompt tokens, especially during prefill, but during decoding the sampler only needs the vocabulary scores for the next token. Computing or retaining logits for every earlier position is usually unnecessary overhead.

What is the unembedding step?

The unembedding step maps a hidden vector back into vocabulary space. A matrix with one row per token is multiplied by the final hidden state, producing one score for each token id. Many language models tie this matrix to the input embedding weights, but conceptually embedding maps token ids into vectors, while unembedding maps vectors into token scores.

Where should I hook code if I want to inspect raw logits?

Hook immediately after the LM head or unembedding projection and before decoding processors if you want raw logits. Hook later if you want the scores actually used by the sampler. The distinction matters because masks, temperature, repetition penalties, and top-k or top-p filtering can substantially change which tokens are eligible or likely.

Logits

Logits are the model’s raw, pre-softmax scores for every token in its vocabulary at a generation step. They are produced by projecting the final hidden state through the unembedding or language-model head, yielding one score per token id before temperature, masking, penalties, sampling, or greedy selection.

A transformer does not directly output the next word. Its internal computation ends with a hidden vector that summarises the current context, but the sampler needs a ranking over concrete token ids. Logits are the bridge between those two spaces. Without them, there is no explicit place to apply decoding rules such as banning tokens, favouring diversity, stopping on EOS, or choosing the most likely next token.

Concretely, the final hidden state for the last sequence position is multiplied by the unembedding matrix, often called the LM head, and optionally shifted by a bias. Each row corresponds to a vocabulary item, so the result is a vector whose entries are scores for token ids. Softmax can turn those scores into probabilities, but the logits themselves are not probabilities and need not sum to anything.

The most common misunderstanding is treating an individual logit as meaningful in isolation. Softmax depends on relative differences, so adding the same constant to every logit changes no probabilities. Temperature rescales those differences: lower temperature makes the largest scores dominate, while higher temperature spreads probability more evenly. Processors such as masks, repetition penalties, top-k, and top-p modify logits or their effective distribution before selection.

The trade-off is that logits are large and transient: every decode step produces a vocabulary-sized float vector per active sequence. Serving systems often avoid keeping more than they need, compute logits mainly for the final position during generation, and may slice or filter the vocabulary. Inspecting logits is also hook-dependent: you must know whether you are seeing raw unembedding output or scores after processors have altered them.

Engineers meet logits at the boundary between model execution and token selection. In Hugging Face-style code they usually appear as the output of an `lm_head`, then pass through logits processors and warpers. In inference servers, a model runner may return hidden states, a logits processor applies the unembedding, and a sampler consumes the resulting scores to choose the next token id.

Go deeper

Inference EngineeringLogitsRead the raw logit vector from a forward pass, understand how it maps vocabulary to scores, and trace the unembedding step.Open the lesson

Common questions

Are logits the same as probabilities?: No. Logits are arbitrary real-valued scores before normalisation. Softmax converts them into probabilities by exponentiating each score and dividing by the total across the vocabulary. This means the gap between logits matters, not their absolute scale. A token with the largest logit is the greedy choice, but sampling can still choose another token.
Why do logits come from only the last position during generation?: In autoregressive generation, the next-token distribution is determined by the hidden state at the current final position. The model may compute hidden states for prompt tokens, especially during prefill, but during decoding the sampler only needs the vocabulary scores for the next token. Computing or retaining logits for every earlier position is usually unnecessary overhead.
What is the unembedding step?: The unembedding step maps a hidden vector back into vocabulary space. A matrix with one row per token is multiplied by the final hidden state, producing one score for each token id. Many language models tie this matrix to the input embedding weights, but conceptually embedding maps token ids into vectors, while unembedding maps vectors into token scores.
Where should I hook code if I want to inspect raw logits?: Hook immediately after the LM head or unembedding projection and before decoding processors if you want raw logits. Hook later if you want the scores actually used by the sampler. The distinction matters because masks, temperature, repetition penalties, and top-k or top-p filtering can substantially change which tokens are eligible or likely.

Every term