Fast Inference from Transformers via Speculative Decoding

Why this one

Read this after Attention Is All You Need and Efficiently Scaling Transformer Inference. The trick is not to make the big model cheaper per token, but to stop asking it one token at a time when a smaller draft model can guess several likely next tokens. The big model then verifies a block in parallel, accepting the guesses that match its own distribution and repairing the first miss. People often describe this as approximate decoding, which misses the point: the paper is valuable because the sampling procedure preserves the target model's output distribution. That distinction matters if you serve models where quality regressions are hard to measure or legally risky. For an engineer, this is the cleanest example of an inference idea that spends extra FLOPs to buy wall clock latency. It also gives you the mental model behind later systems such as Medusa and production draft model stacks.