Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Albert Gu, Tri Dao2023arXiv 2023
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Attention Is All You Need and FlashAttention. Those papers teach you why attention won, then how far careful IO work can push it. Mamba asks a different engineering question: what if the sequence model did not need to look back over every previous token at every step? The trick is not just replacing attention with recurrence, which older models already tried and mostly lost. It is making the state update selective, so the current token controls what gets remembered, overwritten, or passed forward. People get this paper wrong when they treat it as a clean Transformer killer. It is more useful as a proof that content aware memory can be built without a quadratic attention matrix, if the algorithm and hardware path are designed together. Worth your evening because it changes how you think about long context: not as a bigger cache, but as a compression policy learned at every token.
What to take away
- Selection is the core move: SSM parameters depend on the input token, so memory updates become content aware.
- Mamba trades the Transformer KV cache for a fixed recurrent state, making decode cost scale differently with context length.
- The hardware-aware scan is part of the model idea, not an implementation footnote.
Reads with
- Attention Is All You Need
sets the attention baseline that mamba is trying to avoid scaling quadratically
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
shows the io-aware hardware thinking mamba applies to a different sequence primitive
- Long Short-Term Memory
the older gated-memory idea that mamba revives with parallelizable state space updates