Long Short-Term Memory

Why this one

Read this after On the difficulty of training Recurrent Neural Networks and before Attention Is All You Need. The useful idea is not nostalgia for RNNs, it is the gate: give a model a learned way to keep state unchanged, overwrite it, or expose it. That turns memory from a side effect of repeated matrix multiplication into an explicit mechanism you can reason about. People often remember LSTMs as the thing transformers replaced, so they miss the engineering lesson that survived: long context is mostly about controlling information flow and gradients, not just adding more parameters. If you have ever watched a sequence model forget the one token that mattered, this paper gives you the first clean vocabulary for why. It is worth your evening because attention is easier to understand once you have seen the older solution to the same pain: preserve useful signal across many steps without letting optimization destroy it.