Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le2014NeurIPS 2014
Read it on arxiv.org(opens in a new tab)Why this one
Before transformers made everything look like attention blocks, this was the cleanest version of a useful idea: turn an input sequence into a learned state, then decode another sequence from it. People often remember the paper as old machine translation history, but the durable lesson is the interface. Once you accept that text, code, speech, and actions can be modeled as sequence in and sequence out, a lot of modern systems become easier to reason about. The paper is also a good antidote to vague talk about models understanding language. The model is doing a very specific job: compress, condition, and generate. Read it to see the shape of the problem before the transformer solved the bottlenecks better.
What to take away
- The encoder and decoder can be trained end to end even when input and output lengths differ.
- Reversing the input sentence improved optimization by shortening useful dependency paths.
- The fixed vector between encoder and decoder is the bottleneck that later attention mechanisms attack.
Reads with
- Long Short-Term Memory
the encoder and decoder depend on lstms to carry sequence state without vanishing so quickly
- Neural Machine Translation by Jointly Learning to Align and Translate
attention is the next fix for the single vector bottleneck in encoder decoder translation
- Attention Is All You Need
the transformer keeps the sequence in sequence out interface while replacing recurrent compression