Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio2015ICLR 2015
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Attention Is All You Need, not before it. The transformer paper makes attention look inevitable and clean, but Bahdanau, Cho, and Bengio show the original engineering pain: an encoder had to squeeze a whole sentence into one vector, then a decoder had to recover every detail from that bottleneck. The trick here is not magic interpretability, it is giving the decoder a learned way to look back at the input each time it emits a token. People often remember this as the paper that made pretty alignment heatmaps. The more useful lesson is architectural: when a model fails because an intermediate representation must carry too much, do not just make it bigger. Change the information path. That idea still shows up in retrieval, KV caches, cross-attention, and tool use. This is worth your evening because it turns attention from a slogan into a concrete fix for a concrete failure mode.
What to take away
- The decoder computes a fresh context vector for each output token instead of trusting one fixed sentence embedding.
- Alignment is learned with gradient descent, not supplied as a separate translation table or hard preprocessing step.
- The paper is the bridge from RNN encoder-decoders to the attention-first design of transformers.
Reads with
- Sequence to Sequence Learning with Neural Networks
this fixes its single vector encoder bottleneck with a learned input lookup
- Attention Is All You Need
turns this decoder lookup trick into the whole sequence model
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
applies the same information path fix by looking outside the prompt