BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova2019NAACL 2019
Read it on arxiv.org(opens in a new tab)Why this one
If you have used an embedding model, a reranker, a classifier head on top of a transformer, or a masked token objective, you have used the shape of this idea. The useful lesson is not that BERT beat old NLP benchmarks. It is that a transformer encoder can learn a general representation by filling in missing pieces of text, then be adapted with a small task head. People often file BERT as pre ChatGPT history, but that misses the split that still matters in systems today: encoders are built to understand inputs all at once, decoders are built to generate the next token. Reading this will make model choice less mystical. You will know why search stacks, moderation systems, extractors, and many embedding APIs still look more like BERT than like a chatbot.
What to take away
- Masked language modeling trains an encoder to use both left and right context without leaking the target token.
- The CLS token is a learned pooling convention, not a magic sentence meaning vector.
- Fine tuning here means reusing the whole encoder, then adding a small task specific layer on top.
Reads with
- Attention Is All You Need
the transformer encoder is the machinery bert pretrains
- Efficient Estimation of Word Representations in Vector Space
bert turns static predictive word embeddings into contextual sentence representations
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
it generalizes bert-style transfer into a unified text-to-text training recipe