Efficient Estimation of Word Representations in Vector Space
Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean2013ICLR 2013
Read it on arxiv.org(opens in a new tab)Why this one
Embeddings are easy to use and easy to misunderstand. The useful idea here is not that words become points in space, but that a model can learn those points by solving a cheap prediction task at huge scale. Similarity is a side effect of pressure from context, not a hand written dictionary of meaning. That distinction matters when you build retrieval, recommendations, classifiers, or anything that depends on vector search. Read this to see the simplest version of representation learning before transformers added layers, attention, and scale. It will make modern embedding APIs feel less like magic and more like a trained compression of co occurrence.
What to take away
- Skip gram learns word vectors by predicting nearby words from the current word.
- Negative sampling turns a huge softmax problem into a small set of binary decisions.
- Vector arithmetic works because training forces related contexts into nearby directions.
Reads with
- Sequence to Sequence Learning with Neural Networks
uses learned word vectors inside the encoder decoder recipe word2vec helped normalize
- Attention Is All You Need
keeps the embedding idea but makes context dependent representations the core object
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
turns embedding similarity into a systems primitive for fetching knowledge