LoRA: Low-Rank Adaptation of Large Language Models
Edward J. Hu, Yelong Shen, Phillip Wallis, et al.2021ICLR 2022
Read it on arxiv.org(opens in a new tab)Why this one
This is the paper that put fine-tuning back within reach of someone with one GPU, and it is a linear algebra argument before it is a systems one. Freeze the pretrained weight matrix, add a low-rank pair beside it, and train only that pair - because the update a fine-tune actually applies turns out to be well approximated by something of very low rank. Two consequences follow immediately and are the reason to read it rather than just use it: the adapters can be merged back into the base weights so inference costs nothing extra, and a single base model can serve many tasks by swapping a few megabytes. If matrix rank has only ever been an exam topic for you, this is the paper that makes it operational.
What to take away
- The rank hyperparameter is the whole experiment - start small and only raise it when the loss says to.
- Merged adapters mean zero added inference latency, which is not true of most parameter-efficient methods.
- You need SVD intuition to read this properly; do that lesson first.
Reads with
- Attention Is All You Need
it is the transformer substrate whose projection matrices lora modifies
- Language Models are Few-Shot Learners
lora is the cheap adaptation answer to models too large to fully fine-tune
- Training language models to follow instructions with human feedback
instruction tuning is where parameter-efficient adaptation becomes operationally valuable