DeepSeek-V3 Technical Report
DeepSeek-AI, Aixin Liu, Bei Feng, et al.2024arXiv 2024
Read it on arxiv.org(opens in a new tab)Why this one
Read this after Switch Transformers and Efficient Memory Management for Large Language Model Serving with PagedAttention. DeepSeek-V3 is useful because it treats model architecture as a bill of materials, not as a leaderboard trick. The important idea is that sparsity only matters if the serving path can cash it in: fine-grained experts reduce activated compute, and multi-head latent attention attacks the KV cache instead of pretending attention cost is just FLOPs. People often read this as a story about one very large open model. That misses the engineering lesson. The report is an end to end argument about where money goes during training and inference, and which compromises move the curve without making the system unbuildable. If you are designing LLM serving, this is where MoE, cache shape, routing balance, and training stability stop being separate topics.
What to take away
- Multi-head latent attention compresses the KV path, so long-context serving becomes a memory problem you can actually budget.
- Fine-grained MoE makes sparsity less crude: many total parameters, few active per token, and routing that must stay balanced.
- The report ties architecture choices to GPU hours, stability, and deployment cost instead of reporting accuracy alone.
Reads with
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
sets up sparse expert routing as the scaling lever deepseek makes production shaped
- Efficient Memory Management for Large Language Model Serving with PagedAttention
shows why kv cache layout can dominate the economics of serving long generations
- Fast Transformer Decoding: One Write-Head is All You Need
the kv cache pressure it reduces is what multi head latent attention attacks more aggressively