Daily
One paper at a time
A foundational paper, and a note on why it still matters to the code you write. 113 in the archive so far, each one pointing at the module where it stops being background reading.
- Papers
- 113
- Tracks
- 4
- Reading
- 42 hrs
- This month
- 12
Everything in the archive
Newest first. Open one for the note and the takeaways.
- AI Research2022Training Compute-Optimal Large Language ModelsJordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al. - NeurIPS 202228 min readscalingllmstrainingcompute
- Inference Engineering2022Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, et al. - NeurIPS 202218 min readreasoningpromptingllms
- AI Research2020Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, et al. - NeurIPS 202035 min readlanguage-modelspromptingscaling
- AI Research2021Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, et al. - ICML 202128 min readmultimodalembeddingscontrastive-learning
- AI Research2020Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter Abbeel - NeurIPS 202028 min readdiffusiongenerationvision
- AI Research2023Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, et al. - NeurIPS 202328 min readalignmentpreferencefine-tuning
- AI Research2014Sequence to Sequence Learning with Neural NetworksIlya Sutskever, Oriol Vinyals, Quoc V. Le - NeurIPS 201418 min readseq2seqnlpencoder-decoder
- AI Research2019BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova - NAACL 201928 min readtransformersembeddingspretraining
- AI Research2013Efficient Estimation of Word Representations in Vector SpaceTomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean - ICLR 201318 min readembeddingsnlprepresentation
- AI Research2022Training language models to follow instructions with human feedbackLong Ouyang, Jeff Wu, Xu Jiang, et al. - NeurIPS 202228 min readalignmentrlhffine-tuning
- AI Research2020Retrieval-Augmented Generation for Knowledge-Intensive NLP TasksPatrick Lewis, Ethan Perez, Aleksandra Piktus, et al. - NeurIPS 202012 min readretrievallanguage modelsevaluation
- AI Research2021LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, et al. - ICLR 202210 min readfine-tuningefficiencylinear algebra
- System Design1978Time, Clocks, and the Ordering of Events in a Distributed SystemLeslie Lamport - Communications of the ACM14 min readdistributed systemsconsistencyfoundations
- System Design1970Space/Time Trade-offs in Hash Coding with Allowable ErrorsBurton H. Bloom - Communications of the ACM8 min readprobabilistic structuresstoragefoundations
- System Design2014In Search of an Understandable Consensus AlgorithmDiego Ongaro, John Ousterhout - USENIX ATC 201425 min readconsensusreplicationdistributed systems
- System Design2004MapReduce: Simplified Data Processing on Large ClustersJeffrey Dean, Sanjay Ghemawat - OSDI 2004 / CACM 200816 min readbatch processingfault tolerancefoundations
- System Design2007Dynamo: Amazon's Highly Available Key-value StoreGiuseppe DeCandia, Deniz Hastorun, Madan Jampani, et al. - SOSP 200730 min readavailabilitypartitioningreplication
- System Design1996The Log-Structured Merge-Tree (LSM-Tree)Patrick O'Neil, Edward Cheng, Dieter Gawlick, Elizabeth O'Neil - Acta Informatica22 min readstorage engineswrite amplificationindexing
- AI Research2014Adam: A Method for Stochastic OptimizationDiederik P. Kingma, Jimmy Ba - ICLR 201512 min readoptimisationtrainingfoundations
- AI Research2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate ShiftSergey Ioffe, Christian Szegedy - ICML 201514 min readtrainingnormalisationarchitecture
- AI Research2015Deep Residual Learning for Image RecognitionKaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun - CVPR 201616 min readarchitectureoptimisationfoundations
- AI Research2017Attention Is All You NeedAshish Vaswani, Noam Shazeer, Niki Parmar, et al. - NeurIPS 201720 min readtransformersattentionfoundations
- System Design2003The Google File SystemSanjay Ghemawat, Howard Gobioff, Shun-Tak Leung - SOSP 200328 min readstoragedistributed-systemsfault-tolerance
- Inference Engineering2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré - NeurIPS 202224 min readattentiongpumemorytransformers
- AI Research2015Neural Machine Translation by Jointly Learning to Align and TranslateDzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio - ICLR 201524 min readattentiontranslationseq2seq
- ML Mathematics2020Scaling Laws for Neural Language ModelsJared Kaplan, Sam McCandlish, Tom Henighan, et al. - arXiv 202018 min readscalinglanguage-modelscomputeestimation
- System Design2006Bigtable: A Distributed Storage System for Structured DataFay Chang, Jeffrey Dean, Sanjay Ghemawat, et al. - OSDI 200628 min readstoragedatabasesnosqlpartitioning
- Inference Engineering2023Efficient Memory Management for Large Language Model Serving with PagedAttentionWoosuk Kwon, Zhuohan Li, Siyuan Zhuang, et al. - SOSP 202328 min readinferencememorykv-cacheserving
- AI Research2014Generative Adversarial NetworksIan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. - NeurIPS 201418 min readgenerative-modelsoptimizationdeep-learning
- ML Mathematics2018The Matrix Calculus You Need For Deep LearningTerence Parr, Jeremy Howard - arXiv 201818 min readcalculusbackproplinear-algebra
- System Design2012Spanner: Google's Globally-Distributed DatabaseJames C. Corbett, Jeffrey Dean, Michael Epstein, et al. - OSDI 201232 min readdatabasesconsistencydistributed-systems
- Inference Engineering2023Fast Inference from Transformers via Speculative DecodingYaniv Leviathan, Matan Kalman, Yossi Matias - ICML 202318 min readinferencetransformerslatencysampling
- AI Research2014Auto-Encoding Variational BayesDiederik P. Kingma, Max Welling - ICLR 201428 min readgenerative-modelsvariational-inferencelatent-variables
- ML Mathematics2010Understanding the difficulty of training deep feedforward neural networksXavier Glorot, Yoshua Bengio - AISTATS 201018 min readinitializationgradientsvariance
- System Design2001Paxos Made SimpleLeslie Lamport - ACM SIGACT News 200118 min readconsensusdistributed-systemsfault-tolerance
- Inference Engineering2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleTim Dettmers, Mike Lewis, Younes Belkada, Luke Zettlemoyer - NeurIPS 202218 min readquantizationinferencetransformers
- AI Research2022Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient SparsityWilliam Fedus, Barret Zoph, Noam Shazeer - JMLR 202228 min readtransformersmoescalingtraining
- System Design2010ZooKeeper: Wait-free Coordination for Internet-scale SystemsPatrick Hunt, Mahadev Konar, Flavio P. Junqueira, Benjamin Reed - USENIX ATC 201024 min readcoordinationconsensusdistributed-systems
- System Design1997Consistent Hashing and Random Trees: Distributed Caching Protocols for Relieving Hot Spots on the World Wide WebDavid Karger, Eric Lehman, Tom Leighton, et al. - STOC 199718 min readhashingshardingcachingdistributed-systems
- Inference Engineering2023GPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersElias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh - ICLR 202322 min readquantizationinferencellmscompression
- AI Research2014Dropout: A Simple Way to Prevent Neural Networks from OverfittingNitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, et al. - JMLR 201418 min readregularizationneural-networksensembles
- ML Mathematics2013On the difficulty of training Recurrent Neural NetworksRazvan Pascanu, Tomas Mikolov, Yoshua Bengio - ICML 201318 min readrnngradientseigenvalues
- System Design2012CAP Twelve Years Later: How the Rules Have ChangedEric Brewer - IEEE Computer 201212 min readdistributed systemsconsistencyavailability
- Inference Engineering2018Mixed Precision TrainingPaulius Micikevicius, Sharan Narang, Jonah Alben, et al. - ICLR 201818 min readprecisiontrainingnumerics
- AI Research2012ImageNet Classification with Deep Convolutional Neural NetworksAlex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton - NeurIPS 201218 min readvisioncnntraininggpu
- ML Mathematics2019The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural NetworksJonathan Frankle, Michael Carbin - ICLR 201924 min readsparsityinitializationpruning
- System Design2013The Tail at ScaleJeffrey Dean, Luiz André Barroso - Communications of the ACM 201318 min readlatencyreliabilitydistributed-systems
- Inference Engineering2020ZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsSamyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He - SC 202028 min readtrainingmemoryparallelism
- AI Research2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerNoam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, et al. - ICLR 201728 min readmoescalingtransformersrouting
- ML Mathematics2018Neural Ordinary Differential EquationsRicky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud - NeurIPS 201828 min readodedifferentiationcontinuous-depth
- System Design2013Scaling Memcache at FacebookRajesh Nishtala, Hans Fugal, Steven Grimm, et al. - NSDI 201328 min readcachingdistributed-systemsoperations
- Inference Engineering2022Efficiently Scaling Transformer InferenceReiner Pope, Sholto Douglas, Aakanksha Chowdhery, et al. - MLSys 202324 min readinferencetransformersparallelismlatency
- System Design2012Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster ComputingMatei Zaharia, Mosharaf Chowdhury, Tathagata Das, et al. - NSDI 201218 min readsparkdatafault-toleranceclusters
- AI Research1997Long Short-Term MemorySepp Hochreiter, Jurgen Schmidhuber - Neural Computation 199728 min readrnnmemorysequence-models
- ML Mathematics1948A Mathematical Theory of CommunicationClaude E. Shannon - Bell System Technical Journal35 min readentropyinformationprobabilitycompression
- Inference Engineering2015Distilling the Knowledge in a Neural NetworkGeoffrey Hinton, Oriol Vinyals, Jeff Dean - arXiv 201518 min readdistillationcompressioninference
- System Design2015Large-scale cluster management at Google with BorgAbhishek Verma, Luis Pedrosa, Madhukar Korupolu, et al. - EuroSys 201528 min readschedulingcontainersoperations
- AI Research2015U-Net: Convolutional Networks for Biomedical Image SegmentationOlaf Ronneberger, Philipp Fischer, Thomas Brox - MICCAI 201518 min readsegmentationcnndiffusionvision
- ML Mathematics2016Layer NormalizationJimmy Lei Ba, Jamie Ryan Kiros, Geoffrey E. Hinton - arXiv 201618 min readnormalizationoptimizationtransformers
- System Design2010Dapper, a Large-Scale Distributed Systems Tracing InfrastructureBenjamin H. Sigelman, Luiz André Barroso, Mike Burrows, et al. - Google Research 201024 min readtracingobservabilitydistributed-systems
- Inference Engineering2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMohammad Shoeybi, Mostofa Patwary, Raul Puri, et al. - arXiv 201924 min readparallelismtransformersgputraining
- AI Research2013Playing Atari with Deep Reinforcement LearningVolodymyr Mnih, Koray Kavukcuoglu, David Silver, et al. - NeurIPS Deep Learning Workshop 201318 min readreinforcement-learningdeep-learningq-learning
- ML Mathematics1977Maximum Likelihood from Incomplete Data via the EM AlgorithmA. P. Dempster, N. M. Laird, D. B. Rubin - Journal of the Royal Statistical Society B28 min readestimationlatent-variablesoptimization
- System Design2001Chord: A Scalable Peer-to-peer Lookup Service for Internet ApplicationsIon Stoica, Robert Morris, David Karger, et al. - SIGCOMM 200118 min readdistributed-systemshashingp2prouting
- Inference Engineering2019Fast Transformer Decoding: One Write-Head is All You NeedNoam Shazeer - arXiv 201918 min readattentionkv-cachedecodingtransformers
- AI Research2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning AlgorithmDavid Silver, Thomas Hubert, Julian Schrittwieser, et al. - arXiv 201724 min readreinforcement-learningsearchself-play
- System Design2010Cassandra: A Decentralized Structured Storage SystemAvinash Lakshman, Prashant Malik - ACM SIGOPS 201018 min readdatabasesstoragereplication
- ML Mathematics1989Approximation by superpositions of a sigmoidal functionGeorge Cybenko - Mathematics of Control, Signals and Systems18 min readapproximationneural-networkstheory
- Inference Engineering2023SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language ModelsGuangxuan Xiao, Ji Lin, Mickael Seznec, et al. - ICML 202318 min readquantizationinferencellmsserving
- System Design1970A Relational Model of Data for Large Shared Data BanksE. F. Codd - Communications of the ACM22 min readdatabasessqldata-modeling
- AI Research2017Proximal Policy Optimization AlgorithmsJohn Schulman, Filip Wolski, Prafulla Dhariwal, et al. - arXiv 201718 min readreinforcement-learningpolicy-gradientsrlhf
- ML Mathematics1995Support-Vector NetworksCorinna Cortes, Vladimir Vapnik - Machine Learning 199528 min readkernelsclassificationgeometry
- Inference Engineering2023The case for 4-bit precision: k-bit Inference Scaling LawsTim Dettmers, Luke Zettlemoyer - ICML 202322 min readquantizationinferencescaling
- System Design1982The Byzantine Generals ProblemLeslie Lamport, Robert Shostak, Marshall Pease - ACM TOPLAS 198218 min readconsensusfault-tolerancedistributed-systems
- AI Research2020Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerColin Raffel, Noam Shazeer, Adam Roberts, et al. - JMLR 202032 min readtransfer-learningtransformersfine-tuning
- ML Mathematics1997No Free Lunch Theorems for OptimizationDavid H. Wolpert, William G. Macready - IEEE Transactions on Evolutionary Computation28 min readoptimizationinductionml-theory
- System Design1984End-to-End Arguments in System DesignJ. H. Saltzer, D. P. Reed, D. D. Clark - ACM TOCS 198418 min readsystem-designnetworkingreliability
- Inference Engineering2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsJoshua Ainslie, James Lee-Thorp, Michiel de Jong, et al. - EMNLP 202318 min readattentioninferencetransformerskv-cache
- AI Research2021An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al. - ICLR 202118 min readvisiontransformerspretraining
- ML Mathematics2001Random ForestsLeo Breiman - Machine Learning 200128 min readensemblestreesvariancetabular
- System Design1985Impossibility of Distributed Consensus with One Faulty ProcessMichael J. Fischer, Nancy A. Lynch, Michael S. Paterson - Journal of the ACM 198525 min readconsensusdistributed-systemsfault-tolerance
- Inference Engineering2023FlashAttention-2: Faster Attention with Better Parallelism and Work PartitioningTri Dao - arXiv 202318 min readattentiongpukernelsinference
- System Design1988Congestion Avoidance and ControlVan Jacobson - SIGCOMM 198818 min readnetworkingcongestionrate-limiting
- AI Research2021Highly accurate protein structure prediction with AlphaFoldJohn Jumper, Richard Evans, Alexander Pritzel, et al. - Nature 202130 min readbiologyattentionstructurebenchmarks
- Inference Engineering2023Ring Attention with Blockwise Transformers for Near-Infinite ContextHao Liu, Matei Zaharia, Pieter Abbeel - arXiv 202322 min readattentiondistributed-systemslong-contextinference
- System Design2002Practical Byzantine Fault ToleranceMiguel Castro, Barbara Liskov - ACM TOCS 200228 min readconsensusbyzantinereplicationdistributed-systems
- AI Research2023Robust Speech Recognition via Large-Scale Weak SupervisionAlec Radford, Jong Wook Kim, Tao Xu, et al. - ICML 202318 min readspeechdatascalingrobustness
- ML Mathematics2016XGBoost: A Scalable Tree Boosting SystemTianqi Chen, Carlos Guestrin - KDD 201628 min readboostingtreesoptimizationsystems
- Inference Engineering2024AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and AccelerationJi Lin, Jiaming Tang, Haotian Tang, et al. - MLSys 202422 min readquantizationllminferenceedge
- System Design2008Bitcoin: A Peer-to-Peer Electronic Cash SystemSatoshi Nakamoto - bitcoin.org 200825 min readconsensuscryptographydistributed-systems
- AI Research2022Constitutional AI: Harmlessness from AI FeedbackYuntao Bai, Saurav Kadavath, Sandipan Kundu, et al. - arXiv 202228 min readalignmentrlhffeedbacksafety
- ML Mathematics2017Understanding deep learning requires rethinking generalizationChiyuan Zhang, Samy Bengio, Moritz Hardt, et al. - ICLR 201718 min readgeneralizationcapacityregularization
- Inference Engineering2023Mamba: Linear-Time Sequence Modeling with Selective State SpacesAlbert Gu, Tri Dao - arXiv 202328 min readsequence-modelsssminferencearchitecture
- System Design2010Large-scale Incremental Processing Using Distributed Transactions and NotificationsDaniel Peng, Frank Dabek - OSDI 201024 min readindexingtransactionsdistributed-systems
- AI Research2023ReAct: Synergizing Reasoning and Acting in Language ModelsShunyu Yao, Jeffrey Zhao, Dian Yu, et al. - ICLR 202318 min readagentspromptingtools
- System Design2010Dremel: Interactive Analysis of Web-Scale DatasetsSergey Melnik, Andrey Gubarev, Jing Jing Long, et al. - VLDB 201022 min readanalyticsstoragesqlcolumnar
- ML Mathematics2017Wasserstein GANMartin Arjovsky, Soumith Chintala, Leon Bottou - ICML 201728 min readganoptimizationprobability
- Inference Engineering2024SGLang: Efficient Execution of Structured Language Model ProgramsLianmin Zheng, Liangsheng Yin, Zhiqiang Xie, et al. - NeurIPS 202424 min readinferencellmskv-cacheagents
- System Design2012Calvin: Fast Distributed Transactions for Partitioned Database SystemsAlexander Thomson, Thaddeus Diamond, Shu-Chun Weng, et al. - SIGMOD 201218 min readtransactionsdatabasescoordination
- AI Research2023Toolformer: Language Models Can Teach Themselves to Use ToolsTimo Schick, Jane Dwivedi-Yu, Roberto Dessi, et al. - NeurIPS 202324 min readagentstool-uselanguage-models
- ML Mathematics2018Neural Tangent Kernel: Convergence and Generalization in Neural NetworksArthur Jacot, Franck Gabriel, Clement Hongler - NeurIPS 201832 min readkernelsoptimizationgeneralization
- Inference Engineering2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsTianle Cai, Yuhong Li, Zhengyang Geng, et al. - ICML 202424 min readinferencedecodinglatencytransformers
- System Design2015Building a Replicated Logging System with Apache KafkaGuozhang Wang, Joel Koshy, Sriram Subramanian, et al. - VLDB 201518 min readlogsstreamsreplicationkafka
- AI Research2024The Llama 3 Herd of ModelsAaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. - arXiv 202435 min readllmstrainingdataposttraining
- ML Mathematics2020Deep Double Descent: Where Bigger Models and More Data HurtPreetum Nakkiran, Gal Kaplun, Yamini Bansal, et al. - ICLR 202018 min readgeneralizationoverfittingstatistics
- Inference Engineering2024DeepSeek-V3 Technical ReportDeepSeek-AI, Aixin Liu, Bei Feng, et al. - arXiv 202432 min readinferencemoeattentiontraining
- System Design2017Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational DatabasesAlexandre Verbitski, Anurag Gupta, Debanjan Saha, et al. - SIGMOD 201724 min readdatabasesstoragecloudreplication
- AI Research2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI, Daya Guo, Dejian Yang, et al. - arXiv 202528 min readreinforcement-learningreasoningllms
- ML Mathematics2022Grokking: Generalization Beyond Overfitting on Small Algorithmic DatasetsAlethea Power, Yuri Burda, Harri Edwards, et al. - arXiv 202218 min readgeneralizationoptimizationoverfitting
- Inference Engineering2022Efficient Transformers: A SurveyYi Tay, Mostafa Dehghani, Dara Bahri, Donald Metzler - ACM Computing Surveys 202235 min readtransformersattentionefficiencysurvey
- System Design2019Zanzibar: Google's Consistent, Global Authorization SystemRuoming Pang, Ramon Caceres, Mike Burrows, et al. - USENIX ATC 201928 min readauthorizationconsistencydistributed-systemsgraph
- ML Mathematics2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet ClassificationKaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun - ICCV 201518 min readinitializationactivationsvariancecnn
- ML Mathematics2008Visualizing Data using t-SNELaurens van der Maaten, Geoffrey Hinton - JMLR 200818 min readvisualizationembeddingsprobabilitymanifolds