Build GPT from scratch

After this lesson you can: Implement a small working GPT and understand every line.

First: Attention and the transformer, PyTorch, first steps

Watch

• Let's build GPT: from scratch, in code, spelled out. - Andrej Karpathy, 116 min. Exact lecture match: code-first small GPT from scratch by a trusted teacher, aimed at demystifying implementation.
• Let's reproduce GPT-2 (124M) - Andrej Karpathy, 241 min. Strong follow-up for scaling intuition, but GPT-2 reproduction is less beginner-friendly than a small scratch build.

Notes

Building GPT from scratch means assembling a small language model out of ordinary tensor operations: token lookup, positional information, attention, feed-forward layers, residual connections, normalization, and a final prediction head. The goal is not to build a production chatbot; it is to make the transformer stop feeling like a black box by seeing how text becomes integers, integers become vectors, vectors interact through attention, and the model learns to predict the next token.

Unlock the full lesson notes, the exercises and the graded checkpoint - ₹5,000 once.