Build Your Own LLM, Lesson 5: Self-Attention From Scratch
Understand and code the attention mechanism behind transformers: queries, keys, values, the causal mask and softmax — first with NumPy on four tokens, then…
Read article →Category Archives: Build Your Own LLM
A step-by-step course: tokenizer, embeddings, attention and a tiny GPT you train yourself.
Build Your Own LLM, Lesson 4: Embeddings and Your First Neural Language Model
Build Your Own LLM, Lesson 4: Embeddings and Your First Neural Language Model
Rebuild the bigram model as a neural network in PyTorch, train it with gradient descent, and add embeddings — the vectors that let language…
Read article →Build Your Own LLM, Lesson 3: A Language Model Made of Counting
Build Your Own LLM, Lesson 3: A Language Model Made of Counting
Build a bigram language model with plain Python dictionaries, generate text from it, and measure it with loss — the same number every LLM…
Read article →Build Your Own LLM, Lesson 2: Write a BPE Tokenizer in Python
Build Your Own LLM, Lesson 2: Write a BPE Tokenizer in Python
Build the byte-pair encoding tokenizer used by GPT-style models from scratch in about 40 lines of Python: training merges, encoding, decoding and why the…
Read article →Build Your Own LLM, Lesson 1: How a Language Model Actually Works
Build Your Own LLM, Lesson 1: How a Language Model Actually Works
A plain-English tour of what happens inside ChatGPT-style models: tokens, probabilities, training on next-token prediction, and why a model that only guesses the next…
Read article →