🧠 Lesson 3 of 12 · Build Your Own LLM course

Before neural networks, let’s build a language model with nothing but counting. It will write nonsense — but it introduces the three things every LLM has: probabilities for the next token, sampling, and a loss that tells us how good the model is.

The data

Use any plain-text file of at least 100 KB: a public-domain book from Project Gutenberg, your own blog posts, or song titles. We work at character level to keep things visible.

“`python
text = open(“input.txt”, encoding=”utf-8″).read()
chars = sorted(set(text))
print(len(text), “characters,”, len(chars), “unique”)
“`

Counting what follows what

A bigram model looks only at the previous character. For every character we count which characters come next:

“`python
from collections import defaultdict, Counter

counts = defaultdict(Counter)
for a, b in zip(text, text[1:]):
counts[a][b] += 1

print(counts[“q”].most_common(3)) # in English, almost always ‘u’
print(counts[” “].most_common(5)) # common first letters of words
“`

Divide each count by the row total and you have probabilities. That table is the model — its “parameters” are the counts.

Generating text

“`python
import random

def next_char(c):
options = counts[c]
return random.choices(list(options), weights=list(options.values()))[0]

c = “T”
out = [c]
for _ in range(300):
c = next_char(c)
out.append(c)
print(“”.join(out))
“`

You will get word-shaped gibberish like “Thend the ware ofos”. It has the right letter rhythm but no memory beyond one character — exactly the limitation that attention solves later.

Measuring the model: loss

How do we score a language model? Look at real text and ask: how much probability did the model give to what actually came next? We take the negative log of that probability and average it. This is the cross-entropy loss, the same number GPT-style models are trained to minimise.

“`python
import math

totals = {a: sum(c.values()) for a, c in counts.items()}
nll, n = 0.0, 0
for a, b in zip(text, text[1:]):
p = counts[a][b] / totals[a]
nll += -math.log(p)
n += 1
print(“average loss:”, round(nll / n, 3))
print(“random-guess loss:”, round(math.log(len(chars)), 3))
“`

• A model that guesses uniformly at random scores log(vocabulary size) — about 4.2 for 65 characters.
• The bigram model typically lands around 2.4–2.6 on English text: better than random, far from good.
• A perfect model that always gives 100% to the right answer would score 0.

The zero-count problem

We measured on the training text, so every pair had been seen. On new text, an unseen pair gets probability 0 and log(0) is minus infinity. The classic fix is smoothing — pretend every pair was seen once more than it was:

“`python
def prob(a, b):
return (counts[a][b] + 1) / (totals.get(a, 0) + len(chars))
“`

Why not just count longer contexts?

A trigram model (two characters of context) is better, a 5-gram better still. But the table grows explosively: with 65 characters there are 65⁵ ≈ 1.16 billion possible 5-character contexts, and almost all of them never appear in your data. Counting cannot generalise — it cannot guess that “Excel” and “excel” behave alike.

Neural networks fix this by learning embeddings: similar tokens get similar vectors, so knowledge transfers between them. That is Lesson 4.

Exercise

1. Split the text 90/10, build counts on the first part, and measure smoothed loss on the second. Is it higher than training loss? (It should be — this gap is called overfitting.)
2. Build a trigram version with counts[text[i:i+2]][text[i+2]] and compare the generated text.