Build Your Own LLM, Lesson 9: Evaluating a Model — Loss, Perplexity and Test Prompts

📎 This article includes 4 downloadable practice files ↓

“It looks good” isn’t a measurement. Language models are compared with a few standard numbers, plus a set of prompts you check by eye every time you change something.

Loss and perplexity

Training minimises cross-entropy loss: how surprised the model is by the real next token. Perplexity = e^loss, roughly “how many tokens it is choosing between”. A model guessing randomly among 40 characters has perplexity 40; ours scored about 1.1 on held-out text because the sample is so repetitive.

losses = [model(*get_batch(val, 64, 32))[1].item() for _ in range(20)]
avg = sum(losses) / len(losses)
print(avg, math.exp(avg))

Fixed test prompts

Keep 10-20 prompts that matter for your use, generate with a fixed seed and low temperature after each training run, and compare. Regressions (it used to answer, now it rambles) show up immediately.

Beyond loss

• Benchmarks (multiple-choice tests of knowledge and reasoning) for general models.
• Task metrics: accuracy for classification, exact match for extraction.
• Human review for tone, helpfulness and safety, which no number captures well.

Practice

Run after lesson 7. Then retrain for fewer steps and compare the perplexity.