Build Your Own LLM, Lesson 7: Train a Tiny GPT on Your Own Text

📎 This article includes 4 downloadable practice files ↓

Time to train a real (tiny) GPT. The model in tinygpt.py is the same design as GPT-2, only smaller: token embeddings plus position embeddings, a stack of transformer blocks, a final LayerNorm and a linear layer that predicts the next character.

The training loop

model = TinyGPT(tok.vocab_size, d_model=96, n_heads=4, n_layers=3, block_size=64)
opt = torch.optim.AdamW(model.parameters(), lr=3e-3)
for step in range(steps + 1):
    xb, yb = get_batch(train, 64, 32)        # 32 random 64-character windows; yb = xb shifted by one
    _, loss = model(xb, yb)
    opt.zero_grad(); loss.backward(); opt.step()

Every window teaches 64 predictions at once: after “E” comes “x”, after “Ex” comes “c”… That’s why language models train efficiently.

Train vs validation loss

Keep the last 10% of the text aside. If training loss keeps falling but validation loss rises, the model is memorising rather than learning patterns (overfitting): use more text, add dropout, or stop earlier.

In our test the small, very repetitive sample file reached a validation loss of about 0.1 in 300 steps (around 40 seconds on a laptop CPU), meaning the model has essentially memorised the sentences.

Scaling up

Practice

Run the script on sample.txt, then on a book of your choice, and compare the generated text.