Build Your Own LLM, Lesson 11: Instruction Tuning and Chat Templates

📎 This article includes 4 downloadable practice files ↓

A model trained only on books continues text; it doesn’t answer questions. Instruction tuning trains it on many examples of “user asks, assistant answers”, formatted with special markers so it learns the turn-taking.

Chat templates

<|user|>
What does VLOOKUP do?
<|assistant|>
It finds a value in the first column of a table and returns a value from another column.<|end|>

The markers are special tokens the model learns to recognise. Every model family uses its own format, so always apply the template from the model card (Hugging Face tokenizers have apply_chat_template()).

Learn only from the answers

You don’t want the model to learn to write questions; you want answers. Set the labels for prompt positions to −100, which PyTorch’s cross-entropy ignores:

mask = [0] * len(prompt) + [1] * len(answer)
labels = torch.where(mask == 1, ids, torch.tensor(-100))

Where the data comes from

• Curated question-answer sets written by people.
• Your own domain: past support tickets and their resolutions (anonymised).
• Synthetic data generated by a larger model, then reviewed.

After instruction tuning, models are often further aligned with human preference data (RLHF, DPO) to be more helpful and safe.

Practice

Run the script and add two of your own question-answer pairs.