📎 This article includes 4 downloadable practice files ↓
The model outputs a probability for every possible next token. How you choose from those probabilities changes the text dramatically: the same model can be dull, creative or nonsensical.
In this article
Temperature
logits = logits[:, -1, :] / temperature
probs = F.softmax(logits, dim=-1)Below 1, the distribution sharpens (safe, repetitive); above 1 it flattens (varied, eventually gibberish). Near 0 it’s effectively greedy: always the top token.
Top-k
Keep only the k most likely tokens, set the rest to −∞, then sample. Cuts off the long tail of unlikely, weird choices.
Top-p (nucleus)
sp, si = torch.sort(probs, descending=True)
cum = sp.cumsum(-1)
sp[(cum - sp) > top_p] = 0 # keep the smallest set whose total is >= top_pAdapts to the situation: when the model is confident, very few tokens are kept; when unsure, more are.
What the test showed
At t=0.1 and with top-k or top-p, the model reproduced clean training sentences; at t=1.5 it started splicing fragments (“A piCSV file…”). That’s the trade-off every chat product tunes.
Practice
Run lesson 7 first (it saves tinygpt.pt), then this script. Try your own temperatures.
📎 Practice files for this article
⬇ Download all 4 files (ZIP · 4 KB)
- 🐍tinygpt.py (shared model code)The GPT model, tokenizer, batching and generation used in lessons 6-12.⬇ PY · 4 KB
- 🐍Lesson 8 scriptTested script for this lesson.⬇ PY · 700 B
- 📄Expected outputWhat the script printed in our test run (your numbers will be close).⬇ TXT · 406 B
- 📄sample.txtSmall training text; swap in any public-domain book for better results.⬇ TXT · 83 KB
Free to use for learning. Files with macros (.bas) are plain text — import them with Alt+F11 → File → Import File, and always test on a copy.
Stuck on a step? Ask a question and the AI answers using this article.