📎 This article includes 4 downloadable practice files ↓
A 7-billion-parameter model in 32-bit floats needs about 28 GB; in 4-bit it fits in about 4 GB and runs on a laptop. Quantization stores weights with fewer bits at a small cost in quality.
In this article
Try it on the tiny GPT
q = torch.quantization.quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)The script’s linear layers go from 1,423 KB to 449 KB, with the same generated text in our test. On such a tiny model int8 wasn’t faster (0.44 s vs 0.30 s); the speed benefit appears on large models where memory bandwidth is the bottleneck.
Formats you’ll meet
| Format | Use |
|---|---|
| fp16 / bf16 | Standard for GPU inference and training |
| int8 | Halves memory again, little quality loss |
| 4-bit (GPTQ, AWQ, bitsandbytes) | Run big models on one consumer GPU |
| GGUF (Q4_K_M, Q5…) | llama.cpp and Ollama on CPUs and Macs |
Run an open model locally
- Install Ollama (ollama.com) on Windows, Mac or Linux.
ollama run llama3.2(or another model from its library) downloads a quantized GGUF and starts a chat.- It also serves an API on localhost, so your Python scripts can call it without sending data to the cloud.
You’ve finished the course
You built a tokenizer, a counting model, embeddings, attention, a transformer, trained a tiny GPT, sampled, evaluated, fine-tuned with LoRA, prepared chat data and quantized it. That’s the full pipeline behind modern LLMs, in miniature.
Practice
Run after lesson 7, then install Ollama and chat with a small open model.
📎 Practice files for this article
⬇ Download all 4 files (ZIP · 4 KB)
- 🐍tinygpt.py (shared model code)The GPT model, tokenizer, batching and generation used in lessons 6-12.⬇ PY · 4 KB
- 🐍Lesson 12 scriptTested script for this lesson.⬇ PY · 879 B
- 📄Expected outputWhat the script printed in our test run (your numbers will be close).⬇ TXT · 195 B
- 📄sample.txtSmall training text; swap in any public-domain book for better results.⬇ TXT · 83 KB
Free to use for learning. Files with macros (.bas) are plain text — import them with Alt+F11 → File → Import File, and always test on a copy.
Stuck on a step? Ask a question and the AI answers using this article.