Fully retraining a 7-billion-parameter model needs a rack of expensive GPUs. LoRA (Low-Rank Adaptation) lets you adapt the same model on a single consumer GPU — or a free cloud notebook — by training a tiny add-on instead of the whole thing.
In this article
The idea
A model layer is a big matrix W, for example 4096 × 4096 ≈ 16.8 million numbers. LoRA freezes W and learns a correction made of two thin matrices:
W' = W + B × A, where A is r × 4096 and B is 4096 × r
With rank r = 8, the two matrices hold 8 × (4096 + 4096) = 65,536 numbers — about 0.4% of the original. The bet (which works surprisingly well) is that adapting a model to a new task needs only a “low-rank” change.
What you get
- Small memory: gradients and optimiser state only for the adapter.
- Tiny files: an adapter is megabytes, not gigabytes. Keep one per task and swap them.
- The base model stays intact, so you cannot “break” it for other uses.
QLoRA goes further: the frozen base model is loaded in 4-bit, cutting memory again so 7–8B models fit on 16–24 GB GPUs.
Minimal setup with Hugging Face PEFT
“`python
# pip install transformers peft datasets accelerate
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model
name = “Qwen/Qwen2.5-0.5B” # small enough to try on a CPU or free GPU
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name)
config = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
target_modules=[“q_proj”, “v_proj”], task_type=”CAUSAL_LM”)
model = get_peft_model(model, config)
model.print_trainable_parameters() # a fraction of a percent is trainable
“`
From here you train with a normal training loop (or the Trainer / TRL SFTTrainer classes) on your examples, then model.save_pretrained("my-adapter").
Key settings
| Setting | Meaning | Typical |
|---|---|---|
r |
Rank — adapter capacity | 8–16 |
lora_alpha |
Scales the adapter’s effect | 2 × r |
target_modules |
Which layers get adapters | Attention projections; more layers = more capacity |
| Learning rate | Higher than full fine-tuning | 1e-4 to 2e-4 |
Fine-tune, prompt, or RAG?
| Need | Best tool |
|---|---|
| Answers based on your documents, which change | RAG (guide) |
| A consistent style, format or tone | Fine-tuning (LoRA) |
| A narrow task done by a small, cheap model | Fine-tuning |
| Everything else, first | A better prompt |
Data matters more than settings
A few hundred clean, consistent examples usually beat thousands of messy ones. Write them in the exact input/output format you want at the end, and keep 10% aside to test.
We fine-tune a real model step by step in Lesson 10 of the free Build Your Own LLM course.