Build Your Own LLM, Lesson 10: Fine-Tuning a Pre-Trained Model With LoRA

📎 This article includes 4 downloadable practice files ↓

Retraining every weight of a big model needs huge GPUs. LoRA (Low-Rank Adaptation) freezes the original weights and learns small “adapter” matrices instead, often under 1% of the parameters, with results close to full fine-tuning.

The idea

A weight update ΔW for a layer is approximated by two thin matrices: ΔW = B·A, where A is r × in and B is out × r, with a small rank r (4-16). B starts at zero, so the adapted model starts exactly as the original.

class LoRALinear(nn.Module):
    def __init__(self, base, r=4, alpha=8):
        super().__init__()
        self.base = base                                   # frozen
        self.A = nn.Parameter(torch.randn(r, base.in_features) * 0.01)
        self.B = nn.Parameter(torch.zeros(base.out_features, r))
        self.scale = alpha / r
    def forward(self, x):
        return self.base(x) + (x @ self.A.T @ self.B.T) * self.scale

In the script, LoRA on the attention layers trains 6,912 of 355,584 parameters (1.9%), and loss on the new text drops from 6.9 to about 0.15 in 150 steps.

On real open models

Libraries like Hugging Face peft do the same with a few lines (LoraConfig(r=8, target_modules=["q_proj","v_proj"])). Combined with 4-bit loading (QLoRA), 7-8B parameter models can be fine-tuned on a single consumer GPU.

More: LoRA fine-tuning explained.

Practice

Run after lesson 7, then change r to 1 and 16 and compare the trainable parameter count and loss.