Build Your Own LLM, Lesson 2: Write a BPE Tokenizer in Python

📎 This article includes 1 downloadable practice file ↓

🧠 Lesson 2 of 12 · Build Your Own LLM course

In Lesson 1 we said models read tokens, not letters. Today we build the tokenizer. The algorithm GPT models use is byte-pair encoding (BPE), and the core of it is surprisingly small.

Why not just use words or letters?

• Letters: tiny vocabulary, but sequences get very long, and the model must spend effort re-learning that t-h-e is a word.
• Words: short sequences, but the vocabulary explodes (every name, typo and number) and unseen words cannot be represented at all.
• BPE: starts with bytes (so anything can be encoded, including emoji and Hindi) and learns to glue frequent neighbours into bigger tokens. Common words end up as one token; rare words become a few pieces.

The idea in one example

Take the text aaabdaaabac. The most frequent adjacent pair is aa. Replace it with a new symbol Z: ZabdZabac. Now the most common pair is Za → Y: YbdYbac. Each replacement is a merge, and the list of merges is the trained tokenizer.

Step 1: text to bytes

text = "नमस्ते Excel! Excel formulas are excellent."
ids = list(text.encode("utf-8"))
print(len(text), "characters ->", len(ids), "bytes")
print(ids[:12])

UTF-8 turns every character into 1–4 bytes (values 0–255). Hindi characters take 3 bytes each, which is why a byte-level start works for any language. Our base vocabulary is therefore the 256 possible byte values.

Step 2: count pairs and merge

from collections import Counter

def get_pairs(ids):
    """Count how often each adjacent pair appears."""
    return Counter(zip(ids, ids[1:]))

def merge(ids, pair, new_id):
    """Replace every occurrence of `pair` with `new_id`."""
    out, i = [], 0
    while i < len(ids):
        if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

Step 3: training = repeat the most common merge

def train(text, vocab_size):
    ids = list(text.encode("utf-8"))
    merges = {}                              # (a, b) -> new token id
    for new_id in range(256, vocab_size):
        pairs = get_pairs(ids)
        if not pairs:
            break
        pair = pairs.most_common(1)[0][0]
        ids = merge(ids, pair, new_id)
        merges[pair] = new_id
    return merges

corpus = open("my_text.txt", encoding="utf-8").read()   # any few hundred KB of text
merges = train(corpus, vocab_size=512)                  # learn 256 merges

Each loop scans the text once, so this simple version is slow on huge files — fine for learning with a few hundred kilobytes. Production tokenizers use clever data structures, but the result is the same kind of merge list.

Step 4: encoding new text

To encode, start from bytes and apply merges in the order they were learned — earliest merges first, because later merges were built on top of them.

def encode(text, merges):
    ids = list(text.encode("utf-8"))
    while len(ids) >= 2:
        pairs = get_pairs(ids)
        # the pair that was learned earliest has the lowest new id
        pair = min(pairs, key=lambda p: merges.get(p, float("inf")))
        if pair not in merges:
            break                              # nothing left to merge
        ids = merge(ids, pair, merges[pair])
    return ids

Step 5: decoding back to text

def build_vocab(merges):
    vocab = {i: bytes([i]) for i in range(256)}
    for (a, b), idx in merges.items():         # dicts keep insertion order = merge order
        vocab[idx] = vocab[a] + vocab[b]
    return vocab

def decode(ids, vocab):
    return b"".join(vocab[i] for i in ids).decode("utf-8", errors="replace")

vocab = build_vocab(merges)
sample = "Excel formulas are excellent"
tokens = encode(sample, merges)
print(len(sample.encode("utf-8")), "bytes ->", len(tokens), "tokens")
print([vocab[t].decode("utf-8", errors="replace") for t in tokens])
assert decode(tokens, vocab) == sample

errors="replace" matters: a token can end in the middle of a multi-byte character, and a model can generate an invalid byte sequence. Replacing instead of crashing keeps generation going.

What you should see

On English text, 256 merges usually shrink the byte count by roughly a third to a half. Print the vocabulary and you will see the story of your corpus: first spaces plus common letters (" t", "th"), then whole words (" the", " Excel").

for idx in range(256, 276):
    print(idx, repr(vocab[idx].decode("utf-8", errors="replace")))

Vocabulary size is a trade-off

GPT-2 used about 50,000 tokens; many newer models use 100,000–200,000 to handle more languages and code efficiently. For our tiny model we will use a character-level or 512-token vocabulary to keep training fast.

Real tokenizers add a few extras

• Pre-splitting: text is first split with a regular expression (words, numbers, punctuation) so merges never cross those boundaries — you do not want "dog." and "dog!" to become separate tokens.
• Special tokens like <|endoftext|> mark document or message boundaries. Chat models use them to separate the user and assistant turns (Lesson 11).

Exercise

1. Train on 200 KB of English, then encode a Hindi sentence. Why does it use so many more tokens?
2. Add a pre-split step with re.findall(r"\s?\w+|\s?[^\w\s]+|\s+", text) and train merges per chunk.
3. Save merges to JSON so you can reuse the tokenizer in Lesson 4.

Next lesson: we use tokens to build our first language model — using nothing but counting.