🧠 Lesson 2 of 12 · Build Your Own LLM course

In Lesson 1 we said models read tokens, not letters. Today we build the tokenizer. The algorithm GPT models use is byte-pair encoding (BPE), and the core of it is surprisingly small.

Why not just use words or letters?

• Letters: tiny vocabulary, but sequences get very long, and the model must spend effort re-learning that t-h-e is a word.
• Words: short sequences, but the vocabulary explodes (every name, typo and number) and unseen words cannot be represented at all.
• BPE: starts with bytes (so anything can be encoded, including emoji and Hindi) and learns to glue frequent neighbours into bigger tokens. Common words end up as one token; rare words become a few pieces.

The idea in one example

Take the text aaabdaaabac. The most frequent adjacent pair is aa. Replace it with a new symbol Z: ZabdZabac. Now the most common pair is Za → Y: YbdYbac. Each replacement is a merge, and the list of merges is the trained tokenizer.

Step 1: text to bytes

“`python
text = “नमस्ते Excel! Excel formulas are excellent.”
ids = list(text.encode(“utf-8”))
print(len(text), “characters ->”, len(ids), “bytes”)
print(ids[:12])
“`

UTF-8 turns every character into 1–4 bytes (values 0–255). Hindi characters take 3 bytes each, which is why a byte-level start works for any language. Our base vocabulary is therefore the 256 possible byte values.

Step 2: count pairs and merge

“`python
from collections import Counter

def get_pairs(ids):
“””Count how often each adjacent pair appears.”””
return Counter(zip(ids, ids[1:]))

def merge(ids, pair, new_id):
“””Replace every occurrence of `pair` with `new_id`.”””
out, i = [], 0
while i < len(ids): if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair: out.append(new_id) i += 2 else: out.append(ids[i]) i += 1 return out ```

Step 3: training = repeat the most common merge

“`python
def train(text, vocab_size):
ids = list(text.encode(“utf-8”))
merges = {} # (a, b) -> new token id
for new_id in range(256, vocab_size):
pairs = get_pairs(ids)
if not pairs:
break
pair = pairs.most_common(1)[0][0]
ids = merge(ids, pair, new_id)
merges[pair] = new_id
return merges

corpus = open(“my_text.txt”, encoding=”utf-8″).read() # any few hundred KB of text
merges = train(corpus, vocab_size=512) # learn 256 merges
“`

Each loop scans the text once, so this simple version is slow on huge files — fine for learning with a few hundred kilobytes. Production tokenizers use clever data structures, but the result is the same kind of merge list.

Step 4: encoding new text

To encode, start from bytes and apply merges in the order they were learned — earliest merges first, because later merges were built on top of them.

“`python
def encode(text, merges):
ids = list(text.encode(“utf-8”))
while len(ids) >= 2:
pairs = get_pairs(ids)
# the pair that was learned earliest has the lowest new id
pair = min(pairs, key=lambda p: merges.get(p, float(“inf”)))
if pair not in merges:
break # nothing left to merge
ids = merge(ids, pair, merges[pair])
return ids
“`

Step 5: decoding back to text

“`python
def build_vocab(merges):
vocab = {i: bytes([i]) for i in range(256)}
for (a, b), idx in merges.items(): # dicts keep insertion order = merge order
vocab[idx] = vocab[a] + vocab[b]
return vocab

def decode(ids, vocab):
return b””.join(vocab[i] for i in ids).decode(“utf-8″, errors=”replace”)

vocab = build_vocab(merges)
sample = “Excel formulas are excellent”
tokens = encode(sample, merges)
print(len(sample.encode(“utf-8”)), “bytes ->”, len(tokens), “tokens”)
print([vocab[t].decode(“utf-8″, errors=”replace”) for t in tokens])
assert decode(tokens, vocab) == sample
“`

errors="replace" matters: a token can end in the middle of a multi-byte character, and a model can generate an invalid byte sequence. Replacing instead of crashing keeps generation going.

What you should see

On English text, 256 merges usually shrink the byte count by roughly a third to a half. Print the vocabulary and you will see the story of your corpus: first spaces plus common letters (" t", "th"), then whole words (" the", " Excel").

“`python
for idx in range(256, 276):
print(idx, repr(vocab[idx].decode(“utf-8″, errors=”replace”)))
“`

Vocabulary size is a trade-off

GPT-2 used about 50,000 tokens; many newer models use 100,000–200,000 to handle more languages and code efficiently. For our tiny model we will use a character-level or 512-token vocabulary to keep training fast.

Real tokenizers add a few extras

• Pre-splitting: text is first split with a regular expression (words, numbers, punctuation) so merges never cross those boundaries — you do not want "dog." and "dog!" to become separate tokens.
• Special tokens like <|endoftext|> mark document or message boundaries. Chat models use them to separate the user and assistant turns (Lesson 11).

Exercise

1. Train on 200 KB of English, then encode a Hindi sentence. Why does it use so many more tokens?
2. Add a pre-split step with re.findall(r"\s?\w+|\s?[^\w\s]+|\s+", text) and train merges per chunk.
3. Save merges to JSON so you can reuse the tokenizer in Lesson 4.

Next lesson: we use tokens to build our first language model — using nothing but counting.