RAG: Make AI Answer From Your Own Documents

📘 AI Agents & RAG Course · Lesson 3 of 7

A model only knows what it was trained on. Ask it about your leave policy and it will either refuse or, worse, invent something plausible. RAG fixes this: find the relevant parts of your documents first, then give only those parts to the model with the question.

The three steps

1. Chunk — split documents into passages of a few hundred words.
2. Retrieve — score every chunk against the question and keep the top few.
3. Generate — ask the model to answer using only those chunks, and to cite them.

Chunking

from pathlib import Path

def chunks(text, size=180, overlap=40):
    words = text.split()
    step = size - overlap
    for i in range(0, max(len(words) - overlap, 1), step):
        yield " ".join(words[i:i + size])

docs = []
for f in Path("policies").glob("*.txt"):
    for n, c in enumerate(chunks(f.read_text(encoding="utf-8"))):
        docs.append({"source": f"{f.name}#{n+1}", "text": c})
print(len(docs), "chunks")

The overlap means a sentence cut at a boundary still appears whole in one chunk.

Retrieval with BM25

You don’t need a vector database to start. BM25, the classic search-engine scoring, works surprisingly well on policy documents and needs one small library.

pip install rank-bm25

from rank_bm25 import BM25Okapi
import re
tok = lambda s: re.findall(r"\w+", s.lower())
bm25 = BM25Okapi([tok(d["text"]) for d in docs])

def search(question, k=4):
    scores = bm25.get_scores(tok(question))
    best = sorted(range(len(docs)), key=lambda i: -scores[i])[:k]
    return [(scores[i], docs[i]) for i in best]

Generation with sources

import anthropic
client = anthropic.Anthropic()

def answer(question):
    hits = search(question)
    context = "\n\n".join(f"[{d['source']}]\n{d['text']}" for _, d in hits)
    msg = client.messages.create(model="claude-opus-5-5", max_tokens=1500,
        system="Answer only from the provided passages. Cite sources in [brackets]. If the passages do not contain the answer, say you don't know.",
        messages=[{"role": "user", "content": f"Passages:\n{context}\n\nQuestion: {question}"}])
    return msg.content[0].text

print(answer("How many casual leaves do I get per year?"))

Why not just paste all the documents?

For a handful of short documents, you can — modern models have very large context windows. RAG pays off when you have hundreds of pages: it is cheaper per question, faster, and the answer points to the exact passage it came from.