Better Retrieval: Chunk Size, Scores and a “Don’t Know” Threshold

📘 AI Agents & RAG Course · Lesson 4 of 7

When a RAG system gives a bad answer, the cause is usually retrieval, not the model: the right passage was never sent. This lesson makes retrieval measurable.

Build a tiny test set

Write 15–20 real questions and, for each, the source chunk that contains the answer. Include 4–5 questions your documents can’t answer.

tests = [
    ("How many casual leaves per year?", "leave.txt"),
    ("Can I carry forward earned leave?", "leave.txt"),
    ("What is the travel allowance for Mumbai?", "travel.txt"),
    ("What is the office wifi password?", None),   # not in the documents
]

def hit_rate(k=4):
    ok = 0
    for q, src in tests:
        if src is None: continue
        ok += any(d["source"].startswith(src) for _, d in search(q, k))
    return ok / sum(1 for _, s in tests if s)
print(f"hit rate @4: {hit_rate():.0%}")

Try different chunk sizes

Re-chunk at 120, 180 and 300 words and compare hit rates. Small chunks are precise but can lose context; large chunks carry context but dilute the score. For policy-style documents 150–250 words with 20% overlap is a good starting point — but measure on your documents.

The threshold

Print the top score for answerable and unanswerable questions side by side:

for q, src in tests:
    top = search(q, 1)[0][0]
    print(f"{top:6.2f}  {'(no answer)' if src is None else ''}  {q}")

You will usually see a gap: real questions score clearly higher than off-topic ones. Pick a threshold in that gap and refuse below it:

MIN_SCORE = 4.0   # set from YOUR test output

def answer(question):
    hits = [h for h in search(question) if h[0] >= MIN_SCORE]
    if not hits:
        return "I couldn't find this in the policy documents. Please check with HR."
    ...  # same generation code as Lesson 3

When to move to embeddings

BM25 matches words. If users ask “how many days off” while the policy says “leave entitlement”, word matching struggles. Embedding models match meaning instead. A common upgrade is hybrid search: run both, merge the results, and keep the same test set to prove the change helped.