Language models know a lot about the world but nothing about your policies, contracts, SOPs or price lists. Retrieval-augmented generation (RAG) fixes that: when a question comes in, find the most relevant passages from your own files and hand them to the model along with the question. The answer is grounded in your text, with sources you can check.

In this guide we build a complete RAG chatbot in about 80 lines of Python, running fully on your own computer.

How RAG works

1. Load documents (PDF, Word, Excel).
2. Chunk them into passages of a few hundred words.
3. Embed each chunk: turn it into a vector of numbers that captures its meaning.
4. Retrieve: embed the question the same way and find the chunks whose vectors are closest.
5. Generate: send the question plus those chunks to an LLM with the instruction “answer only from this context”.

Steps 1–3 run once (or whenever files change). Steps 4–5 run for every question.

Setup

“`bash
pip install pypdf sentence-transformers numpy pandas openpyxl requests
ollama pull llama3.2
“`

We use sentence-transformers for embeddings (runs locally, small model) and Ollama for the LLM — see running an LLM locally with Ollama. Nothing leaves your machine.

Step 1–2: load and chunk

“`python
import glob
from pypdf import PdfReader
import pandas as pd

def chunk_text(text, source, size=800, overlap=150):
text = ” “.join(text.split()) # normalise whitespace
out = []
for start in range(0, len(text), size – overlap):
piece = text[start:start + size]
if len(piece) > 100:
out.append({“source”: source, “text”: piece})
return out

def load_pdfs(folder):
chunks = []
for path in glob.glob(f”{folder}/*.pdf”):
for n, page in enumerate(PdfReader(path).pages, 1):
chunks += chunk_text(page.extract_text() or “”, f”{path} p.{n}”)
return chunks

def load_excel(path, sheet=0):
df = pd.read_excel(path, sheet_name=sheet).fillna(“”)
# one chunk per row: “Column: value; Column: value”
return [{“source”: f”{path} row {i + 2}”,
“text”: “; “.join(f”{c}: {v}” for c, v in row.items() if v != “”)}
for i, row in df.iterrows()]

chunks = load_pdfs(“docs”) + load_excel(“docs/price_list.xlsx”)
print(len(chunks), “chunks”)
“`

Choosing chunk size

• Too small (a sentence): chunks lose context — “it must be approved by them” means nothing alone.
• Too large (several pages): the relevant sentence is diluted, and fewer chunks fit in the prompt.
• Start with 500–1000 characters and 10–20% overlap, so an idea split across a boundary still appears whole in one chunk.

Keeping the page number or row number in source is what makes citations possible later.

Step 3: embeddings

“`python
import numpy as np
from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer(“all-MiniLM-L6-v2”) # ~90 MB, fast on CPU
vectors = embedder.encode([c[“text”] for c in chunks],
normalize_embeddings=True, show_progress_bar=True)
np.save(“vectors.npy”, vectors) # reuse next time
“`

Each chunk becomes a list of 384 numbers. Texts with similar meaning get similar vectors even if they share no words: “refund timeline” lands near “money returned within 14 days”. That is why this beats Ctrl+F.

Because we normalised the vectors, the dot product of two vectors is their cosine similarity (1 = same meaning, 0 = unrelated).

Step 4: retrieve

“`python
def search(question, k=4):
q = embedder.encode([question], normalize_embeddings=True)[0]
scores = vectors @ q
best = np.argsort(-scores)[:k]
return [chunks[i] | {“score”: float(scores[i])} for i in best]

for hit in search(“How many days do customers have to return an item?”):
print(round(hit[“score”], 2), hit[“source”], hit[“text”][:80])
“`

For a few thousand chunks, this NumPy search takes milliseconds. Beyond roughly a hundred thousand, use a vector database (FAISS, Chroma, pgvector) — same idea, with an index.

Step 5: generate the answer

“`python
import requests

PROMPT = “””Answer the question using ONLY the context below.
If the answer is not in the context, reply “I could not find this in the documents.”
Cite sources in square brackets like [1] after each fact.

Context:
{context}

Question: {question}
Answer:”””

def ask(question, k=4, model=”llama3.2″):
hits = search(question, k)
context = “\n\n”.join(f”[{i + 1}] ({h[‘source’]}) {h[‘text’]}” for i, h in enumerate(hits))
r = requests.post(“http://localhost:11434/api/generate”, timeout=300, json={
“model”: model, “stream”: False, “options”: {“temperature”: 0.1},
“prompt”: PROMPT.format(context=context, question=question)})
return r.json()[“response”], hits

answer, hits = ask(“What is the price of the 27-inch monitor and is there a bulk discount?”)
print(answer)
for i, h in enumerate(hits, 1):
print(f”[{i}] {h[‘source’]}”)
“`

Three details make this reliable:

• “Only the context” plus an explicit way to say “not found” — this is the single biggest defence against hallucination.
• Numbered sources the model can cite, which you print underneath so a person can verify.
• Low temperature, because you want faithful extraction, not creativity.

Step 6: a simple chat loop

“`python
while True:
q = input(“\nQuestion (blank to quit): “).strip()
if not q:
break
answer, hits = ask(q)
print(“\n” + answer)
print(“Sources:”, “, “.join(h[“source”] for h in hits))
“`

Want a web page instead? Wrap ask() in a small Streamlit or Flask app — the RAG logic does not change.

Making it better

Evaluate before you trust it

Write 20–30 real questions with known answers and the source they should come from. Then check two things separately:

“`python
tests = [(“What is the return window?”, “returns_policy.pdf”),
(“Who approves travel above ₹50,000?”, “travel_sop.pdf”)]
for q, expected in tests:
found = any(expected in h[“source”] for h in search(q, k=4))
print(“OK ” if found else “MISS”, q)
“`

1. Retrieval: is the right source in the top k? If not, no prompt can save the answer — fix chunking or search first.
2. Answer: given the right context, is the answer correct and cited? Read them yourself; for bigger test sets you can ask a stronger model to grade.

Security and privacy

• Documents can contain text like “ignore your instructions”. Keep your instructions in the prompt template, treat retrieved text as data, and never let the chatbot take actions based on document text alone.
• Respect permissions: if a user cannot open a file, the chatbot must not quote it. Filter chunks by the user’s access before searching.
• Running locally with Ollama keeps confidential files off third-party servers.

Where RAG fits

RAG is the right tool when answers must come from documents that change. If you need a model to adopt a style or learn a narrow task, look at LoRA fine-tuning. And if you want to understand what the LLM is doing with your context, the free Build Your Own LLM course builds one from scratch.