Build a Simple RAG System with Ollama and Python

RAG (Retrieval-Augmented Generation) lets you ask questions about your own documents using a local LLM. Instead of relying on the model's training data alone, RAG retrieves relevant passages from your document and feeds them to the LLM so it can give grounded, accurate answers.

Privacy: Ollama runs entirely on your machine. No data is sent to the cloud, so your information never leaves your computer.

This tutorial builds a complete RAG pipeline over a PDF in a single Python script — no frameworks, no vector databases, just the essentials. It starts with the basics (talking to a local LLM) and gradually adds the RAG components.

Prerequisites

1. Install Ollama

Ollama runs LLMs locally on your machine.

1. Download and install from ollama.com.
2. Pull the two models you'll need:

ollama pull llama3.2            # the LLM that generates answers
ollama pull nomic-embed-text    # the model that creates embeddings

2. Install Python libraries

pip install ollama pypdf numpy

Library	Purpose
ollama	Python client to communicate with the local Ollama server
pypdf	Extracts text from PDF files
numpy	Computes cosine similarity between vectors

Step-by-step walkthrough

Part 1 — Ask Ollama a question directly

Before doing any RAG, let's just talk to the LLM. This confirms Ollama is working and shows how simple the Python API is.

import ollama

LLM_MODEL = "llama3.2"

question = "What is the capital of France?"
response = ollama.chat(
    model=LLM_MODEL,
    messages=[{"role": "user", "content": question}],
)
print(response["message"]["content"])

Everything runs locally — no API keys, no cloud services, no data leaving your computer.

Part 2 — Read the PDF

Now we load the document we want to ask questions about.

from pypdf import PdfReader

reader = PdfReader("your_document.pdf")

text = ""
for page in reader.pages:
    text += page.extract_text()

PdfReader opens the PDF and gives you access to each page. We loop through every page and concatenate the extracted text into one big string.

Part 3 — Split the text into chunks

CHUNK_SIZE = 500
OVERLAP = 50

def split_text(text, chunk_size, overlap):
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap
    return chunks

chunks = split_text(text, CHUNK_SIZE, OVERLAP)

The full text is too long to send to the LLM at once, and embeddings work better on smaller pieces of text. We split into chunks of 500 characters with a 50-character overlap so sentences at chunk boundaries aren't lost.

Part 4 — Create embeddings

EMBEDDING_MODEL = "nomic-embed-text"

def get_embedding(text):
    response = ollama.embed(model=EMBEDDING_MODEL, input=text)
    return response["embeddings"][0]

embeddings = [get_embedding(chunk) for chunk in chunks]

An embedding converts text into a list of numbers (a vector). Texts with similar meaning end up with similar vectors. We create one embedding per chunk using the nomic-embed-text model. This also runs locally via Ollama.

Part 5 — Find relevant chunks

import numpy as np

def cosine_similarity(vec_a, vec_b):
    a = np.array(vec_a)
    b = np.array(vec_b)
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def find_relevant_chunks(question, chunks, embeddings, top_k=3):
    question_embedding = get_embedding(question)
    scores = []
    for i, emb in enumerate(embeddings):
        score = cosine_similarity(question_embedding, emb)
        scores.append((score, i))
    scores.sort(reverse=True)
    return [chunks[i] for _, i in scores[:top_k]]

When a question comes in, we embed it with the same model and compare it to every chunk embedding using cosine similarity (a standard measure of how similar two vectors are). The top 3 most similar chunks are returned.

Part 6 — Ask a question using RAG

def ask(question):
    relevant_chunks = find_relevant_chunks(question, chunks, embeddings)
    context = "\n".join(relevant_chunks)

    prompt = (
        "You are a helpful assistant. Use ONLY the following context to "
        "answer the question. If the answer is not in the context, say "
        "'I don't have enough information to answer that.'\n"
        f"Context: {context}\n"
        f"Question: {question}\n"
        "Answer:"
    )

    response = ollama.chat(
        model="llama3.2",
        messages=[{"role": "user", "content": prompt}],
    )
    return response["message"]["content"]

Now we put it all together. We retrieve the most relevant chunks, combine them into a context block, and send that along with the question to the LLM. This is the Retrieval-Augmented Generation pattern: the LLM answers based on your document, not just its training data.

Part 7 — Try it

answer = ask("What is this document about?")
print(answer)

Change the question to anything related to your PDF.

Tips for going further

• Use a bigger model — Replace llama3.2 with llama3:70b or another model for better answers.
• Tune chunk size — Smaller chunks (200-300 chars) are more precise; larger chunks (800-1000) give more context per hit.
• Add a vector database — For large document collections, store embeddings in a database like ChromaDB or FAISS instead of a Python list.
• Handle multiple PDFs — Loop over several files, tag each chunk with its source filename, and include the source in the answer.