RAG on a Budget: Building a Knowledge Base with Ollama & ChromaDB

You know what’s fun? Asking ChatGPT a question about your company’s internal docs and watching it confidently hallucinate an answer that sounds right but is completely made up. Real fun. Like trusting a confident stranger for directions in a city they’ve never visited.

RAG fixes this, and you don’t need a cloud subscription, a GPU cluster, or a second mortgage to build one.

In this guide we’re building a fully local Retrieval-Augmented Generation system using Ollama (free, local LLMs) and ChromaDB (free, local vector database). Everything runs on your machine. No API keys, no metered billing, no sending your proprietary data to someone else’s servers.

What Even Is RAG?

RAG stands for Retrieval-Augmented Generation. Instead of relying on an LLM’s training data, which is frozen in time and might be flat wrong about your specific stuff, you first retrieve relevant documents from a knowledge base, then feed those documents to the LLM as context so it can generate an informed answer.

Think of it like hiring a really smart intern. They’re brilliant, well-read, articulate, but they know nothing about your company. RAG is handing them a folder of relevant docs before each question: “Read these first, then answer.” Without RAG, that intern just wings it. With RAG, they have receipts.

The pipeline, in three steps:

1. Ingest: split your documents into chunks, generate embeddings (numerical representations) for each chunk, store them in a vector database.
2. Query: convert the user’s question into an embedding, search the vector database for the most similar chunks, retrieve the top results.
3. Generate: send those chunks plus the question to the LLM, which answers grounded in your actual data.

That’s it. The magic is in the details, which we’re about to cover.

Why Ollama + ChromaDB?

There are approximately nine thousand ways to build a RAG system. Most tutorials point you at a hosted API and a managed vector database. Those work fine if you’re happy paying per token and shipping your data to someone else’s servers.

Here’s why we’re going local instead:

• Ollama runs open-source LLMs (Llama 3.2, Qwen 3, Gemma 3, Mistral, and friends) locally, and handles model management, serving, and, critically, embedding generation. Free. No API key.
• ChromaDB is an open-source vector database built for AI applications: lightweight, runs locally or in Docker, dead-simple Python API.
• Privacy: your data never leaves your machine. Matters for proprietary docs, medical records, legal files, anything you’d rather not upload to the cloud.
• Cost: $0/month, forever. The only cost is your electricity bill.

The trade-off: local models are smaller and less capable than frontier models like GPT-5 or Claude. But for document Q&A over your own knowledge base, they’re more than good enough.

Setting Up the Stack with Docker Compose

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    # Uncomment if you have an NVIDIA GPU
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - capabilities: [gpu]

  chromadb:
    image: chromadb/chroma:latest
    container_name: chromadb
    ports:
      - "8000:8000"
    volumes:
      - chroma_data:/chroma/chroma
    environment:
      - ANONYMIZED_TELEMETRY=FALSE

volumes:
  ollama_data:
  chroma_data:

docker compose up -d

# Chat model: llama3.2 is a solid all-rounder (qwen3 and gemma3 are good alternatives)
docker exec ollama ollama pull llama3.2

# Embedding model: nomic-embed-text is the workhorse for RAG
docker exec ollama ollama pull nomic-embed-text

The embedding model is the unsung hero here. It converts text into vectors (arrays of numbers) that capture semantic meaning. nomic-embed-text produces 768-dimensional vectors and punches way above its weight for something you can run on a laptop.

Verify both models landed:

docker exec ollama ollama list

If you see both, congratulations, you have a local AI inference stack running. That was the hard part, and it wasn’t that hard.

Python Project Setup

mkdir rag-budget && cd rag-budget
python -m venv venv
source venv/bin/activate
pip install chromadb requests langchain langchain-community

requests talks to Ollama’s REST API, chromadb handles the vector store, and langchain provides text-splitting utilities that save a lot of boilerplate.

Document Ingestion: Load, Chunk, Embed

Start simple with text files:

import os

def load_documents(docs_dir: str) -> list[dict]:
    """Load all text files from a directory."""
    documents = []
    for filename in os.listdir(docs_dir):
        if filename.endswith(('.txt', '.md')):
            filepath = os.path.join(docs_dir, filename)
            with open(filepath, 'r', encoding='utf-8') as f:
                content = f.read()
            documents.append({
                'content': content,
                'metadata': {'source': filename, 'filepath': filepath}
            })
    print(f"Loaded {len(documents)} documents")
    return documents

For production, add PDF, Word, and HTML loaders. LangChain has these built in, but this is enough to get moving.

Chunking: The Part Where RAG Systems Quietly Go Wrong

You can’t shove a 50-page document into a vector database and expect good results. You need to split it, but how you split matters enormously. Chunks too big carry too much irrelevant noise and the signal gets diluted; chunks too small lack enough context to be useful. Goldilocks problem.

from langchain.text_splitter import RecursiveCharacterTextSplitter

def chunk_documents(documents: list[dict],
                    chunk_size: int = 500,
                    chunk_overlap: int = 50) -> list[dict]:
    """Split documents into overlapping chunks."""
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", ". ", " ", ""]
    )
    chunks = []
    for doc in documents:
        splits = splitter.split_text(doc['content'])
        for i, split in enumerate(splits):
            chunks.append({
                'content': split,
                'metadata': {**doc['metadata'], 'chunk_index': i, 'chunk_total': len(splits)}
            })
    print(f"Created {len(chunks)} chunks from {len(documents)} documents")
    return chunks

RecursiveCharacterTextSplitter is smart about where it cuts: paragraph breaks first, then sentences, then words. chunk_overlap keeps context alive across chunk boundaries, so don’t set it to zero.

This is the single most impactful tuning knob in your whole RAG system, and most people never touch it.

Embedding and Storing

import requests

OLLAMA_BASE_URL = "http://localhost:11434"

def get_embedding(text: str, model: str = "nomic-embed-text") -> list[float]:
    """Generate embedding for a single text using Ollama."""
    response = requests.post(
        f"{OLLAMA_BASE_URL}/api/embeddings",
        json={"model": model, "prompt": text}
    )
    response.raise_for_status()
    return response.json()["embedding"]

def get_embeddings_batch(texts: list[str], model: str = "nomic-embed-text") -> list[list[float]]:
    embeddings = []
    for i, text in enumerate(texts):
        embeddings.append(get_embedding(text, model))
        if (i + 1) % 50 == 0:
            print(f"  Embedded {i + 1}/{len(texts)} chunks...")
    return embeddings

On a decent CPU, expect 10-30 chunks per second. With a GPU, faster still.

import chromadb

def create_collection(chunks: list[dict], collection_name: str = "knowledge_base"):
    """Create a ChromaDB collection and add document chunks."""
    client = chromadb.HttpClient(host="localhost", port=8000)
    try:
        client.delete_collection(collection_name)
    except Exception:
        pass

    collection = client.create_collection(
        name=collection_name,
        metadata={"hnsw:space": "cosine"}
    )

    documents = [chunk['content'] for chunk in chunks]
    metadatas = [chunk['metadata'] for chunk in chunks]
    ids = [f"chunk_{i}" for i in range(len(chunks))]

    print("Generating embeddings...")
    embeddings = get_embeddings_batch(documents)

    batch_size = 100
    for i in range(0, len(documents), batch_size):
        end = min(i + batch_size, len(documents))
        collection.add(
            documents=documents[i:end],
            embeddings=embeddings[i:end],
            metadatas=metadatas[i:end],
            ids=ids[i:end]
        )
    print(f"Added {len(documents)} chunks to collection '{collection_name}'")
    return collection

def ingest_documents(docs_dir: str = "docs", collection_name: str = "knowledge_base"):
    """Full pipeline: load, chunk, embed, store."""
    documents = load_documents(docs_dir)
    if not documents:
        print("No documents found!")
        return None
    chunks = chunk_documents(documents, chunk_size=500, chunk_overlap=50)
    return create_collection(chunks, collection_name)

collection = ingest_documents("docs")

hnsw:space: cosine tells ChromaDB to use cosine similarity, the standard choice for text embeddings since it measures the angle between vectors rather than raw distance.

Querying: Retrieval and Generation

def retrieve_context(query: str, collection_name: str = "knowledge_base", n_results: int = 5) -> list[dict]:
    """Retrieve the most relevant chunks for a query."""
    client = chromadb.HttpClient(host="localhost", port=8000)
    collection = client.get_collection(collection_name)
    query_embedding = get_embedding(query)

    results = collection.query(
        query_embeddings=[query_embedding],
        n_results=n_results,
        include=["documents", "metadatas", "distances"]
    )

    return [
        {
            'content': results['documents'][0][i],
            'metadata': results['metadatas'][0][i],
            'distance': results['distances'][0][i]
        }
        for i in range(len(results['documents'][0]))
    ]

Always embed the query with the same model used for the documents. Mixing embedding models is the fastest way to get garbage results.

def generate_answer(query: str, contexts: list[dict], model: str = "llama3.2") -> str:
    """Generate an answer using retrieved context."""
    context_text = "\n\n---\n\n".join([
        f"[Source: {ctx['metadata'].get('source', 'unknown')}]\n{ctx['content']}"
        for ctx in contexts
    ])

    prompt = f"""You are a helpful assistant. Answer the user's question based ONLY
on the provided context. If the context doesn't contain enough information to answer
the question, say so. Do not make up information.

CONTEXT:
{context_text}

QUESTION: {query}

ANSWER:"""

    response = requests.post(
        f"{OLLAMA_BASE_URL}/api/generate",
        json={
            "model": model,
            "prompt": prompt,
            "stream": False,
            "options": {"temperature": 0.3, "num_ctx": 4096}
        }
    )
    response.raise_for_status()
    return response.json()["response"]


def ask(query: str, collection_name: str = "knowledge_base", n_results: int = 5, model: str = "llama3.2") -> str:
    """Full RAG pipeline: retrieve, then generate."""
    contexts = retrieve_context(query, collection_name, n_results)
    for i, ctx in enumerate(contexts):
        print(f"  [{i+1}] {ctx['metadata'].get('source', 'unknown')} (similarity: {1 - ctx['distance']:.3f})")
    return generate_answer(query, contexts, model)

ask("What is our refund policy?")

Three things doing the real work in that prompt: “based ONLY on the provided context” is the anti-hallucination clause, without it the model happily fills gaps with invented facts. Temperature 0.3 keeps answers deterministic instead of creative. Source attribution in the context lets the model (and you) trace where an answer came from.

That’s your complete RAG system. All local, all free.

Making It a Real Docs Chatbot

Point it at a company’s actual docs (onboarding, API reference, HR policies) and the only change you need is a recursive loader:

def load_documents_recursive(docs_dir: str) -> list[dict]:
    """Load documents from a nested directory structure."""
    documents = []
    for root, _, files in os.walk(docs_dir):
        for filename in files:
            if filename.endswith(('.txt', '.md')):
                filepath = os.path.join(root, filename)
                rel_path = os.path.relpath(filepath, docs_dir)
                with open(filepath, 'r', encoding='utf-8') as f:
                    content = f.read()
                documents.append({
                    'content': content,
                    'metadata': {
                        'source': filename,
                        'filepath': rel_path,
                        'category': os.path.dirname(rel_path) or "general"
                    }
                })
    return documents

Wrap ask() in a while True input loop with a quit and reindex command, and you have an interactive docs assistant your team can query in plain English, grounded in your actual content instead of vibes.

Performance Tips

• Tune chunk size empirically. Try 300, 500, 800, 1200 on the same test questions and compare answer quality. Dense technical docs favor smaller chunks; narrative policy docs tolerate bigger ones.
• Filter by metadata. ChromaDB supports where={"category": "hr"} on queries. If you know the domain of a question, narrow the search before it happens.
• Re-rank results. Retrieve 10-15 candidates, then have the LLM itself re-rank them before you use the top 5 for generation. Adds latency, meaningfully improves answer quality.
• Cache embeddings. Don’t regenerate them on every restart. ChromaDB persists to disk with this Docker setup, so they survive container restarts on their own.
• Match your hardware. CPU-only: fine, expect a few seconds for embedding and 10-30 seconds for generation on a small model. NVIDIA GPU: uncomment the GPU block, embedding gets near-instant. Apple Silicon: run Ollama natively via brew install ollama instead of Docker for Metal acceleration.

When RAG Beats Fine-Tuning

Choose RAG when:

• Your data changes often (docs get updated, new content lands)
• You need source attribution, showing where an answer came from
• You want to keep the base model general-purpose
• Your data is proprietary and shouldn’t get baked into model weights
• You need something working this week, not this quarter

Choose fine-tuning when:

• You need the model to adopt a specific tone, style, or behavior
• The task is highly specialized (medical coding, legal classification)
• Retrieval latency is unacceptable for your use case
• The knowledge is static and well-defined

Choose both when you want the model to understand your domain deeply (fine-tune) and stay current with changing data (RAG). For most self-hosted setups, RAG wins: faster to build, easier to maintain, and your data stays separate from the model so you can swap models without retraining anything.

Common Pitfalls

1. Mismatched embedding models. Embed docs with nomic-embed-text and queries with something else, and your vectors live in different spaces. Results will be garbage. Always match your models.
2. Ignoring document updates. When a document changes, re-ingest it. A simple mtime or hash check lets you re-index only what changed.
3. Zero chunk overlap. If a critical fact spans two chunks and neither one has the full context, retrieval misses it. That’s the entire reason chunk_overlap exists.
4. Stuffing too much context. More retrieved chunks isn’t always better. Fifteen mediocre chunks bury the LLM in noise; 3-5 well-chosen ones outperform them.
5. Skipping evaluation. Build a small set of question-answer pairs and run your system against them periodically. Without that, you’re tuning blind.

Wrapping Up

You now have everything you need to build a local, private, free RAG system: Ollama for embeddings and generation, ChromaDB for vector storage, some Python glue to tie it together.

It won’t outperform a RAG system built on a frontier cloud model, a managed vector database, and a dedicated ML team. It doesn’t need to. It runs on your laptop, costs nothing, keeps your data private, and is more than capable for internal docs, personal knowledge bases, and proof-of-concept work.

The barrier to entry for AI-powered knowledge systems has dropped to almost nothing. What’s standing between you and an assistant that actually knows your stuff is about an hour of setup and a docker compose up.

Related Reading

• AMD Lemonade: Local LLM Serving for AMD GPUs
• Exploring the Diverse World of LLM Models
• Beyond RAG: When a Virtual Filesystem Works Better
• 1-Bit LLMs: The Quantization Endgame
• Continue.dev vs Cody vs Tabby: AI Code Assistants That Live on Your Machine