Build a Local RAG Chatbot with Ollama and Chroma

You keep answering the same questions from your own docs, and copying the answers by hand. A SaaS assistant fixes the repetition but sends your private documents to a third party and bills you per token. A local RAG stack avoids both: Ollama runs the models on your machine, Chroma stores the vectors, and everything stays offline once the model weights are downloaded.

RAG stands for retrieval-augmented generation. You index a pile of documents, find the ones closest to a question, and hand those to the model as context before it writes an answer. The model never needs your whole archive in its context window, and it can point at the exact text it used.

Prerequisites

• Python 3.10+
• Ollama installed (macOS/Linux: curl -fsSL https://ollama.com/install.sh | sh, or download from ollama.com/download)
• About 4 GB of disk for the two models
• A terminal and pip

Step 1: Pull the models

You need two: one for chat, one for embeddings.

ollama pull llama3.2
ollama pull nomic-embed-text

llama3.2 is a 3B-parameter chat model, roughly 2 GB, and runs fine on 8 GB of RAM. nomic-embed-text is a 274 MB model that turns text into vectors; it cannot chat, it only embeds. Confirm the chat model works:

ollama run llama3.2 "Reply with: ready"

Step 2: Check the embedding output

The embedding endpoint returns a vector for any input text:

curl -s http://localhost:11434/api/embeddings -d '{"model":"nomic-embed-text","prompt":"hello world"}'

If Ollama is running, you get a JSON object with an embedding array of floats. That array is what Chroma indexes.

Step 3: Set up the vector store

pip install ollama chromadb

Chroma has an in-memory client that deletes data when the script exits, and a persistent client that writes to a folder. Use the persistent one so your index survives.

import chromadb

client = chromadb.PersistentClient(path="./chroma")
col = client.get_or_create_collection(name="notes")

Step 4: Full RAG script

Save this as rag.py:

import ollama
import chromadb

EMBED_MODEL = "nomic-embed-text"
CHAT_MODEL = "llama3.2"
COLLECTION = "notes"

client = chromadb.PersistentClient(path="./chroma")
col = client.get_or_create_collection(name=COLLECTION)

def embed(text: str) -> list[float]:
    return ollama.embeddings(model=EMBED_MODEL, prompt=text)["embedding"]

def index(documents: list[str]) -> None:
    ids = [f"doc-{i}" for i in range(len(documents))]
    col.upsert(
        ids=ids,
        documents=documents,
        embeddings=[embed(d) for d in documents],
    )

def retrieve(query: str, k: int = 4) -> list[str]:
    res = col.query(query_embeddings=[embed(query)], n_results=k)
    return res["documents"][0]

def ask(question: str) -> str:
    context = "\n---\n".join(retrieve(question))
    prompt = (
        "Answer the question using only the context below.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}\n\nAnswer:"
    )
    return ollama.chat(model=CHAT_MODEL, messages=[
        {"role": "user", "content": prompt}
    ])["message"]["content"]

if __name__ == "__main__":
    index([
        "Deploys run on Fridays at 18:00 via a GitHub Actions workflow named build-deploy.",
        "Rollback is automatic: if the health check fails for 30 seconds, the pipeline redeploys the previous image.",
        "Secrets live in the project's CI variables, never in the repository.",
    ])
    print(ask("When does the deploy run, and what happens if it fails?"))

Replace the three example documents with your own content, then run:

python rag.py

The script indexes the chunks, retrieves the four closest to the question, and asks llama3.2 to answer using only that context.

How the retrieval works

Three moving parts:

1. Indexing. embed() turns each chunk into a vector, and index() stores it with an id in Chroma.
2. Retrieval. retrieve() embeds the question with the same model and asks Chroma for the k nearest vectors. Chroma returns chunks sorted by distance.
3. Generation. ask() packs the retrieved chunks into a prompt that tells the model to use only that context, then returns the answer.

Keep embeddings and queries on the same model. Mixing nomic-embed-text at index time with a different model at query time quietly breaks retrieval.

When to use a different store

Chroma's persistent client is fine up to a few hundred thousand chunks on one machine. If you need multi-tenant access, running auth, or you already run Postgres, look at:

• Chroma in client-server mode for shared access
• pgvector, the vector extension for Postgres, when you want one database for rows and vectors
• Qdrant when you need higher query throughput or distributed deployments

Same RAG loop either way; only the query call changes.

Gotchas

• nomic-embed-text handles about 2048 tokens of context. Chunk long documents below that, or retrieval quality drops.
• If answers come back hollow, your chunks are probably too long or too few are retrieved. Raise k and shorten chunks before blaming the model.
• The in-memory Chroma client throws everything away on exit. Use PersistentClient unless you only want a demo.

Next steps

• Wrap ask() in a Flask or FastAPI endpoint and give it a web UI.
• Index PDFs and Markdown files directly by splitting them into chunks first.
• Add a reranker, or generate the answer from the top-k chunk plus the raw query, for sharper citations.

References

• Ollama quickstart (official docs)
• Chroma getting started (official docs)
• nomic-embed-text model (Ollama library)