RAG without magic: Auditable knowledge integration for local LLMs

Introduction

In previous articles I have shown how a locally operated LLM can provide support in practice - initially using the example of a task robot (Local LLM in practice - Redmine ticket analysis). I then delved deeper into the subject in Local LLM with depth to better understand the entire supply chain and free my own LLM pipelines from unnecessary or uncontrollable ballast.

The next practical part follows today. This time it’s about a particularly useful aspect of modern LLM architectures: Retrieval Augmented Generation (RAG). A language model not only gains access to its trained knowledge, but also to exactly the information that is really relevant for the respective use case. RAG combines the strengths of an LLM with targeted information searches in its own databases.

The result is answers that are context-rich, up-to-date, accurate and generated entirely within your own security perimeter - a central building block for robust, sovereign and traceable AI systems.

Below I will show how RAG is implemented in practice, which components are involved and how this approach can be seamlessly integrated into local LLM workflows.

Basics: What is Retrieval-Augmented Generation (RAG)?

Briefly summarized:

• Retrieval = Retrieve relevant information from a knowledge base
• Augmented = This information is given to the model in addition to the prompt
• Generation = The LLM then generates its answer

RAG is particularly suitable for:

• technical wikis
• internal documentation
• Policies & Playbooks
• Knowledge bases
• Product catalogs

Task

We run a local LLM (e.g. llama.cpp with a Qwen model) and have stored our internal wiki documents as Markdown files in a directory. This content should be prepared in such a way that the model can search it specifically and use it as a knowledge base.

Solution approach

In order to make our wiki accessible for the LLM, we need four consecutive steps:

1. Read the wiki files, break them down into chunks and convert them into embeddings
2. A suitable database to store these embeddings transparently and locally
3. Select the embed user question and the most relevant wiki chunks
4. Build a context prompt and send it to the llama-server

Below we’ll go through these steps sequentially and in detail.

1. A pipeline for vectorizing the wiki documents (embeddings)

The pipeline handles:

• Read Markdown files
• break the content down into meaningful chunks
• create an embedding for each chunk
• save everything together in one database

2. A database for the embeddings

Once all wiki chunks have been converted into embeddings, they need to be stored in a database so that they can be accessed efficiently later. We deliberately use SQLite for our project because our expected data volume remains manageable:

• realistic 2,000-20,000 chunks,
• fully searchable in RAM,
• no high latency requirements,
• no scaling or clustering requirements.

A classic vector database would be overengineering for this size, as its strengths only come into play with several hundred thousand or million embeddings.

3. Embed the user question and retrieve appropriate wiki chunks

Once a user asks a question:

• the question itself is vectorized (embedding),
• compared against all wiki embeddings,
• the top-k most suitable chunks selected.

4. Build context prompt and send to llama-server

After the relevant chunks are found, they are merged into one Context prompt embedded:

• The prompt contains the user’s question
• plus the Top-k-Chunks from the Wiki
• plus an instruction that the model only is based on this Context may respond.

This prompt is then sent to the local llama-server.

Architecture overview (high-level)

┌──────────────┐   ┌──────────┐   ┌──────────┐   ┌──────────────┐
│ Markdown-Wiki│ → │ Chunking │ → │Embedding │ → │   SQLite     │
└──────────────┘   └──────────┘   └──────────┘   │  (Vectors)   │
                                                 └──────┬───────┘
                                                        ▼
      ┌──────────┐   ┌───────────────────────┐   ┌──────────────┐
      │  Query   │ → │ Query-Embedding       │ → │    Top-k     │
      └──────────┘   │ Cosine Similarity     │   └──────┬───────┘
                     └───────────────────────┘          ▼
                                                 ┌──────────────┐
                                                 │   Prompt     │
                                                 └──────┬───────┘
                                                        ▼
                                                 ┌──────────────┐
                                                 │ llama-server │
                                                 └──────────────┘

The pipeline in detail – step by step (PoC)

In the following Proof-of-Concept, we orchestrate the entire knowledge integration with just two small Python scripts. The files are deliberately kept minimal and are primarily intended to demonstrate the process, not represent a productive architecture:

1. build_index.py — Read Wiki, create chunks, calculate embeddings, fill SQLite
2. ask_rag.py — Embed user question, get top-k chunks, build context prompt and send to llama-server

Step 1 – Read Markdown

from pathlib import Path

def strip_frontmatter(text: str) -> str:
    if text.startswith("---"):
        parts = text.split("---", 2)
        if len(parts) == 3:
            return parts[2].lstrip()
    return text

text = Path("wiki/intro.md").read_text(encoding="utf-8")
cleaned = strip_frontmatter(text)

Step 2 – Chunking

MAX_CHARS = 1200

def chunk_text(text: str):
    paragraphs = text.split("\n\n")   # richtiger Absatztrenner
    current = ""

    for para in paragraphs:
        if len(current) + len(para) + 2 <= MAX_CHARS:
            # Absatz anhängen
            current += ("\n\n" if current else "") + para
        else:
            # aktuelles Chunk ausgeben
            if current:
                yield current.strip()

            # zu großer Absatz -> hart splitten
            while len(para) > MAX_CHARS:
                yield para[:MAX_CHARS].strip()
                para = para[MAX_CHARS:]

            current = para

    # Rest ausgeben
    if current:
        yield current.strip()

Step 3 – Create embeddings

import subprocess
import re

def run_embedding(text: str):
    CMD = "/home/mpeterma/Projekte/ggml-org/llama.cpp/build/bin/llama-embedding"
    MODEL = "/home/mpeterma/Library/models/Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q5_k_m.gguf"

    proc = subprocess.run(
        [CMD, "-m", MODEL, "-p", text],
        capture_output=True, text=True
    )

    if proc.returncode != 0:
        raise RuntimeError(proc.stderr)

    for line in proc.stdout.splitlines():
        line = line.strip()
        if line.startswith("embedding 0:"):
            # nur die Zahlen extrahieren
            numbers = re.findall(r"-?\d+\.\d+", line)
            return [float(n) for n in numbers]

    raise RuntimeError("No embedding vector found in output")

Step 4 – SQLite as a knowledge base

Scheme

CREATE TABLE docs (
    id INTEGER PRIMARY KEY,
    path TEXT NOT NULL,
    chunk_index INTEGER NOT NULL,
    content TEXT NOT NULL,
    embedding TEXT NOT NULL
);

CREATE INDEX idx_docs_path ON docs(path);

Indexer

import sqlite3, json
from pathlib import Path

def index_wiki():
    conn = sqlite3.connect("knowledge.db")
    cur = conn.cursor()

    cur.execute("""
    CREATE TABLE IF NOT EXISTS docs (
        id INTEGER PRIMARY KEY,
        path TEXT NOT NULL,
        chunk_index INTEGER NOT NULL,
        content TEXT NOT NULL,
        embedding TEXT NOT NULL
    )
    """)

    for path in Path("./wiki").rglob("*.md"):
        text = strip_frontmatter(path.read_text(encoding="utf-8"))
        for idx, chunk in enumerate(chunk_text(text)):
            emb = run_embedding(chunk)
            cur.execute(
                "INSERT INTO docs (path, chunk_index, content, embedding) VALUES (?, ?, ?, ?)",
                (str(path), idx, chunk, json.dumps(emb))
            )
        conn.commit()

    conn.close()

if __name__ == "__main__":
    index_wiki()

Step 5 – Retrieval (embedding comparison)

Cosine Similarity

import math

def cosine(a, b):
    dot = sum(x*y for x, y in zip(a, b))
    na = math.sqrt(sum(x*x for x in a))
    nb = math.sqrt(sum(y*y for y in b))
    return dot / (na * nb) if na > 0 and nb > 0 else 0

Retrieval function

import json, sqlite3

def retrieve(query: str, top_k=5):
    # 1) Anfrage als Embedding
    q_emb = run_embedding(query)

    # 2) Alle Chunks aus DB ziehen
    conn = sqlite3.connect("knowledge.db")
    rows = conn.execute(
        "SELECT path, chunk_index, content, embedding FROM docs"
    ).fetchall()
    conn.close()

    # 3) Similarity berechnen
    scored = []
    for path, idx, content, emb_json in rows:
        emb = json.loads(emb_json)
        score = cosine(q_emb, emb)
        scored.append((score, content))

    # 4) Top-k zurückgeben
    scored.sort(reverse=True)
    return [content for score, content in scored[:top_k]]

Step 6 – Prompt construction + request to the llama server

Create promptly

def build_prompt(query: str, chunks: list[str]) -> str:
    context = "\n\n".join(chunks)
    return f"""
Du bist ein Assistent, der ausschließlich auf Basis des folgenden Wikis antwortet.
Wenn etwas nicht im Kontext steht, sag: "Dazu habe ich keine Information."

Kontext:
{context}

Frage: {query}
Antwort:
""".strip()

Request to the llama server

import requests

def ask_llama(query: str):
    chunks = retrieve(query, top_k=4)
    prompt = build_prompt(query, chunks)

    resp = requests.post(
        "http://localhost:8080/v1/completions",
        json={
            "model": "qwen2.5-3b",
            "prompt": prompt,
            "max_tokens": 600
        }
    )
    resp.raise_for_status()
    return resp.json()["choices"][0]["text"]

Step 7 - Main function with CLI arguments

import sys

def main():
    if len(sys.argv) < 2:
        print("Usage: python3 rag.py \"Deine Frage\"")
        sys.exit(1)

    query = " ".join(sys.argv[1:])
    answer = ask_llama(query)
    print(answer)


if __name__ == "__main__":
    main()

End-to-end flow in use

1. Build index (Markdown → Chunks → Embeddings → SQLite)

python3 build_index.py

2. Make a sample request (Query → Retrieval → Context → Response)

python3 ask_rag.py "Wie lege ich in Cyrus IMAP eine neue Mailbox an?"

3. Reply received

Mit cyradm kannst Du für Cyrus IMAP neue Mailboxen anlegen.
...

Conclusion

We have built an end-to-end, transparent knowledge pipeline stack:

• Imported Markdown wiki
• Chunks generated
• Embeddings generated
• Used SQLite as a transparent knowledge base
• Query → Embedding → Cosine Similarity → Top-k
• RAG prompting
• Reply via llama-server