Local LLM with depth - full transparency instead of a black box

Introduction

In my previous articles Local LLM – AI language models without cloud and Local AI in practice – Redmine ticket analysis with Ollama I showed how uncomplicated local AI is today. Simply drag the model, start it, use it – done.

But while I certainly enjoy this lightness, one thought remains: How much of this do I really understand?

Because Ollama’s comfort layer covers a complex infrastructure. And that’s exactly where the exciting part begins: We go below the surface.

The stack under Ollama – what actually happens there?

Ollama is often perceived as “runtime”. In reality, it is a convenient orchestrator that builds on several other projects - particularly llama.cpp.

What Ollama really takes over

• Scheduler & resource management
• Model bundles and registry
• Prompt/template handling
• Daemon + HTTP API
• Backend selection (CUDA/Metal/ggml)
• Cache management

What llama.cpp does

llama.cpp is deliberately minimalist:

• pure C/C++ engine
• open GGUF models
• no bundles, no registry
• full control over quantization, tokenizer, build flags

Ollama expands, abstracts and packages – llama.cpp executes.

Why the manual route is important

Anyone who operates AI in sensitive or productive areas needs transparency:

• Which components are executed?
• Which compilers were used?
• Which quantized variant did I actually load?
• What dependencies exist?
• How trustworthy is the supply chain?

Step 1: Build llama.cpp yourself

To have complete transparency about the inference engine, we build llama.cpp directly from source code. The process consists of three clearly defined sub-steps:

1.1 Clone repository

We get the full, unmodified source code from GitHub:

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp

This means that exactly the version that was published on GitHub is available - including commit hashes and build scripts.

1.2 Install dependencies

Using Ubuntu as an example, the build environment required for llama.cpp can be set up with just a few packages. The advantage: The entire toolchain remains extremely slim and transparent.

sudo apt update
sudo apt install -y \
    build-essential \
    cmake \
    libcurl4-openssl-dev

What’s behind it?

• build-essential Installs the basic compiler toolchain (gcc/g++, make).

• cmake Used by llama.cpp as a build system.

• libcurl4-openssl-dev Small, non-critical system library – necessary for HTTP functionality of individual llama.cpp tools.

With this, the Ubuntu environment is already fully prepared to compile llama.cpp from source code.

1.3 Build build

We create a reproducible release build variant:

cmake -B build
cmake --build build --config Release

Optional: Make binary available globally:

export PATH=$PATH:$(pwd)/build/bin

Step 2: Load models without Ollama

Ollama uses its own, complex bundle formats. For a completely transparent stack, we load pure GGUF models instead - with no additional layers, no hidden metadata and no proprietary structure.

The process consists of three clear individual steps:

2.1 Build download tool

We use a small, lightweight CLI (“hfdownloader”) that only loads HuggingFace’s actual artifacts – recursively, reproducibly and without dependency on Python ecosystems.

git clone https://github.com/bodaay/HuggingFaceModelDownloader.git
cd HuggingFaceModelDownloader

# Binary bauen
make

# Binary systemweit verfügbar machen
cp output/hfdownloader_linux_amd64_2.0.0 ~/bin/hfdownloader

2.2 Set access token

Private or large models require a valid HuggingFace access token:

export HF_TOKEN=hf_xxxxxxxxx

2.3 Download GGUF model

Now we get the model exactly as it is on HuggingFace - without conversion or additional data:

hfdownloader download Qwen/Qwen2.5-3B-Instruct-GGUF -o ./models

The folder structure corresponds exactly to the repository. Important: nothing will be unpacked, merged or changed here.

Step 3: Your own API – with llama-server

With the model and engine ready, all we need is a lightweight, local API that applications can use to communicate with the model. This is exactly the gap that llama-server closes.

llama-server is the official HTTP server component of llama.cpp and provides:

• an OpenAI compatible REST API
• without proprietary layers or bundles
• directly executable as a single binary

This means that any GGUF model can be addressed immediately via curl, script or client library.

llama-server   -m ./models/Qwen/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q5_k_m.gguf   --host 0.0.0.0   --port 8080

Step 4: Test API

Once llama-server is running, it provides an OpenAI compatible REST API. This means: existing tools, scripts and clients work without adjustments - only the host changes.

The key endpoints for initial testing are:

4.1 Chat Completion (multiple messages, roles)

The chat endpoint reflects the classic ChatGPT behavior:

curl http://localhost:8080/v1/chat/completions \
  -d '{
        "model": "qwen2.5-3b",
        "messages": [
          { "role": "user", "content": "Erkläre lokale LLMs." }
        ]
      }'

4.2 Text Completion (simple prompt → response)

For classic prompt-in/text-out cases:

curl http://localhost:8080/v1/completions \
  -d '{
        "model": "qwen2.5-3b",
        "prompt": "Schreibe ein Gedicht."
      }'

This endpoint behaves like the original GPT-3 API model: A pure prompt without roles - ideal for short texts, code, extractions and structured output.

Step 5: Bonus – the new web GUI from llama-server

llama-server comes with a lightweight UI:

Features:

• Browser-based chat interface
• multiple sessions, history, parameter control
• completely local
• Document processing (PDF, TXT, MD, JSON)

Conclusion

We opened the Ollama stack layer by layer and built a completely transparent local LLM workflow:

• the inference engine self-compiled
• Models loaded directly and comprehensibly
• Quantization consciously controlled
• Provided our own, minimalistic OpenAI compatible API
• and added local ease of use with the web UI

Result: Instead of an abstracted black box, there is now a clear, auditable and reproducible AI stack that can be understood and adapted at every level.