Introduction
Large Language Models – LLMs for short – are the driving force of the current AI wave. They generate texts, code, dialogues and answers as if they had minds. Well-known representatives are called GPT-4 (OpenAI), Claude (Anthropic), Gemini (Google), Llama (Meta) or Mistral (Mistral AI).
What they have in common: They live almost exclusively in the cloud. Their intelligence belongs to the infrastructure of the providers - not the user. Every request runs through remote servers, every response is the result of an opaque calculation somewhere between the data center and the business model.
But there are alternatives. A growing part of the community is trying to regain control - through local LLMs that run directly on their own computer. No cloud, no API key, no tracking. Just computing time, electricity and patience.
How an LLM works
An LLM is, at its core, a text prediction system. It analyzed billions of text examples and learned which word was likely to come next. It doesn’t remember sentences, but rather patterns - connections between words, meanings and contexts.
So when you ask it a question, it calculates the most likely next word, character by character. This happens billions of times per answer. A modern laptop can do this, but slowly - and with an audible fan noise.
The models are mostly based on so-called transformers - neural networks that can “pay attention” to many words in a sentence at the same time (“self-attention”). This makes them surprisingly good at capturing structure and meaning – at least statistically.
Risks and dependencies
The large cloud models bring enormous computing power, but also dependencies. You give out data without knowing exactly what happens to it. The models change silently overnight. Access can be restricted, regulated or monetized.
There is also a cultural danger: the idea that voice intelligence is something that can only exist in proprietary data centers - and no longer in your own toolbox.
Local LLMs as an alternative
A locally operated LLM shifts this balance of power. Instead of accessing external servers, the model runs on its own hardware. The only cloud is the water vapor from the CPU cooling.
This has three immediate consequences:
- Data sovereignty: No prompt leaves the computer.
- Independence: No provider can block or throttle access.
- Transparency: The entire process - from download to generation - is visible.
However, this freedom costs computing time. And patience.
Ollama - a local LLM runtime environment
Ollama is an LLM Runtime - a runtime environment for language models. It provides the same range of functions locally that only cloud APIs otherwise offer: Load, manage, chat, create or embed models.
Technically, Ollama consists of three parts:
- Server (
ollama serve) - a local HTTP service on port 11434 that supports commands like/api/chator/api/generateprovides. - CLI Client (
ollama run,ollama pull,ollama create) – the command line tool. - Model Repository - a public catalog from which models such as “mistral”, “llama3” or “phi3” can be downloaded.
Ollama is open source, written in Go, and self-compilable.
Practice: Build Ollama locally and run Mistral model
Dependencies
sudo apt update
sudo apt install -y build-essential git curl pkg-config cmake
Build from source code
git clone https://github.com/ollama/ollama.git
cd ollama
go build
cp ollama ~/bin/
Then start:
ollama serve
The server generates an SSH key on the first run and starts a local API:
[GIN-debug] POST /api/generate
[GIN-debug] GET /v1/models
...
A test shows that everything is working:
curl http://localhost:11434/api/tags
Answer:
{"models":[]}
Download and start Mistral
ollama pull mistral
ollama run mistral
Prompt:
>>> Was bedeutet Unabhängigkeit für Dich?
Answer:
Für mich bedeutet Unabhängigkeit, die Freiheit und Selbstbestimmung ohne externe Kontrolle …
The model runs – slowly, but locally.
Derive your own models
Ollama allows you to create your own variants based on existing models.
You write a small text file called Modelfile:
FROM mistral
SYSTEM """
Du bist ein erfahrener Berater und Sounddesigner für elektronische Musikinstrumente.
Deine Spezialgebiete sind analoge und digitale Synthesizer, FM-Synthese, Modularsysteme
und die Erstellung von Patches in DAWs.
"""
PARAMETER temperature 0.8
PARAMETER top_p 0.9
Then:
ollama create mymodel -f Modelfile
ollama run mymodel
Example prompt:
>>> Welchen Synthesizer kannst Du für warme Flächensounds empfehlen?
Answer:
Für warme Flächensounds empfehle ich den Moog Subsequent 37 oder Software wie Pigments …
A simple model extension – but created completely locally.
FROM mistral
```→ specifies the **base model** from which it is derived.
```text
SYSTEM """ ... """
```→ defines the **system prompt** – i.e. role and style of the model (here: synthesizer advisor).
```text
PARAMETER temperature 0.8
```→ determines the **creativity** of the answers (lower = more precise, higher = more free).
```text
PARAMETER top_p 0.9
```→ controls how **broadly** the model selects from possible words (trade-off between diversity and coherence).
📝 *The model file is basically the recipe that gives an existing LLM a new identity and different response behavior.*
Reality and limits
The whole thing works – with limitations. On a 2021 i5 laptop without GPU, a response takes about five minutes. The calculation runs on the CPU and uses all cores. With a modern graphics card (CUDA, ROCm or Metal) the response time is drastically reduced.
Training such models is on a completely different scale: Thousands of GPUs, runtimes lasting months, megawatts of energy. Here one inevitably remains dependent on published models.
Conclusion
Running an LLM locally is a mix of fascination and disillusionment. Fascinating because you suddenly experience a piece of linguistic intelligence on your own computer – open source, comprehensible, tangible. Sobering because you realize how much energy and computing power these systems need.
The Ollama runtime environment is an important step towards self-determination: She turns models into tools. But the illusion of complete independence remains fragile – as long as the models themselves come from central sources.
Nevertheless, the moment when the terminal writes a meaningful answer after a few minutes is a small triumph. It’s like having a miniature version of the cloud in your own hand.
“The machine responds – slowly, puffing, but independently. Maybe sometimes independence is just that: trading time for control."