Self-Hosted LLM at Home: What Actually Runs

Your ChatGPT Plus renewal just went up again, or you pasted a chunk of work into a public chatbot and immediately wished you hadn't. You already have a homelab box running Proxmox. The obvious question follows: can that box just run the AI itself?

The honest answer depends entirely on which hardware tier you're on, and most articles on this topic never say so.

What "self-hosted LLM" actually means

Ollama is the tool almost everyone starts with. It wraps llama.cpp under the hood, exposes an OpenAI-compatible API on your local network, and handles model quantization for you — mainly 4-bit (Q4), which cuts a model's RAM footprint by roughly 75% at a small, usually acceptable quality cost. Point a chat UI like Open WebUI at it and you have a private ChatGPT-shaped interface with nothing leaving your network.

None of that changes the physics: a language model's speed is bounded by how much data your CPU or GPU can move per second, and that's exactly where hardware tier stops being a footnote and starts being the whole story.

What each hardware tier can actually run

Hardware tier	Usable model size	Tokens/sec	Verdict
Raspberry Pi 5 (8GB)	1–3B (Gemma 3 1B, Llama 3.2 3B)	8–22 tok/s	Usable for tiny models only; needs active cooling or throttles within ~90 seconds
N100/N150 mini PC (16GB)	up to 7B (Q4)	6–9 tok/s	CPU-only, readable but slow; 13B+ is impractical
GPU / Apple Silicon (RTX 4090, M-series)	7B–32B	~40 tok/s (7B), ~15 tok/s (32B)	The tier that actually feels responsive

Raspberry Pi 5: fine for the smallest models, nothing else

On a Pi 5, 1–3B models like Gemma 3 1B and Llama 3.2 3B run at a genuinely usable 8–22 tokens/sec. Push to a 7B model and generation drops below 1 token/sec — technically it loads, but it's not a chatbot at that point, it's a slideshow. The Pi's Arm CPU also throttles fast: without active cooling, sustained inference cuts token speed roughly in half within about 90 seconds as the SoC heats up.

N100/N150 mini PC: the class most homelabs already run

This is the same N100/N150 mini PC tier this site's own homelab guide recommends as the default first Proxmox box, and it's where most homelabbers will actually test this. Ollama runs entirely on CPU here — the integrated UHD graphics aren't used for inference — and a 7B Q4 model lands at 6–9 tokens/sec. That's slow enough to notice, fast enough to read along with, and the practical ceiling: 13B+ models become impractical on this class of hardware.

GPU / Apple Silicon: where it stops feeling like a compromise

A consumer GPU like an RTX 4090, or Apple Silicon with enough unified memory, changes the picture completely: roughly 40 tokens/sec on a 7B Q4 model, and 32GB of unified memory can run a 32B model at around 15 tokens/sec. This is the tier where a local model starts to feel like a real ChatGPT alternative instead of a proof of concept. Most homelabs don't have this tier sitting idle — it usually means a dedicated purchase, not a repurposed box.

The honest answer for a Proxmox mini PC

In my homelab, Proxmox runs on exactly the N100-class mini PC hardware in the table above, with no discrete GPU to pass through to a VM. That means the CPU-only ceiling described for the mini PC tier is not a hypothetical for this box — it's the real one. A 7B model at 6–9 tokens/sec is genuinely useful for light, private tasks. It is not going to replace a frontier model for anything demanding, and pretending otherwise just sets up a disappointing first fifteen minutes.

What a local LLM can replace — and what it can't

A self-hosted model on mini-PC-class hardware is a solid replacement for:

• Drafting and rewriting text you don't want leaving your network
• Summarizing personal notes, emails, or documents
• Light code autocomplete and boilerplate generation
• A simple retrieval-augmented setup over your own files

It is not a replacement for frontier-level reasoning, very large context windows, fast multimodal (image/audio) work, or anything where waiting 6–9 tokens per second for a long answer is genuinely annoying. For those, ChatGPT or Claude remain the better tool — the self-hosted model's job is the private, low-stakes 80%, not the whole 100%.

Running it: Ollama in a Proxmox VM

Consistent with how Docker workloads should sit inside a Proxmox homelab — in a dedicated VM, not directly on the hypervisor host — Ollama and a chat UI drop in as two services in one Docker Compose file:

services:
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama_data:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    ports:
      - "3000:8080"
    depends_on:
      - ollama
    restart: unless-stopped

volumes:
  ollama_data:

Pull a model that matches your hardware tier from the table above — ollama pull llama3.2:3b on a Pi 5, ollama pull llama3.1:7b on an N100 mini PC — and it's answering questions at http://<vm-ip>:3000 within minutes. Readers building a Proxmox box from scratch can start with this site's homelab beginner's guide before adding this as a second service.

What it actually costs to run

Using the site's standard German household rate of 37.0 ct/kWh (BDEW, April 2026): a mini PC idling at 10W between requests draws 10 × 24 × 365 ÷ 1000 = 87.6 kWh a year, which comes to about €32/year — consistent with the idle-power math already worked out in this site's mini PC buyer's guide. The model itself only pulls the higher 12–18W load figure during active generation, which for occasional personal use adds a few extra euros at most. Electricity is not the reason to skip this; the tokens/sec ceiling in the table above is.

Frequently asked questions

Is running a self-hosted LLM free?

The software (Ollama, Open WebUI, the models themselves) is free and open-source. The only cost is the hardware you already own or buy, plus a few euros a year in electricity, as calculated above.

Do I need a GPU to self-host an LLM?

No. A Raspberry Pi 5 or an N100/N150 mini PC will run small models entirely on CPU, just at the slower tokens/sec speeds shown in the comparison table. A GPU or Apple Silicon device is only required if you want 7B+ models to feel fast.

Is a self-hosted LLM actually private?

Yes, in the sense that matters most: nothing you type is sent to an external company by default. Ollama serves the model locally on your network. You are still responsible for securing that network the same way you would any other self-hosted service.

Can a self-hosted LLM replace ChatGPT or Claude?

For private drafting, summarization, and light code help on mini-PC-class hardware, largely yes. For frontier reasoning, very large context windows, or fast multimodal work, no — those still favor a hosted model, and mini-PC-class hardware simply isn't built for that tier of workload.