Skip to content

Ollama

Ollama is a tool for running large language models locally — download and run models like Llama, Mistral, Qwen, Gemma, and DeepSeek on your own machine with a single command. It's the most popular on-ramp to self-hosted inference, and the sovereignty answer to "do I have to send everything to OpenAI/Anthropic?"

Why it matters for owned AI infrastructure

Local models mean prompts and data never leave your hardware — the ultimate privacy stance, and it removes per-token API bills for suitable workloads. The tradeoff: local models are weaker than frontier hosted models and need real hardware (GPU/RAM) for good performance.

Key facts

  • Single-command model pull/run: ollama run llama3.
  • Bundles model weights, config, and a REST API (OpenAI-compatible endpoint) so apps/agents can point at it.
  • Runs on macOS, Linux, Windows; GPU-accelerated where available, CPU fallback.
  • Model library covers most popular open-weight families; supports custom Modelfile builds and quantized variants.
  • Can be self-hosted via docker-compose and fronted by caddy like any service.

What it replaces

Per-token calls to hosted LLM APIs (OpenAI/Anthropic) for workloads a smaller local model can handle. Not a full frontier-model replacement — a complement.