Running a language model on your own server keeps prompts and data private, avoids per-token fees, and gives you an API you control. Ollama makes it simple: one command installs it, another downloads and runs a model. With Open WebUI on top, you get a ChatGPT-style interface for your team.

Set realistic expectations

A typical VPS has no GPU, so models run on the CPU. That works well for small models (roughly 1–8 billion parameters, quantized):

Model size (4-bit quantized) RAM needed (approx.) Speed on a 4–8 vCPU VPS
1–3B (e.g. Llama 3.2 3B, Qwen 2.5 3B, Gemma small variants) 2–4 GB Comfortable for chat and automation
7–8B (e.g. Llama 3.1 8B, Qwen 2.5 7B, Mistral 7B) 6–8 GB Usable; a few tokens per second
14B+ 12 GB+ Slow on CPU; better with a GPU

Good CPU-friendly uses: summarizing, classification, extraction, drafting, private chat, embeddings for search (RAG). For large models at high speed, you need GPU hardware.

1. Install Ollama

curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollama

The installer creates a systemd service that listens on 127.0.0.1:11434.

2. Run a model

ollama pull llama3.2:3b
ollama run llama3.2:3b "Explain what a reverse proxy is in two sentences."
ollama list

Browse available models and sizes in the Ollama library and pick a quantized variant that fits your RAM with room to spare.

3. Use the API

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.2:3b",
  "prompt": "Write a haiku about servers.",
  "stream": false
}'

Ollama also offers an OpenAI-compatible endpoint at /v1/chat/completions, so many existing libraries and tools work by just changing the base URL.

4. Add Open WebUI

docker run -d --name open-webui --restart unless-stopped \
  -p 127.0.0.1:3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

For the container to reach Ollama, Ollama must listen on an address the container can reach. Edit the service:

systemctl edit ollama
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
systemctl restart ollama
ufw deny 11434          # keep the API off the internet

Then put Open WebUI behind Nginx with HTTPS (see reverse proxy for Docker containers). The first account you create becomes the admin; disable open sign-ups afterwards in the admin settings.

5. Secure it

  • Never expose port 11434 publicly. The Ollama API has no authentication.
  • Access the web UI only via HTTPS, with strong passwords — or behind SSO (Authentik) or a VPN (WireGuard).
  • Keep Ollama and Open WebUI updated.

6. Tune performance

  • Give it RAM: the model must fit in memory, plus the context window.
  • OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS control concurrency — keep them low on small servers.
  • OLLAMA_KEEP_ALIVE controls how long a model stays loaded after use (default a few minutes); a longer value avoids reload delays.
  • Smaller context windows are faster.
  • Watch htop while generating — all cores should be busy.

Using cloud AI APIs instead (or as well)

If you need frontier-model quality, call hosted APIs from your server and use local models for private or high-volume simple tasks. Some AI providers restrict access by region; our Japan VPS is in a region supported by major AI services, and the USA VPS works well for US-based APIs.

Sizing

For a 3B model with Open WebUI, start with 4 vCPU and 8 GB RAM; for 7–8B models, 6–8 vCPU and 12–16 GB. More cores mean faster generation. See our Docker VPS plans, and self-hosted Telegram bots for one way to put your model to work.

  • AI
  • Ollama
  • LLM
  • Docker