How to Run Ollama on a VPS: Self-Hosted LLM Inference for Llama, Mistral, and Qwen

Ollama is an open-source tool that makes running large language models locally as simple as ollama run llama3 — it handles model downloading, quantization selection, hardware detection, and API serving. Running Ollama on a VPS gives you a private LLM inference endpoint: no API rate limits, no per-token billing, no data sent to OpenAI or Anthropic, and an OpenAI-compatible REST API your existing tools can use without code changes.

VPS Requirements by Model Size

Rule of thumb: Model parameters (B) × 0.6 ≈ RAM needed in GB for Q4 quantization. A 8 GB RAM VPS comfortably runs 7B–8B models. CPU inference is slow but works — a 7B model generates ~3–8 tokens/second on 4 vCPU.

Step 1: Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version
sudo systemctl status ollama

Step 2: Download and Run Models

<code"># Pull a model (downloads to /usr/share/ollama/.ollama/models)
ollama pull llama3.2          # 3B — fastest, uses ~3 GB RAM
ollama pull mistral           # 7B — good balance of quality and speed
ollama pull qwen2.5:14b       # 14B — requires 8+ GB RAM
ollama pull nomic-embed-text  # Embedding model for RAG pipelines

# List available models
ollama list

# Test interactively
ollama run mistral
# Type your prompt, Ctrl+D or /bye to exit

# Run with system prompt
ollama run mistral "Explain VPS hosting in one paragraph"

Step 3: Configure Ollama to Listen on All Interfaces

By default, Ollama only listens on localhost. To expose the API (secured behind Nginx with auth):

sudo nano /etc/systemd/system/ollama.service.d/override.conf

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
# Keep bound to localhost — Nginx handles external access with auth

sudo systemctl daemon-reload && sudo systemctl restart ollama

Step 4: Nginx with Basic Auth (Protect the API)

<code"># Create password file
sudo apt install -y apache2-utils
sudo htpasswd -c /etc/nginx/.ollama_htpasswd yourusername

sudo nano /etc/nginx/sites-available/ollama

server {
    listen 443 ssl http2;
    server_name llm.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/llm.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.yourdomain.com/privkey.pem;

    # Basic auth to protect the API
    auth_basic "Ollama API";
    auth_basic_user_file /etc/nginx/.ollama_htpasswd;

    location / {
        proxy_pass http://127.0.0.1:11434;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        # Long timeout for LLM streaming responses
        proxy_read_timeout 600s;
        proxy_buffering off;
    }
}

sudo certbot --nginx -d llm.yourdomain.com
sudo ln -s /etc/nginx/sites-available/ollama /etc/nginx/sites-enabled/
sudo systemctl reload nginx

Step 5: Use the OpenAI-Compatible API

Ollama’s REST API is compatible with the OpenAI SDK — swap the base URL and use existing code:

<code"># Direct API call
curl http://localhost:11434/api/generate \
  -d '{"model":"mistral","prompt":"What is KVM virtualization?","stream":false}' | \
  python3 -m json.tool

# OpenAI-compatible endpoint
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral",
    "messages": [{"role": "user", "content": "Explain Docker in one paragraph"}]
  }' | python3 -m json.tool

<code"># Python with OpenAI SDK — just change base_url
from openai import OpenAI

client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama',  # Required but ignored by Ollama
)

response = client.chat.completions.create(
    model='mistral',
    messages=[
        {'role': 'system', 'content': 'You are a helpful VPS administrator.'},
        {'role': 'user', 'content': 'Write a bash script to monitor disk usage'}
    ],
    stream=True,
)

for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end='', flush=True)

Step 6: Install Open WebUI for a Chat Interface

<code">docker run -d \
  --name open-webui \
  --restart always \
  -p 127.0.0.1:3000:8080 \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open_webui_data:/app/backend/data \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/open-webui/open-webui:main

# Add Nginx site for Open WebUI on a different subdomain
# (same config as ollama but proxy_pass to port 3000)

Step 7: Embedding Models for RAG

<code"># Pull an embedding model
ollama pull nomic-embed-text

# Generate embeddings via API
curl http://localhost:11434/api/embeddings \
  -d '{"model":"nomic-embed-text","prompt":"VPS hosting providers comparison"}'

# Python embedding usage
response = client.embeddings.create(
    model='nomic-embed-text',
    input='Your text to embed here',
)
vector = response.data[0].embedding   # 768-dimensional vector

Performance Tuning

<code"># Environment variables for performance
sudo nano /etc/systemd/system/ollama.service.d/override.conf

[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
# Number of models to keep in memory simultaneously
Environment="OLLAMA_MAX_LOADED_MODELS=2"
# Keep model in memory for N minutes after last use (0 = immediately unload)
Environment="OLLAMA_KEEP_ALIVE=5m"
# Number of parallel requests (limited by RAM)
Environment="OLLAMA_NUM_PARALLEL=1"

Getting Started

For 7B–8B models, a VPS at VPS.DO with 8 GB RAM and 4 vCPU handles Ollama inference at 3–8 tokens/second — adequate for developer tools, internal chatbots, and document processing pipelines where response latency isn’t critical. NVMe storage matters for model loading time: a 4.7 GB model loads in ~10 seconds from NVMe versus 60+ seconds from HDD.

Conclusion

Ollama on a VPS provides private, uncensored LLM inference with no per-token costs and no data leaving your infrastructure. The OpenAI-compatible API means existing tools (LangChain, LlamaIndex, Cursor, Continue.dev) work without code changes. For teams using LLMs for internal document search, code review, or customer support drafting, self-hosted Ollama on a 8 GB VPS is cost-effective versus $20–$100/month OpenAI API bills.