LocalAI on a VPS: OpenAI-Compatible Multi-Model Gateway for LLMs, Embeddings, and Image Generation

LocalAI is a free, open-source alternative to the OpenAI API — it provides a fully compatible OpenAI REST API endpoint that serves local models. Unlike Ollama (focused on model management UX) or vLLM (focused on throughput), LocalAI is an API gateway that supports the broadest range of model formats (GGUF, GPTQ, AWQ), backends (llama.cpp, whisper.cpp, diffusion), and endpoints (chat, embeddings, image generation, audio transcription, TTS) — a single server that covers everything OpenAI’s API offers.

LocalAI vs Ollama vs vLLM

Step 1: Docker Compose Setup

<code">mkdir -p /opt/localai/models && cd /opt/localai
nano docker-compose.yml

<code">version: '3.8'

services:
  localai:
    image: localai/localai:latest-aio-cpu   # All-in-one CPU image
    # For GPU: localai/localai:latest-aio-gpu-nvidia-cuda-12
    container_name: localai
    restart: always
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      MODELS_PATH: /models
      THREADS: 4                    # Match your VPS vCPU count
      CONTEXT_SIZE: 4096
      F16: "true"                   # Use float16 for better performance
      DEBUG: "false"
      # Optional: pre-load a model at startup
      # PRELOAD_MODELS: "mistral-7b"
    volumes:
      - ./models:/models
      # LocalAI gallery cache
      - localai_cache:/root/.cache/huggingface

volumes:
  localai_cache:

<code">docker compose up -d
docker compose logs -f localai   # Wait for "LocalAI API is ready"

Step 2: Nginx with API Key Auth

<code">sudo nano /etc/nginx/sites-available/localai

<code">server {
    listen 443 ssl http2;
    server_name ai.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/ai.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ai.yourdomain.com/privkey.pem;

    location / {
        # Validate API key from Authorization header
        if ($http_authorization != "Bearer YOUR_STRONG_API_KEY") {
            return 401 '{"error":{"message":"Invalid API key","type":"invalid_request_error"}}';
        }

        proxy_pass http://127.0.0.1:8080;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_read_timeout 600s;
        proxy_buffering off;
        proxy_request_buffering off;
    }
}

<code">sudo certbot --nginx -d ai.yourdomain.com
sudo ln -s /etc/nginx/sites-available/localai /etc/nginx/sites-enabled/
sudo systemctl reload nginx

Step 3: Install Models via Gallery

<code"># LocalAI has a model gallery — install models by name
# List available gallery models:
curl http://localhost:8080/models/available | python3 -m json.tool | grep '"id"' | head -20

# Install a model from gallery (downloads and configures automatically)
curl -X POST http://localhost:8080/models/apply \
  -H "Content-Type: application/json" \
  -d '{"id": "mistral-7b-instruct-v0.2-q4_K_M"}'

# Check installation status:
curl http://localhost:8080/models/jobs | python3 -m json.tool

Step 4: Manual GGUF Model Configuration

<code"># Download a GGUF model manually:
pip install huggingface_hub
huggingface-cli download \
    bartowski/Mistral-7B-Instruct-v0.3-GGUF \
    Mistral-7B-Instruct-v0.3-Q4_K_M.gguf \
    --local-dir /opt/localai/models/

<code">nano /opt/localai/models/mistral.yaml

<code">name: mistral-7b
backend: llama-cpp
parameters:
  model: Mistral-7B-Instruct-v0.3-Q4_K_M.gguf
context_size: 4096
f16: true
threads: 4
gpu_layers: 0      # 0 = CPU only; set higher for GPU VRAM offload

template:
  chat_message: |
    [INST] {{if .SystemPrompt}}{{.SystemPrompt}}\n{{end}}{{.Input}} [/INST]
  completion: "{{.Input}}"

Step 5: Embedding Model Setup

<code">nano /opt/localai/models/embeddings.yaml

<code">name: text-embedding-ada-002   # Use OpenAI's name for drop-in compatibility
backend: bert-embeddings
parameters:
  model: bert-base-uncased
embeddings: true
mmap: true

<code"># Test embeddings:
curl http://localhost:8080/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"input": "The quick brown fox", "model": "text-embedding-ada-002"}'

Step 6: Use with OpenAI SDK (Drop-In)

<code">pip install openai

<code">from openai import OpenAI

# Point to LocalAI — zero code changes beyond base_url and api_key
client = OpenAI(
    api_key="YOUR_STRONG_API_KEY",
    base_url="https://ai.yourdomain.com/v1",
)

# Chat completion — identical to OpenAI usage
response = client.chat.completions.create(
    model="mistral-7b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"},
    ],
    temperature=0.7,
    max_tokens=200,
    stream=True,
)
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

# Embeddings — identical to OpenAI usage
embeddings = client.embeddings.create(
    model="text-embedding-ada-002",
    input=["Hello world", "OpenAI compatible"],
)
print(embeddings.data[0].embedding[:5])   # First 5 dimensions

# List available models
models = client.models.list()
for m in models.data:
    print(m.id)

Step 7: Image Generation (Stable Diffusion)

<code">nano /opt/localai/models/stablediffusion.yaml

<code">name: stable-diffusion
backend: diffusers
parameters:
  model: stabilityai/stable-diffusion-2-1
step: 20

<code"># Generate image via API (OpenAI-compatible endpoint):
curl http://localhost:8080/v1/images/generations \
  -H "Content-Type: application/json" \
  -d '{
    "model": "stable-diffusion",
    "prompt": "a golden retriever on a sunny beach, photorealistic",
    "size": "512x512",
    "n": 1
  }' | python3 -m json.tool

Getting Started

LocalAI on CPU runs GGUF models efficiently via llama.cpp. A 4 vCPU / 8 GB RAM Ubuntu VPS at VPS.DO serves 7B models at 3–8 tokens/second — suitable for document processing, batch analysis, and internal tools. The AIO Docker image includes all backends (llama.cpp, whisper, diffusion) pre-installed, making it the fastest path to a multi-capability local AI server.

Conclusion

LocalAI provides a unified OpenAI-compatible API gateway for local models covering chat, embeddings, image generation, and audio — every endpoint the OpenAI API offers, served locally. The YAML model configuration and gallery system make it easy to add and configure new models. For teams building applications against the OpenAI API who want to add a local fallback, reduce costs, or ensure data privacy, LocalAI is the most compatible self-hosted alternative.