LocalAI is an open-source server that exposes an OpenAI-compatible API (chat completions, embeddings, audio transcription, image generation, text-to-speech) backed by models running on your own hardware. Apps written for the OpenAI API can point at LocalAI by changing the base URL. No data leaves your server, and there are no per-token fees.

CPU vs GPU: set expectations

Most VPS plans have no GPU. LocalAI runs on CPU, with realistic limits:

Model size (quantised) RAM needed CPU speed (typical VPS) Good for
1–3B parameters (Q4) 2–4 GB Fast enough for short chats Classification, extraction, simple Q&A
7–8B (Q4) 6–8 GB A few tokens/second General chat, drafting, light RAG
13B+ 12–16 GB+ Slow on CPU Better quality, impatient users suffer
Embedding models < 1 GB Fast Semantic search, RAG
Whisper (small/base) 1–2 GB Faster than real time for short clips Transcription

CPU inference uses all cores while generating. Pick a plan with enough vCPU and RAM, and don’t run it beside latency-sensitive services. For large models, or many users, use GPU servers or hosted APIs. See how much RAM a VPS needs.

Install with Docker (CPU)

# /opt/localai/compose.yaml
services:
  localai:
    image: localai/localai:latest-cpu
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      THREADS: 4                     # ≈ number of physical cores
      CONTEXT_SIZE: 4096
      API_KEY: change-me-long-random   # require a Bearer key
    volumes:
      - ./models:/build/models
cd /opt/localai && docker compose up -d
docker compose logs -f

(AIO images come with preconfigured models. GPU images exist for NVIDIA CUDA and others.)

Install models

Use the built-in gallery (web UI at http://127.0.0.1:8080, via an SSH tunnel), or the API:

curl http://127.0.0.1:8080/models/apply -H "Authorization: Bearer $KEY" \
  -H "Content-Type: application/json" -d '{"id": "[email protected]:q4_k_m"}'

You can also place GGUF model files in ./models with a YAML config. Prefer quantised GGUF models (Q4_K_M is a good balance) sized to your RAM. Check model licences for commercial use.

Use the OpenAI-compatible API

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"model": "llama-3.2-3b-instruct:q4_k_m", "messages": [{"role":"user","content":"Summarise RAID 1 in one sentence."}]}'

With the official OpenAI SDKs, set the base URL:

from openai import OpenAI
client = OpenAI(base_url="https://ai.example.com/v1", api_key="change-me-long-random")
resp = client.chat.completions.create(model="llama-3.2-3b-instruct:q4_k_m", messages=[{"role":"user","content":"Hi"}])

Other endpoints: /v1/embeddings (for RAG and vector search), /v1/audio/transcriptions (Whisper), /v1/images/generations and /v1/audio/speech. See vector databases and Whisper transcription server.

Expose it securely

  • Keep the port on localhost or a private network. LocalAI happily burns CPU for anyone who can reach it.
  • Set API_KEY, and put Nginx with HTTPS in front if apps on other servers need access. Restrict by IP or VPN where possible. See Nginx reverse proxy and WireGuard.
  • Disable buffering for streaming responses (proxy_buffering off;).

Front-ends and integrations

LocalAI vs Ollama

LocalAI Ollama
API OpenAI-compatible (plus more modalities) Own API + OpenAI-compatible chat
Modalities Text, embeddings, audio, images, TTS Mostly text and embeddings
Ease More configuration Very easy model management

Quick reference

Item Value
Port 8080
Chat API /v1/chat/completions
Embeddings /v1/embeddings
Auth API_KEY → Authorization: Bearer …
Models ./models (GGUF + YAML), gallery
Threads THREADS ≈ physical cores

Related: vector databases and CPU benchmarks.

  • AI
  • Self-Hosting
  • Docker