vLLM on a VPS: High-Throughput LLM Inference Server with OpenAI-Compatible API

vLLM is a high-throughput LLM inference engine from UC Berkeley, optimized for serving language models to multiple concurrent users efficiently. Its key innovation — PagedAttention — manages GPU/CPU memory like an operating system manages virtual memory, enabling 24× higher throughput than naive inference implementations. For teams serving LLMs to multiple users simultaneously, vLLM significantly outperforms Ollama in throughput while maintaining the same OpenAI-compatible API.

vLLM vs Ollama: When to Choose Each

Server Requirements

• GPU (optimal): NVIDIA GPU with 8+ GB VRAM for 7B models, CUDA 12.1+
• CPU (supported): 16+ GB RAM for 7B models, 4+ vCPU; uses AVX2/AVX-512 for optimized inference
• Python 3.9+, 50+ GB disk for model storage

Step 1: Install vLLM (CPU Mode for Standard VPS)

<code"># For standard VPS (CPU inference):
sudo apt install -y python3.11 python3.11-venv python3-pip
python3.11 -m venv /opt/vllm-env
source /opt/vllm-env/bin/activate

# Install vLLM with CPU support
pip install vllm --extra-index-url https://download.pytorch.org/whl/cpu

# Verify installation
python -c "import vllm; print(vllm.__version__)"

# For GPU VPS (NVIDIA with CUDA):
# pip install vllm   # Default install uses CUDA

Step 2: Download a Model

<code"># Install HuggingFace Hub CLI
pip install huggingface_hub

# Log in (free account at huggingface.co)
huggingface-cli login

# Download a model (example: Mistral 7B Instruct, quantized for CPU)
huggingface-cli download \
    mistralai/Mistral-7B-Instruct-v0.3 \
    --local-dir /opt/models/mistral-7b-instruct

# Or use a GGUF-quantized model for faster CPU inference:
huggingface-cli download \
    bartowski/Mistral-7B-Instruct-v0.3-GGUF \
    Mistral-7B-Instruct-v0.3-Q4_K_M.gguf \
    --local-dir /opt/models/

# Lighter model for limited RAM VPS:
huggingface-cli download \
    microsoft/Phi-3-mini-4k-instruct \
    --local-dir /opt/models/phi3-mini

Step 3: Start vLLM Server

<code"># Serve Mistral 7B (CPU — will be slow but functional)
source /opt/vllm-env/bin/activate

python -m vllm.entrypoints.openai.api_server \
    --model /opt/models/mistral-7b-instruct \
    --host 127.0.0.1 \
    --port 8000 \
    --dtype bfloat16 \
    --max-model-len 4096 \      # Limit context window to save RAM
    --served-model-name mistral-7b \
    --device cpu \              # Explicit CPU mode
    --num-cpu-blocks 512        # KV cache blocks for CPU

Step 4: Systemd Service

<code">sudo nano /etc/systemd/system/vllm.service

<code">[Unit]
Description=vLLM OpenAI-Compatible Server
After=network.target

[Service]
Type=simple
User=ubuntu
WorkingDirectory=/opt
ExecStart=/opt/vllm-env/bin/python -m vllm.entrypoints.openai.api_server \
    --model /opt/models/mistral-7b-instruct \
    --host 127.0.0.1 \
    --port 8000 \
    --dtype bfloat16 \
    --max-model-len 4096 \
    --device cpu
Restart=on-failure
RestartSec=10
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

<code">sudo systemctl daemon-reload
sudo systemctl enable vllm
sudo systemctl start vllm
journalctl -u vllm -f

Step 5: Nginx with API Key Authentication

<code">sudo nano /etc/nginx/sites-available/vllm

<code">server {
    listen 443 ssl http2;
    server_name llm.yourdomain.com;

    ssl_certificate /etc/letsencrypt/live/llm.yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.yourdomain.com/privkey.pem;

    location / {
        # Validate API key
        if ($http_authorization != "Bearer YOUR_STRONG_API_KEY") {
            return 401 '{"error": "Unauthorized"}';
        }

        proxy_pass http://127.0.0.1:8000;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_read_timeout 600s;
        proxy_buffering off;
    }
}

Step 6: Use the OpenAI-Compatible API

<code"># Direct API call (identical to OpenAI API)
curl https://llm.yourdomain.com/v1/chat/completions \
  -H "Authorization: Bearer YOUR_STRONG_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral-7b",
    "messages": [{"role": "user", "content": "Explain vLLM in one sentence."}],
    "max_tokens": 100
  }'

# Python with OpenAI SDK — just change base_url:
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_STRONG_API_KEY",
    base_url="https://llm.yourdomain.com/v1",
)

response = client.chat.completions.create(
    model="mistral-7b",
    messages=[{"role": "user", "content": "Write a Python function to sort a list"}],
    stream=True,
)
for chunk in response:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end='', flush=True)

Step 7: Serving Multiple Models

<code"># vLLM can serve multiple models using multiple instances
# Run on different ports, proxy with Nginx:

# Instance 1: General model
python -m vllm.entrypoints.openai.api_server \
    --model /opt/models/mistral-7b-instruct \
    --port 8000 --served-model-name general &

# Instance 2: Code model
python -m vllm.entrypoints.openai.api_server \
    --model /opt/models/codestral \
    --port 8001 --served-model-name code &

# Nginx routes by model name in request path

Performance: CPU vs GPU

• CPU (4 vCPU): 3–8 tokens/second for 7B models — adequate for async use cases
• CPU (16 vCPU): 8–20 tokens/second with AVX-512 optimization
• GPU (RTX 4090): 80–120 tokens/second — required for real-time interactive use
• For interactive chat, CPU inference is usable; for document processing pipelines, token rate is less critical

Getting Started

vLLM on CPU works on any Ubuntu VPS at VPS.DO, though model inference speed depends heavily on CPU count and RAM. For CPU inference of 7B models, 16 GB RAM and 8+ vCPUs provide acceptable performance for batch document processing use cases. vLLM’s OpenAI-compatible API means it integrates with every tool that supports OpenAI — LangChain, LlamaIndex, Open WebUI, and custom applications.

Conclusion

vLLM provides production-grade LLM inference with continuous batching, PagedAttention memory management, and an OpenAI-compatible API. For teams serving LLMs to multiple concurrent users, vLLM’s throughput advantage over sequential inference is significant. CPU deployment on a high-core-count VPS is viable for batch processing workloads; GPU deployment is required for interactive, real-time applications. The drop-in OpenAI API compatibility makes migration from cloud LLM APIs straightforward.