“AI hosting” covers very different workloads. Training large models needs racks of GPUs, but a lot of practical AI work runs perfectly well on a regular CPU VPS: the back end of an AI app that calls model APIs, retrieval (RAG) pipelines, vector databases, transcription with small models, and small local language models. Here’s what fits, and what doesn’t.

What runs well on a CPU VPS

Workload Why it fits
AI app back ends (calling OpenAI, Anthropic, Gemini and other APIs) The heavy compute happens at the provider; your server handles users, data and orchestration
Chat front ends (Open WebUI, LibreChat) using APIs Light web apps. See LibreChat
RAG pipelines and document chat Embedding + retrieval is moderate work. See AnythingLLM
Vector databases (Qdrant, pgvector, Weaviate, Chroma) RAM-bound, not GPU-bound. See vector databases
Embedding models (small sentence transformers) Fast enough on CPU for moderate volumes
Speech-to-text with small Whisper models Faster than real time on a few cores. See Whisper server
Small LLMs (1–8B parameters, quantised) Usable for private, low-volume tasks. See LocalAI
Automation (n8n, agents, scheduled AI jobs) Orchestration, not compute. See n8n

What needs a GPU

  • Training or fine-tuning anything beyond tiny models.
  • Image generation at usable speed (Stable Diffusion, Flux). See Stable Diffusion on a server.
  • Serving 13B+ language models to many users with low latency.
  • High-volume transcription, or video AI.

For those, use GPU servers or hosted inference APIs. A CPU VPS can still run the web app, queue and storage around them.

Sizing a CPU VPS for local models

Model (quantised, e.g. Q4) RAM Realistic use
1–3B 2–4 GB Classification, extraction, short answers
7–8B 6–8 GB Private chat, drafting, light RAG, a few tokens/second
13–14B 12–16 GB Better quality, slow generation
Embeddings < 1 GB Semantic search for thousands to millions of chunks
  • RAM is the first limit: the whole model must fit in memory, plus context.
  • vCPU count and speed set tokens per second. Inference uses all available cores while generating. See vCPU explained.
  • Storage: models are several GB each. SSD storage speeds up loading.

Runtimes: Ollama (easiest), llama.cpp (most efficient on CPU), LocalAI (OpenAI-compatible API, many modalities).

API-based AI apps: what matters

For apps that call hosted models, the server needs are ordinary web app needs:

  • low latency to your users and to the AI provider’s API;
  • proxy_buffering off in Nginx for streamed (token-by-token) responses;
  • background workers and queues for long jobs;
  • secure storage of API keys. See secrets management.

Provider availability: some AI APIs are restricted in certain regions. If your users or team are in Asia, VPS.DO’s Tokyo VPS plans support access to major AI APIs. See Japan VPS.

Cost comparison

Approach Good when
Hosted APIs + small VPS Most apps; pay per token, best quality
Local small models on CPU VPS Privacy, predictable cost, low volume, simple tasks
GPU servers High volume, large models, image or video generation

Many teams combine them: hosted frontier models for hard tasks, local small models or embeddings for cheap, high-volume steps.

Security

  • Don’t expose model servers (Ollama, LocalAI) publicly without authentication. They’re compute you pay for. Bind to localhost or a VPN. See Tailscale.
  • Rate-limit public AI endpoints, and set spending limits at API providers. See API rate limiting.
  • Treat prompts and documents as sensitive data: encrypt backups, and limit retention.

Quick reference

Question Answer
AI app calling APIs? Any VPS that fits the web app
Local 7–8B model? 8 GB+ RAM, 4+ vCPU, expect a few tokens/second
RAG / vector DB? RAM for the index + embeddings on CPU
Training, image gen, big models? GPU server or hosted API

Related: LocalAI, Open WebUI, and Dify.

  • AI
  • VPS Guide
  • Self-Hosting