Whisper is an open-source speech recognition model released by OpenAI. It transcribes audio in about 100 languages, adds punctuation, and can translate speech to English. Unlike image generation, Whisper runs acceptably on CPU when you use an optimised implementation and a smaller model, so a regular VPS can be a private transcription server for meetings, podcasts, voice notes, or subtitles.

Pick an implementation

Implementation Strengths
faster-whisper (CTranslate2) Up to ~4× faster than the original, int8 on CPU, Python API
whisper.cpp C/C++, no Python, very efficient on CPU, small binaries
Original openai-whisper Reference implementation, slowest on CPU
Servers: faster-whisper-server / Speaches, LocalAI, whisper.cpp server HTTP API, often OpenAI-compatible

Model size vs speed on CPU

Model RAM (int8) Relative speed on CPU Accuracy
tiny / base < 1 GB Much faster than real time OK for clear speech
small ~1–2 GB Faster than real time on 4+ vCPU Good
medium ~2–4 GB Around real time or slower Very good
large-v3 / large-v3-turbo 3–6 GB Slow on CPU (turbo helps) Best

Rule of thumb: start with small (or large-v3-turbo if you have 8+ vCPU and patience), and measure on your own audio. English-only .en variants are a bit better for English. See vCPU explained.

Quick start: faster-whisper (Python)

sudo apt install -y python3-venv ffmpeg
python3 -m venv /opt/whisper && . /opt/whisper/bin/activate
pip install faster-whisper
# transcribe.py
from faster_whisper import WhisperModel
import sys

model = WhisperModel("small", device="cpu", compute_type="int8", cpu_threads=4)
segments, info = model.transcribe(sys.argv[1], vad_filter=True, beam_size=5)
print(f"Language: {info.language}")
for s in segments:
    print(f"[{s.start:7.1f} → {s.end:7.1f}] {s.text.strip()}")
python transcribe.py meeting.mp3

vad_filter=True skips silence, which speeds things up and reduces hallucinated text in quiet parts.

Quick start: whisper.cpp

sudo apt install -y build-essential cmake ffmpeg git
git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp
cmake -B build && cmake --build build -j --config Release
./models/download-ggml-model.sh small
ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le input.wav
./build/bin/whisper-cli -m models/ggml-small.bin -f input.wav -osrt -t 4

-osrt writes subtitles, and -otxt or -ovtt give other formats.

Run an OpenAI-compatible API

An API lets apps send audio over HTTP. Many speech servers mimic OpenAI’s /v1/audio/transcriptions:

services:
  whisper:
    image: ghcr.io/speaches-ai/speaches:latest-cpu
    restart: unless-stopped
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - ./hf-cache:/home/ubuntu/.cache/huggingface
curl http://127.0.0.1:8000/v1/audio/transcriptions \
  -F [email protected] -F model=Systran/faster-whisper-small

Check the project’s README for the current image name and model IDs. LocalAI also serves Whisper on the same endpoint.

Security: keep it on localhost or a VPN, or put Nginx with HTTPS and authentication in front. Raise client_max_body_size for large uploads, and lengthen proxy timeouts. See Nginx reverse proxy.

Batch transcription

For many files, use a simple queue: drop files into a folder, and let a worker transcribe them one at a time (so jobs don’t fight over CPU cores).

for f in /data/inbox/*.mp3; do
  python transcribe.py "$f" > "/data/out/$(basename "${f%.*}").txt" && mv "$f" /data/done/
done

Run it from a systemd timer or cron. See cron jobs.

Tips for better results

  • Convert to 16 kHz mono first, and trim long silences.
  • Set the language when you know it. Auto-detection on the first 30 seconds can guess wrong.
  • Use an initial_prompt with names and jargon to improve spelling.
  • Speaker labels need diarisation (for example pyannote or WhisperX), which is heavier.

Privacy

Self-hosting keeps audio on your server: useful for interviews, medical or legal recordings, or internal meetings. Delete source files when they’re no longer needed, and encrypt backups.

Quick reference

Task Command
Install pip install faster-whisper (+ ffmpeg)
CPU settings device="cpu", compute_type="int8"
Convert audio ffmpeg -i in.mp3 -ar 16000 -ac 1 out.wav
Subtitles whisper.cpp -osrt
API /v1/audio/transcriptions

Related: how much RAM a VPS needs and Stable Diffusion on a server.

  • AI
  • Self-Hosting
  • Docker