AnythingLLM is an open-source, all-in-one app for chatting with your own documents (retrieval-augmented generation, RAG). You upload PDFs, Word files, web pages or code into workspaces, and AnythingLLM splits them into chunks, embeds them in a vector database, and lets an LLM answer questions with citations from your content. The Docker version runs on a VPS and supports multiple users.

How it works

  1. Ingest: documents are converted to text and split into chunks.
  2. Embed: each chunk is turned into a vector by an embedding model.
  3. Store: vectors go into a vector database (LanceDB is built in, or use Qdrant, Chroma, Milvus, PGVector and others).
  4. Query: your question is embedded, similar chunks are retrieved, and the LLM answers using them.

See vector databases.

Requirements

  • 2 vCPU and 2–4 GB RAM when using a hosted LLM API (OpenAI, Anthropic, Gemini, OpenRouter and others).
  • Much more CPU and RAM if you also run the LLM locally (Ollama or LocalAI). See LocalAI.
  • Disk for documents and vectors.

Install with Docker

sudo mkdir -p /opt/anythingllm/storage
sudo touch /opt/anythingllm/.env
sudo chown -R 1000:1000 /opt/anythingllm
# /opt/anythingllm/compose.yaml
services:
  anythingllm:
    image: mintplexlabs/anythingllm:latest
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3001"
    cap_add: [SYS_ADMIN]
    environment:
      STORAGE_DIR: /app/server/storage
    volumes:
      - ./storage:/app/server/storage
      - ./.env:/app/server/.env
cd /opt/anythingllm && docker compose up -d

SYS_ADMIN is used by the built-in web scraper (headless Chromium). Leave it out if you don’t need scraping, and check the docs for the current recommendation.

HTTPS with Nginx

server {
    server_name docs-ai.example.com;
    client_max_body_size 100M;
    location / {
        proxy_pass http://127.0.0.1:3001;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_buffering off;
        proxy_read_timeout 600s;
    }
    # listen 443 ssl; certbot --nginx -d docs-ai.example.com
}

WebSockets and unbuffered streaming are needed for agents and streamed answers. See Let’s Encrypt.

First-run setup

Open the site, and the onboarding wizard asks for:

  1. LLM provider: an API key for a hosted model, or the URL of Ollama, LocalAI, or any OpenAI-compatible server.
  2. Embedding model: the built-in local embedder (runs on CPU, fine for small collections) or a provider’s embedding API.
  3. Vector database: the built-in LanceDB is fine to start.

Then enable multi-user mode right away (Settings → Security). Set an admin password, create user accounts, and assign roles. A single-user instance on the internet without a password is open to anyone.

Workspaces and documents

  • Create a workspace per topic or team. Each has its own documents, prompt and settings.
  • Upload files, or add web pages, GitHub repositories and other data connectors.
  • Tune chunk size, similarity threshold, and the number of retrieved chunks if answers miss context.
  • Use query mode (answer only from documents) for support or knowledge-base use, to reduce made-up answers.

Embedding a chat widget

AnythingLLM can generate an embeddable chat widget for a website, backed by a workspace. Restrict the allowed domains, set rate limits, and only use documents that are fine to make public.

API

A developer API (Settings → API keys) lets other apps upload documents and query workspaces. Keep keys server-side. See API security.

Backups and updates

  • Everything lives in the storage directory (database, documents, vectors) plus .env. Stop the container, archive it, then start it again. See backup strategy.
  • Update with docker compose pull && docker compose up -d. See updating self-hosted apps.

Privacy note

Documents stay on your server, but retrieved chunks are sent to your LLM provider with each question. For sensitive data, use a local model, or a provider with suitable data terms.

Quick reference

Item Value
Port 3001
Data storage/ + .env
Default vector DB LanceDB (built in)
Multi-user Settings → Security
Streaming via Nginx proxy_buffering off + WebSocket headers

Related: LibreChat, Open WebUI, and Dify.

  • AI
  • Self-Hosting
  • Docker