VPS Log Management with Grafana Loki: Centralize, Search, and Alert on Logs

Scattered log files across /var/log/nginx/, /var/log/syslog, and Docker container logs are difficult to correlate during incident response. Grafana Loki is a log aggregation system designed alongside Prometheus — it stores logs indexed by labels (not full-text indexed like Elasticsearch), ships logs via Promtail agents, and integrates into the same Grafana dashboard as your metrics. The result: a single pane for both metric graphs and log streams.

Architecture Overview

• Promtail: Agent that tails log files and forwards entries to Loki with labels
• Loki: Storage backend — keeps logs compressed on disk, indexed only by labels
• Grafana: Visualization — queries Loki with LogQL, shows logs alongside Prometheus metrics

Loki is significantly cheaper to operate than Elasticsearch because it doesn’t build full-text indexes — it stores compressed log chunks and only indexes labels (host, job, level). The trade-off: LogQL is label-first (filter by source, then search content).

Step 1: Docker Compose Setup

mkdir -p /opt/loki && cd /opt/loki
nano docker-compose.yml

version: '3.8'

services:
  loki:
    image: grafana/loki:latest
    container_name: loki
    restart: always
    ports:
      - "127.0.0.1:3100:3100"
    volumes:
      - loki_data:/loki
      - ./loki-config.yml:/etc/loki/local-config.yaml:ro
    command: -config.file=/etc/loki/local-config.yaml

  promtail:
    image: grafana/promtail:latest
    container_name: promtail
    restart: always
    volumes:
      - /var/log:/var/log:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - ./promtail-config.yml:/etc/promtail/config.yml:ro
    command: -config.file=/etc/promtail/config.yml
    depends_on:
      - loki

volumes:
  loki_data:

Step 2: Loki Configuration

nano /opt/loki/loki-config.yml

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 1h
  max_chunk_age: 1h
  chunk_target_size: 1048576

schema_config:
  configs:
    - from: 2024-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks

limits_config:
  retention_period: 30d    # Keep logs for 30 days
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  retention_enabled: true

Step 3: Promtail Configuration

nano /opt/loki/promtail-config.yml

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  # SSH authentication logs
  - job_name: auth
    static_configs:
      - targets: [localhost]
        labels:
          job: auth
          host: YOUR_HOSTNAME
          __path__: /var/log/auth.log

  # Nginx access and error logs
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          host: YOUR_HOSTNAME
          __path__: /var/log/nginx/*.log
    pipeline_stages:
      - regex:
          expression: '^(?P<remote_addr>\S+) .* \[.*\] "(?P<method>\S+) (?P<path>\S+).*" (?P<status>\d+)'
      - labels:
          status:
          method:

  # System log
  - job_name: syslog
    static_configs:
      - targets: [localhost]
        labels:
          job: syslog
          host: YOUR_HOSTNAME
          __path__: /var/log/syslog

  # Docker container logs
  - job_name: docker
    static_configs:
      - targets: [localhost]
        labels:
          job: docker
          host: YOUR_HOSTNAME
          __path__: /var/lib/docker/containers/*/*.log
    pipeline_stages:
      - json:
          expressions:
            log: log
            stream: stream
            container_name: attrs.name
      - labels:
          stream:
          container_name:

docker compose up -d
docker compose logs -f   # Watch startup

Step 4: Add Loki as Grafana Data Source

In Grafana (assuming Grafana is running at port 3000):

1. Connections → Data Sources → Add data source → Loki
2. URL: http://localhost:3100
3. Save & Test → should show “Data source connected and labels found”

Step 5: Essential LogQL Queries

# View all Nginx access logs
{job="nginx"}

# Filter to HTTP 500 errors only
{job="nginx"} |= "\" 5"

# Count 404s per minute (rate query)
count_over_time({job="nginx"} |= "\" 404 "[1m])

# Failed SSH login attempts
{job="auth"} |= "Failed password"

# Firewall blocks (UFW)
{job="syslog"} |= "UFW BLOCK"

# Logs from a specific Docker container
{job="docker", container_name="myapp"}

# Application errors across all sources
{host="your-vps"} |= "ERROR" | logfmt | level="error"

# Rate of errors in last 5 minutes (for alerts)
rate({job="myapp"} |= "ERROR" [5m])

Step 6: Create a Log Dashboard in Grafana

1. Create new dashboard → Add panel
2. Data source: Loki
3. Query: {job="nginx"} | logfmt | status >= 400
4. Visualization: Logs
5. Add another panel with count_over_time({job="nginx"} |= "\" 5" [5m]) as a Time series

Step 7: Log-Based Alerts

1. Grafana → Alerting → Alert Rules → New alert rule
2. Data source: Loki
3. Query: count_over_time({job="auth"} |= "Failed password" [5m])
4. Threshold: value > 20 (alert on 20+ failed SSH attempts in 5 minutes)
5. Contact point: your Telegram bot or email

Getting Started

Loki + Promtail uses approximately 200–400 MB RAM — suitable for running alongside Prometheus and Grafana on a monitoring VPS, or co-located with your application stack. Ubuntu VPS at VPS.DO with NVMe storage keeps Loki’s compressed log chunks fast to write and query. A 2 GB VPS can run the full Prometheus + Grafana + Loki monitoring stack alongside lightweight applications.

Conclusion

Grafana Loki with Promtail centralizes all VPS logs — Nginx, SSH, firewall, system events, and Docker containers — into a single queryable interface in the same Grafana dashboard as your metrics. Label-based indexing keeps Loki resource-efficient compared to Elasticsearch, while LogQL provides powerful filtering and aggregation for both real-time monitoring and incident investigation.