Once you run more than a couple of servers, SSHing into each one to grep logs gets old fast. Grafana Loki collects logs from all of them into one place, so you can search everything from Grafana, correlate it with metrics, and alert on log patterns. Unlike Elasticsearch, Loki doesn’t index the full text. It indexes only a few labels (host, service, environment), and stores compressed log chunks. That makes it far cheaper to run on a VPS.

Architecture

servers: Grafana Alloy / Promtail  ──push──▶  Loki  ◀──query──  Grafana
         (tail files, journald, Docker)       (stores chunks + label index)
  • Agents on each server read logs and push them to Loki, with labels.
  • Loki stores them, on local disk or object storage (S3, MinIO).
  • Grafana queries them with LogQL.

Promtail was the classic agent. Grafana Alloy is its successor, and handles logs, metrics and traces in one agent. Both work.

Run Loki (single binary, Docker)

# /opt/loki/compose.yaml
services:
  loki:
    image: grafana/loki:3.3.2
    restart: unless-stopped
    command: -config.file=/etc/loki/config.yaml
    ports:
      - "10.0.0.5:3100:3100"           # private interface only
    volumes:
      - ./config.yaml:/etc/loki/config.yaml:ro
      - ./data:/loki

A minimal config.yaml with local storage and 30-day retention:

auth_enabled: false
server:
  http_listen_port: 3100
common:
  path_prefix: /loki
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h
limits_config:
  retention_period: 720h
compactor:
  working_directory: /loki/compactor
  retention_enabled: true
  delete_request_store: filesystem

auth_enabled: false means there’s no built-in authentication. Keep Loki on a private network or VPN, or put an authenticating proxy in front. See WireGuard.

Ship logs with Promtail

On each server (Docker example):

# promtail-config.yaml
server:
  http_listen_port: 9080
positions:
  filename: /tmp/positions.yaml
clients:
  - url: http://10.0.0.5:3100/loki/api/v1/push
scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
        host: web1
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit
  - job_name: nginx
    static_configs:
      - targets: [localhost]
        labels:
          job: nginx
          host: web1
          __path__: /var/log/nginx/*.log

Alloy uses its own configuration language with equivalent components (loki.source.file, loki.source.journal, loki.write). For Docker containers, use the Docker service discovery, or the Loki Docker logging driver. See docker logs and journalctl.

Labels: keep them few and low-cardinality

Labels decide how logs are grouped and indexed. Good labels: host, job/service, env, unit, level. Bad labels: user IDs, request IDs, IPs, full URLs. These create millions of streams and hurt performance badly. Search those values in the log content instead.

Query with LogQL

In Grafana: Explore → Loki.

{job="nginx", host="web1"}                                   # all Nginx logs from web1
{job="nginx"} |= " 500 "                                      # lines containing " 500 "
{unit="ssh.service"} |~ "Failed password|Invalid user"        # regex filter
{job="nginx"} | pattern `<ip> - - [<_>] "<method> <path> <_>" <status> <_>` | status >= 500
sum by (host) (rate({job="nginx"} |= " 500 " [5m]))           # error rate per host (metric from logs)
topk(10, sum by (ip) (count_over_time({job="nginx"} | pattern `<ip> <_>` [1h])))   # top client IPs

Parsers (json, logfmt, pattern, regexp) extract fields at query time.

Alert on logs

Loki’s ruler (or Grafana alerting) evaluates LogQL metric queries:

- alert: NginxErrorsHigh
  expr: sum(rate({job="nginx"} |= " 500 " [5m])) > 1
  for: 10m
- alert: SSHBruteForce
  expr: sum by (host) (count_over_time({unit="ssh.service"} |= "Failed password" [5m])) > 50

See server security monitoring.

Retention and storage

  • Set retention_period, and enable the compactor, or the disk fills up.
  • Loki compresses well, but plan for roughly 10–20% of the raw log volume on disk.
  • For more data, or durability, use S3-compatible object storage (MinIO, cloud storage). See MinIO.

Loki vs alternatives

Need Tool
Cheap central logs, Grafana integration Loki
Full-text search, complex analytics OpenSearch / Elasticsearch. See OpenSearch
Simple forwarding and archiving rsyslog to a log server. See rsyslog
Security analytics (SIEM) Wazuh. See Wazuh

Quick reference

Task Command / setting
Loki port 3100 (/loki/api/v1/push)
Retention limits_config.retention_period + compactor
Agents Grafana Alloy or Promtail
Filter lines |= "text", |~ "regex"
Rate from logs rate({…} |= "x" [5m])

Related: Prometheus and Grafana and logrotate.

  • Monitoring
  • Logging
  • Self-Hosting