OpenTelemetry (OTel) is the open, vendor-neutral standard for collecting traces, metrics and logs from applications. You instrument your code once (often automatically), and send the data to whichever backend you like: self-hosted Jaeger, Grafana Tempo and Prometheus, or a commercial service. Changing backends later doesn’t require touching the code.

Background: observability basics.

The pieces

Component Role
API / SDK Libraries in your app (Python, Node.js, Java, Go, .NET, PHP…) that create spans and metrics
Auto-instrumentation Agents and plugins that instrument frameworks (HTTP servers, DB drivers, HTTP clients) without code changes
OTLP The protocol for sending telemetry (gRPC port 4317, HTTP port 4318)
Collector A service that receives, processes (batching, sampling, filtering) and exports telemetry to backends
Backends Jaeger/Tempo (traces), Prometheus (metrics), Loki (logs)

Run a small backend stack with Docker

# /opt/otel/compose.yaml
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./collector.yaml:/etc/otelcol/config.yaml:ro
    ports:
      - "127.0.0.1:4317:4317"     # OTLP gRPC
      - "127.0.0.1:4318:4318"     # OTLP HTTP
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "127.0.0.1:16686:16686"   # Jaeger UI (use an SSH tunnel)
# /opt/otel/collector.yaml
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
      http: { endpoint: 0.0.0.0:4318 }
processors:
  batch: {}
exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls: { insecure: true }
  prometheus:
    endpoint: 0.0.0.0:8889
service:
  pipelines:
    traces:  { receivers: [otlp], processors: [batch], exporters: [otlp/jaeger] }
    metrics: { receivers: [otlp], processors: [batch], exporters: [prometheus] }

Prometheus scrapes the collector on port 8889. In production, Grafana Tempo is a common trace backend, with Grafana for dashboards. See Grafana + Prometheus. Pin image versions instead of latest. See Docker image tags best practices.

Instrument an app (Python example, zero-code)

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install          # installs instrumentations for detected libraries

export OTEL_SERVICE_NAME=billing-api
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317
export OTEL_TRACES_SAMPLER=parentbased_traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.2          # keep 20% of traces
opentelemetry-instrument gunicorn app:app

Node.js uses @opentelemetry/auto-instrumentations-node, and Java a -javaagent JAR. Same idea, same environment variables. Requests, database queries and outgoing HTTP calls now appear as spans in Jaeger.

Add custom spans

from opentelemetry import trace
tracer = trace.get_tracer("billing")

def generate_invoice(order_id):
    with tracer.start_as_current_span("generate_invoice") as span:
        span.set_attribute("order.id", order_id)
        ...

Context propagation

OTel passes trace context between services in HTTP headers (traceparent, the W3C Trace Context standard). Auto-instrumented clients and servers handle it, so one trace spans your frontend API, background workers, and downstream services. Make sure proxies forward these headers (Nginx does by default).

Connect logs

Include trace_id and span_id in your structured logs (logging instrumentations can inject them), so you can jump from a slow trace to its log lines. See log management.

Sampling and cost

  • Head sampling (decide at the start; e.g. 10%) is simple and cheap.
  • Tail sampling in the collector (keep all errors and slow traces, sample the rest) gives better data, but needs more collector memory.
  • Avoid high-cardinality metric attributes (user IDs, full URLs).

Security

Keep OTLP ports and backend UIs on localhost or a private network. Use SSH tunnels or a VPN for the Jaeger and Grafana UIs, and TLS plus authentication if collectors receive data over the internet. See SSH tunnelling.

Quick reference

Item Value
OTLP ports 4317 (gRPC), 4318 (HTTP)
Service name OTEL_SERVICE_NAME
Endpoint OTEL_EXPORTER_OTLP_ENDPOINT
Python auto-instrument opentelemetry-instrument <cmd>
Sampling OTEL_TRACES_SAMPLER(_ARG)
Trace UI Jaeger :16686 / Grafana + Tempo

Related: alerting best practices and SRE basics.

  • Observability
  • DevOps
  • Monitoring