Metrics and health

The node offers health endpoints for container runtimes and load balancers, and an optional Prometheus endpoint for monitoring.

Health endpoints

Two endpoints tell whether the node is alive and whether it is ready to serve.

Endpoint	Answers	Use it for
GET /healthz	Always 200 while the process serves requests: {"status":"ok","voice":"running"}. voice is the state of the supervised LiveKit process (disabled, stopped, starting, running, restarting); voice_problem appears when the operator has to fix something in the voice setup (e.g. none of the announced hostnames resolves).	Liveness.
GET /readyz	200 or 503 with {"ready":…,"database":…,"livekit":…,"cache":…}. ready requires the database and, with voice on, LiveKit answering on its local port (livekit is null with voice off). cache reports Valkey and the cache bus (null without TAVRN_CACHE_URL) but is not part of ready.	Readiness, load balancer health checks, external monitoring.

The image’s Docker HEALTHCHECK runs tavrn-node health every 30 seconds: one request to /healthz on the listen port inside the container with a 3-second timeout, nothing printed while healthy. The container turns unhealthy when the server stops answering (hung, crashed, the listener gone); a database or LiveKit outage does not restart it (tavrn-node health --ready probes /readyz instead, for a setup that wants that). It does not replace a /readyz check from outside, and it never contacts the identity provider (check-config does, run it by hand).

Prometheus metrics

TAVRN_METRICS=on exposes GET /metrics in the Prometheus text format on the node’s HTTP port (TAVRN_LISTEN).

Access:

• TAVRN_METRICS_TOKEN set (16–256 printable ASCII characters without spaces): scrapers must send Authorization: Bearer <token>.
• TAVRN_METRICS_TOKEN empty: /metrics answers only to requests from loopback and from the addresses in TAVRN_TRUSTED_PROXIES: the TCP peer and every address in X-Forwarded-For must be one of them, so public requests through a reverse proxy on the same host are refused; everything else gets the same 404 as an unknown path. Set a token when TAVRN_TRUSTED_PROXIES lists networks that untrusted hosts share with the node (the compose file lists Docker’s private ranges).
• Block /metrics at the reverse proxy for the public anyway (nginx: location = /metrics { return 404; }) and scrape the instances directly, not through the public host name.

Metrics (all prefixed tavrn_):

Metric	Type	Labels	Meaning
tavrn_http_requests_total	counter	method, route, status	HTTP requests. route is the route template such as /api/v1/rooms/{room_id}, never an id.
tavrn_http_request_duration_seconds	histogram	method, route	Request duration.
tavrn_websocket_connections	gauge		Open WebSocket connections of users.
tavrn_connected_devices	gauge		Devices with at least one open connection.
tavrn_messages_total	counter	encrypted="true"|"false"	Messages created on this instance.
tavrn_upload_bytes_total	counter		Bytes of uploaded attachments and files.
tavrn_voice_participants	gauge		Participants in voice channels.
tavrn_calls_active	gauge		Active calls in DMs and group DMs.
tavrn_federation_deliveries_total	counter	result="success"|"failure"	Delivery attempts of the federation outbox.
tavrn_push_sent_total	counter		Web pushes accepted by a push service.
tavrn_push_failed_total	counter		Web pushes that failed.
tavrn_db_pool_connections	gauge		Open database connections.
tavrn_db_pool_idle	gauge		Idle database connections.
tavrn_cache_up	gauge		1 Valkey reachable, 0 not; only with TAVRN_CACHE_URL.
tavrn_housekeeping_last_run_timestamp_seconds	gauge		Unix time of the last finished housekeeping round on this instance; 0 before the first (hourly, the first one hour after the start; only the leader runs it with several instances).
tavrn_build_info	gauge	version	Always 1; the label carries the node version.

Each instance exposes its own metrics: scrape every instance. A Prometheus scrape configuration with the token in a file:

scrape_configs:
  - job_name: tavrn
    metrics_path: /metrics
    scheme: http
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/tavrn-metrics-token   # the value of TAVRN_METRICS_TOKEN
    static_configs:
      - targets: ["tavrn-1.internal:8080", "tavrn-2.internal:8080"]

Dashboards and alerts

Graph request rate and latency per route, WebSocket connections, voice participants and the database pool in Grafana or a similar tool.

Alerts worth having (adapt thresholds to your node):

groups:
  - name: tavrn
    rules:
      - alert: TavrnHighErrorRate          # more than 5 % server errors
        expr: sum(rate(tavrn_http_requests_total{status=~"5.."}[5m])) / sum(rate(tavrn_http_requests_total[5m])) > 0.05
        for: 10m
      - alert: TavrnPushFailing            # more failed than delivered pushes
        expr: sum(rate(tavrn_push_failed_total[15m])) > sum(rate(tavrn_push_sent_total[15m]))
        for: 30m
      - alert: TavrnFederationFailing      # more than 20 % failed deliveries to other nodes
        expr: sum(rate(tavrn_federation_deliveries_total{result="failure"}[15m])) / sum(rate(tavrn_federation_deliveries_total[15m])) > 0.2
        for: 30m
      - alert: TavrnHousekeepingStalled    # no housekeeping run for more than 2 hours
        expr: time() - max(tavrn_housekeeping_last_run_timestamp_seconds) > 7200 and max(tavrn_housekeeping_last_run_timestamp_seconds) > 0
        for: 15m
      - alert: TavrnCacheDown
        expr: tavrn_cache_up == 0
        for: 2m

Add a probe of /readyz per instance (e.g. with the Prometheus blackbox exporter) and alert when it fails. With several instances only the leader runs the housekeeping, hence max() across the instances in the housekeeping rule.

---

This page is generated from tavrn-node/docs/HANDBOOK.md (AGPL-3.0-only) when the website is built; its links were adapted for this site.