Health Checks

Health checks are available in config-driven mode only. Each [[upstream]] — and, for WebSocket backends, each [[ws_proxy]] — that includes a [*.health_check] section gets a dedicated daemon thread (health-{name}) that probes every backend on a regular interval and updates the live backend list without any restart. Both use the exact same checker (start_health_checker), so everything on this page applies to [ws_proxy.health_check] too, unless noted otherwise.

Configuration

[[upstream]]
name     = "api"
backends = ["api-1:3000", "api-2:3000", "api-3:3000"]

[upstream.health_check]
path                 = "/healthz"   # GET path (default: "/health")
interval_secs        = 15           # probe interval in seconds (default: 30)
timeout_ms           = 3000         # connect + read timeout per probe (default: 5000)
healthy_threshold    = 2            # consecutive successes to restore (default: 2)
unhealthy_threshold  = 3            # consecutive failures to remove (default: 3)

Same fields for a WebSocket proxy:

[[ws_proxy]]
name     = "chat"
listen   = "0.0.0.0:9000"
backends = ["wss://chat-a:8443", "wss://chat-b:8443"]

[ws_proxy.health_check]
path                 = "/healthz"
interval_secs        = 10
timeout_ms           = 2000
healthy_threshold    = 2
unhealthy_threshold  = 3

The probe is a plain HTTP GET against the backend’s host:port — not a real WebSocket handshake — connected over TLS when the backend uses wss://. This is deliberate: nginx and Traefik health-check WebSocket upstreams the same way, because a WS backend is almost always a regular HTTP server with an upgrade route, not a WS-only listener, so a plain HTTP endpoint is enough signal that the process and its network stack are up.

How it works

Startup state

All backends start as live. The health checker assumes backends are healthy until proven otherwise.

Probe request

Every interval_secs seconds the checker sends a minimal HTTP/1.1 request to each backend:

GET /healthz HTTP/1.1
Host: api-1
Connection: close

Both the TCP connect and the response read are bounded by timeout_ms. A backend is considered healthy if it replies with a 2xx status code (the checker reads only the first 16 bytes of the response — just enough for HTTP/1.1 2).

Failure tracking

Per-backend counters track consecutive successes and failures independently:

backend api-2:
  consecutive failures = 1  →  still live
  consecutive failures = 2  →  still live
  consecutive failures = 3  →  REMOVED from live list  (unhealthy_threshold reached)

backend api-2 later:
  consecutive successes = 1  →  still dead
  consecutive successes = 2  →  RESTORED to live list  (healthy_threshold reached)

The counters reset on state transition: a success resets the failure counter to 0, and vice versa.

Live list update

After probing all backends, the checker atomically replaces the shared live list:

Arc<RwLock<Vec<String>>>  // written by health checker; read by DynamicProxy

DynamicProxy acquires a read lock on every request, which is concurrent-safe. The health checker acquires a write lock only when publishing the new list.

Log output

State changes are logged to stderr:

[health] upstream=api backend=api-2:3000 removed (3x fail)
[health] upstream=api backend=api-2:3000 restored (2x ok)

All backends unhealthy

If all backends fail their health checks, the live list becomes empty. DynamicProxy returns 502 Bad Gateway for every HTTP/gRPC request until at least one backend recovers; a [[ws_proxy]] returns 503 Service Unavailable for every new WebSocket upgrade attempt instead (the connection never gets far enough to be a “gateway” in the HTTP sense — it’s rejected before any backend is contacted).

No health checks in library mode — except WsProxy

ReverseProxy does not have built-in health checks. Use BackendPool with a File or Dns source (see Service Discovery) and CircuitBreaker (see Circuit Breaker) to replicate the behaviour programmatically.

WsProxy is the exception: WsProxy::with_live_backends(all_backends, live) takes an Arc<RwLock<Vec<String>>> you update from a health-check thread you write yourself — useful if you want a probe other than a plain HTTP GET (e.g. a real WebSocket handshake). See WebSocket Proxy.

Implementation reference

The health checker lives in src/proxy_config/health.rs:

• start_health_checker(upstream_name, backends, live, config) — spawns the daemon thread.
• check_backend(backend, path, timeout) — sends a single probe, returns true on 2xx.
• parse_backend_url(backend) — strips the https:///http:///h2:///wss:///ws:// prefix (or none) and returns (host, port, tls).

WsProxy’s side of this lives in src/ws_proxy/mod.rs:

• WsProxy::with_live_backends(all_backends, live) — the constructor [ws_proxy.health_check] uses internally, also public for library use.
• pick_backend() reads the current live list on every new connection and returns None (→ 503) if it’s empty.