# Metrics

> Built-in Prometheus metrics — server-wide counters always active, per-route histograms via MetricsLayer middleware.

Source: https://rws8.tech/docs/features/metrics/ · Last updated 2026-11-16

`rust-web-server` exposes two tiers of Prometheus-compatible metrics. Server-wide counters are always active at the server-core level with no configuration. Per-route counters and latency histograms are opt-in via `MetricsLayer` middleware. All metrics are served at `GET /metrics` in the standard Prometheus text format.

## Server-wide counters

These atomics are updated by `Server::process()` and the connection dispatch loop automatically. No middleware registration is required.

| Symbol | Type | Description |
|--------|------|-------------|
| `REQUESTS_TOTAL` | counter | Total HTTP requests handled across all connections and protocols |
| `ERRORS_TOTAL` | counter | Requests where `app.execute()` returned `Err` |
| `ACTIVE_CONNECTIONS` | gauge | Number of currently open TCP/QUIC connections |
| `SERVER_READY` | `AtomicBool` | `true` after server setup; `false` during shutdown; drives `/readyz` |

### Helper functions

```rust
use rust_web_server::metrics;

metrics::record_request();   // increments REQUESTS_TOTAL
metrics::record_error();     // increments ERRORS_TOTAL
metrics::connection_open();  // increments ACTIVE_CONNECTIONS
metrics::connection_close(); // decrements ACTIVE_CONNECTIONS
```

These functions are called by the server core, but you can also call them from custom instrumentation code.

## `GET /metrics` endpoint

The built-in endpoint returns all metrics in Prometheus text format. No setup is needed — it is registered automatically by `App::new()`.

It answers only clients in `metrics_allow` (`RWS_CONFIG_METRICS_ALLOW`):
loopback and the private ranges by default, where Prometheus scrapes pods
from, so the internet gets a `404` — the metrics name upstreams and backend
addresses (since 18.27.0; before, anyone). Behind a load balancer the client
is the one in its `X-Forwarded-For` ([the client address](/proxy/reverse-proxy/#the-client-address)),
so a public visitor through the balancer is refused too. A scraper that
reaches rws over a public address needs it listed; `"*"` serves everyone.

```toml
metrics_allow = ["10.0.0.0/8", "203.0.113.7"]   # RWS_CONFIG_METRICS_ALLOW
```

```
# HELP rws_requests_total Total HTTP requests handled
# TYPE rws_requests_total counter
rws_requests_total 12483

# HELP rws_errors_total HTTP requests that returned an application error
# TYPE rws_errors_total counter
rws_errors_total 3

# HELP rws_active_connections Currently open connections
# TYPE rws_active_connections gauge
rws_active_connections 7
```

When `MetricsLayer` is active, per-route metrics are appended to the same response body.

A scraper whose `Accept` header asks for `application/vnd.google.protobuf; proto=io.prometheus.client.MetricFamily` (Prometheus does when `scrape_native_histograms` is on) gets the same metrics in the Prometheus protobuf format instead — the only format that carries [native histograms](#native-histograms).

## HTTP server duration (OpenTelemetry)

Every request on every protocol — HTTP/1.1, HTTP/2 (TLS and h2c), HTTP/3 — is recorded by the server core in `http_server_request_duration_seconds`, the OpenTelemetry [`http.server.request.duration`](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/#metric-httpserverrequestduration) metric under its Prometheus name. No layer or setting is needed.

| Label | Values |
|---|---|
| `http_request_method` | `GET`, `POST`, … (the nine standard methods); anything else `_OTHER` |
| `http_response_status_code` | the status sent |
| `network_protocol_version` | `1.0`, `1.1`, `2`, `3` |
| `url_scheme` | `http` or `https` |

There is no path label, so the number of series stays bounded however many URLs are requested ([`MetricsLayer`](#per-route-metrics-with-metricslayer) has per-path series). Buckets are the convention's advised boundaries: 5 ms, 10 ms, 25 ms, 50 ms, 75 ms, 100 ms, 250 ms, 500 ms, 750 ms, 1 s, 2.5 s, 5 s, 7.5 s, 10 s. The duration runs from when the request is handed to the application until its response is ready to write — the transfer of a streamed body (a large file, a proxied stream, SSE) isn't included. A request the application fails is counted with the status rws sends for it (`400` on HTTP/1.1, `500` on HTTP/2 and HTTP/3); a request rejected before it reaches the application (malformed, too large, timed out) isn't.

```promql
# p99 per protocol
histogram_quantile(0.99, sum by (le, network_protocol_version) (rate(http_server_request_duration_seconds_bucket[5m])))

# 5xx ratio
sum(rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
  / sum(rate(http_server_request_duration_seconds_count[5m]))
```

### Native histograms

`http_server_request_duration_seconds` also has exponential buckets (schema 3: eight per power of two, each about 9% wider than the last), served as a [native histogram](https://prometheus.io/docs/specs/native_histograms/) when Prometheus scrapes in the protobuf format. Prometheus 3.9 and later need `scrape_native_histograms` set on the job:

```yaml
scrape_configs:
  - job_name: rws
    scrape_native_histograms: true
    static_configs:
      - targets: ["localhost:7878"]
```

Prometheus then stores the native form only (unless `always_scrape_classic_histograms` is set), and queries drop `_bucket` and `le`:

```promql
histogram_quantile(0.99, sum(rate(http_server_request_duration_seconds[5m])))
histogram_count(rate(http_server_request_duration_seconds{http_response_status_code="200"}[5m]))
```

Without `scrape_native_histograms` nothing changes: Prometheus asks for the text format and gets the classic buckets.

## Per-route metrics with `MetricsLayer`

Wrap your application once at startup to enable per-route request counts and latency histograms:

```rust
use rust_web_server::app::App;
use rust_web_server::core::New;
use rust_web_server::metrics::MetricsLayer;

let app = App::new().wrap(MetricsLayer);
```

`MetricsLayer` records two metric families per `(method, path)` pair after each request completes.

### `rws_route_requests_total`

A counter labelled by method, path, and HTTP status code.

```
# HELP rws_route_requests_total Total requests handled per route
# TYPE rws_route_requests_total counter
rws_route_requests_total{method="GET",path="/api/users",status="200"} 9401
rws_route_requests_total{method="GET",path="/api/users",status="404"} 12
rws_route_requests_total{method="POST",path="/api/users",status="201"} 530
```

Query strings are stripped from the path label automatically (`/users?page=2` → `/users`).

### `rws_route_duration_seconds`

A cumulative histogram labelled by method and path, with 11 predefined buckets matching the default Prometheus/Go client boundaries:

`0.005`, `0.01`, `0.025`, `0.05`, `0.1`, `0.25`, `0.5`, `1.0`, `2.5`, `5.0`, `10.0` (seconds)

```
# HELP rws_route_duration_seconds Request duration in seconds per route
# TYPE rws_route_duration_seconds histogram
rws_route_duration_seconds_bucket{method="GET",path="/api/users",le="0.005"} 8200
rws_route_duration_seconds_bucket{method="GET",path="/api/users",le="0.01"} 9100
...
rws_route_duration_seconds_bucket{method="GET",path="/api/users",le="+Inf"} 9413
rws_route_duration_seconds_sum{method="GET",path="/api/users"} 47.231000000
rws_route_duration_seconds_count{method="GET",path="/api/users"} 9413
```

### Behaviour notes

- A handler that returns `Err` is recorded with status `500`.
- The per-route section is absent from `/metrics` until at least one route has been observed.
- `metrics::record_route(method, path, status, elapsed_secs)` records a request directly, for custom instrumentation without `MetricsLayer`:

```rust
use rust_web_server::metrics;

metrics::record_route("GET", "/jobs/run", 200, 0.042);
```

## TLS

With a certificate configured, the TLS listener reports what each handshake negotiated and when the certificates it serves expire:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `rws_tls_handshakes_total` | Counter | `tls_protocol_version`, `tls_cipher`, `tls_group` | Completed handshakes (HTTP/1.1 and HTTP/2 over TLS). `tls_group` is the key exchange: `X25519MLKEM768` (post-quantum hybrid, preferred when the client offers it), `X25519`, `secp256r1`, … |
| `rws_tls_handshake_failures_total` | Counter | — | Handshakes that failed (not counting clients that never finished) |
| `rws_tls_certificate_expiry_timestamp_seconds` | Gauge | `name` | The leaf certificate's notAfter, Unix seconds: `default`, each `[[virtual_host]]` domain, each [on-demand](/features/acme/) name; updated when certificates are reloaded (SIGHUP, ACME renewal) |

HTTP/3 handshakes aren't counted: the QUIC library doesn't expose what its TLS session negotiated.

```promql
# Certificates expiring within 14 days
rws_tls_certificate_expiry_timestamp_seconds - time() < 14 * 86400

# Share of handshakes with a post-quantum key exchange
sum(rate(rws_tls_handshakes_total{tls_group=~".*MLKEM.*"}[1h])) / sum(rate(rws_tls_handshakes_total[1h]))
```

## Circuit breaker state

`rws_circuit_breaker_state{backend}` is included automatically — no `MetricsLayer` or other opt-in needed — for every backend known to [`circuit_breaker::global()`](/proxy/circuit-breaker/):

```
# HELP rws_circuit_breaker_state Circuit breaker state per backend (0=closed, 1=half_open, 2=open)
# TYPE rws_circuit_breaker_state gauge
rws_circuit_breaker_state{backend="api-1:8080"} 0
rws_circuit_breaker_state{backend="api-2:8080"} 2
```

The gauge value is `0` (Closed/healthy), `1` (HalfOpen/probing), or `2` (Open/unhealthy). Wiring `ReverseProxy::with_circuit_breaker(Arc::new(circuit_breaker::global()))` gets you this metric "for free" alongside the automatic proxy integration, since both read from the same breaker instance. `RedisCircuitBreaker` state doesn't appear here — the minimal hand-rolled RESP client has no `SCAN`/`KEYS` support to enumerate its keys.

## Upstream metrics

Every `[[upstream]]` of a [config-file proxy](/proxy/config-driven/) is measured per backend, with no opt-in:

| Metric | Type | Labels | Description |
|---|---|---|---|
| `rws_upstream_requests_total` | Counter | `upstream`, `backend`, `code` | Requests sent to the backend by response status; `code="error"` when none came back |
| `rws_upstream_request_duration_seconds` | Histogram | `upstream`, `backend` | Time to the backend's response head |
| `rws_upstream_retries_total` | Counter | `upstream` | Requests sent again to another backend (`[upstream.retry]`) |
| `rws_upstream_backend_up` | Gauge | `upstream`, `backend` | `1` while in rotation (passing health checks, breaker not open), else `0` |
| `rws_upstream_circuit_breaker_state` | Gauge | `upstream`, `backend` | `0`=closed, `1`=half_open, `2`=open; only for upstreams with `[upstream.circuit_breaker]` |
| `rws_upstream_backend_open_requests` | Gauge | `upstream`, `backend` | Requests running on the backend right now; a streamed response or a WebSocket counts until it ends (since 18.38.0) |
| `rws_upstream_backend_drained` | Gauge | `upstream`, `backend` | `1` while the backend is [drained](/deployment/admin/), else `0` (since 18.38.0) |

```promql
# Backend error ratio
sum by (upstream, backend) (rate(rws_upstream_requests_total{code=~"5..|error"}[5m]))
  / sum by (upstream, backend) (rate(rws_upstream_requests_total[5m]))

# Backends out of rotation
rws_upstream_backend_up == 0

# Drained backends that are safe to stop: nothing running on them on any replica
sum by (upstream, backend) (rws_upstream_backend_open_requests) == 0
  and on (upstream, backend) max by (upstream, backend) (rws_upstream_backend_drained) == 1
```

## `SERVER_READY` and `/readyz`

The built-in `GET /readyz` controller returns `200 OK` when `SERVER_READY` is `true` and `503 Service Unavailable` when it is `false`. The flag is set to `true` after `Server::setup()` completes and back to `false` when a shutdown signal is received, so Kubernetes stops routing traffic before the pod exits.

```rust
use rust_web_server::metrics::SERVER_READY;
use std::sync::atomic::Ordering;

// Check readiness programmatically.
let ready = SERVER_READY.load(Ordering::Relaxed);
```

## Scraping with Prometheus

```yaml
# prometheus.yml
scrape_configs:
  - job_name: rws
    static_configs:
      - targets: ["localhost:7878"]
    metrics_path: /metrics
```
