Kubernetes Gateway API
rws as a Kubernetes Gateway API controller — every route kind, the standard-channel extended features, shared and provisioned modes, and how to install it.
rws passes the Gateway API v1.7.0 conformance suite (standard channel) for the GATEWAY-HTTP, GATEWAY-GRPC, GATEWAY-TLS, GATEWAY-TCP and GATEWAY-UDP profiles with every standard-channel extended feature, in both deployment modes; CI runs the suite on every change (.github/workflows/gateway-api.yml). With the experimental-channel CRDs and RWS_CONFIG_GATEWAY_API_EXPERIMENTAL=true (Helm gatewayAPI.experimental) it also handles and claims the experimental features: destination-port matching, listener filters, ClusterTrustBundle CA references and address routability (see Experimental features). Design and history: spec/GATEWAY_API.md.
The Linux release binaries and the container image (ghcr.io/bohdaq/rws, bohdaq/rws) are built with the gateway-api feature; macOS and Windows binaries aren't (build those with --features gateway-api). With it and RWS_CONFIG_GATEWAY_API_ENABLED=true, rws watches the cluster's Gateway API objects, serves the Gateways whose GatewayClass names its controller, and reports what it accepted in their status. Every replica serves traffic; one, holding the rws-gateway-controller Lease, writes status and manages Services.
What it serves
-
GatewayClass, Gateway, HTTPRoute, ReferenceGrant status: acceptance, listener conditions (protocol, route kinds, certificates, conflicts), attached-route counts, route parent conditions (listener allowance by namespace, hostname intersection,
sectionName/port), backend resolution (Service kind, ReferenceGrants, existence),observedGeneration. -
A Service per Gateway (
rws-<gateway>,LoadBalancerby default), in the Gateway's namespace: one port per listener port, each mapped to a container port fromRWS_CONFIG_GATEWAY_API_PORT_RANGE. A Service only selects pods in its own namespace, so it has no selector: rws writes its EndpointSlice (rws-<gateway>-ipv4, and-ipv6when pods have IPv6 addresses) with its own serving pods — those in its namespace matchingRWS_CONFIG_GATEWAY_API_POD_SELECTOR— and their readiness. The Service's load-balancer address becomes the Gateway'sstatus.addresses;spec.infrastructurelabels and annotations are put on it. -
HTTP and HTTPS listeners, the latter with the certificate from the listener's TLS Secret (by SNI between listeners).
-
HTTPRoute routing: hostnames,
Exact/PathPrefixpaths, exact header and query-parameter matches, methods, the spec's precedence between routes and rules (between routes on one listener, the route's own hostname decides — an exact name before a wildcard — not its intersection with the listener's), weighted backends over the Services' ready endpoints. The client'sHostis kept. -
HTTPRoute filters, applied in the order listed:
RequestHeaderModifierandResponseHeaderModifier(setreplaces,addappends,removedrops — names case-insensitive),RequestHeaderModifierandURLRewriteon abackendRef(only for requests sent to that backend; aReplacePrefixMatchthere replaces the rule's matched prefix),RequestRedirect(scheme, hostname, port,ReplaceFullPath/ReplacePrefixMatchpath, status 301/302/303/307/308; the port defaults to the scheme's well-known port when the scheme is set, else the listener's, and a well-known port isn't written intoLocation),URLRewrite(hostname, path) andCORS(preflights answered with204by rws, actual requests getAccess-Control-Allow-Originand the exposed headers;*in an origin matches one or more labels). AnExtensionRefor unknown filter answers500, as the spec requires. -
Timeouts and retries:
timeouts.requestbounds the whole exchange, retries included, andtimeouts.backendRequesteach attempt; past either the client gets504, and0sturns a timeout off.retry.attemptsis the number of retries (the first try isn't counted) on the listedretry.codes, on connection errors and on backend timeouts,retry.backoffapart (25 ms when unset). -
Backend protocols: a Service port with
appProtocol: kubernetes.io/h2cis reached over cleartext HTTP/2 (prior knowledge); WebSocket and otherUpgraderequests are forwarded on a connection of their own, and a backend's101turns the client's connection into a byte relay with it.
The GatewayClass's status.supportedFeatures lists what rws claims; the conformance job checks the same list.
-
Mirroring: each
RequestMirrorfilter sends a copy of the request, as the rule's other request filters left it, to one ready endpoint of its Service, forpercentorfractionof requests (all when neither is set); the copy's response is dropped and never delays the client's. -
GRPCRoute: attached like HTTPRoutes (HTTP and HTTPS listeners carry both), matched on
method(Exactservice and/or method;RegularExpressionisn't supported) and headers, with the header-modifier and mirror filters, weighted backends reached over h2c with trailers passed through. A call no rule matches gets HTTP404, which gRPC clients report asUNIMPLEMENTED. -
TLSRoute on
TLSlisteners: inPassthroughmode rws reads the ClientHello's SNI, picks the route whose hostname matches best and relays the still-encrypted stream (the backend presents its own certificate); inTerminatemode (the default) it decrypts with the listener's certificate and relays the plaintext over TCP. Listeners of both modes can share a port when their hostnames differ: the ClientHello's SNI picks the listener, then its mode. -
TCPRoute on
TCPlisteners and UDPRoute onUDPlisteners: bytes, or datagrams, relayed to a backend of the route's first rule by weight. When several routes attach to one listener, the oldest serves it (TLSRoutes: the best hostname match first). Each UDP client gets its own socket to the backend, closed after 60 s without traffic. -
ListenerSet: a Gateway whose
spec.allowedListenersadmits a ListenerSet's namespace (Same,All,Selector; the default admits none) serves its listeners after its own, older sets first, on the Gateway's Service. A set's listener that conflicts with an earlier one (same port and hostname, or another protocol on the port) isn't accepted; the set is accepted while any of its listeners is, and the Gateway counts the attached sets instatus.attachedListenerSets. Routes reach a set's listeners only with aListenerSetparentRef. -
Status written alongside other controllers: Gateway and listener conditions with a domain-qualified type (
example.com/Ready) that another controller set are kept as they are; status is written with the object'sresourceVersion, so a write from a stale cache is refused and redone. -
Misdirected requests: over HTTPS, a request whose
Hostbelongs to a different listener than the one its SNI chose (and whose certificate it was given) is answered421 Misdirected Request. -
Client certificates (
spec.tls.frontend): the validation for a listener's port (perPort, elsedefault) names ConfigMaps whoseca.crtholds the CA certificates (another namespace needs a ReferenceGrant). InAllowValidOnlymode (the default) the handshake requires a client certificate those CAs signed;AllowInsecureFallbackasks for one but lets the connection through either way, and the Gateway reportsInsecureFrontendValidationMode. A reference that doesn't resolve leaves the listener unaccepted (NoValidCACertificate) — rws never serves a port without the validation it was asked for. The controller also watches ConfigMaps for this. -
BackendTLSPolicy: a Service (or one of its ports, by
sectionName) a policy targets is reached over TLS, trusting the CA certificates in the referenced ConfigMaps'ca.crt(or the platform's roots withwellKnownCACertificates: System), withhostnameas SNI and as the name checked — or, whensubjectAltNamesare listed, any of those (HostnameorURI) instead. Between two policies for the same target the older wins and the other isConflicted; a port's own policy beats one for the whole Service. A policy that can't be used (a missing or wrong-kind CA reference) turns its backends' requests into500s rather than plaintext. GRPCRoute backends (andkubernetes.io/h2cones) under a policy are reached over HTTP/2 on TLS; a TLSRoute a listener terminates is encrypted again to a backend a policy covers. Status is reported per Gateway whose routes reach a targeted Service. A Gateway'sspec.tls.backend.clientCertificateRef(a TLS Secret, or one in another namespace a ReferenceGrant opens) is the client certificate its connections to these backends present; a reference that doesn't resolve is reported in the Gateway'sResolvedRefs. -
Static addresses (
spec.addresses): the firstIPAddressis asked of the load balancer as the Service'sloadBalancerIP; the Gateway is programmed once every requested address is the Service's (elseAddressNotUsable). Another address type isn't accepted (UnsupportedAddress). An address with only a type asks for any address.
Deployment modes
- shared (
RWS_CONFIG_GATEWAY_API_MODE=shared, the default): the pods of the rws Deployment that runs the controller serve every Gateway. One Deployment, scaled as one. - provisioned (
provisioned; HelmgatewayAPI.mode: provisioned): each Gateway gets its own Deployment,rws-gw-<namespace>-<name>, in the controller's namespace, with the controller's image (RWS_CONFIG_GATEWAY_API_IMAGE) and ServiceAccount; the Gateway'sspec.infrastructurelabels and annotations go on its pods. Those pods (started withRWS_CONFIG_GATEWAY_API_GATEWAY=<namespace>/<name>) watch the cluster like the controller but serve only their Gateway and never write; the controller's own pods serve nothing. Gateways don't share pods, so one's load or failure doesn't reach another. A Gateway's Deployment is deleted with it (by the controller: an object can't own one in another namespace).
Either way the Gateway's Service lists the serving pods in its EndpointSlice.
Settings
| Setting | Default | Meaning |
|---|---|---|
RWS_CONFIG_GATEWAY_API_ENABLED |
false |
Run the controller |
RWS_CONFIG_GATEWAY_API_CONTROLLER_NAME |
rws8.tech/gateway-controller |
The GatewayClasses' controllerName to handle |
RWS_CONFIG_GATEWAY_API_POD_SELECTOR |
app.kubernetes.io/name=rws |
Labels of the rws pods, selected by every Gateway's Service |
RWS_CONFIG_GATEWAY_API_PORT_RANGE |
10000-10999 |
Container ports for listeners |
RWS_CONFIG_GATEWAY_API_SERVICE_TYPE |
LoadBalancer |
Type of each Gateway's Service |
RWS_CONFIG_GATEWAY_API_LEASE_NAMESPACE |
the pod's namespace | Where the leader Lease lives |
Also as [gateway_api] in rws.config.toml (Config file).
Permissions
The controller needs to list and watch GatewayClasses, Gateways, HTTPRoutes, GRPCRoutes, TLSRoutes, TCPRoutes, UDPRoutes, ReferenceGrants, BackendTLSPolicies, ListenerSets, Services, EndpointSlices, Namespaces, Secrets and ConfigMaps; patch the status of GatewayClasses, Gateways and every route kind; create and patch Services and EndpointSlices; read its own namespace's Pods; and get, create and update Leases in its namespace. The Gateway API CRDs (standard channel, v1.6 or v1.7) must be installed in the cluster first.
Benchmarks
rws 18.8.1 against five other Gateway API implementations, each at its chart defaults with one data-plane replica, in one kind cluster on one GitHub Actions runner (run 40681376798). The same HTTPRoute sends every request to the same nginx backend; oha loads it for 20 s per row with keep-alive and no compression, from the runner. CPU is what the gateway's pods (controller and data plane) used per request, from the kubelet. Runner hardware varies between runs, so compare rows of this run only. The harness is bench/gateway/run.sh (workflow gateway-bench.yml, manual).
Versions: Envoy Gateway v1.9.2, NGINX Gateway Fabric 2.7.2, Traefik (chart 41.7.1), kgateway v2.4.6, Istio 1.30.5.
Requests per second, and p99 latency in ms (64 connections unless noted):
| Scenario | rws | kgateway | Istio | Traefik | Envoy Gateway | NGINX GF |
|---|---|---|---|---|---|---|
| HTTP/1.1, 1 KiB | 38 501 · 5.8 | 20 225 · 9.5 | 16 274 · 9.3 | 17 539 · 14 | 16 685 · 15 | 10 951 · 20 |
| HTTP/1.1, 1 KiB, 256 conns | 39 230 · 23 | 21 103 · 36 | 17 795 · 23 | 18 484 · 45 | 17 757 · 67 | 10 814 · 86 |
| HTTP/1.1, 1 MiB | 2 467 · 71 | 2 009 · 93 | 2 057 · 186 | 1 879 · 81 | 1 556 · 191 | 841 · 200 |
| HTTPS, 1 KiB | 34 792 · 5.3 | 19 085 · 10 | 15 946 · 6.7 | 17 941 · 13 | 16 372 · 16 | 10 121 · 22 |
| HTTP/2 + TLS, 1 KiB | 32 120 · 5.5 | 17 226 · 11 | 14 063 · 7.6 | 13 158 · 14 | 14 817 · 15 | 9 419 · 25 |
| Header modifiers + rewrite, 1 KiB | 36 821 · 5.0 | 19 573 · 8.9 | 16 272 · 6.7 | 18 004 · 14 | 16 907 · 16 | —¹ |
CPU µs per request:
| Scenario | rws | kgateway | Istio | Traefik | Envoy Gateway | NGINX GF |
|---|---|---|---|---|---|---|
| HTTP/1.1, 1 KiB | 40 | 106 | 116 | 96 | 80 | 117 |
| HTTP/1.1, 1 KiB, 256 conns | 19 | 82 | 66 | 175 | 171 | 191 |
| HTTP/1.1, 1 MiB | 928 | 1182 | 1254 | 597 | 1098 | 1365 |
| HTTPS, 1 KiB | 33 | 73 | 64 | 165 | 121 | 171 |
| HTTP/2 + TLS, 1 KiB | 63 | 152 | 209 | 178 | 188 | 292 |
| Header modifiers + rewrite, 1 KiB | 27 | 129 | 77 | 96 | 116 | —¹ |
Every gateway answered every request with 200 except as noted. Memory (working set of the gateway's pods after the runs): rws 42–118 MiB (mimalloc keeps freed memory, so it stays at its peak after the 1 MiB run), NGINX Gateway Fabric 60–74, the others 70–200.
¹ NGINX Gateway Fabric answered 404 through the rewriting route in this setup, so the row isn't comparable; it also had 160 cancelled requests over HTTP/2.
What this doesn't measure: many routes or Gateways, configuration churn, connection setup (every scenario reuses connections), and real backends.
Experimental features
These need the experimental-channel CRDs (experimental-install.yaml) and RWS_CONFIG_GATEWAY_API_EXPERIMENTAL=true (Helm gatewayAPI.experimental=true); with the standard CRDs the API server would drop the fields rws reports, so they stay off. The GatewayClass then lists them in status.supportedFeatures.
-
Listener filters (
listeners[].filters.requests, GEP-5224): run in order on every request an HTTP or HTTPS listener receives, before a route is picked, so a route can match on what they add. rws runsExternalAuth:protocol: HTTP: the authorization server gets the request's method and path (afterhttp.path),Host,Authorization,Content-Length, thehttp.allowedHeadersand the body up toforwardBody.maxSize. A200lets the request through with the server's response headers (http.allowedResponseHeaders, else all of them butHostand framing headers); any other answer is sent to the client as it is.protocol: GRPC: Envoy'sext_authzCheckcall over HTTP/2, with the request's method, path, host, scheme, headers (grpc.allowedHeaders, else all) and body.OKlets it through with the response's header changes; anything else answers with the denied response's status (403by default), headers and body.- A filter can't change
Hostor:authority: the listener was chosen by them. - An authorization server that can't be reached, or whose Service doesn't resolve, answers
403: the filter fails closed. A BackendTLSPolicy on its Service is honoured. ExtensionRef, or a filter on a listener that isn't HTTP or HTTPS, leaves the listener unaccepted (UnsupportedValue).
-
Address routability (
spec.addresses[].routability, GEP-5093). Each entry is matched on its own, by a distinct address:""(or unset): a load-balancer address of the Gateway's Service;Cluster: the Service's ClusterIP (a value asks the Service for thatclusterIP, which can only be set when the Service is created);testing.x-k8s.io/sentinel: reported as asked;- any other scope can't be satisfied; rws never substitutes another.
Every status address carries its
routability. TheAddressesAssignedcondition saysAddressesAssigned,PartiallyAssignedorNotAssigned(and lists what's missing); with nothing assigned the Gateway isn't programmed (AddressNotAssigned). -
ClusterTrustBundle CA references (GEP-5075):
caCertificateRefsentries{group: certificates.k8s.io, kind: ClusterTrustBundle, name}in BackendTLSPolicies and in client-certificate validation trust the bundle'sspec.trustBundle.- rws watches ClusterTrustBundles, and advertises the feature, only where the cluster serves the API (
v1, elsev1beta1; Kubernetes before 1.37 needs theClusterTrustBundlefeature gate). - A missing, empty or unparsable bundle gives
ResolvedRefs=False InvalidCACertificateRef; without the API it'sInvalidKind. - A BackendTLSPolicy left without a CA fails its requests. A client-certificate validation keeps its other sources and reports the bad bundle.
- rws watches ClusterTrustBundles, and advertises the feature, only where the cluster serves the API (
-
Destination port matching (
HTTPRouteDestinationPortMatching): a parentRefportthat no listener of the Gateway has leaves the route unattached (NoMatchingParent).