[
https://issues.apache.org/jira/browse/HDDS-16360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-16360:
-------------------------------------
Assignee: Andrey Yarovoy
> Add a health check endpoint to S3 Gateway for load balancer probes
> ------------------------------------------------------------------
>
> Key: HDDS-16360
> URL: https://issues.apache.org/jira/browse/HDDS-16360
> Project: Apache Ozone
> Issue Type: Improvement
> Components: s3gateway
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h3. Problem
> S3 Gateway is horizontally scaled and fronted by a load balancer (the
> distribution ships {{s3-haproxy.cfg}} for exactly this). Today S3G exposes no
> purpose-built health endpoint, so operators fall back to probing an
> incidental URL:
> * The S3 listener on {{9878}} — but {{GET /}} is {{ListBuckets}} and any
> other path is interpreted as a bucket/object, so there's no path that
> reliably means "am I healthy" without colliding with a bucket name, and in
> secure mode it requires AWS SigV4.
> * An admin servlet on {{19878}} ({{{}/conf{}}}, {{{}/jmx{}}}, {{{}/prom{}}})
> — but these report only that the Jetty process is up; a 200 there does *not*
> confirm the gateway can actually serve S3 (e.g. its {{{}OzoneClient{}}}/OM
> connection may be broken). The bundled {{s3-haproxy.cfg}} in fact does no
> HTTP health check at all — it balances raw TCP.
> The result: a load balancer can keep routing traffic to an S3G instance that
> is listening but cannot reach OM, or that is still starting up.
>
> h3. Goal
> Provide a stable, documented, unauthenticated HTTP endpoint that a load
> balancer can poll to decide whether an S3G instance should receive traffic,
> reflecting both process liveness and the gateway's ability to serve S3
> requests.
> h3. Proposed approach
> Add a lightweight health servlet on {{S3GatewayWebAdminServer}} (the plain
> Jetty admin server, port {{{}19878{}}}) at a fixed path (e.g.
> {{{}/health{}}}), returning {{200}} when healthy and a non-2xx (e.g.
> {{{}503{}}}) when not, with a tiny plaintext/JSON body for humans.
> Serving it on the admin server rather than the S3 listener avoids the S3
> path-router bucket-name collision entirely and keeps it off the
> SigV4-authenticated path. Load balancers can still gate the S3 traffic port
> on this endpoint: HAProxy in {{mode tcp}} performs Layer 7 HTTP health checks
> independently of the traffic mode and can target a different port than the
> one it balances ({{{}check port 19878{}}}), so it balances raw S3 TCP on
> {{9878}} while HTTP-probing {{/health}} on {{{}19878{}}}.
> The endpoint reports *Readiness* — the gateway can serve S3: its cached
> {{OzoneClient}} is initialized and a cheap OM reachability check succeeds.
> This is what the LB actually needs to gate traffic on.
> The endpoint must be reachable by an LB without Kerberos/SPNEGO. Requires
> explicitly excluding the health path from the S3G admin-server auth filters.
> New {{@Config}} keys (e.g. enable/disable, probe timeout), defaulting to
> enabled.
> h3. Scope
> * New health servlet + registration on {{{}S3GatewayWebAdminServer{}}}.
> * Auth-filter exclusion for the health path in secure mode.
> * {{@Config}} keys with getters (self-documenting; no {{ozone-default.xml}}
> edits).
> * Update {{s3-haproxy.cfg}} (and {{{}common/s3-haproxy.cfg{}}}) to add an
> {{option httpchk}} against {{/health}} on the admin port, as a worked example.
> h3. Out of scope
> * Per-bucket / OM-side health, cluster-wide readiness aggregation.
> * A health path on the S3 listener ({{{}9878{}}}) itself — unnecessary given
> HAProxy can check the admin port.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
