This is an automated email from the ASF dual-hosted git repository.
jerryshao pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gravitino.git
The following commit(s) were added to refs/heads/main by this push:
new c619ba08f6 [#13066]improvement(server): Report unhealthy status after
observed out-of-memory errors (#13067)
c619ba08f6 is described below
commit c619ba08f63c7e7eb527ae067318deefea1175b2
Author: Qi Yu <[email protected]>
AuthorDate: Thu Sep 10 22:31:34 2026 +0800
[#13066]improvement(server): Report unhealthy status after observed
out-of-memory errors (#13067)
### What changes were proposed in this pull request?
Record observed `OutOfMemoryError`s and make the Gravitino, Iceberg
REST, and Lance REST health endpoints and root aliases return HTTP 503
with a `jvm` failure until process restart. Healthy checks and response
formats remain unchanged.
Detect direct and wrapped OOM through a servlet filter installed before
other filters, authentication error handling, Jersey listeners and
mappers, request helpers including the built-in IdP, and health-probe
tasks, including failures after timeout. The Jetty worker
uncaught-exception handler remains an additional fallback. The marker
retains no throwable and allocates no visited-cause collection.
### Why are the changes needed?
After heap or Metaspace OOM, already-loaded endpoints and entity-store
lookups can still succeed while catalog operations fail. Health probes
must stop advertising the degraded process as healthy.
Fix: #13066
### Does this PR introduce _any_ user-facing change?
Yes. Once OOM is recorded, health endpoints return 503 with a `jvm`
check and `OutOfMemoryError; restart required`. Successful requests do
not clear the state, and subsequent health checks skip the entity store.
No new configuration. JSON status values remain lowercase for
Gravitino/Lance and uppercase for Iceberg REST.
Embedded services in the same JVM share the health marker through the
default auxiliary classloaders; separate JVMs record OOM independently.
Each service port still needs its own probes for HTTP availability and
initialization. Detection follows the throwable and its cause chain, not
suppressed exceptions or errors swallowed entirely by
connectors/background executors. Severe memory exhaustion may prevent
any HTTP response.
### How was this patch tested?
- Unit tests cover direct/wrapped OOM, cyclic causes, ordinary errors,
request helpers including IdP error conversion, Jersey error handling,
and health-probe failures before/after timeout.
- HTTP regression tests exercise the production Jetty filter
installation with failures in downstream filters and non-Jersey
servlets, as well as Jersey resource failures. All seven main-server
health paths return 503 after OOM while a warm endpoint still succeeds.
- Authentication HTTP tests cover direct/wrapped OOM and both exception
handlers, checking that OOM is recorded before an overridden
error-response method runs. Ordinary failures retain their responses
without marking the server unhealthy.
- Iceberg and Lance tests cover their real error mappers, health checks,
and actual JSON status serialization.
- Errors are simulated; the test JVM is not exhausted.
- `spotlessApply` passed. All 1,459 tests passed across server-common,
server, Iceberg REST, Lance REST, and idp-basic; Docker and integration
tests were excluded.
```bash
SKIP_DOCKER_TESTS=true ./gradlew :server-common:test :server:test \
:iceberg:iceberg-rest-server:test :lance:lance-rest-server:test \
:plugins:idp-basic:test -PskipITs -PskipDockerTests=true -PskipWeb=true
```
---
docs/gravitino-server-config.md | 26 ++-
docs/health-and-readiness.md | 89 ++++++--
docs/iceberg-rest-service.md | 12 +-
docs/lance-rest-service.md | 11 +
.../org/apache/gravitino/iceberg/RESTService.java | 2 +
.../iceberg/service/IcebergExceptionMapper.java | 2 +
.../service/rest/IcebergHealthOperations.java | 35 ++-
.../service/TestIcebergExceptionMapper.java | 24 +-
.../service/rest/TestIcebergHealthOperations.java | 110 ++++++++-
.../apache/gravitino/lance/LanceRESTService.java | 2 +
.../lance/service/LanceExceptionMapper.java | 2 +
.../lance/service/rest/LanceHealthOperations.java | 35 ++-
.../service/rest/TestLanceHealthOperations.java | 110 ++++++++-
.../org/apache/gravitino/idp/web/IdpRESTUtils.java | 3 +
.../apache/gravitino/idp/web/TestIdpRESTUtils.java | 56 +++++
.../authentication/AuthenticationFilter.java | 5 +
.../apache/gravitino/server/web/JettyServer.java | 4 +
.../server/web/OutOfMemoryErrorFilter.java | 55 +++++
.../server/web/OutOfMemoryErrorListener.java | 62 ++++++
.../apache/gravitino/server/web/ServerHealth.java | 82 +++++++
.../org/apache/gravitino/server/web/Utils.java | 27 ++-
.../TestAuthenticationOutOfMemoryHttp.java | 118 ++++++++++
.../gravitino/server/web/TestJettyServer.java | 19 ++
.../server/web/TestOutOfMemoryErrorListener.java | 46 ++++
.../gravitino/server/web/TestServerHealth.java | 67 ++++++
.../gravitino/server/web/TestUtilsOutOfMemory.java | 90 ++++++++
.../apache/gravitino/server/GravitinoServer.java | 2 +
.../server/web/mapper/ErrorExceptionMapper.java | 18 ++
.../server/web/rest/HealthOperations.java | 43 +++-
.../server/TestGravitinoServerOutOfMemoryHttp.java | 210 ++++++++++++++++++
.../web/mapper/TestErrorExceptionMapper.java | 5 +-
.../server/web/rest/TestHealthOperations.java | 77 ++++++-
.../server/web/rest/TestOutOfMemoryHealthHttp.java | 247 +++++++++++++++++++++
33 files changed, 1639 insertions(+), 57 deletions(-)
diff --git a/docs/gravitino-server-config.md b/docs/gravitino-server-config.md
index 8b0725382f..7da4f85cfe 100644
--- a/docs/gravitino-server-config.md
+++ b/docs/gravitino-server-config.md
@@ -208,27 +208,28 @@ Gravitino exposes three health endpoints following
of them are exempt from authentication, so Kubernetes probes, load balancers,
and traffic managers
reach them without credentials.
-| Endpoint | Root Alias | Description
| HTTP Status |
-|-------------------------|---------------------|---------------------------------------------------------------------------------------------------------------------------------------------|-------------|
-| `GET /api/health/live` | `GET /health/live` | Liveness. Returns 200 as
long as an HTTP server thread can respond. Use it to decide whether to restart
a pod. | 200 |
-| `GET /api/health/ready` | `GET /health/ready` | Readiness. Returns 200 when
the entity store answers within the probe timeout, 503 when it is unavailable
or slow. Use it to route traffic. | 200 or 503 |
-| `GET /api/health` | `GET /health` | Aggregate. Returns 200 when
both of the above pass. Also aliased as `GET /health.html`.
| 200 or 503 |
+| Endpoint | Root Alias | Description
| HTTP Status |
+|-------------------------|---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|-------------|
+| `GET /api/health/live` | `GET /health/live` | Liveness. Returns 200 if an
HTTP thread can respond and no OOM has been observed; otherwise 503. Use it to
decide whether to restart a pod. | 200 or 503 |
+| `GET /api/health/ready` | `GET /health/ready` | Readiness. Returns 200 when
no OOM has been observed and the entity store answers within the probe timeout;
otherwise 503. Use it to route traffic. | 200 or 503 |
+| `GET /api/health` | `GET /health` | Aggregate. Returns 200 when
both of the above pass. Also aliased as `GET /health.html`.
| 200 or 503 |
| Configuration Item | Description
| Default Value |
|------------------------------------------------------|---------------------------------------------------------------------|---------------|
| `gravitino.server.health.entityStore.probeTimeoutMs` | Timeout in
milliseconds for the entity store probe behind `/ready`. | `2000` |
Every endpoint returns the same JSON shape, but not the same checks. `code` is
always `0`,
-`status` is `UP` or `DOWN`, and `checks` carries one entry per component
probed. `/live` reports
-`httpServer` alone, `/ready` reports `entityStore` alone, and the aggregate
endpoint reports both:
+`status` is `up` or `down`, and `checks` carries one entry per component
probed. `/live` reports
+`httpServer` alone, `/ready` reports `entityStore` alone, and the aggregate
endpoint reports both
+while no OOM has been observed:
```json
{
"code": 0,
- "status": "DOWN",
+ "status": "down",
"checks": [
- { "name": "httpServer", "status": "UP", "details": {} },
- { "name": "entityStore", "status": "DOWN", "details": { "reason":
"timeout" } }
+ { "name": "httpServer", "status": "up", "details": {} },
+ { "name": "entityStore", "status": "down", "details": { "reason":
"timeout" } }
]
}
```
@@ -236,6 +237,11 @@ Every endpoint returns the same JSON shape, but not the
same checks. `code` is a
A failing `entityStore` check reports `timeout`, `interrupted`,
`probe-rejected`,
`entity store not initialized`, or the simple class name of an unexpected
exception.
+After an observed `OutOfMemoryError` (including Metaspace OOM), all three
endpoints and their root
+aliases return 503 with a single `jvm: down` check and the reason
`OutOfMemoryError; restart required`.
+This state persists until process restart; successful requests do not reset
it. See
+[Out-of-memory failures](./health-and-readiness.md#out-of-memory-failures) for
detection scope.
+
#### JVM Memory
`GRAVITINO_MEM` sets the heap and metaspace flags. The launch scripts append
it to `JAVA_OPTS`, and
diff --git a/docs/health-and-readiness.md b/docs/health-and-readiness.md
index afe265c5a2..877d8c59df 100644
--- a/docs/health-and-readiness.md
+++ b/docs/health-and-readiness.md
@@ -10,15 +10,17 @@ license: "This software is licensed under the Apache
License version 2."
---
Gravitino exposes separate liveness and readiness endpoints so that a caller
can tell "restart this
-process" apart from "send traffic somewhere else." Liveness answers whether
the server can respond
-at all. Readiness answers whether it can reach the entity store and therefore
do useful work.
+process" apart from "send traffic somewhere else." Liveness checks whether the
server can respond
+and has not observed an out-of-memory error. Readiness also checks whether it
can reach the entity
+store.
The endpoints follow MicroProfile Health semantics. A healthy check returns
200 and an unhealthy one
returns 503, both with a JSON body naming the individual checks that ran.
## Quick Start
-**1. Check liveness.** This returns 200 whenever an HTTP thread is able to
answer.
+**1. Check liveness.** This returns 200 when an HTTP thread can answer and no
out-of-memory error
+has been observed.
```shell
GRAVITINO_URL=http://localhost:8090
@@ -26,7 +28,8 @@ GRAVITINO_URL=http://localhost:8090
curl -i "${GRAVITINO_URL}/api/health/live"
```
-**2. Check readiness.** This returns 200 only when the entity store responds.
+**2. Check readiness.** This returns 200 only when the entity store responds
and no out-of-memory
+error has been observed.
```shell
curl -i "${GRAVITINO_URL}/api/health/ready"
@@ -41,11 +44,11 @@ curl -i "${GRAVITINO_URL}/api/health"
## Endpoints
-| Path | Checks | Returns 503 when
|
-|---------------------|------------------------------|-----------------------------------|
-| `/api/health/live` | HTTP server | Never, if the request
is answered |
-| `/api/health/ready` | Entity store | The entity store check
fails |
-| `/api/health` | HTTP server and entity store | Either check fails
|
+| Path | Checks | Returns 503
when |
+|---------------------|-----------------------------------------|---------------------------------------------------------------------|
+| `/api/health/live` | HTTP server and OOM state | An
out-of-memory error was observed |
+| `/api/health/ready` | Entity store and OOM state | An
out-of-memory error was observed or the entity store check fails |
+| `/api/health` | HTTP server, entity store and OOM state | Any check
fails |
Each path is also served at the root of the server, without the `/api` prefix,
for load balancers
and traffic managers that require probes at well-known locations. The root
aliases are `/health`,
@@ -53,8 +56,63 @@ and traffic managers that require probes at well-known
locations. The root alias
endpoint rather than to a check of its own.
The response body carries an overall status and a list of individual checks.
Each check has a name,
-a status of UP or DOWN, and a details map that explains a failure. On the
Gravitino server the two
-check names are `httpServer` and `entityStore`.
+a status of `up` or `down`, and a details map that explains a failure. On the
Gravitino server the two
+normal check names are `httpServer` and `entityStore`. After an observed
out-of-memory error, all
+three endpoints instead report the `jvm` failure described below.
+
+## Out-of-memory Failures
+
+A Metaspace or heap `OutOfMemoryError` can leave already-loaded endpoints
responding successfully
+while other operations fail. A successful HTTP response or entity-store lookup
therefore does not
+prove recovery after OOM.
+
+The Gravitino, Iceberg REST, and Lance REST servers record OOM observed by
their Jersey exception
+listeners, error mappers, and a servlet filter installed before other filters
and servlets.
+Authentication error handling and request execution/error-response helpers
(including the built-in
+IdP helpers) also record errors they consume. The main server also records
failures in health-probe
+tasks. The Jetty worker uncaught-exception handler is an additional fallback,
not the request
+exception boundary.
+Wrapped causes are checked too. Once recorded, the affected service’s health
endpoints and root
+aliases return HTTP 503. Gravitino and Lance REST serialize status values as
`up`/`down`;
+Iceberg REST uses `UP`/`DOWN`. The following body shows the Gravitino and
Lance REST format
+(the main server uses the `/api/health` prefix); Iceberg REST uses `"DOWN"`
for both status fields:
+
+```json
+{
+ "code": 0,
+ "status": "down",
+ "checks": [
+ {
+ "name": "jvm",
+ "status": "down",
+ "details": { "reason": "OutOfMemoryError; restart required" }
+ }
+ ]
+}
+```
+
+This state lasts until process restart, even if subsequent ordinary API
requests succeed. Health
+checks skip the entity-store probe once OOM is recorded. A database outage,
ordinary HTTP 500,
+`StackOverflowError`, or missing connector class alone does not set this state.
+
+This policy also applies to an OOM caused by a single request, such as an
oversized list response
+or `Requested array size exceeds VM limit`. The server does not distinguish
recoverable allocation
+failures from persistent memory exhaustion: even if memory becomes available
again, the health
+state remains unhealthy until restart. If liveness probes trigger automatic
restarts, repeatedly
+retrying the same oversized request against different replicas can cause those
replicas to restart
+in succession. Account for this behavior when configuring request limits and
retry policies.
+
+Detection covers errors reaching these server boundaries; it cannot detect an
OOM swallowed
+entirely by a connector or unrelated background executor. This is not a
JVM-wide OOM trap. If the
+JVM cannot allocate enough memory to answer a probe, the probe may fail
without a JSON response.
+Only the throwable itself and its cause chain are inspected. An OOM present
only in suppressed
+exceptions (for example, from resource cleanup) is not detected, avoiding
defensive array copies
+while examining failures.
+
+When Iceberg REST and Lance REST run embedded in the main server, the default
auxiliary
+classloaders share the same `ServerHealth` marker. An OOM recorded by any of
these services makes
+all of their health endpoints report unhealthy. Services running in separate
JVM processes track
+OOM independently.
## What Readiness Actually Tests
@@ -69,9 +127,12 @@ and queues at most twenty probes before rejecting further
ones.
## Iceberg REST and Lance REST Endpoints
The Iceberg REST service and the Lance REST service each run their own HTTP
server on their own
-port, including when they run inside the Gravitino server process, so the
Gravitino server's
-endpoints do not report on them. A deployment that runs either service needs
probes against its
-port as well.
+port, including when they run inside the Gravitino server process. Embedded
services share the
+OOM marker, but HTTP availability and initialization checks remain specific to
each service.
+A deployment that runs either service therefore needs probes against its port
as well.
+
+Both services return 503 from all health endpoints and root aliases after
observing OOM, with the
+`jvm` failure described above, until restart. Before OOM, their existing
initialization checks apply.
| Server | Default Port | Health Path Prefix | Readiness Check
|
|----------------------|--------------|--------------------|-------------------------|
diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md
index f3c55cdafd..8eeffbcd8d 100644
--- a/docs/iceberg-rest-service.md
+++ b/docs/iceberg-rest-service.md
@@ -722,11 +722,11 @@ Gravitino provides the built-in
`org.apache.gravitino.iceberg.service.cache.Loca
The Iceberg REST server exposes three health check endpoints following the
same [MicroProfile
Health](https://microprofile.io/project/eclipse/microprofile-health) semantics
as the main Gravitino server. All endpoints are exempt from authentication. The
readiness probe checks whether the `IcebergCatalogWrapperManager` has been
initialized. It performs no I/O and has no configurable timeout.
-| Endpoint | Description
|
HTTP status |
-|-----------------------------|----------------------------------------------------------------------------------------------------------------------------|-------------|
-| `GET /iceberg/health/live` | Liveness probe. Returns 200 as long as the
HTTP server thread can respond.
| 200 |
-| `GET /iceberg/health/ready` | Readiness probe. Returns 200 when the catalog
wrapper manager is initialized; 503 when initialization is not yet complete. |
200 / 503 |
-| `GET /iceberg/health` | Aggregate check. Returns 200 when both
liveness and readiness pass; 503 when any check fails.
| 200 / 503 |
+| Endpoint | Description
|
HTTP status |
+|-----------------------------|---------------------------------------------------------------------------------------------------------------------------|-------------|
+| `GET /iceberg/health/live` | Liveness probe. Returns 200 when the HTTP
thread can respond and no OOM has been observed; 503 after an observed OOM.
| 200 / 503 |
+| `GET /iceberg/health/ready` | Readiness probe. Returns 200 when the catalog
wrapper manager is initialized and no OOM has been observed; 503 otherwise. |
200 / 503 |
+| `GET /iceberg/health` | Aggregate check. Returns 200 when both
liveness and readiness pass; 503 when any check fails.
| 200 / 503 |
Root-level aliases are also available for global traffic managers that require
probes at well-known root paths:
@@ -739,6 +739,8 @@ Root-level aliases are also available for global traffic
managers that require p
**Response format:**
+After an observed `OutOfMemoryError`, all health endpoints and root aliases
return 503 with a `jvm` failure until restart. See [out-of-memory
failures](health-and-readiness.md#out-of-memory-failures) for detection scope.
+
All endpoints return a JSON body with the same shape as the main Gravitino
server. The `code` field is always `0`. `status` is `UP` or `DOWN`. Liveness
reports `httpServer` and readiness reports `catalogWrapperManager`.
Healthy response (HTTP 200):
diff --git a/docs/lance-rest-service.md b/docs/lance-rest-service.md
index 0b464c5451..7186243bb2 100644
--- a/docs/lance-rest-service.md
+++ b/docs/lance-rest-service.md
@@ -218,6 +218,17 @@ Access the service at `http://localhost:9101`.
- **Optional:** Other variables can use default values unless you have
specific requirements
:::
+## Health Checks
+
+The Lance REST service exposes `/lance/health/live`, `/lance/health/ready`, and
+`/lance/health` on its own port. Root aliases `/health/live`, `/health/ready`,
`/health`,
+and `/health.html` are also available. Readiness checks whether the namespace
wrapper
+is initialized.
+
+After an observed `OutOfMemoryError`, all these endpoints return HTTP 503 with
a
+`jvm` failure until restart, even if the wrapper remains initialized. See
+[health and readiness](health-and-readiness.md) for response details and
detection scope.
+
## Usage Guidelines
When using Lance REST service with Gravitino backend, keep the following
considerations in mind:
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
index 78a58f71b7..3710708c57 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/RESTService.java
@@ -61,6 +61,7 @@ import org.apache.gravitino.server.web.HttpAuditFilter;
import org.apache.gravitino.server.web.HttpServerMetricsSource;
import org.apache.gravitino.server.web.JettyServer;
import org.apache.gravitino.server.web.JettyServerConfig;
+import org.apache.gravitino.server.web.OutOfMemoryErrorListener;
import org.apache.gravitino.server.web.RequestContextFilter;
import
org.apache.gravitino.server.web.filter.IcebergRESTAuthInterceptionService;
import org.glassfish.hk2.api.InterceptionService;
@@ -105,6 +106,7 @@ public class RESTService implements
GravitinoAuxiliaryService {
config.register(IcebergObjectMapperProvider.class).register(JacksonFeature.class);
config.register(IcebergExceptionMapper.class);
+ config.register(new OutOfMemoryErrorListener());
HttpServerMetricsSource httpServerMetricsSource =
new
HttpServerMetricsSource(MetricsSource.ICEBERG_REST_SERVER_METRIC_NAME, config,
server);
metricsSystem.register(httpServerMetricsSource);
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergExceptionMapper.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergExceptionMapper.java
index c69a9d6dfb..940bf785e7 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergExceptionMapper.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/IcebergExceptionMapper.java
@@ -29,6 +29,7 @@ import
org.apache.gravitino.exceptions.IllegalNameIdentifierException;
import org.apache.gravitino.exceptions.NoSuchCatalogException;
import org.apache.gravitino.exceptions.TokenExpiredException;
import org.apache.gravitino.exceptions.UnauthorizedException;
+import org.apache.gravitino.server.web.ServerHealth;
import org.apache.iceberg.exceptions.AlreadyExistsException;
import org.apache.iceberg.exceptions.BadRequestException;
import org.apache.iceberg.exceptions.CommitFailedException;
@@ -133,6 +134,7 @@ public class IcebergExceptionMapper implements
ExceptionMapper<Throwable> {
}
public static Response toRESTResponse(Throwable ex) {
+ ServerHealth.getInstance().recordFailure(ex);
int status =
EXCEPTION_ERROR_CODES.getOrDefault(
ex.getClass(), Status.INTERNAL_SERVER_ERROR.getStatusCode());
diff --git
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergHealthOperations.java
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergHealthOperations.java
index e91fe5e0a8..27304041b6 100644
---
a/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergHealthOperations.java
+++
b/iceberg/iceberg-rest-server/src/main/java/org/apache/gravitino/iceberg/service/rest/IcebergHealthOperations.java
@@ -34,6 +34,7 @@ import org.apache.gravitino.dto.HealthCheckDTO;
import org.apache.gravitino.dto.responses.HealthResponse;
import org.apache.gravitino.iceberg.service.IcebergCatalogWrapperManager;
import org.apache.gravitino.metrics.MetricNames;
+import org.apache.gravitino.server.web.ServerHealth;
import org.apache.gravitino.server.web.Utils;
/**
@@ -41,13 +42,14 @@ import org.apache.gravitino.server.web.Utils;
* semantics as the main Gravitino server.
*
* <ul>
- * <li>{@code GET /iceberg/health/live} — liveness, 200 as long as the HTTP
thread can respond
+ * <li>{@code GET /iceberg/health/live} — liveness, 200 when the HTTP thread
can respond and no
+ * OOM has been observed
* <li>{@code GET /iceberg/health/ready} — readiness, 200 when the catalog
wrapper manager is
* initialized
* <li>{@code GET /iceberg/health} — aggregate, 200 when both pass
* </ul>
*
- * All endpoints return 503 with a JSON body describing the failed check(s)
when unhealthy.
+ * All endpoints return 503 after an observed OOM until process restart, or
when their checks fail.
*/
@Path("/health")
@Produces(MediaType.APPLICATION_JSON)
@@ -56,21 +58,32 @@ public class IcebergHealthOperations {
private static final String CHECK_HTTP_SERVER = "httpServer";
private static final String CHECK_CATALOG_WRAPPER_MANAGER =
"catalogWrapperManager";
+ private final ServerHealth serverHealth;
+
@Inject private IcebergCatalogWrapperManager catalogWrapperManager;
/** Default constructor for Jersey auto-discovery. */
- public IcebergHealthOperations() {}
+ public IcebergHealthOperations() {
+ this(ServerHealth.getInstance());
+ }
+
+ IcebergHealthOperations(ServerHealth serverHealth) {
+ this.serverHealth = serverHealth;
+ }
/**
- * Liveness probe. Returns 200 as long as the HTTP thread can respond.
+ * Liveness probe. Returns 200 when the HTTP thread can respond and no OOM
has been observed.
*
- * @return 200 OK with an UP {@link HealthResponse}
+ * @return 200 OK when live, or 503 with a JVM failure after an observed OOM
*/
@GET
@Path("/live")
@Timed(name = "iceberg.health.live." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
@ResponseMetered(name = "iceberg.health.live", absolute = true)
public Response live() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO check = up(CHECK_HTTP_SERVER, Collections.emptyMap());
return Utils.ok(new HealthResponse(HealthCheckDTO.Status.UP,
Collections.singletonList(check)));
}
@@ -86,7 +99,13 @@ public class IcebergHealthOperations {
@Timed(name = "iceberg.health.ready." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
@ResponseMetered(name = "iceberg.health.ready", absolute = true)
public Response ready() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO managerCheck = checkCatalogWrapperManager();
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO.Status overall = managerCheck.getStatus();
HealthResponse body = new HealthResponse(overall,
Collections.singletonList(managerCheck));
return overall == HealthCheckDTO.Status.UP ? Utils.ok(body) :
Utils.serviceUnavailable(body);
@@ -101,9 +120,15 @@ public class IcebergHealthOperations {
@Timed(name = "iceberg.health." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
@ResponseMetered(name = "iceberg.health", absolute = true)
public Response health() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
List<HealthCheckDTO> checks = new ArrayList<>(2);
checks.add(up(CHECK_HTTP_SERVER, Collections.emptyMap()));
checks.add(checkCatalogWrapperManager());
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO.Status overall =
checks.stream().anyMatch(c -> c.getStatus() ==
HealthCheckDTO.Status.DOWN)
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/TestIcebergExceptionMapper.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/TestIcebergExceptionMapper.java
index 354f4d26bd..4226dfdf5e 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/TestIcebergExceptionMapper.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/TestIcebergExceptionMapper.java
@@ -21,6 +21,7 @@ package org.apache.gravitino.iceberg.service;
import javax.ws.rs.NotFoundException;
import javax.ws.rs.core.Response;
import org.apache.gravitino.exceptions.TokenExpiredException;
+import org.apache.gravitino.server.web.ServerHealth;
import org.apache.iceberg.exceptions.AlreadyExistsException;
import org.apache.iceberg.exceptions.CommitFailedException;
import org.apache.iceberg.exceptions.CommitStateUnknownException;
@@ -36,6 +37,8 @@ import org.apache.iceberg.exceptions.ValidationException;
import org.apache.iceberg.rest.responses.ErrorResponse;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
public class TestIcebergExceptionMapper {
private final IcebergExceptionMapper icebergExceptionMapper = new
IcebergExceptionMapper();
@@ -71,14 +74,19 @@ public class TestIcebergExceptionMapper {
new NoClassDefFoundError("catalog class"), new
AssertionError("assertion")
}) {
error.initCause(new IllegalStateException("root cause"));
- try (Response response = icebergExceptionMapper.toResponse(error)) {
- Assertions.assertEquals(500, response.getStatus());
- ErrorResponse entity = (ErrorResponse) response.getEntity();
- Assertions.assertEquals(error.getClass().getSimpleName(),
entity.type());
- Assertions.assertEquals(error.getMessage(), entity.message());
- Assertions.assertTrue(
- String.join("\n", entity.stack())
- .contains("Caused by: java.lang.IllegalStateException: root
cause"));
+ try (MockedStatic<ServerHealth> shared =
Mockito.mockStatic(ServerHealth.class)) {
+ ServerHealth health = new ServerHealth();
+ shared.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response = icebergExceptionMapper.toResponse(error)) {
+ Assertions.assertEquals(500, response.getStatus());
+ ErrorResponse entity = (ErrorResponse) response.getEntity();
+ Assertions.assertEquals(error.getClass().getSimpleName(),
entity.type());
+ Assertions.assertEquals(error.getMessage(), entity.message());
+ Assertions.assertTrue(
+ String.join("\n", entity.stack())
+ .contains("Caused by: java.lang.IllegalStateException: root
cause"));
+ }
+ Assertions.assertEquals(error instanceof OutOfMemoryError,
health.hasOutOfMemoryError());
}
}
}
diff --git
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergHealthOperations.java
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergHealthOperations.java
index 376479140c..3610b85da6 100644
---
a/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergHealthOperations.java
+++
b/iceberg/iceberg-rest-server/src/test/java/org/apache/gravitino/iceberg/service/rest/TestIcebergHealthOperations.java
@@ -20,18 +20,48 @@ package org.apache.gravitino.iceberg.service.rest;
import static org.mockito.Mockito.mock;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.util.List;
+import java.util.function.Supplier;
import javax.ws.rs.core.Response;
import org.apache.gravitino.dto.HealthCheckDTO;
import org.apache.gravitino.dto.responses.HealthResponse;
import org.apache.gravitino.iceberg.service.IcebergCatalogWrapperManager;
+import org.apache.gravitino.iceberg.service.IcebergExceptionMapper;
+import org.apache.gravitino.iceberg.service.IcebergObjectMapper;
+import org.apache.gravitino.server.web.ServerHealth;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
public class TestIcebergHealthOperations {
+ /** Verifies the documented status casing with the service's actual JSON
mapper. */
+ @Test
+ public void testSerializedHealthStatus() throws Exception {
+ ServerHealth health = new ServerHealth();
+ IcebergHealthOperations operations = new IcebergHealthOperations(health);
+ ObjectMapper mapper = IcebergObjectMapper.getInstance();
+ try (Response response = operations.live()) {
+ JsonNode json =
mapper.readTree(mapper.writeValueAsString(response.getEntity()));
+ Assertions.assertEquals("UP", json.path("status").asText());
+ Assertions.assertEquals("UP",
json.path("checks").get(0).path("status").asText());
+ }
+ health.recordFailure(new OutOfMemoryError("Metaspace"));
+ try (Response response = operations.live()) {
+ JsonNode json =
mapper.readTree(mapper.writeValueAsString(response.getEntity()));
+ Assertions.assertEquals(503, response.getStatus());
+ Assertions.assertEquals("DOWN", json.path("status").asText());
+ Assertions.assertEquals("DOWN",
json.path("checks").get(0).path("status").asText());
+ Assertions.assertEquals("jvm",
json.path("checks").get(0).path("name").asText());
+ }
+ }
+
private static IcebergHealthOperations operationsWithManager(
IcebergCatalogWrapperManager manager) {
- return new IcebergHealthOperations() {
+ return new IcebergHealthOperations(new ServerHealth()) {
@Override
IcebergCatalogWrapperManager getCatalogWrapperManager() {
return manager;
@@ -91,4 +121,82 @@ public class TestIcebergHealthOperations {
body.getChecks().stream().anyMatch(c ->
"catalogWrapperManager".equals(c.getName()));
Assertions.assertTrue(hasCatalogCheck);
}
+
+ /** Verifies mapped direct and wrapped OOM disable all health probes. */
+ @Test
+ public void testMappedOutOfMemoryMakesAllProbesUnhealthy() {
+ for (Throwable failure :
+ new Throwable[] {
+ new OutOfMemoryError("Metaspace"),
+ new IllegalStateException(new OutOfMemoryError("Java heap space"))
+ }) {
+ ServerHealth health = new ServerHealth();
+ IcebergHealthOperations ops =
+ new IcebergHealthOperations(health) {
+ @Override
+ IcebergCatalogWrapperManager getCatalogWrapperManager() {
+ Assertions.fail("Readiness must skip initialization checks after
OOM");
+ return null;
+ }
+ };
+ try (MockedStatic<ServerHealth> shared =
Mockito.mockStatic(ServerHealth.class)) {
+ shared.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response =
IcebergExceptionMapper.toRESTResponse(failure)) {
+ Assertions.assertEquals(500, response.getStatus());
+ }
+ }
+ for (Supplier<Response> probe :
+ List.<Supplier<Response>>of(ops::live, ops::ready, ops::health)) {
+ try (Response response = probe.get()) {
+ Assertions.assertEquals(503, response.getStatus());
+ HealthResponse body = (HealthResponse) response.getEntity();
+ Assertions.assertEquals(HealthCheckDTO.Status.DOWN,
body.getStatus());
+ Assertions.assertEquals(1, body.getChecks().size());
+ Assertions.assertEquals("jvm", body.getChecks().get(0).getName());
+ Assertions.assertEquals(
+ "OutOfMemoryError; restart required",
+ body.getChecks().get(0).getDetails().get("reason"));
+ }
+ }
+ Assertions.assertTrue(health.hasOutOfMemoryError());
+ }
+ }
+
+ /** Verifies an ordinary mapped failure leaves liveness healthy. */
+ @Test
+ public void testOrdinaryMappedFailureDoesNotPoisonLiveness() {
+ ServerHealth health = new ServerHealth();
+ Throwable failure = new IllegalStateException("ordinary failure");
+ try (MockedStatic<ServerHealth> shared =
Mockito.mockStatic(ServerHealth.class)) {
+ shared.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response = IcebergExceptionMapper.toRESTResponse(failure))
{
+ Assertions.assertEquals(500, response.getStatus());
+ }
+ }
+ try (Response response = new IcebergHealthOperations(health).live()) {
+ Assertions.assertEquals(200, response.getStatus());
+ }
+ }
+
+ /** Verifies OOM recorded during initialization checks overrides their
successful result. */
+ @Test
+ public void testOutOfMemoryObservedDuringReadinessOverridesSuccess() {
+ for (boolean aggregate : new boolean[] {false, true}) {
+ ServerHealth health = new ServerHealth();
+ IcebergCatalogWrapperManager dependency =
mock(IcebergCatalogWrapperManager.class);
+ IcebergHealthOperations ops =
+ new IcebergHealthOperations(health) {
+ @Override
+ IcebergCatalogWrapperManager getCatalogWrapperManager() {
+ health.recordFailure(new OutOfMemoryError("Metaspace"));
+ return dependency;
+ }
+ };
+ try (Response response = aggregate ? ops.health() : ops.ready()) {
+ Assertions.assertEquals(503, response.getStatus());
+ HealthResponse body = (HealthResponse) response.getEntity();
+ Assertions.assertEquals("jvm", body.getChecks().get(0).getName());
+ }
+ }
+ }
}
diff --git
a/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/LanceRESTService.java
b/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/LanceRESTService.java
index 6d0777a477..bfb27e04d6 100644
---
a/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/LanceRESTService.java
+++
b/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/LanceRESTService.java
@@ -48,6 +48,7 @@ import org.apache.gravitino.server.web.HttpAuditFilter;
import org.apache.gravitino.server.web.HttpServerMetricsSource;
import org.apache.gravitino.server.web.JettyServer;
import org.apache.gravitino.server.web.JettyServerConfig;
+import org.apache.gravitino.server.web.OutOfMemoryErrorListener;
import org.apache.gravitino.server.web.RequestContextFilter;
import org.glassfish.hk2.api.InterceptionService;
import org.glassfish.hk2.utilities.binding.AbstractBinder;
@@ -110,6 +111,7 @@ public class LanceRESTService implements
GravitinoAuxiliaryService {
resourceConfig.register(JacksonFeature.class);
resourceConfig.packages(LANCE_REST_SPEC_PACKAGE);
resourceConfig.register(LanceExceptionMapper.class);
+ resourceConfig.register(new OutOfMemoryErrorListener());
resourceConfig.register(
new AbstractBinder() {
@Override
diff --git
a/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/LanceExceptionMapper.java
b/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/LanceExceptionMapper.java
index 47d0b7521a..a3de0c46d6 100644
---
a/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/LanceExceptionMapper.java
+++
b/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/LanceExceptionMapper.java
@@ -25,6 +25,7 @@ import javax.ws.rs.ext.ExceptionMapper;
import javax.ws.rs.ext.Provider;
import org.apache.gravitino.exceptions.NoSuchTableException;
import org.apache.gravitino.exceptions.NotFoundException;
+import org.apache.gravitino.server.web.ServerHealth;
import org.lance.namespace.errors.ConcurrentModificationException;
import org.lance.namespace.errors.InternalException;
import org.lance.namespace.errors.InvalidInputException;
@@ -47,6 +48,7 @@ public class LanceExceptionMapper implements
ExceptionMapper<Throwable> {
private static final Logger LOG =
LoggerFactory.getLogger(LanceExceptionMapper.class);
public static Response toRESTResponse(String instance, Throwable ex) {
+ ServerHealth.getInstance().recordFailure(ex);
LanceNamespaceException lanceException =
ex instanceof LanceNamespaceException
? (LanceNamespaceException) ex
diff --git
a/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/rest/LanceHealthOperations.java
b/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/rest/LanceHealthOperations.java
index 4c280d9f8b..e98b1d04de 100644
---
a/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/rest/LanceHealthOperations.java
+++
b/lance/lance-rest-server/src/main/java/org/apache/gravitino/lance/service/rest/LanceHealthOperations.java
@@ -34,6 +34,7 @@ import org.apache.gravitino.dto.HealthCheckDTO;
import org.apache.gravitino.dto.responses.HealthResponse;
import org.apache.gravitino.lance.common.ops.NamespaceWrapper;
import org.apache.gravitino.metrics.MetricNames;
+import org.apache.gravitino.server.web.ServerHealth;
import org.apache.gravitino.server.web.Utils;
/**
@@ -41,12 +42,13 @@ import org.apache.gravitino.server.web.Utils;
* as the main Gravitino server.
*
* <ul>
- * <li>{@code GET /lance/health/live} — liveness, 200 as long as the HTTP
thread can respond
+ * <li>{@code GET /lance/health/live} — liveness, 200 when the HTTP thread
can respond and no OOM
+ * has been observed
* <li>{@code GET /lance/health/ready} — readiness, 200 when the namespace
wrapper is initialized
* <li>{@code GET /lance/health} — aggregate, 200 when both pass
* </ul>
*
- * All endpoints return 503 with a JSON body describing the failed check(s)
when unhealthy.
+ * All endpoints return 503 after an observed OOM until process restart, or
when their checks fail.
*/
@Path("/health")
@Produces(MediaType.APPLICATION_JSON)
@@ -55,21 +57,32 @@ public class LanceHealthOperations {
private static final String CHECK_HTTP_SERVER = "httpServer";
private static final String CHECK_NAMESPACE_WRAPPER = "namespaceWrapper";
+ private final ServerHealth serverHealth;
+
@Inject private NamespaceWrapper namespaceWrapper;
/** Default constructor for Jersey auto-discovery. */
- public LanceHealthOperations() {}
+ public LanceHealthOperations() {
+ this(ServerHealth.getInstance());
+ }
+
+ LanceHealthOperations(ServerHealth serverHealth) {
+ this.serverHealth = serverHealth;
+ }
/**
- * Liveness probe. Returns 200 as long as the HTTP thread can respond.
+ * Liveness probe. Returns 200 when the HTTP thread can respond and no OOM
has been observed.
*
- * @return 200 OK with an UP {@link HealthResponse}
+ * @return 200 OK when live, or 503 with a JVM failure after an observed OOM
*/
@GET
@Path("/live")
@Timed(name = "lance.health.live." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
@ResponseMetered(name = "lance.health.live", absolute = true)
public Response live() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO check = up(CHECK_HTTP_SERVER, Collections.emptyMap());
HealthResponse healthResponse =
new HealthResponse(HealthCheckDTO.Status.UP,
Collections.singletonList(check));
@@ -86,7 +99,13 @@ public class LanceHealthOperations {
@Timed(name = "lance.health.ready." + MetricNames.HTTP_PROCESS_DURATION,
absolute = true)
@ResponseMetered(name = "lance.health.ready", absolute = true)
public Response ready() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO namespaceCheck = checkNamespaceWrapper();
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO.Status overall = namespaceCheck.getStatus();
HealthResponse body = new HealthResponse(overall,
Collections.singletonList(namespaceCheck));
return overall == HealthCheckDTO.Status.UP ? Utils.ok(body) :
Utils.serviceUnavailable(body);
@@ -101,9 +120,15 @@ public class LanceHealthOperations {
@Timed(name = "lance.health." + MetricNames.HTTP_PROCESS_DURATION, absolute
= true)
@ResponseMetered(name = "lance.health", absolute = true)
public Response health() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
List<HealthCheckDTO> checks = new ArrayList<>(2);
checks.add(up(CHECK_HTTP_SERVER, Collections.emptyMap()));
checks.add(checkNamespaceWrapper());
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO.Status overall =
checks.stream().anyMatch(c -> c.getStatus() ==
HealthCheckDTO.Status.DOWN)
diff --git
a/lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/service/rest/TestLanceHealthOperations.java
b/lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/service/rest/TestLanceHealthOperations.java
index 24256fd67d..7c4883e6dc 100644
---
a/lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/service/rest/TestLanceHealthOperations.java
+++
b/lance/lance-rest-server/src/test/java/org/apache/gravitino/lance/service/rest/TestLanceHealthOperations.java
@@ -21,17 +21,46 @@ package org.apache.gravitino.lance.service.rest;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.util.List;
+import java.util.function.Supplier;
import javax.ws.rs.core.Response;
import org.apache.gravitino.dto.HealthCheckDTO;
import org.apache.gravitino.dto.responses.HealthResponse;
import org.apache.gravitino.lance.common.ops.NamespaceWrapper;
+import org.apache.gravitino.lance.service.LanceExceptionMapper;
+import org.apache.gravitino.server.web.ServerHealth;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+import org.mockito.Mockito;
public class TestLanceHealthOperations {
+ /** Verifies the documented status casing with the service's actual JSON
mapper. */
+ @Test
+ public void testSerializedHealthStatus() throws Exception {
+ ServerHealth health = new ServerHealth();
+ LanceHealthOperations operations = new LanceHealthOperations(health);
+ ObjectMapper mapper = new
JsonNullableMapperProvider().getContext(HealthResponse.class);
+ try (Response response = operations.live()) {
+ JsonNode json =
mapper.readTree(mapper.writeValueAsString(response.getEntity()));
+ Assertions.assertEquals("up", json.path("status").asText());
+ Assertions.assertEquals("up",
json.path("checks").get(0).path("status").asText());
+ }
+ health.recordFailure(new OutOfMemoryError("Metaspace"));
+ try (Response response = operations.live()) {
+ JsonNode json =
mapper.readTree(mapper.writeValueAsString(response.getEntity()));
+ Assertions.assertEquals(503, response.getStatus());
+ Assertions.assertEquals("down", json.path("status").asText());
+ Assertions.assertEquals("down",
json.path("checks").get(0).path("status").asText());
+ Assertions.assertEquals("jvm",
json.path("checks").get(0).path("name").asText());
+ }
+ }
+
private static LanceHealthOperations operationsWithWrapper(NamespaceWrapper
wrapper) {
- return new LanceHealthOperations() {
+ return new LanceHealthOperations(new ServerHealth()) {
@Override
NamespaceWrapper getNamespaceWrapper() {
return wrapper;
@@ -93,4 +122,83 @@ public class TestLanceHealthOperations {
body.getChecks().stream().anyMatch(c ->
"namespaceWrapper".equals(c.getName()));
Assertions.assertTrue(hasNamespaceCheck);
}
+
+ /** Verifies mapped direct and wrapped OOM disable all health probes. */
+ @Test
+ public void testMappedOutOfMemoryMakesAllProbesUnhealthy() {
+ for (Throwable failure :
+ new Throwable[] {
+ new OutOfMemoryError("Metaspace"),
+ new IllegalStateException(new OutOfMemoryError("Java heap space"))
+ }) {
+ ServerHealth health = new ServerHealth();
+ LanceHealthOperations ops =
+ new LanceHealthOperations(health) {
+ @Override
+ NamespaceWrapper getNamespaceWrapper() {
+ Assertions.fail("Readiness must skip initialization checks after
OOM");
+ return null;
+ }
+ };
+ try (MockedStatic<ServerHealth> shared =
Mockito.mockStatic(ServerHealth.class)) {
+ shared.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response = LanceExceptionMapper.toRESTResponse("test",
failure)) {
+ Assertions.assertEquals(500, response.getStatus());
+ }
+ }
+ for (Supplier<Response> probe :
+ List.<Supplier<Response>>of(ops::live, ops::ready, ops::health)) {
+ try (Response response = probe.get()) {
+ Assertions.assertEquals(503, response.getStatus());
+ HealthResponse body = (HealthResponse) response.getEntity();
+ Assertions.assertEquals(HealthCheckDTO.Status.DOWN,
body.getStatus());
+ Assertions.assertEquals(1, body.getChecks().size());
+ Assertions.assertEquals("jvm", body.getChecks().get(0).getName());
+ Assertions.assertEquals(
+ "OutOfMemoryError; restart required",
+ body.getChecks().get(0).getDetails().get("reason"));
+ }
+ }
+ Assertions.assertTrue(health.hasOutOfMemoryError());
+ }
+ }
+
+ /** Verifies an ordinary mapped failure leaves liveness healthy. */
+ @Test
+ public void testOrdinaryMappedFailureDoesNotPoisonLiveness() {
+ ServerHealth health = new ServerHealth();
+ Throwable failure = new IllegalStateException("ordinary failure");
+ try (MockedStatic<ServerHealth> shared =
Mockito.mockStatic(ServerHealth.class)) {
+ shared.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response = LanceExceptionMapper.toRESTResponse("test",
failure)) {
+ Assertions.assertEquals(500, response.getStatus());
+ }
+ }
+ try (Response response = new LanceHealthOperations(health).live()) {
+ Assertions.assertEquals(200, response.getStatus());
+ }
+ }
+
+ /** Verifies OOM recorded during initialization checks overrides their
successful result. */
+ @Test
+ public void testOutOfMemoryObservedDuringReadinessOverridesSuccess() {
+ for (boolean aggregate : new boolean[] {false, true}) {
+ ServerHealth health = new ServerHealth();
+ NamespaceWrapper dependency = mock(NamespaceWrapper.class);
+ when(dependency.isInitialized()).thenReturn(true);
+ LanceHealthOperations ops =
+ new LanceHealthOperations(health) {
+ @Override
+ NamespaceWrapper getNamespaceWrapper() {
+ health.recordFailure(new OutOfMemoryError("Metaspace"));
+ return dependency;
+ }
+ };
+ try (Response response = aggregate ? ops.health() : ops.ready()) {
+ Assertions.assertEquals(503, response.getStatus());
+ HealthResponse body = (HealthResponse) response.getEntity();
+ Assertions.assertEquals("jvm", body.getChecks().get(0).getName());
+ }
+ }
+ }
}
diff --git
a/plugins/idp-basic/src/main/java/org/apache/gravitino/idp/web/IdpRESTUtils.java
b/plugins/idp-basic/src/main/java/org/apache/gravitino/idp/web/IdpRESTUtils.java
index b843e5670f..35a2831b5a 100644
---
a/plugins/idp-basic/src/main/java/org/apache/gravitino/idp/web/IdpRESTUtils.java
+++
b/plugins/idp-basic/src/main/java/org/apache/gravitino/idp/web/IdpRESTUtils.java
@@ -30,6 +30,7 @@ import org.apache.gravitino.exceptions.AlreadyExistsException;
import org.apache.gravitino.exceptions.ForbiddenException;
import org.apache.gravitino.exceptions.NonEmptyEntityException;
import org.apache.gravitino.exceptions.NotFoundException;
+import org.apache.gravitino.server.web.ServerHealth;
import org.apache.gravitino.utils.PrincipalUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -71,6 +72,7 @@ public final class IdpRESTUtils {
public static Response handleException(
String resourceType, IdpOperationType op, String name, Exception e) {
+ ServerHealth.getInstance().recordFailure(e);
String errorMsg =
String.format(
"Failed to operate built-in IdP %s [%s] operation [%s], reason
[%s]",
@@ -109,6 +111,7 @@ public final class IdpRESTUtils {
}
public static Response internalError(String message, Throwable throwable) {
+ ServerHealth.getInstance().recordFailure(throwable);
return json(
Response.Status.INTERNAL_SERVER_ERROR,
ErrorResponse.internalError(message, throwable));
}
diff --git
a/plugins/idp-basic/src/test/java/org/apache/gravitino/idp/web/TestIdpRESTUtils.java
b/plugins/idp-basic/src/test/java/org/apache/gravitino/idp/web/TestIdpRESTUtils.java
index 3b8840fd14..9d1ed61f11 100644
---
a/plugins/idp-basic/src/test/java/org/apache/gravitino/idp/web/TestIdpRESTUtils.java
+++
b/plugins/idp-basic/src/test/java/org/apache/gravitino/idp/web/TestIdpRESTUtils.java
@@ -19,15 +19,71 @@
package org.apache.gravitino.idp.web;
import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
+import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.core.Response;
import org.apache.gravitino.dto.responses.ErrorConstants;
import org.apache.gravitino.dto.responses.ErrorResponse;
import org.apache.gravitino.exceptions.NonEmptyEntityException;
+import org.apache.gravitino.server.web.ServerHealth;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.mockito.MockedStatic;
class TestIdpRESTUtils {
+ @ParameterizedTest
+ @ValueSource(booleans = {true, false})
+ void testRecordsWrappedOomInRequestHelpers(boolean oom) {
+ for (boolean viaDoAs : new boolean[] {true, false}) {
+ ServerHealth health = new ServerHealth();
+ // A mapped 400 must still record an OOM; internalError alone is
insufficient.
+ IllegalArgumentException failure =
+ new IllegalArgumentException(
+ "invalid request",
+ oom ? new OutOfMemoryError("Metaspace") : new
IllegalStateException("ordinary"));
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response =
+ viaDoAs
+ ? IdpRESTUtils.doAs(
+ mock(HttpServletRequest.class),
+ () -> {
+ throw failure;
+ },
+ "group",
+ IdpOperationType.GET,
+ "group")
+ : IdpRESTUtils.handleException("group", IdpOperationType.GET,
"group", failure)) {
+ assertEquals(400, response.getStatus());
+ assertEquals(oom, health.hasOutOfMemoryError());
+ }
+ }
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"direct", "wrapped", "ordinary"})
+ void testRecordsOomInInternalError(String kind) {
+ ServerHealth health = new ServerHealth();
+ Throwable failure =
+ kind.equals("direct")
+ ? new OutOfMemoryError("Metaspace")
+ : kind.equals("wrapped")
+ ? new RuntimeException(new OutOfMemoryError("Metaspace"))
+ : new IllegalStateException("ordinary");
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response = IdpRESTUtils.internalError("failure", failure))
{
+ assertEquals(500, response.getStatus());
+ assertEquals(!kind.equals("ordinary"), health.hasOutOfMemoryError());
+ }
+ }
+ }
+
@Test
void testUnsupportedOperationReturnsNotImplemented() {
Response response =
diff --git
a/server-common/src/main/java/org/apache/gravitino/server/authentication/AuthenticationFilter.java
b/server-common/src/main/java/org/apache/gravitino/server/authentication/AuthenticationFilter.java
index 6f75837212..28024d6075 100644
---
a/server-common/src/main/java/org/apache/gravitino/server/authentication/AuthenticationFilter.java
+++
b/server-common/src/main/java/org/apache/gravitino/server/authentication/AuthenticationFilter.java
@@ -42,6 +42,7 @@ import org.apache.gravitino.exceptions.ForbiddenException;
import org.apache.gravitino.exceptions.UnauthorizedException;
import org.apache.gravitino.server.web.HealthCheckPathMatcher;
import org.apache.gravitino.server.web.ObjectMapperProvider;
+import org.apache.gravitino.server.web.ServerHealth;
import org.apache.gravitino.utils.PrincipalUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -50,6 +51,8 @@ public class AuthenticationFilter implements Filter {
private static final Logger LOG =
LoggerFactory.getLogger(AuthenticationFilter.class);
+ private final ServerHealth health = ServerHealth.getInstance();
+
private final List<Authenticator> filterAuthenticators;
/**
@@ -131,6 +134,7 @@ public class AuthenticationFilter implements Filter {
return null;
});
} catch (UnauthorizedException ue) {
+ health.recordFailure(ue);
HttpServletResponse resp = (HttpServletResponse) response;
if (!ue.getChallenges().isEmpty()) {
// For some authentication, HTTP response can provide some challenge
information
@@ -144,6 +148,7 @@ public class AuthenticationFilter implements Filter {
}
sendAuthErrorResponse(resp, ue);
} catch (Exception e) {
+ health.recordFailure(e);
HttpServletResponse resp = (HttpServletResponse) response;
sendAuthErrorResponse(resp, e);
}
diff --git
a/server-common/src/main/java/org/apache/gravitino/server/web/JettyServer.java
b/server-common/src/main/java/org/apache/gravitino/server/web/JettyServer.java
index 2385b49a1f..7a507eb74c 100644
---
a/server-common/src/main/java/org/apache/gravitino/server/web/JettyServer.java
+++
b/server-common/src/main/java/org/apache/gravitino/server/web/JettyServer.java
@@ -179,6 +179,9 @@ public class JettyServer {
webUiEnabled = false;
}
+ // Install before authentication, custom filters, and servlet mappings on
every service.
+ addFilter(new OutOfMemoryErrorFilter(), "/*");
+
MetricsSystem metricsSystem = GravitinoEnv.getInstance().metricsSystem();
// Metrics System could be null in UT.
if (metricsSystem != null) {
@@ -484,6 +487,7 @@ public class JettyServer {
thread.setName(getName() + "-" + thread.getId());
thread.setUncaughtExceptionHandler(
(t, throwable) -> {
+ ServerHealth.getInstance().recordFailure(throwable);
LOG.error("{} uncaught exception:", t.getName(),
throwable);
});
// JettyServer maybe used by Gravitino server and Iceberg
REST server with
diff --git
a/server-common/src/main/java/org/apache/gravitino/server/web/OutOfMemoryErrorFilter.java
b/server-common/src/main/java/org/apache/gravitino/server/web/OutOfMemoryErrorFilter.java
new file mode 100644
index 0000000000..06272ba336
--- /dev/null
+++
b/server-common/src/main/java/org/apache/gravitino/server/web/OutOfMemoryErrorFilter.java
@@ -0,0 +1,55 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web;
+
+import java.io.IOException;
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+
+/** Records OOM escaping downstream filters and servlets before Jetty consumes
the failure. */
+public final class OutOfMemoryErrorFilter implements Filter {
+ private final ServerHealth health = ServerHealth.getInstance();
+
+ /** Creates a filter using the shared server health state. */
+ public OutOfMemoryErrorFilter() {}
+
+ /** {@inheritDoc} */
+ @Override
+ public void init(FilterConfig filterConfig) {}
+
+ /** {@inheritDoc} */
+ @Override
+ public void destroy() {}
+
+ /** {@inheritDoc} */
+ @Override
+ public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain)
+ throws IOException, ServletException {
+ try {
+ chain.doFilter(request, response);
+ } catch (Throwable failure) {
+ health.recordFailure(failure);
+ throw failure;
+ }
+ }
+}
diff --git
a/server-common/src/main/java/org/apache/gravitino/server/web/OutOfMemoryErrorListener.java
b/server-common/src/main/java/org/apache/gravitino/server/web/OutOfMemoryErrorListener.java
new file mode 100644
index 0000000000..d4538ed48b
--- /dev/null
+++
b/server-common/src/main/java/org/apache/gravitino/server/web/OutOfMemoryErrorListener.java
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web;
+
+import org.glassfish.jersey.server.monitoring.ApplicationEvent;
+import org.glassfish.jersey.server.monitoring.ApplicationEventListener;
+import org.glassfish.jersey.server.monitoring.RequestEvent;
+import org.glassfish.jersey.server.monitoring.RequestEventListener;
+
+/** Records out-of-memory failures before Jersey maps them to an HTTP
response. */
+public final class OutOfMemoryErrorListener
+ implements ApplicationEventListener, RequestEventListener {
+ private final ServerHealth health;
+
+ /** Creates a listener using the shared server health state. */
+ public OutOfMemoryErrorListener() {
+ this(ServerHealth.getInstance());
+ }
+
+ /**
+ * Creates a listener using the supplied health state.
+ *
+ * @param health the state to update on an out-of-memory failure
+ */
+ public OutOfMemoryErrorListener(ServerHealth health) {
+ this.health = health;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void onEvent(ApplicationEvent event) {}
+
+ /** {@inheritDoc} */
+ @Override
+ public RequestEventListener onRequest(RequestEvent event) {
+ return this;
+ }
+
+ /** {@inheritDoc} */
+ @Override
+ public void onEvent(RequestEvent event) {
+ if (event.getType() == RequestEvent.Type.ON_EXCEPTION) {
+ health.recordFailure(event.getException());
+ }
+ }
+}
diff --git
a/server-common/src/main/java/org/apache/gravitino/server/web/ServerHealth.java
b/server-common/src/main/java/org/apache/gravitino/server/web/ServerHealth.java
new file mode 100644
index 0000000000..7e5ff1662b
--- /dev/null
+++
b/server-common/src/main/java/org/apache/gravitino/server/web/ServerHealth.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web;
+
+import javax.annotation.Nullable;
+
+/**
+ * Records out-of-memory failures observed by the server. Once recorded, the
failure remains until
+ * the process restarts; a successful request does not establish that the JVM
has recovered.
+ *
+ * <p>The failure path only sets a flag. It does not retain the error, format
its stack trace, or
+ * allocate a collection while walking its causes.
+ */
+public final class ServerHealth {
+ private static final ServerHealth INSTANCE = new ServerHealth();
+
+ private volatile boolean outOfMemory;
+
+ /** Creates an independent health state, initially healthy. */
+ public ServerHealth() {}
+
+ /**
+ * Returns the shared state used by the server's request and health-check
paths.
+ *
+ * @return the shared health state
+ */
+ public static ServerHealth getInstance() {
+ return INSTANCE;
+ }
+
+ /**
+ * Records an out-of-memory error, including one wrapped in another
throwable. Suppressed
+ * exceptions are not inspected, avoiding the array copies made by {@link
+ * Throwable#getSuppressed()}.
+ *
+ * @param failure the observed failure, or null
+ */
+ public void recordFailure(@Nullable Throwable failure) {
+ Throwable slow = failure;
+ boolean advanceSlow = false;
+ while (failure != null) {
+ if (failure instanceof OutOfMemoryError) {
+ outOfMemory = true;
+ return;
+ }
+ failure = failure.getCause();
+ if (advanceSlow) {
+ slow = slow.getCause();
+ }
+ advanceSlow = !advanceSlow;
+ // Throwable cause chains can be cyclic. Detect cycles without
allocating a visited set.
+ if (failure == slow) {
+ return;
+ }
+ }
+ }
+
+ /**
+ * Returns whether this server has observed an out-of-memory error.
+ *
+ * @return true after an out-of-memory error has been recorded
+ */
+ public boolean hasOutOfMemoryError() {
+ return outOfMemory;
+ }
+}
diff --git
a/server-common/src/main/java/org/apache/gravitino/server/web/Utils.java
b/server-common/src/main/java/org/apache/gravitino/server/web/Utils.java
index e2165637bb..0f5196c9ae 100644
--- a/server-common/src/main/java/org/apache/gravitino/server/web/Utils.java
+++ b/server-common/src/main/java/org/apache/gravitino/server/web/Utils.java
@@ -21,6 +21,7 @@ package org.apache.gravitino.server.web;
import com.google.common.collect.Maps;
import java.lang.reflect.Parameter;
import java.security.PrivilegedExceptionAction;
+import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.Optional;
@@ -37,7 +38,9 @@ import org.apache.gravitino.audit.FilesetDataOperation;
import org.apache.gravitino.audit.InternalClientType;
import org.apache.gravitino.auth.AuthConstants;
import org.apache.gravitino.credential.CredentialConstants;
+import org.apache.gravitino.dto.HealthCheckDTO;
import org.apache.gravitino.dto.responses.ErrorResponse;
+import org.apache.gravitino.dto.responses.HealthResponse;
import org.apache.gravitino.utils.PrincipalUtils;
public class Utils {
@@ -100,6 +103,7 @@ public class Utils {
}
public static Response internalError(String message, Throwable throwable) {
+ ServerHealth.getInstance().recordFailure(throwable);
return Response.status(Response.Status.INTERNAL_SERVER_ERROR)
.entity(ErrorResponse.internalError(message, throwable))
.type(MediaType.APPLICATION_JSON)
@@ -260,6 +264,21 @@ public class Utils {
.build();
}
+ /**
+ * Returns the health response used after an observed out-of-memory failure.
+ *
+ * @return HTTP 503 with a JVM failure requiring process restart
+ */
+ public static Response outOfMemoryResponse() {
+ HealthCheckDTO check =
+ new HealthCheckDTO(
+ "jvm",
+ HealthCheckDTO.Status.DOWN,
+ Collections.singletonMap("reason", "OutOfMemoryError; restart
required"));
+ return serviceUnavailable(
+ new HealthResponse(HealthCheckDTO.Status.DOWN,
Collections.singletonList(check)));
+ }
+
public static Response doAs(
HttpServletRequest httpRequest, PrivilegedExceptionAction<Response>
action) throws Exception {
UserPrincipal principal =
@@ -268,7 +287,13 @@ public class Utils {
if (principal == null) {
principal = new UserPrincipal(AuthConstants.ANONYMOUS_USER);
}
- return PrincipalUtils.doAs(principal, action);
+ try {
+ return PrincipalUtils.doAs(principal, action);
+ } catch (Exception | Error failure) {
+ // Record before a resource converts a wrapped failure into an ordinary
error response.
+ ServerHealth.getInstance().recordFailure(failure);
+ throw failure;
+ }
}
public static Map<String, String>
filterFilesetAuditHeaders(HttpServletRequest httpRequest) {
diff --git
a/server-common/src/test/java/org/apache/gravitino/server/authentication/TestAuthenticationOutOfMemoryHttp.java
b/server-common/src/test/java/org/apache/gravitino/server/authentication/TestAuthenticationOutOfMemoryHttp.java
new file mode 100644
index 0000000000..af300d029f
--- /dev/null
+++
b/server-common/src/test/java/org/apache/gravitino/server/authentication/TestAuthenticationOutOfMemoryHttp.java
@@ -0,0 +1,118 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.authentication;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.util.Collections;
+import java.util.concurrent.atomic.AtomicBoolean;
+import javax.servlet.Filter;
+import javax.servlet.http.HttpServletResponse;
+import org.apache.gravitino.Config;
+import org.apache.gravitino.exceptions.UnauthorizedException;
+import org.apache.gravitino.rest.RESTUtils;
+import org.apache.gravitino.server.web.JettyServer;
+import org.apache.gravitino.server.web.JettyServerConfig;
+import org.apache.gravitino.server.web.ServerHealth;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.mockito.MockedStatic;
+
+/** Verifies authentication failures are recorded across the real HTTP filter
chain. */
+class TestAuthenticationOutOfMemoryHttp {
+ @ParameterizedTest
+ @ValueSource(strings = {"direct", "wrapped", "unauthorized", "ordinary",
"ordinary-unauthorized"})
+ void recordsOomBeforeConvertingAuthenticationErrors(String kind) throws
Exception {
+ ServerHealth health = new ServerHealth();
+ AtomicBoolean recordedBeforeConversion = new AtomicBoolean();
+ Authenticator authenticator = mock(Authenticator.class);
+ when(authenticator.supportsToken(any())).thenReturn(true);
+ when(authenticator.isDataFromToken()).thenReturn(true);
+ Throwable failure;
+ switch (kind) {
+ case "direct":
+ failure = new OutOfMemoryError("Metaspace");
+ break;
+ case "wrapped":
+ failure = new IllegalStateException(new OutOfMemoryError("Java heap
space"));
+ break;
+ case "unauthorized":
+ failure =
+ new UnauthorizedException(new OutOfMemoryError("Metaspace"),
"authentication failed");
+ break;
+ case "ordinary-unauthorized":
+ failure = new UnauthorizedException("invalid credentials");
+ break;
+ default:
+ failure = new IllegalStateException("ordinary failure");
+ }
+ when(authenticator.authenticateToken(any())).thenThrow(failure);
+ int port = RESTUtils.findAvailablePort(0, 0);
+ Config config = new Config(false) {};
+ config.set(JettyServerConfig.WEBSERVER_HTTP_PORT, port);
+ JettyServer server =
+ new JettyServer() {
+ /** {@inheritDoc} */
+ @Override
+ protected Filter createAuthenticationFilter() {
+ return new
AuthenticationFilter(Collections.singletonList(authenticator)) {
+ /** {@inheritDoc} */
+ @Override
+ protected void sendAuthErrorResponse(
+ HttpServletResponse response, Exception exception) throws
IOException {
+ recordedBeforeConversion.set(health.hasOutOfMemoryError());
+ super.sendAuthErrorResponse(response, exception);
+ }
+ };
+ }
+ };
+ try {
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ server.initialize(JettyServerConfig.fromConfig(config),
"authentication-oom-test", false);
+ server.addSystemFilters("/*");
+ }
+ server.start();
+ HttpResponse<String> response =
+ HttpClient.newHttpClient()
+ .send(
+ HttpRequest.newBuilder(URI.create("http://localhost:"; + port
+ "/test"))
+ .header("Authorization", "test")
+ .GET()
+ .build(),
+ HttpResponse.BodyHandlers.ofString());
+ assertEquals(kind.contains("unauthorized") ? 401 : 500,
response.statusCode());
+ assertEquals(!kind.startsWith("ordinary"), health.hasOutOfMemoryError());
+ if (!kind.equals("direct")) {
+ assertEquals(!kind.startsWith("ordinary"),
recordedBeforeConversion.get());
+ }
+ } finally {
+ server.stop();
+ }
+ }
+}
diff --git
a/server-common/src/test/java/org/apache/gravitino/server/web/TestJettyServer.java
b/server-common/src/test/java/org/apache/gravitino/server/web/TestJettyServer.java
index 3ff3cfc2cc..21b0bfc58d 100644
---
a/server-common/src/test/java/org/apache/gravitino/server/web/TestJettyServer.java
+++
b/server-common/src/test/java/org/apache/gravitino/server/web/TestJettyServer.java
@@ -20,16 +20,20 @@ package org.apache.gravitino.server.web;
import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
import java.io.IOException;
import javax.servlet.Filter;
import javax.servlet.Servlet;
import org.apache.gravitino.Config;
import org.apache.gravitino.rest.RESTUtils;
+import org.eclipse.jetty.util.thread.QueuedThreadPool;
import org.junit.jupiter.api.AfterEach;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
public class TestJettyServer {
@@ -95,4 +99,19 @@ public class TestJettyServer {
public void testStartWithoutInitialise() throws InterruptedException {
assertThrows(RuntimeException.class, () -> jettyServer.start());
}
+ /** Jetty worker failures update health before logging the uncaught error. */
+ @Test
+ public void testUncaughtOutOfMemoryUpdatesHealth() throws IOException {
+ ServerHealth health = new ServerHealth();
+ Config config = new Config(false) {};
+ jettyServer.initialize(JettyServerConfig.fromConfig(config), "test",
false);
+ Thread worker = ((QueuedThreadPool)
jettyServer.getThreadPool()).newThread(() -> {});
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ worker
+ .getUncaughtExceptionHandler()
+ .uncaughtException(worker, new OutOfMemoryError("Metaspace"));
+ assertTrue(health.hasOutOfMemoryError());
+ }
+ }
}
diff --git
a/server-common/src/test/java/org/apache/gravitino/server/web/TestOutOfMemoryErrorListener.java
b/server-common/src/test/java/org/apache/gravitino/server/web/TestOutOfMemoryErrorListener.java
new file mode 100644
index 0000000000..933f8d05d7
--- /dev/null
+++
b/server-common/src/test/java/org/apache/gravitino/server/web/TestOutOfMemoryErrorListener.java
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import org.glassfish.jersey.server.monitoring.RequestEvent;
+import org.junit.jupiter.api.Test;
+
+/** Tests detection of errors that Jersey may wrap or map to ordinary
responses. */
+class TestOutOfMemoryErrorListener {
+ @Test
+ void recordsWrappedErrorsOnlyOnExceptionEvents() {
+ ServerHealth health = new ServerHealth();
+ OutOfMemoryErrorListener listener = new OutOfMemoryErrorListener(health);
+ RequestEvent event = mock(RequestEvent.class);
+ when(event.getException()).thenReturn(new RuntimeException(new
OutOfMemoryError("Metaspace")));
+ when(event.getType()).thenReturn(RequestEvent.Type.START);
+ assertSame(listener, listener.onRequest(event));
+ listener.onEvent(event);
+ assertFalse(health.hasOutOfMemoryError());
+ when(event.getType()).thenReturn(RequestEvent.Type.ON_EXCEPTION);
+ listener.onEvent(event);
+ assertTrue(health.hasOutOfMemoryError());
+ }
+}
diff --git
a/server-common/src/test/java/org/apache/gravitino/server/web/TestServerHealth.java
b/server-common/src/test/java/org/apache/gravitino/server/web/TestServerHealth.java
new file mode 100644
index 0000000000..f49d630c3d
--- /dev/null
+++
b/server-common/src/test/java/org/apache/gravitino/server/web/TestServerHealth.java
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web;
+
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.junit.jupiter.api.Test;
+
+/** Tests the sticky out-of-memory state and cause-chain handling. */
+class TestServerHealth {
+ @Test
+ void ordinaryFailuresDoNotPoisonHealth() {
+ ServerHealth health = new ServerHealth();
+ health.recordFailure(null);
+ health.recordFailure(new IllegalStateException("database unavailable"));
+ health.recordFailure(new StackOverflowError());
+ health.recordFailure(new NoClassDefFoundError("missing connector"));
+ assertFalse(health.hasOutOfMemoryError());
+ }
+
+ @Test
+ void heapAndMetaspaceFailuresRemainRecorded() {
+ for (String message :
+ new String[] {"Java heap space", "Metaspace", "unable to create native
thread"}) {
+ ServerHealth health = new ServerHealth();
+ health.recordFailure(new RuntimeException(new
OutOfMemoryError(message)));
+ health.recordFailure(null);
+ health.recordFailure(new IllegalArgumentException());
+ assertTrue(health.hasOutOfMemoryError());
+ }
+ }
+
+ @Test
+ void cyclicCausesTerminateAndStillFindOutOfMemory() {
+ Throwable first = new RuntimeException();
+ Throwable second = new RuntimeException(first);
+ first.initCause(second);
+ ServerHealth health = new ServerHealth();
+ health.recordFailure(first);
+ assertFalse(health.hasOutOfMemoryError());
+
+ Throwable root = new RuntimeException();
+ Throwable oom = new OutOfMemoryError("Metaspace");
+ Throwable middle = new RuntimeException(oom);
+ root.initCause(middle);
+ oom.initCause(root);
+ health.recordFailure(root);
+ assertTrue(health.hasOutOfMemoryError());
+ }
+}
diff --git
a/server-common/src/test/java/org/apache/gravitino/server/web/TestUtilsOutOfMemory.java
b/server-common/src/test/java/org/apache/gravitino/server/web/TestUtilsOutOfMemory.java
new file mode 100644
index 0000000000..c2d50285c0
--- /dev/null
+++
b/server-common/src/test/java/org/apache/gravitino/server/web/TestUtilsOutOfMemory.java
@@ -0,0 +1,90 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertSame;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
+
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.core.Response;
+import org.junit.jupiter.api.Test;
+import org.mockito.MockedStatic;
+
+/** Tests recording errors before resource-level exception handling hides
them. */
+class TestUtilsOutOfMemory {
+ @Test
+ void doAsRecordsAndRethrowsTheOriginalError() {
+ ServerHealth health = new ServerHealth();
+ OutOfMemoryError failure = new OutOfMemoryError("Metaspace");
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ assertSame(
+ failure,
+ assertThrows(
+ OutOfMemoryError.class,
+ () ->
+ Utils.doAs(
+ mock(HttpServletRequest.class),
+ () -> {
+ throw failure;
+ })));
+ assertTrue(health.hasOutOfMemoryError());
+ }
+ }
+
+ @Test
+ void doAsRecordsWrappedFailuresBeforeTheyAreMappedToResponses() {
+ ServerHealth health = new ServerHealth();
+ RuntimeException failure = new RuntimeException(new OutOfMemoryError("Java
heap space"));
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ assertThrows(
+ Exception.class,
+ () ->
+ Utils.doAs(
+ mock(HttpServletRequest.class),
+ () -> {
+ throw failure;
+ }));
+ assertTrue(health.hasOutOfMemoryError());
+ }
+ }
+
+ @Test
+ void internalErrorRecordsWrappedOomButNotOrdinaryServerErrors() {
+ ServerHealth health = new ServerHealth();
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ try (Response response = Utils.internalError("unavailable", new
IllegalStateException())) {
+ assertEquals(500, response.getStatus());
+ assertFalse(health.hasOutOfMemoryError());
+ }
+ try (Response response =
+ Utils.internalError("failed", new RuntimeException(new
OutOfMemoryError()))) {
+ assertEquals(500, response.getStatus());
+ assertTrue(health.hasOutOfMemoryError());
+ }
+ }
+ }
+}
diff --git
a/server/src/main/java/org/apache/gravitino/server/GravitinoServer.java
b/server/src/main/java/org/apache/gravitino/server/GravitinoServer.java
index 04ba74a209..0fb4b007b6 100644
--- a/server/src/main/java/org/apache/gravitino/server/GravitinoServer.java
+++ b/server/src/main/java/org/apache/gravitino/server/GravitinoServer.java
@@ -60,6 +60,7 @@ import
org.apache.gravitino.server.web.HttpServerMetricsSource;
import org.apache.gravitino.server.web.JettyServer;
import org.apache.gravitino.server.web.JettyServerConfig;
import org.apache.gravitino.server.web.ObjectMapperProvider;
+import org.apache.gravitino.server.web.OutOfMemoryErrorListener;
import org.apache.gravitino.server.web.RequestContextFilter;
import org.apache.gravitino.server.web.VersioningFilter;
import org.apache.gravitino.server.web.filter.AccessControlNotAllowedFilter;
@@ -188,6 +189,7 @@ public class GravitinoServer extends ResourceConfig {
}
});
register(JsonProcessingExceptionMapper.class);
+ register(new OutOfMemoryErrorListener());
register(ErrorExceptionMapper.class);
register(JsonParseExceptionMapper.class);
register(JsonMappingExceptionMapper.class);
diff --git
a/server/src/main/java/org/apache/gravitino/server/web/mapper/ErrorExceptionMapper.java
b/server/src/main/java/org/apache/gravitino/server/web/mapper/ErrorExceptionMapper.java
index 892cc8f699..41ec7c63df 100644
---
a/server/src/main/java/org/apache/gravitino/server/web/mapper/ErrorExceptionMapper.java
+++
b/server/src/main/java/org/apache/gravitino/server/web/mapper/ErrorExceptionMapper.java
@@ -22,6 +22,7 @@ import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import javax.ws.rs.ext.ExceptionMapper;
import org.apache.gravitino.dto.responses.ErrorResponse;
+import org.apache.gravitino.server.web.ServerHealth;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -29,6 +30,22 @@ import org.slf4j.LoggerFactory;
public class ErrorExceptionMapper implements ExceptionMapper<Error> {
private static final Logger LOG =
LoggerFactory.getLogger(ErrorExceptionMapper.class);
+ private final ServerHealth health;
+
+ /** Creates a mapper using the shared server health state. */
+ public ErrorExceptionMapper() {
+ this(ServerHealth.getInstance());
+ }
+
+ /**
+ * Creates a mapper using the supplied health state.
+ *
+ * @param health the state to update before constructing an error response
+ */
+ public ErrorExceptionMapper(ServerHealth health) {
+ this.health = health;
+ }
+
/**
* Returns a server error response retaining the original error type and
complete stack trace.
*
@@ -37,6 +54,7 @@ public class ErrorExceptionMapper implements
ExceptionMapper<Error> {
*/
@Override
public Response toResponse(Error error) {
+ health.recordFailure(error);
String message = "Server error while processing request: " + error;
LOG.error(message, error);
return Response.status(Response.Status.INTERNAL_SERVER_ERROR)
diff --git
a/server/src/main/java/org/apache/gravitino/server/web/rest/HealthOperations.java
b/server/src/main/java/org/apache/gravitino/server/web/rest/HealthOperations.java
index ea2aa6ffc0..1a505be127 100644
---
a/server/src/main/java/org/apache/gravitino/server/web/rest/HealthOperations.java
+++
b/server/src/main/java/org/apache/gravitino/server/web/rest/HealthOperations.java
@@ -47,6 +47,7 @@ import org.apache.gravitino.dto.HealthCheckDTO;
import org.apache.gravitino.dto.responses.HealthResponse;
import org.apache.gravitino.metrics.MetricNames;
import org.apache.gravitino.server.ServerConfig;
+import org.apache.gravitino.server.web.ServerHealth;
import org.apache.gravitino.server.web.Utils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -57,8 +58,10 @@ import org.slf4j.LoggerFactory;
* managers can distinguish "restart this pod" from "route traffic elsewhere."
*
* <ul>
- * <li>{@code GET /api/health/live} — liveness, 200 as long as the HTTP
thread can respond
- * <li>{@code GET /api/health/ready} — readiness, 200 when entity store is
reachable
+ * <li>{@code GET /api/health/live} — liveness, 200 when the HTTP thread can
respond and no OOM
+ * has been observed
+ * <li>{@code GET /api/health/ready} — readiness, 200 when entity store is
reachable and no OOM
+ * has been observed
* <li>{@code GET /api/health} — aggregate, 200 when both pass
* </ul>
*
@@ -96,12 +99,20 @@ public class HealthOperations {
private static final String CHECK_HTTP_SERVER = "httpServer";
private static final String CHECK_ENTITY_STORE = "entityStore";
+ private final ServerHealth serverHealth;
+
/**
* Default constructor for Jersey auto-discovery. The entity store is
resolved lazily at request
* time via {@link #getEntityStore()} so that probes issued before {@link
GravitinoEnv} has
* finished initializing report DOWN rather than throwing
NullPointerException.
*/
- public HealthOperations() {}
+ public HealthOperations() {
+ this(ServerHealth.getInstance());
+ }
+
+ HealthOperations(ServerHealth serverHealth) {
+ this.serverHealth = serverHealth;
+ }
@GET
@Path("/live")
@@ -109,6 +120,9 @@ public class HealthOperations {
@Timed(name = "health.live." + MetricNames.HTTP_PROCESS_DURATION, absolute =
true)
@ResponseMetered(name = "health.live", absolute = true)
public Response live() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO check = up(CHECK_HTTP_SERVER, Collections.emptyMap());
return Utils.ok(new HealthResponse(HealthCheckDTO.Status.UP,
Collections.singletonList(check)));
}
@@ -119,7 +133,13 @@ public class HealthOperations {
@Timed(name = "health.ready." + MetricNames.HTTP_PROCESS_DURATION, absolute
= true)
@ResponseMetered(name = "health.ready", absolute = true)
public Response ready() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO entityStoreCheck = checkEntityStore();
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO.Status overall = entityStoreCheck.getStatus();
HealthResponse body = new HealthResponse(overall,
Collections.singletonList(entityStoreCheck));
return overall == HealthCheckDTO.Status.UP ? Utils.ok(body) :
Utils.serviceUnavailable(body);
@@ -130,9 +150,15 @@ public class HealthOperations {
@Timed(name = "health." + MetricNames.HTTP_PROCESS_DURATION, absolute = true)
@ResponseMetered(name = "health", absolute = true)
public Response health() {
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
List<HealthCheckDTO> checks = new ArrayList<>(2);
checks.add(up(CHECK_HTTP_SERVER, Collections.emptyMap()));
checks.add(checkEntityStore());
+ if (serverHealth.hasOutOfMemoryError()) {
+ return Utils.outOfMemoryResponse();
+ }
HealthCheckDTO.Status overall =
checks.stream().anyMatch(c -> c.getStatus() ==
HealthCheckDTO.Status.DOWN)
@@ -158,8 +184,14 @@ public class HealthOperations {
try {
return entityStore.exists(
NameIdentifier.of(HEALTH_PROBE_SENTINEL),
EntityType.METALAKE);
- } catch (IOException e) {
- throw new RuntimeException(e);
+ } catch (IOException failure) {
+ serverHealth.recordFailure(failure);
+ throw new RuntimeException(failure);
+ } catch (RuntimeException | Error failure) {
+ // CompletableFuture captures Errors too. Record here even
if the caller has
+ // already timed out and will never inspect the future's
exception.
+ serverHealth.recordFailure(failure);
+ throw failure;
}
},
HEALTH_PROBE_EXECUTOR);
@@ -184,6 +216,7 @@ public class HealthOperations {
return down(CHECK_ENTITY_STORE, "reason", "interrupted");
} catch (ExecutionException e) {
+ serverHealth.recordFailure(e);
Throwable cause = e.getCause() != null ? e.getCause() : e;
// Unwrap RuntimeException wrappers introduced by supplyAsync tunneling
checked exceptions.
if (cause instanceof RuntimeException && cause.getCause() != null) {
diff --git
a/server/src/test/java/org/apache/gravitino/server/TestGravitinoServerOutOfMemoryHttp.java
b/server/src/test/java/org/apache/gravitino/server/TestGravitinoServerOutOfMemoryHttp.java
new file mode 100644
index 0000000000..3a81d3d993
--- /dev/null
+++
b/server/src/test/java/org/apache/gravitino/server/TestGravitinoServerOutOfMemoryHttp.java
@@ -0,0 +1,210 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.io.File;
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import java.time.Duration;
+import java.util.Map;
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServletRequest;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.Produces;
+import javax.ws.rs.core.Response;
+import org.apache.gravitino.Configs;
+import org.apache.gravitino.GravitinoEnv;
+import org.apache.gravitino.auxiliary.AuxiliaryServiceManager;
+import org.apache.gravitino.rest.RESTUtils;
+import org.apache.gravitino.server.web.JettyServerConfig;
+import org.apache.gravitino.server.web.ObjectMapperProvider;
+import org.apache.gravitino.server.web.ServerHealth;
+import org.junit.jupiter.api.io.TempDir;
+import org.junit.jupiter.api.parallel.Isolated;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+
+/** Exercises OOM reporting through the production server's filters and Jersey
providers. */
+@Isolated(
+ "Uses the production server environment and restores its shared OOM marker
after shutdown")
+class TestGravitinoServerOutOfMemoryHttp {
+ private static final String[] HEALTH_PATHS = {
+ "/api/health",
+ "/api/health/live",
+ "/api/health/ready",
+ "/health",
+ "/health/live",
+ "/health/ready",
+ "/health.html"
+ };
+
+ @TempDir File temporaryDirectory;
+
+ /** Adds only failure injection; all health resources and providers come
from production. */
+ @Path("/oom-wiring/{kind}")
+ @Produces("application/vnd.gravitino.v1+json")
+ public static class FailingResource {
+ /**
+ * Simulates a failed allocation without exhausting the test JVM.
+ *
+ * @param kind the failure to simulate
+ * @return a successful response when no failure was requested
+ */
+ @GET
+ public Response get(@PathParam("kind") String kind) {
+ fail(kind);
+ return Response.ok().build();
+ }
+ }
+
+ /** Injects failures downstream of the production request-context and audit
filters. */
+ public static class FailingFilter implements Filter {
+ /** {@inheritDoc} */
+ @Override
+ public void init(FilterConfig config) {}
+
+ /** {@inheritDoc} */
+ @Override
+ public void destroy() {}
+
+ /** {@inheritDoc} */
+ @Override
+ public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain)
+ throws IOException, ServletException {
+ String path = ((HttpServletRequest) request).getRequestURI();
+ String prefix = "/api/oom-wiring/filter-";
+ if (path.startsWith(prefix)) {
+ fail(path.substring(prefix.length()));
+ }
+ chain.doFilter(request, response);
+ }
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"oom", "wrapped", "filter-oom", "filter-wrapped"})
+ void testProductionWiringRecordsOutOfMemory(String kind) throws Exception {
+ ServerHealth health = ServerHealth.getInstance();
+ assertFalse(health.hasOutOfMemoryError());
+ // Real Jersey resources may be constructed on HTTP worker threads. A
thread-local static
+ // mock would not isolate those callers, so restore the real marker only
after Jetty stops.
+ Field marker = ServerHealth.class.getDeclaredField("outOfMemory");
+ marker.setAccessible(true);
+ int port = RESTUtils.findAvailablePort(0, 0);
+ ServerConfig config = new ServerConfig();
+ config.loadFromMap(
+ Map.of(
+ GravitinoServer.WEBSERVER_CONF_PREFIX +
JettyServerConfig.WEBSERVER_HTTP_PORT.getKey(),
+ String.valueOf(port),
+ GravitinoServer.WEBSERVER_CONF_PREFIX +
JettyServerConfig.CUSTOM_FILTERS.getKey(),
+ FailingFilter.class.getName(),
+ Configs.ENTITY_RELATIONAL_JDBC_BACKEND_PATH.getKey(),
+ temporaryDirectory.toPath().resolve("jdbc").toString(),
+ AuxiliaryServiceManager.GRAVITINO_AUX_SERVICE_PREFIX
+ + AuxiliaryServiceManager.AUX_SERVICE_NAMES,
+ ""),
+ entry -> true);
+ GravitinoServer server = new GravitinoServer(config,
GravitinoEnv.getInstance());
+ try {
+ server.initialize();
+ server.register(FailingResource.class);
+ server.start();
+ HttpClient client =
HttpClient.newBuilder().connectTimeout(Duration.ofSeconds(10)).build();
+ assertHealth(client, port, 200);
+ for (String ordinary : new String[] {"ordinary", "filter-ordinary"}) {
+ assertEquals(500, get(client, port, "/api/oom-wiring/" +
ordinary).statusCode());
+ assertHealth(client, port, 200);
+ assertFalse(health.hasOutOfMemoryError());
+ }
+ HttpResponse<String> failure = get(client, port, "/api/oom-wiring/" +
kind);
+ assertEquals(500, failure.statusCode(), failure.body());
+ if (kind.equals("oom")) {
+ // Prove that Jersey instantiated the production class-registered
ErrorExceptionMapper.
+ JsonNode body =
ObjectMapperProvider.objectMapper().readTree(failure.body());
+ assertEquals("OutOfMemoryError", body.path("type").asText());
+ assertEquals(
+ "Server error while processing request:
java.lang.OutOfMemoryError: Requested array size exceeds VM limit",
+ body.path("message").asText());
+ }
+ assertTrue(health.hasOutOfMemoryError());
+ assertHealth(client, port, 503);
+ assertEquals(200, get(client, port,
"/api/oom-wiring/warm").statusCode());
+ assertHealth(client, port, 503);
+ } finally {
+ try {
+ server.stop();
+ } finally {
+ marker.setBoolean(health, false);
+ }
+ }
+ }
+
+ private static void fail(String kind) {
+ switch (kind) {
+ case "oom":
+ throw new OutOfMemoryError("Requested array size exceeds VM limit");
+ case "wrapped":
+ throw new IllegalStateException(new OutOfMemoryError("Java heap
space"));
+ case "ordinary":
+ throw new IllegalStateException("ordinary failure");
+ default:
+ break;
+ }
+ }
+
+ private static void assertHealth(HttpClient client, int port, int status)
throws Exception {
+ for (String path : HEALTH_PATHS) {
+ HttpResponse<String> response = get(client, port, path);
+ assertEquals(status, response.statusCode(), path + ": " +
response.body());
+ JsonNode body =
ObjectMapperProvider.objectMapper().readTree(response.body());
+ assertEquals(status == 200 ? "up" : "down",
body.path("status").asText(), path);
+ if (status == 503) {
+ assertEquals(1, body.path("checks").size(), path);
+ assertEquals("jvm", body.path("checks").get(0).path("name").asText(),
path);
+ assertEquals("down",
body.path("checks").get(0).path("status").asText(), path);
+ }
+ }
+ }
+
+ private static HttpResponse<String> get(HttpClient client, int port, String
path)
+ throws Exception {
+ return client.send(
+ HttpRequest.newBuilder(URI.create("http://127.0.0.1:"; + port + path))
+ .timeout(Duration.ofSeconds(10))
+ .header("Accept", "application/vnd.gravitino.v1+json")
+ .GET()
+ .build(),
+ HttpResponse.BodyHandlers.ofString());
+ }
+}
diff --git
a/server/src/test/java/org/apache/gravitino/server/web/mapper/TestErrorExceptionMapper.java
b/server/src/test/java/org/apache/gravitino/server/web/mapper/TestErrorExceptionMapper.java
index 78d264db46..ca0960a343 100644
---
a/server/src/test/java/org/apache/gravitino/server/web/mapper/TestErrorExceptionMapper.java
+++
b/server/src/test/java/org/apache/gravitino/server/web/mapper/TestErrorExceptionMapper.java
@@ -21,6 +21,7 @@ package org.apache.gravitino.server.web.mapper;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import org.apache.gravitino.dto.responses.ErrorResponse;
+import org.apache.gravitino.server.web.ServerHealth;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -35,7 +36,9 @@ public class TestErrorExceptionMapper {
new NoClassDefFoundError("catalog class"), new
AssertionError("assertion")
}) {
error.initCause(new IllegalStateException("root cause"));
- try (Response response = new ErrorExceptionMapper().toResponse(error)) {
+ ServerHealth health = new ServerHealth();
+ try (Response response = new
ErrorExceptionMapper(health).toResponse(error)) {
+ Assertions.assertEquals(error instanceof OutOfMemoryError,
health.hasOutOfMemoryError());
Assertions.assertEquals(500, response.getStatus());
Assertions.assertEquals(MediaType.APPLICATION_JSON_TYPE,
response.getMediaType());
ErrorResponse entity = (ErrorResponse) response.getEntity();
diff --git
a/server/src/test/java/org/apache/gravitino/server/web/rest/TestHealthOperations.java
b/server/src/test/java/org/apache/gravitino/server/web/rest/TestHealthOperations.java
index 9f5262a2fb..996d99a439 100644
---
a/server/src/test/java/org/apache/gravitino/server/web/rest/TestHealthOperations.java
+++
b/server/src/test/java/org/apache/gravitino/server/web/rest/TestHealthOperations.java
@@ -29,6 +29,7 @@ import javax.ws.rs.core.Response;
import org.apache.gravitino.EntityStore;
import org.apache.gravitino.dto.HealthCheckDTO;
import org.apache.gravitino.dto.responses.HealthResponse;
+import org.apache.gravitino.server.web.ServerHealth;
import org.junit.jupiter.api.Test;
import org.mockito.Mockito;
@@ -39,7 +40,11 @@ public class TestHealthOperations {
}
private HealthOperations newOps(EntityStore store, long probeTimeoutMs) {
- return new HealthOperations() {
+ return newOps(store, probeTimeoutMs, new ServerHealth());
+ }
+
+ private HealthOperations newOps(EntityStore store, long probeTimeoutMs,
ServerHealth health) {
+ return new HealthOperations(health) {
@Override
EntityStore getEntityStore() {
return store;
@@ -146,4 +151,74 @@ public class TestHealthOperations {
HealthResponse response = new HealthResponse();
assertThrows(IllegalArgumentException.class, response::validate);
}
+ /** All probes stay down after OOM, even if the entity store would answer
successfully. */
+ @Test
+ public void testAllEndpointsStayDownAfterOutOfMemory() {
+ ServerHealth health = new ServerHealth();
+ EntityStore store = Mockito.mock(EntityStore.class);
+ HealthOperations ops = newOps(store, 2000L, health);
+ health.recordFailure(new OutOfMemoryError("Metaspace"));
+ for (Response response : new Response[] {ops.live(), ops.ready(),
ops.health()}) {
+ try (Response ignored = response) {
+ assertEquals(503, response.getStatus());
+ HealthResponse body = (HealthResponse) response.getEntity();
+ assertEquals(HealthCheckDTO.Status.DOWN, body.getStatus());
+ assertEquals("jvm", body.getChecks().get(0).getName());
+ assertEquals(
+ "OutOfMemoryError; restart required",
+ body.getChecks().get(0).getDetails().get("reason"));
+ }
+ }
+ Mockito.verifyNoInteractions(store);
+ }
+
+ /** A captured executor OOM must poison liveness too, and must not recover
on the next probe. */
+ @Test
+ public void testProbeOutOfMemoryIsSticky() throws IOException {
+ ServerHealth health = new ServerHealth();
+ EntityStore store = Mockito.mock(EntityStore.class);
+ Mockito.when(store.exists(Mockito.any(), Mockito.any()))
+ .thenThrow(new OutOfMemoryError("Java heap space"))
+ .thenReturn(false);
+ HealthOperations ops = newOps(store, 2000L, health);
+ try (Response first = ops.ready();
+ Response live = ops.live();
+ Response next = ops.ready()) {
+ assertEquals(503, first.getStatus());
+ assertEquals(503, live.getStatus());
+ assertEquals(503, next.getStatus());
+ }
+ Mockito.verify(store).exists(Mockito.any(), Mockito.any());
+ }
+
+ /** OOM from a timed-out probe must still be recorded when the abandoned
task finishes. */
+ @Test
+ public void testOutOfMemoryAfterProbeTimeout() throws IOException {
+ ServerHealth health = new ServerHealth();
+ EntityStore store = Mockito.mock(EntityStore.class);
+ CountDownLatch release = new CountDownLatch(1);
+ Mockito.when(store.exists(Mockito.any(), Mockito.any()))
+ .thenAnswer(
+ invocation -> {
+ release.await();
+ throw new OutOfMemoryError("Metaspace");
+ });
+ try {
+ try (Response response = newOps(store, 100L, health).ready()) {
+ assertEquals(503, response.getStatus());
+ HealthResponse body = (HealthResponse) response.getEntity();
+ assertEquals("timeout",
body.getChecks().get(0).getDetails().get("reason"));
+ }
+ } finally {
+ release.countDown();
+ }
+ // If the first worker has not recorded the OOM yet, this probe queues
behind it.
+ HealthOperations ops = newOps(store, 2000L, health);
+ try (Response ready = ops.ready();
+ Response live = ops.live()) {
+ assertEquals(503, ready.getStatus());
+ assertEquals(503, live.getStatus());
+ assertTrue(health.hasOutOfMemoryError());
+ }
+ }
}
diff --git
a/server/src/test/java/org/apache/gravitino/server/web/rest/TestOutOfMemoryHealthHttp.java
b/server/src/test/java/org/apache/gravitino/server/web/rest/TestOutOfMemoryHealthHttp.java
new file mode 100644
index 0000000000..46779eb92d
--- /dev/null
+++
b/server/src/test/java/org/apache/gravitino/server/web/rest/TestOutOfMemoryHealthHttp.java
@@ -0,0 +1,247 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.gravitino.server.web.rest;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.mockito.ArgumentMatchers.any;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.mockStatic;
+import static org.mockito.Mockito.when;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import com.fasterxml.jackson.databind.ObjectMapper;
+import java.io.IOException;
+import java.net.URI;
+import java.net.http.HttpClient;
+import java.net.http.HttpRequest;
+import java.net.http.HttpResponse;
+import javax.servlet.Filter;
+import javax.servlet.FilterChain;
+import javax.servlet.FilterConfig;
+import javax.servlet.ServletException;
+import javax.servlet.ServletRequest;
+import javax.servlet.ServletResponse;
+import javax.servlet.http.HttpServlet;
+import javax.servlet.http.HttpServletRequest;
+import javax.servlet.http.HttpServletResponse;
+import javax.ws.rs.GET;
+import javax.ws.rs.Path;
+import javax.ws.rs.PathParam;
+import javax.ws.rs.core.Response;
+import javax.ws.rs.ext.ExceptionMapper;
+import org.apache.gravitino.Config;
+import org.apache.gravitino.EntityStore;
+import org.apache.gravitino.rest.RESTUtils;
+import org.apache.gravitino.server.web.HealthAliasServlet;
+import org.apache.gravitino.server.web.JettyServer;
+import org.apache.gravitino.server.web.JettyServerConfig;
+import org.apache.gravitino.server.web.ObjectMapperProvider;
+import org.apache.gravitino.server.web.OutOfMemoryErrorListener;
+import org.apache.gravitino.server.web.ServerHealth;
+import org.apache.gravitino.server.web.mapper.ErrorExceptionMapper;
+import org.glassfish.jersey.jackson.JacksonFeature;
+import org.glassfish.jersey.server.ResourceConfig;
+import org.glassfish.jersey.servlet.ServletContainer;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.mockito.MockedStatic;
+
+/** Exercises an OOM on a separate HTTP request before checking every health
alias. */
+class TestOutOfMemoryHealthHttp {
+ private static final String[] HEALTH_PATHS = {
+ "/api/health",
+ "/api/health/live",
+ "/api/health/ready",
+ "/health",
+ "/health/live",
+ "/health/ready",
+ "/health.html"
+ };
+
+ /** A warm endpoint and failing requests, independent of the health
resources. */
+ @Path("/test/{kind}")
+ public static class FailingResource {
+ /**
+ * Simulates errors without exhausting the test worker's memory.
+ *
+ * @param kind the outcome to simulate
+ * @return the successful or ordinary server error response
+ */
+ @GET
+ public Response get(@PathParam("kind") String kind) {
+ switch (kind) {
+ case "oom":
+ throw new OutOfMemoryError("Metaspace");
+ case "wrapped":
+ throw new IllegalStateException(new OutOfMemoryError("Java heap
space"));
+ case "ordinary":
+ return Response.serverError().build();
+ default:
+ return Response.ok().build();
+ }
+ }
+ }
+
+ /** Simulates a resource-specific mapper that consumes wrapped errors. */
+ public static class RuntimeMapper implements
ExceptionMapper<IllegalStateException> {
+ /** {@inheritDoc} */
+ @Override
+ public Response toResponse(IllegalStateException error) {
+ return Response.serverError().build();
+ }
+ }
+
+ /** Supplies a reachable store so an unrelated readiness failure cannot mask
the result. */
+ public static class TestHealthResource extends HealthOperations {
+ private final EntityStore store;
+
+ TestHealthResource(ServerHealth health, EntityStore store) {
+ super(health);
+ this.store = store;
+ }
+
+ @Override
+ EntityStore getEntityStore() {
+ return store;
+ }
+
+ @Override
+ long getProbeTimeoutMs() {
+ return 2000L;
+ }
+ }
+
+ @Test
+ void directOomPoisonsAllHealthPathsWhileWarmEndpointsStillRespond() throws
Exception {
+ assertHealthAfterFailure("oom");
+ }
+
+ @Test
+ void mappedWrappedOomAlsoPoisonsAllHealthPaths() throws Exception {
+ assertHealthAfterFailure("wrapped");
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"filter-oom", "filter-wrapped", "servlet-oom",
"servlet-wrapped"})
+ void errorsOutsideJerseyPoisonAllHealthPaths(String kind) throws Exception {
+ assertHealthAfterFailure(kind);
+ }
+
+ private void assertHealthAfterFailure(String failureKind) throws Exception {
+ ServerHealth health = new ServerHealth();
+ EntityStore store = mock(EntityStore.class);
+ when(store.exists(any(), any())).thenReturn(false);
+ ResourceConfig config =
+ new ResourceConfig()
+ .register(new TestHealthResource(health, store))
+ .register(FailingResource.class)
+ .register(new OutOfMemoryErrorListener(health))
+ .register(new ErrorExceptionMapper(health))
+ .register(RuntimeMapper.class)
+ .register(ObjectMapperProvider.class)
+ .register(JacksonFeature.class);
+ int port = RESTUtils.findAvailablePort(0, 0);
+ Config serverConfig = new Config(false) {};
+ serverConfig.set(JettyServerConfig.WEBSERVER_HTTP_PORT, port);
+ JettyServer server = new JettyServer();
+ // Capture an independent health state when the production filter is
constructed.
+ try (MockedStatic<ServerHealth> state = mockStatic(ServerHealth.class)) {
+ state.when(ServerHealth::getInstance).thenReturn(health);
+ server.initialize(JettyServerConfig.fromConfig(serverConfig),
"oom-test", false);
+ }
+ server.addServlet(new ServletContainer(config), "/api/*");
+ server.addServlet(new HealthAliasServlet(), "/health/*");
+ server.addServlet(new HealthAliasServlet(), "/health.html");
+ server.addFilter(
+ new Filter() {
+ /** {@inheritDoc} */
+ @Override
+ public void init(FilterConfig config) {}
+
+ /** {@inheritDoc} */
+ @Override
+ public void destroy() {}
+
+ /** {@inheritDoc} */
+ @Override
+ public void doFilter(ServletRequest request, ServletResponse
response, FilterChain chain)
+ throws IOException, ServletException {
+ String path = ((HttpServletRequest) request).getRequestURI();
+ if (path.endsWith("filter-oom")) {
+ throw new OutOfMemoryError("Metaspace");
+ }
+ if (path.endsWith("filter-wrapped")) {
+ throw new IllegalStateException(new OutOfMemoryError("Java heap
space"));
+ }
+ if (path.endsWith("filter-ordinary")) {
+ throw new IllegalStateException("ordinary filter failure");
+ }
+ chain.doFilter(request, response);
+ }
+ },
+ "/*");
+ server.addServlet(
+ new HttpServlet() {
+ /** {@inheritDoc} */
+ @Override
+ protected void doGet(HttpServletRequest request, HttpServletResponse
response)
+ throws IOException {
+ if (request.getRequestURI().endsWith("servlet-oom")) {
+ throw new OutOfMemoryError("Metaspace");
+ }
+ if (request.getRequestURI().endsWith("servlet-wrapped")) {
+ throw new IllegalStateException(new OutOfMemoryError("Java heap
space"));
+ }
+ throw new IOException("ordinary servlet failure");
+ }
+ },
+ "/outside/*");
+ try {
+ server.start();
+ HttpClient client = HttpClient.newHttpClient();
+ for (String path : HEALTH_PATHS) {
+ assertEquals(200, get(client, port, path).statusCode(), path);
+ }
+ assertEquals(500, get(client, port, "/api/test/ordinary").statusCode());
+ assertEquals(500, get(client, port,
"/api/test/filter-ordinary").statusCode());
+ assertEquals(500, get(client, port, "/outside/ordinary").statusCode());
+ assertEquals(200, get(client, port, "/api/health").statusCode());
+ String failurePath = failureKind.startsWith("servlet-") ? "/outside/" :
"/api/test/";
+ assertEquals(500, get(client, port, failurePath +
failureKind).statusCode());
+ // This is the production failure mode: a successful warm endpoint is
not proof of recovery.
+ assertEquals(200, get(client, port, "/api/test/warm").statusCode());
+ for (String path : HEALTH_PATHS) {
+ HttpResponse<String> response = get(client, port, path);
+ assertEquals(503, response.statusCode(), path);
+ JsonNode body = new ObjectMapper().readTree(response.body());
+ assertEquals("down", body.path("status").asText());
+ assertEquals("jvm", body.path("checks").get(0).path("name").asText());
+ }
+ } finally {
+ server.stop();
+ }
+ }
+
+ private HttpResponse<String> get(HttpClient client, int port, String path)
throws Exception {
+ return client.send(
+ HttpRequest.newBuilder(URI.create("http://localhost:"; + port +
path)).GET().build(),
+ HttpResponse.BodyHandlers.ofString());
+ }
+}
