yuqi1129 opened a new pull request, #13067:
URL: https://github.com/apache/gravitino/pull/13067
### What changes were proposed in this pull request?
Record observed `OutOfMemoryError`s and make the main server's health,
liveness, readiness, and root aliases return HTTP 503 with a `jvm` failure
until process restart. Preserve the existing checks and response shape while
healthy.
Record direct and wrapped OOM at Jersey exception handling, shared request
helpers, health-probe tasks (including failures after timeout), and the
existing Jetty worker uncaught-exception handler. The marker does not retain
the throwable or allocate a visited-cause collection.
### Why are the changes needed?
After Metaspace or heap OOM, already-loaded endpoints and entity-store
lookups can keep working while catalog operations fail. The current health
checks therefore keep advertising the server as healthy and allow traffic to
reach a degraded process.
Fix: #13066
### Does this PR introduce _any_ user-facing change?
Yes. Once OOM is observed, all main-server health endpoints report 503 with
`jvm: down` and `OutOfMemoryError; restart required`. Successful requests do
not clear the state, and subsequent health checks skip the entity store. No new
configuration.
Detection covers errors reaching the instrumented server boundaries. It
cannot detect OOM swallowed entirely by an unrelated background executor or
connector. Iceberg and Lance REST health checks are unchanged. Severe memory
exhaustion may prevent any HTTP response.
### How was this patch tested?
- Unit tests for direct/wrapped OOM, cyclic causes, ordinary errors, request
helpers, the error mapper, Jetty worker handling, and health-probe OOM
before/after timeout.
- HTTP regression tests using Jetty and Jersey: warm every health alias,
send a request that throws direct or wrapped OOM, verify another endpoint still
returns 200, then verify all seven health paths return 503. Errors are
simulated; the test JVM is not actually exhausted.
- `spotlessApply` completed.
- Full `:server-common:test` passed (325 tests); full `:server:test` passed
(468 tests). Docker/IT suites were excluded.
```bash
SKIP_DOCKER_TESTS=true ./gradlew :server-common:test :server:test \
-PskipITs -PskipDockerTests=true -PskipWeb=true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
