yuqi1129 opened a new issue, #13044:
URL: https://github.com/apache/gravitino/issues/13044
### What would you like to be improved?
`BaseIT.startServer()` decides the deploy-mode server is ready by polling
`/metrics` until an HTTP server answers:
```java
Awaitility.await()
.atMost(60, TimeUnit.SECONDS)
.pollInterval(1, TimeUnit.SECONDS)
.until(() -> HttpUtils.isHttpServerUp(checkServerUrl));
```
Nothing checks that the process answering is the one
`GravitinoITUtils.startGravitinoServer()` just launched. Any other Gravitino
already listening on that port satisfies the probe, so the suite runs against a
stranger with unrelated configuration.
This is easy to hit: a previous integration-test run that was interrupted
leaves a server behind, and `gravitino.sh` fails to launch a replacement
whenever a port it needs is taken. The health check then passes on the leftover
process.
The resulting failures point away from the cause. In my case the leftover
server had authorization enabled, so `createMetalake` came back as:
```
ForbiddenException: Forbidden error :User 'anonymous' is not authorized to
perform
operation 'createMetalake' : Only service admins can create metalakes
```
even though the conf under test had `gravitino.authorization.enable = false`
and `gravitino.authorization.serviceAdmins = anonymous`. I spent several rounds
chasing the authorization configuration before finding the real story in
`logs/gravitino-server.out`, where our own JVM had never started:
```
ERROR: transport error 202: bind failed: Address already in use
ERROR: JDWP Transport dt_socket failed to initialize, TRANSPORT_INIT(510)
```
### How should we improve?
Make the readiness probe identify the instance, not just the port. A few
options, cheapest first:
- Have `startGravitinoServer()` fail loudly when the launch script cannot
start a JVM, rather than leaving the outcome to the port probe.
- Give the probe an identity to match: for instance have the server write
its PID to `$GRAVITINO_HOME/logs`, or pass a per-run nonce and expose it, and
require the probe to see that value before declaring the server up.
- Detect a pre-existing listener on the configured port before launching and
fail with a message naming the port and the process, so the cause is visible
immediately.
Any of these turns a misleading downstream failure into an accurate one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
