Siyao Meng created HDDS-17288:
---------------------------------
Summary: Intermittent failure in test-repair-tools.sh waiting for
OM container to stop
Key: HDDS-17288
URL: https://issues.apache.org/jira/browse/HDDS-17288
Project: Apache Ozone
Issue Type: Sub-task
Components: Tools, test
Reporter: Siyao Meng
Assignee: Siyao Meng
{code:title=acceptance (tools), ozonesecure-ha/test-repair-tools.sh}
Ratis-Transaction-Repair :: Test recovering from OM crash due to t... | PASS |
Waiting for container 'ozonesecure-ha-om1-1' to stop...
Timeout: Container 'ozonesecure-ha-om1-1' did not stop within 60 seconds.
ERROR: Test execution of ozonesecure-ha/test-repair-tools.sh is FAILED!!!!
{code}
h3. Root cause
{{repair/ratis-transaction-repair.robot}} uses Byteman to make
{{OMBucketCreateRequest#validateAndUpdateCache}} throw, so each OM terminates
from its apply transaction thread. The OM shutdown hook then blocks in
{{omRatisServer.stop()}} until the shutdown hook timeout
({{ozone.service.shutdown.timeout}}, default 60s) fires. This happens on every
run, also on green {{master}} runs:
{code}
20:36:42,114 [om1-OMStateMachineApplyTransactionThread - 0] ERROR
ratis.OzoneManagerStateMachine: Terminating with exit status 1: ...
20:36:46,182 [shutdown-hook-0] INFO server.GrpcServicesImpl: om1: Shutdown
GrpcServerProtocolService successfully
20:37:42,125 [Thread-17] WARN util.ShutdownHookManager: ShutdownHook
'OzoneManagerStarter$OMStarterHelper$$Lambda...' timeout,
java.util.concurrent.TimeoutException
{code}
So each OM container needs a bit over 60s to exit after the crash, while
{{repair_and_restart_om}} in {{test-repair-tools.sh}} waits at most 60s. The
test passes only when the wait starts a few seconds after the crash. In the
failed run, JVM pauses on the runner made om1's hook time out 62s after the
crash, and the wait expired first.
h3. Fix
Wait up to 120s for the OM container to stop, so the wait covers the 60s
shutdown hook timeout.
- https://github.com/apache/ozone/actions/runs/39665766133/job/119456369743
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
