[
https://issues.apache.org/jira/browse/HDDS-17288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17288:
------------------------------
Status: Patch Available (was: Open)
> Intermittent failure in test-repair-tools.sh waiting for OM container to stop
> -----------------------------------------------------------------------------
>
> Key: HDDS-17288
> URL: https://issues.apache.org/jira/browse/HDDS-17288
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test, Tools
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code:title=acceptance (tools), ozonesecure-ha/test-repair-tools.sh}
> Ratis-Transaction-Repair :: Test recovering from OM crash due to t... | PASS |
> Waiting for container 'ozonesecure-ha-om1-1' to stop...
> Timeout: Container 'ozonesecure-ha-om1-1' did not stop within 60 seconds.
> ERROR: Test execution of ozonesecure-ha/test-repair-tools.sh is FAILED!!!!
> {code}
> h3. Root cause
> {{repair/ratis-transaction-repair.robot}} uses Byteman to make
> {{OMBucketCreateRequest#validateAndUpdateCache}} throw, so each OM terminates
> from its apply transaction thread. The OM shutdown hook then blocks in
> {{omRatisServer.stop()}} until the shutdown hook timeout
> ({{ozone.service.shutdown.timeout}}, default 60s) fires. This happens on
> every run, also on green {{master}} runs:
> {code}
> 20:36:42,114 [om1-OMStateMachineApplyTransactionThread - 0] ERROR
> ratis.OzoneManagerStateMachine: Terminating with exit status 1: ...
> 20:36:46,182 [shutdown-hook-0] INFO server.GrpcServicesImpl: om1: Shutdown
> GrpcServerProtocolService successfully
> 20:37:42,125 [Thread-17] WARN util.ShutdownHookManager: ShutdownHook
> 'OzoneManagerStarter$OMStarterHelper$$Lambda...' timeout,
> java.util.concurrent.TimeoutException
> {code}
> So each OM container needs a bit over 60s to exit after the crash, while
> {{repair_and_restart_om}} in {{test-repair-tools.sh}} waits at most 60s. The
> test passes only when the wait starts a few seconds after the crash. In the
> failed run, JVM pauses on the runner made om1's hook time out 62s after the
> crash, and the wait expired first.
> h3. Fix
> Wait up to 120s for the OM container to stop, so the wait covers the 60s
> shutdown hook timeout.
> - https://github.com/apache/ozone/actions/runs/39665766133/job/119456369743
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
