[
https://issues.apache.org/jira/browse/HDDS-17288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17288:
------------------------------
Description:
{code:title=acceptance (tools), ozonesecure-ha/test-repair-tools.sh}
Ratis-Transaction-Repair :: Test recovering from OM crash due to t... | PASS |
Waiting for container 'ozonesecure-ha-om1-1' to stop...
Timeout: Container 'ozonesecure-ha-om1-1' did not stop within 60 seconds.
ERROR: Test execution of ozonesecure-ha/test-repair-tools.sh is FAILED!!!!
{code}
h3. Root cause
{{repair/ratis-transaction-repair.robot}} uses Byteman to make
{{OMBucketCreateRequest#validateAndUpdateCache}} throw, so each OM terminates
from its apply transaction thread. The OM shutdown hook then blocks in
{{omRatisServer.stop()}} until the shutdown hook timeout
({{ozone.service.shutdown.timeout}}, default 60s) fires. This happens on every
run, also on green {{master}} runs:
{code}
20:36:42,114 [om1-OMStateMachineApplyTransactionThread - 0] ERROR
ratis.OzoneManagerStateMachine: Terminating with exit status 1: ...
20:36:46,182 [shutdown-hook-0] INFO server.GrpcServicesImpl: om1: Shutdown
GrpcServerProtocolService successfully
20:37:42,125 [Thread-17] WARN util.ShutdownHookManager: ShutdownHook
'OzoneManagerStarter$OMStarterHelper$$Lambda...' timeout,
java.util.concurrent.TimeoutException
{code}
So each OM container needs a bit over 60s to exit after the crash, while
{{repair_and_restart_om}} in {{test-repair-tools.sh}} waits at most 60s. The
test passes only when the wait starts a few seconds after the crash. In the
failed run, JVM pauses on the runner made om1's hook time out 62s after the
crash, and the wait expired first.
h3. Fix
Set {{ozone.service.shutdown.timeout}} to 20s for the OMs in
{{ozonesecure-ha/debug-tools.yaml}}, so a crashed OM exits well within the
existing 60s wait. The blocked hook is idle until the timeout fires, so the
shorter timeout only shortens that wait.
- https://github.com/apache/ozone/actions/runs/39665766133/job/119456369743
was:
{code:title=acceptance (tools), ozonesecure-ha/test-repair-tools.sh}
Ratis-Transaction-Repair :: Test recovering from OM crash due to t... | PASS |
Waiting for container 'ozonesecure-ha-om1-1' to stop...
Timeout: Container 'ozonesecure-ha-om1-1' did not stop within 60 seconds.
ERROR: Test execution of ozonesecure-ha/test-repair-tools.sh is FAILED!!!!
{code}
h3. Root cause
{{repair/ratis-transaction-repair.robot}} uses Byteman to make
{{OMBucketCreateRequest#validateAndUpdateCache}} throw, so each OM terminates
from its apply transaction thread. The OM shutdown hook then blocks in
{{omRatisServer.stop()}} until the shutdown hook timeout
({{ozone.service.shutdown.timeout}}, default 60s) fires. This happens on every
run, also on green {{master}} runs:
{code}
20:36:42,114 [om1-OMStateMachineApplyTransactionThread - 0] ERROR
ratis.OzoneManagerStateMachine: Terminating with exit status 1: ...
20:36:46,182 [shutdown-hook-0] INFO server.GrpcServicesImpl: om1: Shutdown
GrpcServerProtocolService successfully
20:37:42,125 [Thread-17] WARN util.ShutdownHookManager: ShutdownHook
'OzoneManagerStarter$OMStarterHelper$$Lambda...' timeout,
java.util.concurrent.TimeoutException
{code}
So each OM container needs a bit over 60s to exit after the crash, while
{{repair_and_restart_om}} in {{test-repair-tools.sh}} waits at most 60s. The
test passes only when the wait starts a few seconds after the crash. In the
failed run, JVM pauses on the runner made om1's hook time out 62s after the
crash, and the wait expired first.
h3. Fix
Wait up to 120s for the OM container to stop, so the wait covers the 60s
shutdown hook timeout.
- https://github.com/apache/ozone/actions/runs/39665766133/job/119456369743
> Intermittent failure in test-repair-tools.sh waiting for OM container to stop
> -----------------------------------------------------------------------------
>
> Key: HDDS-17288
> URL: https://issues.apache.org/jira/browse/HDDS-17288
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: test, Tools
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code:title=acceptance (tools), ozonesecure-ha/test-repair-tools.sh}
> Ratis-Transaction-Repair :: Test recovering from OM crash due to t... | PASS |
> Waiting for container 'ozonesecure-ha-om1-1' to stop...
> Timeout: Container 'ozonesecure-ha-om1-1' did not stop within 60 seconds.
> ERROR: Test execution of ozonesecure-ha/test-repair-tools.sh is FAILED!!!!
> {code}
> h3. Root cause
> {{repair/ratis-transaction-repair.robot}} uses Byteman to make
> {{OMBucketCreateRequest#validateAndUpdateCache}} throw, so each OM terminates
> from its apply transaction thread. The OM shutdown hook then blocks in
> {{omRatisServer.stop()}} until the shutdown hook timeout
> ({{ozone.service.shutdown.timeout}}, default 60s) fires. This happens on
> every run, also on green {{master}} runs:
> {code}
> 20:36:42,114 [om1-OMStateMachineApplyTransactionThread - 0] ERROR
> ratis.OzoneManagerStateMachine: Terminating with exit status 1: ...
> 20:36:46,182 [shutdown-hook-0] INFO server.GrpcServicesImpl: om1: Shutdown
> GrpcServerProtocolService successfully
> 20:37:42,125 [Thread-17] WARN util.ShutdownHookManager: ShutdownHook
> 'OzoneManagerStarter$OMStarterHelper$$Lambda...' timeout,
> java.util.concurrent.TimeoutException
> {code}
> So each OM container needs a bit over 60s to exit after the crash, while
> {{repair_and_restart_om}} in {{test-repair-tools.sh}} waits at most 60s. The
> test passes only when the wait starts a few seconds after the crash. In the
> failed run, JVM pauses on the runner made om1's hook time out 62s after the
> crash, and the wait expired first.
> h3. Fix
> Set {{ozone.service.shutdown.timeout}} to 20s for the OMs in
> {{ozonesecure-ha/debug-tools.yaml}}, so a crashed OM exits well within the
> existing 60s wait. The blocked hook is idle until the timeout fires, so the
> shorter timeout only shortens that wait.
> - https://github.com/apache/ozone/actions/runs/39665766133/job/119456369743
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
