[
https://issues.apache.org/jira/browse/HDDS-17648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy reassigned HDDS-17648:
-------------------------------------
Assignee: Andrey Yarovoy
> Key write hangs when its Ratis client is closed while a watch-for-commit is
> still pending
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-17648
> URL: https://issues.apache.org/jira/browse/HDDS-17648
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> Since HDDS-17491, `TestKeyInputStream#readAfterReplication(boolean)[2]`
> intermittently fails on master with
> `readAfterReplication(boolean) timed out after 5 minutes` (integration
> (client), run 41300247417 on 20da41e535).
> Locally it fails in 2 of 6 runs on 20da41e535 and in 0 of 6 runs on the
> commit before it (4ade0f1e9b). The test
> thread is stuck writing the test key, not reading it.
> **What happens** (from the test log of a failing local run):
> 1. The write goes to a pipeline that still includes the datanode the previous
> test case stopped, so a
> watch-for-commit fails (`ALL_COMMITTED way commit failed`,
> `RaftRetryFailureException`), and
> `BlockOutputStream.handleFlush(true)` fails ("Failed to flush").
> 2. `handleFlush(true)` calls `cleanup(false)` in its `finally` block, which
> releases the Ratis client, while the watch
> for the other putBlock of the same key is still being retried by that client.
> 3. The released client has no references left, and its `XceiverClientManager`
> cache entry has already expired:
> `expireAfterAccess` counts from the last cache access, which was the start of
> the write. The periodic
> `cleanUp()` added by HDDS-17491 removes the entry, and the client is closed.
> 4. The pending watch then fails:
> ```
> GrpcClientRpc - client-3F9447C48097: Failed
> RaftClientRequest:client-3F9447C48097->...
> org.apache.ratis.protocol.exceptions.AlreadyClosedException:
> client-3F9447C48097 is already CLOSED
> XceiverClientRatis - ALL_COMMITTED way commit failed on pipeline ...
> AlreadyClosedException: org.apache.ratis.client.impl.RaftClientImpl@144b857d
> is closed
> ```
> 5. `XceiverClientRatis.watchForCommit` handles the failure by sending a
> MAJORITY_COMMITTED watch on the same,
> now closed, client, from inside its `exceptionally` handler. The future it
> returns is never completed.
> 6. `BlockOutputStream.waitForAllPendingFlushes()` waits for all flush futures
> with an untimed
> `allPendingFlushFutures.get()`, so the write never returns. In the test,
> JUnit interrupts it after 5 minutes;
> the write then retries on a new block and succeeds.
> Which of the two ways leaves the future incomplete is not confirmed yet:
> either Ratis returns a watch future that
> never completes on a closed client, or the call throws inside the
> `exceptionally` handler, where the exception is
> swallowed and `replyFuture` is never completed.
> **Why it starts with HDDS-17491:** releasing a client while its async work is
> still pending (step 2) predates
> HDDS-17491. Before it, an expired entry was only removed during a later cache
> access, which did not happen in this
> window in a single-client test, so the client stayed open. HDDS-17491 removes
> expired entries every idle threshold
> (10 s by default), so the window is hit regularly. In a busy client, any
> cache access after the threshold could
> already trigger the same hang before HDDS-17491.
> **Impact:** a key write that hits a failed datanode can hang indefinitely
> instead of retrying on another block.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
