Sergey Soldatov created HDDS-16241:
--------------------------------------
Summary: gRPC deadline kills long-lived block streams after 30
seconds and the client never recovers
Key: HDDS-16241
URL: https://issues.apache.org/jira/browse/HDDS-16241
Project: Apache Ozone
Issue Type: Bug
Components: Ozone Client
Affects Versions: 2.3.0
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
XceiverClientGrpc.initStreamRead sets a gRPC deadline of
ozone.client.read.timeout (30 seconds) on the streaming ReadBlock call. A gRPC
deadline bounds the whole call, and since HDDS-13974 one stream stays open per
block for the lifetime of the input stream, so every stream is cancelled with
DEADLINE_EXCEEDED 30 seconds after it opens, even when healthy. The client then
never recovers: the failed reader is never rebuilt, so every later read of that
block replays the same stale error for as long as the file stays open; a
failure arriving while the reader waits for a response is mistaken for a normal
end of stream and shows up as a wrong early EOF; and the wrapped exception
chain is not seen as retryable, so the client never fails over to another
datanode. Applications that keep files open for a long time, such as HBase
RegionServers, are hit hardest: in an HBase on Ozone YCSB test about 42 percent
of all reads failed this way.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
