[
https://issues.apache.org/jira/browse/HDDS-16207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz-wo Sze updated HDDS-16207:
------------------------------
Status: In Progress (was: Patch Available)
The pull request is now merged. Thanks, [~ssa] !
> Streaming read hangs until read timeout on out-of-range ReadBlock offsets
> -------------------------------------------------------------------------
>
> Key: HDDS-16207
> URL: https://issues.apache.org/jira/browse/HDDS-16207
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Client, Ozone Datanode
> Affects Versions: 2.3.0
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Major
> Labels: pull-request-available
>
> When a streaming read (StreamBlockInputStream / ReadBlock) requests an offset
> at or past the end of the block, the client does not get an error or an EOF —
> it blocks until the stream read timeout (default PT10S) expires. Three
> defects combine to cause this:
> 1. Client pre-read overshoot. StreamBlockInputStream.readBlock() adds the
> pre-read size (up to 32MB) to the required length without clamping against
> blockLength, so requestedLength can run past the physical end of the block.
> The next pre-read request then goes out with an offset beyond EOF — on a
> perfectly normal read near the end of a block.
> 2. Datanode silent failure. For such a request,
> KeyValueHandler.readBlockImpl computes requiredLength <= 0 and simply falls
> through the read loop: no data response, no error, no half-close. The stream
> stays open and idle. A misaligned variant is also possible: an offset past
> EOF whose checksum-aligned floor is still inside the block streams back data
> that ends before the requested offset, which leaves the client waiting for
> bytes that never arrive.
> 3. Client drops streamed errors. Even when the datanode does report a
> problem, StreamingReader ignores it: onError() only handles CANCELLED
> StatusRuntimeExceptions (any other gRPC status is silently discarded without
> failing the stream), and onNext() reads the payload without checking the
> response result, so an error response proto (e.g. CONTAINER_NOT_FOUND, token
> verification failure) is treated as an empty data message.
> In every case the pending poll() on the response queue just times out,
> surfacing as a 10-second stall / TimeoutIOException instead of an immediate
> error.
> This was discovered during testing of HDDS-15422 using YCSB HBase workload.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
