Weijie Guo created FLINK-30189:
----------------------------------
Summary: HsSubpartitionFileReader may load data that has been
consumed from memory
Key: FLINK-30189
URL: https://issues.apache.org/jira/browse/FLINK-30189
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.16.0, 1.17.0
Reporter: Weijie Guo
Assignee: Weijie Guo
In order to solve the problem that data cannot be read from the disk correctly
after failover, we changed the calculation logical of the buffer's readable
state in FLINK-29238. Buffers that are greater than consumingOffset and have
been released can be pre-load from file. However, the update of consumingOffset
is asynchronous, If it lags behind the actual consumption progress, the buffer
will have a chance to be load from the disk again.
IMO, we can record the consumed status of buffer by each consumer in the
InternalRegion. Only the buffers that have not been consumed and have been
released will be considered as readable. In the case of failover, a new
consumerId will be generated, so all buffers will be considered as unconsumed
and can be correctly read from the disk too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
