Yingjie Cao created FLINK-25774:
-----------------------------------
Summary: Restrict the maximum number of buffers can be used per
result partition for blocking shuffle
Key: FLINK-25774
URL: https://issues.apache.org/jira/browse/FLINK-25774
Project: Flink
Issue Type: Sub-task
Components: Runtime / Network
Reporter: Yingjie Cao
Fix For: 1.15.0
Currently, for blocking shuffle, the maximum number of buffers can be used per
result partition is Integer.MAX_VALUE. For hash-shuffle, the maximum number of
buffers to be used is (numSubpartition + 1), because the hash-shuffle
implementation always flush the previous buffer after a new buffer is added, so
setting the maximum number of buffers can be used to Integer.MAX_VALUE is
meaningless. For sort-shuffle, if too many buffers are taken by one result
partition, other result partitions and input gates may spend too much time
waiting for buffers which can influence performance. This ticket aims to
restrict the maximum number of buffers can be used per result partition and the
selected value is an empirical one based on the TPC-DS test results.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
