Anton Kalashnikov created FLINK-23457:
-----------------------------------------
Summary: Sending the buffer of the right size for broadcast
Key: FLINK-23457
URL: https://issues.apache.org/jira/browse/FLINK-23457
Project: Flink
Issue Type: Sub-task
Reporter: Anton Kalashnikov
It is not enough to know just the number of available buffers (credits) for the
downstream because the size of these buffers can be different. So we are
proposing to resolve this problem in the following way: If the downstream
buffer size is changed then the upstream should send the buffer of the size not
greater than the new one regardless of how big the current buffer on the
upstream. (pollBuffer should receive parameters like bufferSize and return
buffer not greater than it)
Downstream will be able to support any buffer size < max buffer size, so it
should be just good enough to request BufferBuilder with new size after getting
announcement, and leaving existing BufferBuilder/BufferConsumers unchanged. In
other words code in {{PipelinedSubpartition(View)}} doesn’t need to be changed
(apart of forwarding new buffer size to the {{BufferWritingResultPartition}}).
All buffer size adjustments can be implemented exclusively in
{{BufferWritingResultPartition}}.
If different downstream subtasks have different throughput and hence different
desired buffer sizes, then a single upstream subtask has to support having two
different subpartitions with different buffer sizes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
