Re: Network Buffers

2023-06-06 Thread weijie guo
Hi Pritam, The legacy config option `taskmanager.network.numberOfBuffers` is deprecated and will be ignored, so please do not refer to it. The exact size of network memory calculated through `taskmanager.memory.network.fraction`, while also ensuring that it is within the constraints of min and max

Re: Network Buffers

2023-06-06 Thread Hangxiang Yu
Hi, Pritam. IIUC, the number is TM scope and just calculated by "available network memory / buffer size". For example, if the fraction is 0.1, the number may be about ( 8 * 0.1 * 1024 * 1024 * 1024 / 32768). On Tue, Jun 6, 2023 at 3:14 PM Pritam Agarwala wrote: > Thanks for answering Hangxiang!

Re: Network Buffers

2023-06-06 Thread Pritam Agarwala
Thanks for answering Hangxiang! Still confused. How did Flink get this number 22773 ? Couldn't find the default value of "taskmanager.network.numberOfBuffers" config. and According to the formula it should be around 1000. ( #slots-per-TM^2 * #TMs * 4 = 4^2 * 16 * 4 = 1024) I have a total of 6000

Re: Network Buffers

2023-06-05 Thread Hangxiang Yu
Hi, Pritam. This error message indicates that the current configuration of the network buffer is not enough to handle the current workload. > What is the meaning of this exception (The total number of network buffers > is currently set to 22773 of 32768 bytes each)? > This just provides some infor