I am
>> temporarily using taskmanager.memory.network.batch-shuffle-read.size in
>> my PR now. Any suggestions about that?
>>
>> Best,
>> Yingjie (Kevin)
>>
>> --------------
>> 发件人：Guowei Ma
>> 日 期：2021年03月09日 17:28:35
&g
-
> 发件人：Guowei Ma
> 日 期：2021年03月09日 17:28:35
> 收件人：曹英杰(北牧)
> 抄 送：Till Rohrmann; Stephan Ewen;
> dev; user; Xintong Song<
> tonysong...@gmail.com>
> 主 题：Re: Re: [DISCUSSION] Introduce a separated memory pool for the TM
> merge shuffle
>
> Hi,
e a proper default value.
>>
>> Best,
>> Yingjie
>>
>> --------------
>> 发件人：Till Rohrmann
>> 日 期：2021年03月05日 23:03:10
>> 收件人：Stephan Ewen
>> 抄 送：dev; user; Xintong Song<
>> ton
han Ewen
> 抄 送：dev; user; Xintong Song<
> tonysong...@gmail.com>; 曹英杰(北牧); Guowei Ma<
> guowei@gmail.com>
> 主 题：Re: [DISCUSSION] Introduce a separated memory pool for the TM merge
> shuffle
>
> Thanks for this proposal Guowei. +1 for it.
>
> Concerning the defaul
Thanks for this proposal Guowei. +1 for it.
Concerning the default size, maybe we can run some experiments and see how
the system behaves with different pool sizes.
Cheers,
Till
On Fri, Mar 5, 2021 at 2:45 PM Stephan Ewen wrote:
> Thanks Guowei, for the proposal.
>
> As discussed offline alrea
Thanks Guowei, for the proposal.
As discussed offline already, I think this sounds good.
One thought is that 16m sounds very small for a default read buffer pool.
How risky do you think it is to increase this to 32m or 64m?
Best,
Stephan
On Fri, Mar 5, 2021 at 4:33 AM Guowei Ma wrote:
> Hi, a
Hi, all
In the Flink 1.12 we introduce the TM merge shuffle. But the out-of-the-box
experience of using TM merge shuffle is not very good. The main reason is
that the default configuration always makes users encounter OOM [1]. So we
hope to introduce a managed memory pool for TM merge shuffle to
