It's really an amazing job to fill in the defects of flink in batch shuffle. I really appreciate the work done in io scheduling, the sequential reading of the shuffle reader can greatly improve the disk IO performance and stability. Sort-based shuffle realizes this feature in a concise and efficient way. By the way, the default shuffle implementation in the batch mode of flink is still hash-based, maybe we can consider using the new shuffle implementation by default later. Last but not least, thank Yingjie Cao (Kevin) and Daisy Tsang for publishing this blog.
Lijie Wang <wangdachui9...@gmail.com> 于2021年11月3日周三 下午4:17写道： > Thanks Daisy and Kevin for bringing this blog, it is very helpful for > understanding the principle of sort shuffle. > > > Best, > > Lijie > > Guowei Ma <guowei....@gmail.com> 于2021年11月3日周三 下午2:57写道： > >> >> Thank Daisy& Kevin much for your introduction to the improvement of TM >> blocking shuffle, credit base+io scheduling is indeed a very interesting >> thing. At the same time, I look forward to this as a default setting for tm >> blocking shuffle. >> >> Best, >> Guowei >> >> >> On Wed, Nov 3, 2021 at 2:46 PM Gen Luo <luogen...@gmail.com> wrote: >> >>> Thanks Daisy and Kevin! The benchmark results look really exciting! >>> >>> On Tue, Nov 2, 2021 at 4:38 PM David Morávek <d...@apache.org> wrote: >>> >>>> Thanks Daisy and Kevin for a great write up! ;) Especially the 2nd part >>>> was really interesting, I really like the idea of the single spill file >>>> with a custom scheduling of read requests. >>>> >>>> Best, >>>> D. >>>> >>>> On Mon, Nov 1, 2021 at 10:01 AM Daisy Tsang <da...@ververica.com> >>>> wrote: >>>> >>>>> Hey everyone, we have a new two-part post published on the Apache >>>>> Flink blog about the sort-based blocking shuffle implementation in Flink. >>>>> It covers benchmark results, design and implementation details, and more! >>>>> We hope you like it and welcome any sort of feedback on it. :) >>>>> >>>>> >>>>> https://flink.apache.org/2021/10/26/sort-shuffle-part1.html >>>>> https://flink.apache.org/2021/10/26/sort-shuffle-part2.html >>>>> >>>>
