+1 for this proposal. We have been prototyping this feature internally at Alibaba for a couple of months. Yuxin, I think we can also publish the prototype codes so the community can better understand and help with it.
Best, Xintong On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <tanyuxinw...@gmail.com> wrote: > Hi all, > > I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle > integration with > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between > memory, disk, and > remote storage to improve performance and job stability. Concurrently, > Apache Celeborn > provides a stable, performant, scalable remote shuffle service. This > integration proposal is to > harness the benefits from both hybrid shuffle and Celeborn simultaneously. > > Note that this proposal has two parts. > 1. The Flink-side modifications are in FLIP-459[1]. > 2. The Celeborn-side changes are in CIP-6[2]. > > Looking forward to everyone's feedback and suggestions. Thank you! > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > [2] > > https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn > > Best, > Yuxin >
