Hi, Venkatakrishnan, Thanks for joining the discussion. We appreciate your interest in contributing to the work. Once the FLIP and CIP proposals have been approved, we will create some JIRA tickets in Flink and Celeborn projects. Please feel free to take a look at the tickets and select any that resonate with your interests.
Best, Yuxin Venkatakrishnan Sowrirajan <vsowr...@asu.edu> 于2024年5月31日周五 23:11写道： > Thanks for this FLIP. We are also interested in learning/contributing to > the hybrid shuffle integration with celeborn for batch executions. > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <tanyuxinw...@gmail.com> wrote: > > > Hi, Xintong, > > > > > I think we can also publish the prototype codes so the > > community can better understand and help with it. > > > > Ok, I agree on the point. I will prepare and publish the code > > recently. > > > > Rui, > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded. > > > > Thanks for the reminder. I've updated the images. > > > > > > Best, > > Yuxin > > > > > > Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道： > > > > > Thanks Yuxin for driving this proposal! > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded. > > > > > > [1] > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$ > > > > > > Best, > > > Rui > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > +1 for this proposal. > > > > > > > > We have been prototyping this feature internally at Alibaba for a > > couple > > > of > > > > months. Yuxin, I think we can also publish the prototype codes so the > > > > community can better understand and help with it. > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <tanyuxinw...@gmail.com> > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I would like to start a discussion on FLIP-459 Support Flink hybrid > > > > shuffle > > > > > integration with > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions > between > > > > > memory, disk, and > > > > > remote storage to improve performance and job stability. > > Concurrently, > > > > > Apache Celeborn > > > > > provides a stable, performant, scalable remote shuffle service. > This > > > > > integration proposal is to > > > > > harness the benefits from both hybrid shuffle and Celeborn > > > > simultaneously. > > > > > > > > > > Note that this proposal has two parts. > > > > > 1. The Flink-side modifications are in FLIP-459[1]. > > > > > 2. The Celeborn-side changes are in CIP-6[2]. > > > > > > > > > > Looking forward to everyone's feedback and suggestions. Thank you! > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-459*3A*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;JSsrKysrKysr!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9MaOGE7hQ$ > > > > > [2] > > > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$ > > > > > > > > > > Best, > > > > > Yuxin > > > > > > > > > > > > > > >
