Hi all, I would like to start a discussion about FLIP-608: Flink on Ray — Ray Resource Backend For Flink [1]. This FLIP proposes a new Ray resource backend to allow running Flink jobs natively on Ray. The hybrid approach takes the best of both worlds —battle-tested streaming and batch unific computing plus heterogeneous scheduling and the Ray AI eco-system (Data/Serve/Train). One can therefore setup the entire real-time AI workflow in a single cluster with higher resource efficiency and lower maintenance cost. The proposal focuses on supporting running Flink jobs on Ray in Application/Session mode by translating Flink resource requirements to Ray. The Flink core mechanisms like RPC, shuffle, checkpointing, failover and slot management are preserved. A follow-up FLIP will focus on the interoperability between PyFlink and Ray by supporting conversion between Flink Dataframe/Datastream/Table and Ray Dataset. Looking forward to your feedback! [1] https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286348/FLIP-608+Flink+on+Ray+%E2%80%94+Ray+Resource+Backend+For+Flink
Best, Zhanghao Chen
