Hi all,
I would like to start a discussion about FLIP-608: Flink on Ray — Ray Resource
Backend For Flink [1].
This FLIP proposes a new Ray resource backend to allow running Flink jobs
natively on Ray. The hybrid approach takes the best of both worlds
—battle-tested streaming and batch unific computing plus heterogeneous
scheduling and the Ray AI eco-system (Data/Serve/Train). One can therefore
setup the entire real-time AI workflow in a single cluster with higher resource
efficiency and lower maintenance cost.
The proposal focuses on supporting running Flink jobs on Ray in
Application/Session mode by translating Flink resource requirements to Ray. The
Flink core mechanisms like RPC, shuffle, checkpointing, failover and slot
management are preserved.
A follow-up FLIP will focus on the interoperability between PyFlink and Ray by
supporting conversion between Flink Dataframe/Datastream/Table and Ray Dataset.
Looking forward to your feedback!
[1]
https://cwiki.apache.org/confluence/spaces/FLINK/pages/449286348/FLIP-608+Flink+on+Ray+%E2%80%94+Ray+Resource+Backend+For+Flink
Best,
Zhanghao Chen
