Hi everyone, This FLIP [1] introduces a new REST API for declaring resource requirements for the Adaptive Scheduler. There seems to be a clear need for this API based on the discussion on the "Reworking the Rescale API" [2] thread.
Before we get started, this work is heavily based on the prototype [3] created by Till Rohrmann, and the FLIP is being published with his consent. Big shoutout to him! Last and not least, thanks to Chesnay and Roman for the initial reviews and discussions. The best start would be watching a short demo [4] that I've recorded, which illustrates newly added capabilities (rescaling the running job, handing back resources to the RM, and session cluster support). The intuition behind the FLIP is being able to define resource requirements ("resource boundaries") externally that the AdaptiveScheduler can navigate within. This is a building block for higher-level efforts such as an external Autoscaler. The natural extension of this work would be to allow to specify per-vertex ResourceProfiles. Looking forward to your thoughts; any feedback is appreciated! [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-291%3A+Externalized+Declarative+Resource+Management [2] https://lists.apache.org/thread/2f7dgr88xtbmsohtr0f6wmsvw8sw04f5 [3] https://github.com/tillrohrmann/flink/tree/autoscaling [4] https://drive.google.com/file/d/1Vp8W-7Zk_iKXPTAiBT-eLPmCMd_I57Ty/view Best, D.