Max recently moved the autoscaler logic in a separate submodule, did you see that?
https://github.com/apache/flink-kubernetes-operator/commit/5bb8e9dc4dd29e10f3ba7c8ce7cefcdffbf92da4 Gyula On Thu, Feb 16, 2023 at 10:27 AM Samrat Deb <decordea...@gmail.com> wrote: > Hi , > > *Context:* > Auto Scaling was introduced in Flink as part of FLIP-271[1]. > It discusses one of the important aspects to provide a robust default > scaling algorithm. > a. Ensure scaling yields effective usage of assigned task slots. > b. Ramp up in case of any backlog to ensure it gets processed in a > timely manner > c. Minimize the number of scaling decisions to prevent costly rescale > operation > The flip intends to add an auto scaling framework based on 6 major metrics > and contains different types of threshold to trigger the scaling. > > Thread[2] discusses a different problem: why autoscaler is part of the > operator instead of jobmanager at runtime. > The Community decided to keep the autoscaling logic in the > flink-kubernetes-operator. > > *Proposal: * > In this discussion, I want to put forward a thought of extracting out the > auto scaling logic into a new submodule in flink-kubernetes-operator > repository[3], > which will be independent of any resource manager/Operator. > Currently the Autoscaling algorithm is very tightly coupled with the > kubernetes API. > This makes the autoscaling core algorithm not so easily extensible for > different available resource managers like YARN, Mesos etc. > A Separate autoscaling module inside the flink kubernetes operator will > help other resource managers to leverage the autoscaling logic. > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling > [2] https://lists.apache.org/thread/pvfb3fw99mj8r1x8zzyxgvk4dcppwssz > [3] https://github.com/apache/flink-kubernetes-operator > > > Bests, > Samrat >
