Hi Diljeet, Yes, I think we have similar requirements to make autoscaler even more powerful to handle some customized requirements. The quick PoC makes sense to me. Let's get some more feedback from the community.
Best Regards Peter Huang On Mon, Aug 25, 2025 at 2:37 PM Peter Huang <huangzhenqiu0...@gmail.com> wrote: > Just try to combine the discussion into one thread. > > @Diljeet Singh > Posted a quick PoC for the proposal > https://github.com/apache/flink-kubernetes-operator/pull/1020. > > > > > On Mon, Aug 25, 2025 at 7:52 AM Peter Huang <huangzhenqiu0...@gmail.com> > wrote: > >> Hi Community, >> >> Our org has been heavily using the Flink autoscaling algorithm. It >> greatly reduced our operation overhead and improved cost efficiency >> as users always over provision resources when onboard. Recently, we have >> had some requirements to customize the auto scaling algorithm >> for different scenarios, for example, during the holiday season large but >> predictable traffic spike, increase checkpoint interval together with >> scale up for streaming ingestion use cases. >> >> We search through the discussion about the topic in the mail list >> including the existing FLIP-514 >> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler>. >> Looks like the discussion is not finalized yet. >> To accelerate the process, we adopt and combine the >> existing opinions from the community and create a proposal in FLIP-543 >> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm>. >> The basic idea >> is to make some core components of autoscaler pluggable, for example, >> MetricsCollector, Metrics Evaluator, and ScalingRealizer, at the same >> keep the core logic skeleton (which is already well justified in large >> amount of users) of autoscaler untouched. >> >> Looking forward to any feedback and opinions on FLIP-543. >> >> [1] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm >> [2] >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler >> [3] Other related discussion thread >> >> https://lists.apache.org/thread/749l74z1h5jylkxrw3rtjmxcj2t9p7ws >> >> https://lists.apache.org/thread/mcd7jcn4kz6oqtyqq5hfycjf9mqh6c53 >> >> >> Best Regards >> Peter Huang >> >
