Hi Peter, First of all, this is a great initiative. Flink Autoscaling definitely needs more points of extension. We recently added support for hooking into the metric evaluation (FLIP-514), but clearly that is just one extension point.
That said, I think we will need to revise the approach a bit. I'm not sure, we should be replacing core components. As Gyula mentioned, replacing those will easily break the entire autoscaler. Instead, we should be adding extension points which allow for meaningful additions without breaking the scaling logic. There is already the option to replace the entire autoscaling module, if users really want to roll out a completely custom version. What usually works best is to formulate the use case first, then figure out what autoscaler customization would be necessary to implement the use case. As for making the ScalingRealizer pluggable (https://github.com/apache/flink-kubernetes-operator/pull/1020/files), I do think that makes sense for some scenarios. Cheers, Max On Tue, Aug 26, 2025 at 8:59 AM Gyula Fóra <gyula.f...@gmail.com> wrote: > > Hi Peter & Diljeet! > > My general feedback is that we should try to introduce extension plugins > instead of plugins that completely replace key parts of the autoscaler code. > > Let me give you a concrete example through FLIP-514 and FLIP-543 using the > MetricsEvaluator pluggability. > The MetricsEvaluator in the autoscaler is responsible for > evaluating/deriving/calculating metrics from the collected metrics. It has to > calculate everything in a more or less specific way otherwise other parts of > the autoscaler that depend on these metrics may not work. It doesn't seem > very practical/resonable to completely reimplement this just because someone > wants to extend the logic, this is extremely error prone and fragile > especially if the autoscaler logic later evolves. > > FLIP-514 takes the approach to extend the metric evaluator with a new method > that allows users to at the end modify the evaluated metrics and define > custom ones. This is the right approach here as it makes a new extension very > simple to build and maintain without interfering with existing logic. > > The approach in FLIP-543 and in Diljeet's example PR takes the replacement > approach to completely substitute the entire parts of the implementation (the > entire evaluator, scaling realizer etc). I think this is not very good for > either the community or the actual user. From a community perspective it > makes it harder to extend the logic with nice small additions and from a > user's perspective it is very error probe if the operator autoscaler logic > changes as it basically exposes a lot of internal logic on a user interface. > > So at this point, -1 for the approach in FLIP-543 from my side, but I would > love to hear the opinion of others as well. > > Cheers > Gyula > > On Mon, Aug 25, 2025 at 11:44 PM Peter Huang <huangzhenqiu0...@gmail.com> > wrote: >> >> Hi Diljeet, >> >> Yes, I think we have similar requirements to make autoscaler even more >> powerful to handle some customized requirements. >> The quick PoC makes sense to me. Let's get some more feedback from the >> community. >> >> >> >> Best Regards >> Peter Huang >> >> >> >> On Mon, Aug 25, 2025 at 2:37 PM Peter Huang <huangzhenqiu0...@gmail.com> >> wrote: >> >> > Just try to combine the discussion into one thread. >> > >> > @Diljeet Singh >> > Posted a quick PoC for the proposal >> > https://github.com/apache/flink-kubernetes-operator/pull/1020. >> > >> > >> > >> > >> > On Mon, Aug 25, 2025 at 7:52 AM Peter Huang <huangzhenqiu0...@gmail.com> >> > wrote: >> > >> >> Hi Community, >> >> >> >> Our org has been heavily using the Flink autoscaling algorithm. It >> >> greatly reduced our operation overhead and improved cost efficiency >> >> as users always over provision resources when onboard. Recently, we have >> >> had some requirements to customize the auto scaling algorithm >> >> for different scenarios, for example, during the holiday season large but >> >> predictable traffic spike, increase checkpoint interval together with >> >> scale up for streaming ingestion use cases. >> >> >> >> We search through the discussion about the topic in the mail list >> >> including the existing FLIP-514 >> >> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler>. >> >> Looks like the discussion is not finalized yet. >> >> To accelerate the process, we adopt and combine the >> >> existing opinions from the community and create a proposal in FLIP-543 >> >> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm>. >> >> The basic idea >> >> is to make some core components of autoscaler pluggable, for example, >> >> MetricsCollector, Metrics Evaluator, and ScalingRealizer, at the same >> >> keep the core logic skeleton (which is already well justified in large >> >> amount of users) of autoscaler untouched. >> >> >> >> Looking forward to any feedback and opinions on FLIP-543. >> >> >> >> [1] >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm >> >> [2] >> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler >> >> [3] Other related discussion thread >> >> >> >> https://lists.apache.org/thread/749l74z1h5jylkxrw3rtjmxcj2t9p7ws >> >> >> >> https://lists.apache.org/thread/mcd7jcn4kz6oqtyqq5hfycjf9mqh6c53 >> >> >> >> >> >> Best Regards >> >> Peter Huang >> >> >> >
