Hello Peter, To start with, great initiative! But I echo the same concern raised about creating too many extension points can compromise the autoscaler functionality. When we proposed FLIP-514 [1] and a custom evaluator, the aim was twofold: provide the required extension point and ship practical strategies as pluggables. At the same time, we wanted to preserve flexibility for advanced, highly specific scenarios—like predictive scaling—that differ by ecosystem, platform, and company. The custom evaluator strikes that balance was the thought process: it lets users adjust the evaluated metrics—especially TARGET_DATA_RATE—that drive the scale-factor calculation, enabling useful out-of-the-box behavior without constraining bespoke implementations. One of the desired outcomes we had set for FLIP-514 was to ship a scheduled-scaling strategy as a pluggable, leveraging a baseline period and explicit scheduled windows to drive planned capacity changes. I’ve been away since last month due to personal commitments. I plan to resume after first week of September and will complete the scheduled-scaling plugin to wrap up the custom evaluator. Having the ScalingRealizer pluggable (https://github.com/apache/flink-kubernetes-operator/pull/1020/files), definitely sounds helpful for certain scenarios. But I totally agree with the general approach suggested by Gyula, about solving specific issues independently in the "best possible way" and then coming to a good solution regarding pluggability that could be foundation for future use-cases.
Thanks and Regards Pradeepta > On 26 Aug 2025, at 6:05 PM, ctrlaltd...@icloud.com.invalid > <ctrlaltd...@icloud.com.INVALID> wrote: > > From the ScalingRealizer, I think having before/after hooks for > `realizeParallelismOverrides` and `realizeConfigOverrides` would be good. We > can support these hooks from plugins, thoughts? > > > Best, > Diljeet(DJ) Singh > > On 2025/08/26 08:24:33 Maximilian Michels wrote: >> Hi Peter, >> >> First of all, this is a great initiative. Flink Autoscaling definitely >> needs more points of extension. We recently added support for hooking >> into the metric evaluation (FLIP-514), but clearly that is just one >> extension point. >> >> That said, I think we will need to revise the approach a bit. I'm not >> sure, we should be replacing core components. As Gyula mentioned, >> replacing those will easily break the entire autoscaler. Instead, we >> should be adding extension points which allow for meaningful additions >> without breaking the scaling logic. There is already the option to >> replace the entire autoscaling module, if users really want to roll >> out a completely custom version. >> >> What usually works best is to formulate the use case first, then >> figure out what autoscaler customization would be necessary to >> implement the use case. >> >> As for making the ScalingRealizer pluggable >> (https://github.com/apache/flink-kubernetes-operator/pull/1020/files), >> I do think that makes sense for some scenarios. >> >> Cheers, >> Max >> >> On Tue, Aug 26, 2025 at 8:59 AM Gyula Fóra <gy...@gmail.com> wrote: >>> >>> Hi Peter & Diljeet! >>> >>> My general feedback is that we should try to introduce extension plugins >>> instead of plugins that completely replace key parts of the autoscaler code. >>> >>> Let me give you a concrete example through FLIP-514 and FLIP-543 using the >>> MetricsEvaluator pluggability. >>> The MetricsEvaluator in the autoscaler is responsible for >>> evaluating/deriving/calculating metrics from the collected metrics. It has >>> to calculate everything in a more or less specific way otherwise other >>> parts of the autoscaler that depend on these metrics may not work. It >>> doesn't seem very practical/resonable to completely reimplement this just >>> because someone wants to extend the logic, this is extremely error prone >>> and fragile especially if the autoscaler logic later evolves. >>> >>> FLIP-514 takes the approach to extend the metric evaluator with a new >>> method that allows users to at the end modify the evaluated metrics and >>> define custom ones. This is the right approach here as it makes a new >>> extension very simple to build and maintain without interfering with >>> existing logic. >>> >>> The approach in FLIP-543 and in Diljeet's example PR takes the replacement >>> approach to completely substitute the entire parts of the implementation >>> (the entire evaluator, scaling realizer etc). I think this is not very good >>> for either the community or the actual user. From a community perspective >>> it makes it harder to extend the logic with nice small additions and from a >>> user's perspective it is very error probe if the operator autoscaler logic >>> changes as it basically exposes a lot of internal logic on a user interface. >>> >>> So at this point, -1 for the approach in FLIP-543 from my side, but I >>> would love to hear the opinion of others as well. >>> >>> Cheers >>> Gyula >>> >>> On Mon, Aug 25, 2025 at 11:44 PM Peter Huang <hu...@gmail.com> wrote: >>>> >>>> Hi Diljeet, >>>> >>>> Yes, I think we have similar requirements to make autoscaler even more >>>> powerful to handle some customized requirements. >>>> The quick PoC makes sense to me. Let's get some more feedback from the >>>> community. >>>> >>>> >>>> >>>> Best Regards >>>> Peter Huang >>>> >>>> >>>> >>>> On Mon, Aug 25, 2025 at 2:37 PM Peter Huang <hu...@gmail.com> >>>> wrote: >>>> >>>>> Just try to combine the discussion into one thread. >>>>> >>>>> @Diljeet Singh >>>>> Posted a quick PoC for the proposal >>>>> https://github.com/apache/flink-kubernetes-operator/pull/1020. >>>>> >>>>> >>>>> >>>>> >>>>> On Mon, Aug 25, 2025 at 7:52 AM Peter Huang <hu...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Community, >>>>>> >>>>>> Our org has been heavily using the Flink autoscaling algorithm. It >>>>>> greatly reduced our operation overhead and improved cost efficiency >>>>>> as users always over provision resources when onboard. Recently, we have >>>>>> had some requirements to customize the auto scaling algorithm >>>>>> for different scenarios, for example, during the holiday season large but >>>>>> predictable traffic spike, increase checkpoint interval together with >>>>>> scale up for streaming ingestion use cases. >>>>>> >>>>>> We search through the discussion about the topic in the mail list >>>>>> including the existing FLIP-514 >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler>. >>>>>> Looks like the discussion is not finalized yet. >>>>>> To accelerate the process, we adopt and combine the >>>>>> existing opinions from the community and create a proposal in FLIP-543 >>>>>> <https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm>. >>>>>> The basic idea >>>>>> is to make some core components of autoscaler pluggable, for example, >>>>>> MetricsCollector, Metrics Evaluator, and ScalingRealizer, at the same >>>>>> keep the core logic skeleton (which is already well justified in large >>>>>> amount of users) of autoscaler untouched. >>>>>> >>>>>> Looking forward to any feedback and opinions on FLIP-543. >>>>>> >>>>>> [1] >>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-543%3A+Support+Customized+Autoscale+Algorithm >>>>>> [2] >>>>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-514%3A+Custom+Evaluator+plugin+for+Flink+Autoscaler >>>>>> [3] Other related discussion thread >>>>>> >>>>>> https://lists.apache.org/thread/749l74z1h5jylkxrw3rtjmxcj2t9p7ws >>>>>> >>>>>> https://lists.apache.org/thread/mcd7jcn4kz6oqtyqq5hfycjf9mqh6c53 >>>>>> >>>>>> >>>>>> Best Regards >>>>>> Peter Huang >>>>>> >>>>>
