Thanks a lot for reviewing the FLIP! Absolutely agree with the point of having some general strategies inside autoscaler itself. cc. @Max since you had similar thoughts on this, I’d love to get your inputs as well on following :)!
One of the general strategies that could be beneficial to users could be scheduled scaling. Where users could define time range based cron schedule/s. When this scaling schedule is active, we could use a target data rate derived from the scalingHistory from the previous matching scheduled durations?! Some normalisation strategies could also be beneficial at certain scenarios like percentile based, time weight based etc. But I feel, some form of it can be achieved using configs like scale up and scale down factors, scale down interval etc. For more specific requirements would it be more practical to implement via the plugin? In the experiment we had performed, linear regression based models were not proving very accurate and for more complex predictive algorithms, the challenge increases as they typically require a larger window of historical metrics, pre-trained models, and caching mechanisms. Given these additional requirements, would it be more practical to let users implement them as a separate service and merge it via the custom evaluator hook? Looking forward to hearing your thoughts! Also if you think some other strategies can be generalised, please add! Thanks Pradeepta > On 2 Apr 2025, at 7:00 AM, Rui Fan <1996fan...@gmail.com> wrote: > > +1 for this proposal, it allows advanced Users to introduce some specific > strategies. > > Also, if some strategies are general enough, it's better to put them inside > the autoscaler. > > Best, > Rui > > On Sat, Mar 29, 2025 at 3:09 PM <gyula.f...@gmail.com> wrote: > >> Hi Venkata, >> >> Let’s keep this discussion about the FLIP and not other unrelated topics. >> >> You are welcome to post the question in a new thread . >> >> Thanks >> Gyula >> >> Sent from my iPhone >> >>> On 29 Mar 2025, at 02:58, Venkatakrishnan Sowrirajan <vsowr...@asu.edu> >> wrote: >>> >>> Hi Pradeepta, >>> >>> Thanks for the nice proposal. I am curious, how does the current >> autoscaler >>> as well as the newly proposed custom evaluator plugin work for batch >>> execution mode? >>> >>> Regards >>> Venkata krishnan >>> >>> >>>> On Thu, Mar 20, 2025 at 11:33 PM Pradeepta Choudhury >>>> <pchoudhur...@apple.com.invalid> wrote: >>>> >>>> Hi Max, Good morning! Thanks so much for your review and feedback! >>>> >>>> I agree on proposal being generic and might not benefit a normal user >>>> without having a decent understanding of how autoscaler’s logic works. >>>> I had actually planned to contribute some predictive implementations >> using >>>> the custom evaluator plugin. That said, having some built-in commonly >> used >>>> predictive logic as part of the autoscaler sounds like a great idea. We >> can >>>> probably introduce a scaling metric to denote predicted metric for >>>> evaluated metrics to increase transparency that way?! I’ll work on a >>>> follow-up for this. Thanks! >>>> >>>> Thanks >>>> Pradeepta >>>> >>>> >>>>>> On 20 Mar 2025, at 10:36 PM, Maximilian Michels <m...@apache.org> >> wrote: >>>>> >>>>> Hi Pradeepta, >>>>> >>>>> Generally, the proposal looks good to me. The only issue I see is that >>>>> it is fairly generic. On the one hand, it allows plugging in >>>>> additional evaluation logic into the autoscaler, which allows for many >>>>> new ways to extend the current autoscaling logic. On the other hand, >>>>> normal users won't benefit from this. I wish we could build some >>>>> predictive logic directly into the autoscaler. So far, we refrained >>>>> from doing so because we wanted to get the basics right first >>>>> (something that most Flink autoscaling solutions never got around to). >>>>> >>>>> +1 for the proposal as it is meaningful. Hopefully, we can also make >>>>> some progress towards a built-in predictive logic as part of a new >>>>> FLIP :) >>>>> >>>>> -Max >>>>> >>>>> On Thu, Mar 20, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> >> wrote: >>>>>> >>>>>> Hi! >>>>>> >>>>>> I think this is a very nice proposal, +1 from me! >>>>>> >>>>>> The only practical challenge is to actually be able to identify what >> job >>>>>> vertex is what in the job graph to make good predictions / feed in >>>> external >>>>>> information but this is definitely out of scope here. >>>>>> The autoscaler in general needs to have a good way to associate user >>>> code / >>>>>> operators with vertex ids for these types of use-cases that we need to >>>>>> investigate as a followup. >>>>>> >>>>>> Cheers >>>>>> Gyula >>>>>> >>>>>> On Wed, Mar 12, 2025 at 6:24 PM Pradeepta Choudhury >>>>>> <pchoudhur...@apple.com.invalid> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> I would like to start a discussion of FLIP-514: Custom Evaluator >> plugin >>>>>>> for Flink Autoscaler [1]. >>>>>>> >>>>>>> Feel free to share your thoughts and suggestions to make this feature >>>>>>> better. >>>>>>> >>>>>>> >>>>>>> [1] >>>>>>> >>>> >> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-514*3A*Custom*Evaluator*plugin*for*Flink*Autoscaler__;JSsrKysrKw!!IKRxdwAv5BmarQ!a8jmPJ4UU5fJGp6O_sfSJJZ83bAz23j3xUcWGIFnnmw1qsSC51WvTx8hWy0zDa1S7Y4Ceesdfd1tsYzhP-vFSlVHZNiaihur$ >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks >>>>>>> Pradeepta >>>> >>>> >>
