Hi Venkata, Let’s keep this discussion about the FLIP and not other unrelated topics.
You are welcome to post the question in a new thread . Thanks Gyula Sent from my iPhone > On 29 Mar 2025, at 02:58, Venkatakrishnan Sowrirajan <vsowr...@asu.edu> wrote: > > Hi Pradeepta, > > Thanks for the nice proposal. I am curious, how does the current autoscaler > as well as the newly proposed custom evaluator plugin work for batch > execution mode? > > Regards > Venkata krishnan > > >> On Thu, Mar 20, 2025 at 11:33 PM Pradeepta Choudhury >> <pchoudhur...@apple.com.invalid> wrote: >> >> Hi Max, Good morning! Thanks so much for your review and feedback! >> >> I agree on proposal being generic and might not benefit a normal user >> without having a decent understanding of how autoscaler’s logic works. >> I had actually planned to contribute some predictive implementations using >> the custom evaluator plugin. That said, having some built-in commonly used >> predictive logic as part of the autoscaler sounds like a great idea. We can >> probably introduce a scaling metric to denote predicted metric for >> evaluated metrics to increase transparency that way?! I’ll work on a >> follow-up for this. Thanks! >> >> Thanks >> Pradeepta >> >> >>>> On 20 Mar 2025, at 10:36 PM, Maximilian Michels <m...@apache.org> wrote: >>> >>> Hi Pradeepta, >>> >>> Generally, the proposal looks good to me. The only issue I see is that >>> it is fairly generic. On the one hand, it allows plugging in >>> additional evaluation logic into the autoscaler, which allows for many >>> new ways to extend the current autoscaling logic. On the other hand, >>> normal users won't benefit from this. I wish we could build some >>> predictive logic directly into the autoscaler. So far, we refrained >>> from doing so because we wanted to get the basics right first >>> (something that most Flink autoscaling solutions never got around to). >>> >>> +1 for the proposal as it is meaningful. Hopefully, we can also make >>> some progress towards a built-in predictive logic as part of a new >>> FLIP :) >>> >>> -Max >>> >>> On Thu, Mar 20, 2025 at 8:28 AM Gyula Fóra <gyula.f...@gmail.com> wrote: >>>> >>>> Hi! >>>> >>>> I think this is a very nice proposal, +1 from me! >>>> >>>> The only practical challenge is to actually be able to identify what job >>>> vertex is what in the job graph to make good predictions / feed in >> external >>>> information but this is definitely out of scope here. >>>> The autoscaler in general needs to have a good way to associate user >> code / >>>> operators with vertex ids for these types of use-cases that we need to >>>> investigate as a followup. >>>> >>>> Cheers >>>> Gyula >>>> >>>> On Wed, Mar 12, 2025 at 6:24 PM Pradeepta Choudhury >>>> <pchoudhur...@apple.com.invalid> wrote: >>>> >>>>> Hi All, >>>>> >>>>> I would like to start a discussion of FLIP-514: Custom Evaluator plugin >>>>> for Flink Autoscaler [1]. >>>>> >>>>> Feel free to share your thoughts and suggestions to make this feature >>>>> better. >>>>> >>>>> >>>>> [1] >>>>> >> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/FLINK/FLIP-514*3A*Custom*Evaluator*plugin*for*Flink*Autoscaler__;JSsrKysrKw!!IKRxdwAv5BmarQ!a8jmPJ4UU5fJGp6O_sfSJJZ83bAz23j3xUcWGIFnnmw1qsSC51WvTx8hWy0zDa1S7Y4Ceesdfd1tsYzhP-vFSlVHZNiaihur$ >>>>> >>>>> >>>>> >>>>> Thanks >>>>> Pradeepta >> >>
