Please give input for below. From: Kamal Mittal via user <user@flink.apache.org> Sent: 16 May 2025 06:42 To: Zhanghao Chen <zhanghao.c...@outlook.com>; user@flink.apache.org Subject: RE: Flink task manager PODs autoscaling - K8s installation
Thanks for describing. Just to know that why K8s HPA doesn't work well with Flink? Any limitations? Also instead of HPA, Kubernetes Event Driven auto scaler (KEDA) can be used? From: Zhanghao Chen <zhanghao.c...@outlook.com<mailto:zhanghao.c...@outlook.com>> Sent: 14 May 2025 06:47 To: user@flink.apache.org<mailto:user@flink.apache.org>; Kamal Mittal <kamal.mit...@ericsson.com<mailto:kamal.mit...@ericsson.com>> Subject: Re: Flink task manager PODs autoscaling - K8s installation Hi Kamal, Autoscaling consists of two parts: 1. policy: when should we scale up/down the job and for how much? 2. mechanism: how to rescale the job. The adaptive scheduler just gives you the mechanism part, which allows rescaling the job without kill-and-start the whole job. You shall use it together with the operator or K8s HPA which provides the policy part (there's a subtlety that K8s HPA needs to be used with the Reactive Mode [1], a special mode for Adaptive Scheduler). It is totally fine to apply Autoscaling without the K8s Operator at the cost of extra complexity, e.g. the just mentioned K8s HPA + the Reactive Mode pair. However, the K8s HPA policy is more targeted towards stateless workloads without complex internal topology, and it does not perform quite well on Flink. The K8s Operator's approach [2] is more suited. You can mimic its algorithm to get the suggested operator parallelisms, and use [3] to apply the new parallelisms. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#reactive-mode [2] https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/autoscaler/ [3] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/#externalized-declarative-resource-management Best, Zhanghao Chen ________________________________ From: Kamal Mittal via user <user@flink.apache.org<mailto:user@flink.apache.org>> Sent: Monday, May 12, 2025 13:35 To: user@flink.apache.org<mailto:user@flink.apache.org> <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Flink task manager PODs autoscaling - K8s installation Hello, For Flink Task Manager K8s PODs autoscaling, it is mandatory to use Flink Kubernetes Operator/HPA or only flink provided scheduler is sufficient as below? jobmanager.scheduler: adaptive Also what is the additional work done by below configuration? job.autoscaler.enabled: "true" Please give documentation link if available without Flink Kubernetes Operator? Rgds, Kamal
