Hi Madan, The next release of Flink Kubernetes Operator will come with a built-in autoscaler. Unfortunately, there's no official support for autoscaling Flink applications on YARN yet. You can take a look at https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling and try building one on your own. FLIP-271: Autoscaling - Apache Flink - Apache Software Foundation<https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling> Motivation. One of the most challenging aspects of running always-on streaming pipeline is the correct sizing of Flink deployments. Too few resources lead to unstable Flink jobs which requires users to scale up their deployments manually. cwiki.apache.org
Best, Zhanghao Chen ________________________________ From: Madan D via user <user@flink.apache.org> Sent: Saturday, January 28, 2023 8:56 To: user@flink.apache.org <user@flink.apache.org> Subject: Rescale Flink - Adjust Parallelism based on TPS Hello Team, I would like to understand if there's a way flink can rescale automatically by adjusting parallelism based on incoming traffic. For example, when a job starts incoming TPS is around 100 which can be processed with parallelism 4, and when TPS increased 1000 I might need to rescale flink to use parallelism 10 or 20. Is there a way we can set min and max parallelism to flink application which can adjust automatically based on incoming load with stop and starting job with new config. I assume Kubernetes automatically does by changing replicas but all our applications are running on yarn as of today. Regards, Madan
