Hi Bianca. I think you are missing the point here. HPA can scale any resource based on the given metric. And, in the example you point to, they are scaling FlinkDeployments. Basically, they would be launching multiple instances of the same pipeline, working on the same target(s). If such parallelism is supported by the resources they are working on.
That would not increase the number of Task Managers, since they are defined within one Flink Deployment. Unless you meant the overall number of TMs, since the number of Flink Deployments should go up. I think you are better off looking into Flink native Kubernetes mode, where Job Manager contacts Kubernetes for provisioning. I think it can be configured to react on the TM load, not just task slot availability. Perhaps look into this: https://github.com/apache/flink-kubernetes-operator/blob/main/examples/autoscaling/autoscaling-dynamic.yaml I am not sure if that can affect the parallelism of your processors, though. And I suspect that would be your goal. Nix. From: Bianca Falcone <bfalc...@medallia.com> Date: Friday, February 28, 2025 at 5:15 PM To: user@flink.apache.org <user@flink.apache.org> Subject: Implement HPA with Flink operator Hi everyone, I been working with flink operator on using HPA with a FlinkDeployment to scale up and down the amount of TaskManagers based on memory consumption and been testing with the following configuration as per documentation: https://github.com/apache/flink-kubernetes-operator/blob/main/examples/hpa/basic-hpa.yaml FlinkDeployment: jobManager: resource: memory: "3Gi" cpu: 2 taskManager: resource: memory: "6Gi" cpu: 3 job: HPA: --- apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: flink-hpa namespace: flink spec: scaleTargetRef: apiVersion: flink.apache.org/v1beta1<http://flink.apache.org/v1beta1> kind: FlinkDeployment name: flink minReplicas: 1 maxReplicas: 5 metrics: - type: Resource resource: name: memory target: type: Utilization averageUtilization: 60 Given that it didn't trigger the creation of the new TaskManagers I removed the HPA and proceed to define the amount of replicas directly on the Flink deployment but it didn't work either: jobManager: resource: memory: "3Gi" cpu: 2 taskManager: resource: replicas: 3 memory: "6Gi" cpu: 3 job: There is any other variables/approaches to take into account to achieve this? Confidentiality Notice: This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. Additionally, this email and any files transmitted with it may not be disseminated, distributed or copied. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. If you are not the intended recipient, you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited. [cid:~WRD0000.jpg]<https://www.medallia.com/experience/>
