Flink relies on JDK's methods to get the amount of resources available for an TM. If you are using an outdated version of JDK (<8u191 for JDK8 with cgroup-v1 based containerization for example), the exact amount of resources available in a containerized environment cannot be retrieved, and the resources of the underlying host are returned instead. Check [1] for more information.
[1] https://jvmaware.com/container-aware-jvm/ Best, Zhanghao Chen ________________________________ From: Oliver Schmied <uncharted...@gmx.at> Sent: Wednesday, September 11, 2024 16:49 To: user@flink.apache.org <user@flink.apache.org> Subject: TaskManager Using more CPU Cores in Kubernetes Deployment than configured Dear Flink Community, I am currently running Apache Flink Stateful Functions (version 3.2.0) on a Kubernetes cluster. I’ve configured the statefun-master and statefun-worker deployments with the following resource limits: resources: requests: memory: "2Gi" cpu: "1" limits: memory: "2Gi" cpu: "1" However, when I check the Flink UI (see screenshot in attachment), I see that the statefun-worker TaskManager is using 8 CPU cores, despite the fact that I’ve set the CPU requests and limits to 1 core in the Kubernetes deployment. Below are the relevant settings in the flink-conf.yaml file: taskmanager.numberOfTaskSlots: 2 classloader.parent-first-patterns.additional: org.apache.flink.statefun;org.apache.kafka;com.google.protobuf state.backend: rocksdb state.backend.rocksdb.timer-service.factory: ROCKSDB state.backend.incremental: true parallelism.default: 1 jobmanager.memory.process.size: 2g taskmanager.memory.process.size: 2g kubernetes.taskmanager.cpu: 1 kubernetes.jobmanager.cpu: 1 taskmanager.cpu.cores: 1 I would appreciate any guidance on why Flink might be utilizing more CPU cores than specified and how I can ensure that the TaskManager adheres to the configured resource limits. Any insights or suggestions would be greatly appreciated. Best regards, Oliver
