Hi Dennis, Thanks for the detailed response. This aligns closely with the approach we’ve been exploring.
We already use GKE’s optimize-utilization profile. We haven’t marked TaskManagers as safe-to-evict because we’re still testing a controlled restart process and currently trigger consolidation manually. This gives us more control over which jobs are disrupted, and a better understanding of the problem until we move to an automated approach. Do you have experience comparing controlled restarts with GKE-driven eviction using safe-to-evict, particularly in terms of reclaimed capacity and disruption? Overall, your recommendation reinforces our current approach: shape pods to fit the available machines and use controlled restarts to eliminate accumulated fragmentation, while making recovery as smooth as possible. We’re also open to other approaches the community has found effective at scale. Thanks again, Yanis On 2026/08/28 13:23:53 Dennis-Mircea Ciupitu wrote: > Hi Yanis, > > Based on the information that you provided, I assume you run under this > setup: Flink Kubernetes Operator in native mode, with deployments in > application mode. Having this in mind, I'm framing the following response. > > 1. Resource-shape mismatch > > I see this as a provisioning problem more than a Flink one. Either the node > shape has to match the JM + TM shape, or JM + TM must be designed to run > under the ratio that the node provides. There is no Flink core or operator > feature that fixes this automatically today. > > The operator's memory autotuning goes in that direction, since it moves TM > memory requests toward real usage, meaning that in the end the job will run > at the best CPU/memory ratio, but right now it only runs on scaling > decisions. Decoupling it from scaling is tracked in FLINK-40455 [1], and > once that matures (to include CPU tuning as well), it should help exactly > this case. > > 2. Fragmentation > > There are 2 important things here: > > - The default GKE scheduler prefers emptier / balanced nodes, so it > spreads your TMs, and nothing ever moves them back together. > - Once the TM pods are spread, nothing in Kubernetes ever moves them > back together. What I would try, based on what I know about GKE and how > native mode behaves, is to switch the cluster autoscaler to the packing > profile (optimize-utilization) so new pods land on busy nodes, and by > marking the TM pods as safe to evict and JM pods with the opposite > annotation. Both are plain pod annotations, so the operator's podTemplate > can carry them. Eviction is workable for Flink because the JM notices the > lost TM and requests a replacement, which is then scheduled onto the packed > nodes. > > The real cost is that each job's task on the TM that was hosted on the > consolidated nodes will need to be restarted from a checkpoint. With the > adaptive scheduler in place, this will be made much smoother, since the job > can keep running with fewer slots while the replacement comes up. So it > ends up as a tradeoff between wasted capacity and how much disruption each > job can accept. > > In the long term, I think the operator could play an important role here, > as it already takes jobs through full redeploys for spec upgrades, and > those moments recreate all the TM pods anyway, so they are natural windows > to let the cluster repack without paying any extra disruption. Besides > this, as I said, there will be some nice features added within the operator > that will help those use cases. > > Best, > Dennis > > [1] https://issues.apache.org/jira/browse/FLINK-40455 > > > On Thu, Aug 27, 2026 at 2:14 AM Yanis Djeridi via dev <[email protected]> > wrote: > > > Hi all, > > > > We run hundreds of Flink pipelines across several shared GKE clusters and > > experience significant cluster overhead. By cluster overhead, we mean > > allocatable CPU and memory available on active nodes that remain > > unrequested but cannot be reclaimed because the nodes still host pods. > > > > > > This comes mainly from two sources: > > > > > > - > > > > TaskManager CPU/memory request ratios do not always align with our > > machine shapes, leaving capacity stranded when either CPU or memory is > > exhausted first. > > - > > > > Deployments and autoscaling gradually scatter TaskManagers across nodes. > > Because TaskManagers are bare pods, GKE cannot evict them for > > consolidation, and Kubernetes does not proactively repack them. > > > > > > As a result, partially occupied nodes can remain running indefinitely > > despite substantial aggregate free capacity. > > > > Have others operating large shared Flink clusters encountered this? How do > > you manage resource-shape mismatch and fragmentation? > > > > > > Thanks, > > > > Yanis > > >
