flink-kubernetes-operator auto-tuning

[Ifat Afek (Nokia) via user]

Hi,

I’m trying to understand the behavior flink auto-tuning.
I wrote a simple pipeline that reads from kafka and stores events in a state 
that keeps growing. The pipeline is not busy, it has a low input rate, the only 
problem is with the memory.

I see that initially, flink-kubernetes-operator configures the pipeline memory 
as follows:

2024-08-15 06:36:51,059 o.a.f.k.o.l.AuditUtils [INFO 
][nwdaf-edge/pipeline-mece-ip-session] >>> Event  | Info| SPECCHANGED | 
UPGRADE change(s) detected (Diff: FlinkDeployment
Spec[taskManager.resource.memory : 1g -> 775639740, 
flinkConfiguration.taskmanager.memory.jvm-metaspace.size : null -> 127315831 
bytes, flinkConfiguration.pipeline.jobvertex-parallelism-over
rides : null -> 
9f09ef6ac7374093337e9834d0187fdd:2,ec567f3cb83b94e2e606155604dd608f:2,09b5b5f6bcf99a2d0b90160db0a18af0:2,9940eb4ca57add3596d0bc686a58cd2e:2,95c07c0f4a30d6ba565d4901b0ed9289:2
,10d9de4416be230c379396ba05a3261c:2,a6eb160f7f31aaa477e53b8363b2d263:2, 
flinkConfiguration.taskmanager.memory.task.heap.size : 80m -> null, 
flinkConfiguration.taskmanager.memory.process.size
: null -> 775639740 bytes, flinkConfiguration.taskmanager.memory.network.min : 
32m -> 76416 kb, flinkConfiguration.taskmanager.memory.network.max : 64m -> 
76416 kb, flinkConfiguration.taskm
anager.memory.jvm-overhead.fraction : null -> 0.26, 
flinkConfiguration.taskmanager.memory.managed.size : 50m -> null, 
flinkConfiguration.taskmanager.memory.framework.heap.size : null -> 0 by
tes, flinkConfiguration.taskmanager.memory.managed.fraction : null -> 0.0]), 
starting reconciliation.


However, I don’t see any change/update in the memory configuration later, after 
the state grows.
Is it something that is supposed to be handled?
Is the flink-kubernetes-operator expected to update the memory configurations, 
even if the pipeline does not appear as “busy”?
Does anyone have an example of a case where the memory configuration changes 
over time?

Thanks,
Ifat

Apache Beam Summit 2024 is September 4 2024!

[Talat Uyarer via user]

Hey Flink Users,

Heads up! The Apache Beam Summit 2024 is happening next month( September
04, 2024) at the Google Campus in Sunnyvale, CA. It's two days packed with
learning and networking with the Beam community.

Key highlights:

- Improving Stability for Running Python SDK with Flink Runner: improve the
stability of our Flink application using the Python Beam SDK runner, with a
particular focus on memory tuning.
- Beam for Large-Scale ML Inference: See how Google uses Beam to handle
massive ML workloads.
- Project Shield: Learn how Beam helps protect against DDoS attacks and
safeguard critical information.
- MLOps with Generative AI: Get insights into scaling MLOps with large
models and resource management.

Full program and details: https://beamsummit.org/

Talat