Hi Eric,I believe you might be referring to use of the adaptive scheduler which should support these “in-place” scaling operations via:jobmanager.scheduler: adaptiveYou can see the documentation for Elastic Scaling here for additional details and configuration.On Jun 24, 2024, at 11:56 PM, Enric Ot
Hi, Ashish.
Can you confirm whether, on the subtask label page of this sink materializer
node, the input records for each subtask are approximately the same?
If the input records for subtask number 5 are significantly larger compared to
the others, it signifies a serious data skew, and it wou
You can try session mode with only one job, but still with adaptive scheduler
disabled. When stopping a session job, the TMs won't be released immediately
and can be reused later.
Best,
Zhanghao Chen
From: Chetas Joshi
Sent: Tuesday, June 25, 2024 1:59
To: Zhang
Hello,Community:
I??ve recently started using the Flink Kubernetes Operator,and I'd like
to know if CPU and Job Parallelism autoscaling are supported without restarting
the whole job,if it??s supported, please tell me how to configure and deploy
it.
Thanks.
Hello,
After disabling the adaptive scheduler, I was able to have the operator stop
the job with a savepoint, and resume the job from that savepoint after the
upgrade. However I observed that the upgrade life cycle is quite slow as it
takes down and then brings back up all the task managers. I am
Hi Ashish,
Can you check a few things.
1. Is your source broker count also 20 for both topics?
2. You can try increasing the state operation memory and reduce the disk
I/O.
-
- Increase the number of CU resources in a single slot.
- Set optimization parameters:
- task
Hi all,
We are facing backpressure in the flink sql job from the sink and the
backpressure only comes from a single task. This causes the checkpoint to
fail despite enabling unaligned checkpoints and using debloating buffers.
We enabled flamegraph and the task spends most of the time doing rocksdb
