Hi Vignesh,
if I understand you correctly, then you have a job like:
KafkaSources(parallelism = 64) => Mapper(parallelism = 16) => something else
Moreover, you probably have slot sharing enabled which means that a
KafkaSource and a Mapper can be deployed into the same slot.
So what happens befo
I think currently flink doesn't support your case, and another idea is that
you can set the parallelism of all operators to 64, then it will be evenly
distributed to the two taskmanagers.
Vignesh Ramesh 于2021年3月25日周四 上午1:05写道：
> Hi Matthias,
>
> Thanks for your reply. In my case, yes the upstrea
Hi Matthias,
Thanks for your reply. In my case, yes the upstream operator for the
operator which is not distributed evenly among task managers is a flink
Kafka connector with a rebalance(shuffling).
Regards,
Vignesh
On Tue, 23 Mar, 2021, 6:48 pm Matthias Pohl, wrote:
> There was a similar disc
There was a similar discussion recently in this mailing list about
distributing the work onto different TaskManagers. One finding Xintong
shared there [1] was that the parameter cluster.evenly-spread-out-slots is
used to evenly allocate slots among TaskManagers but not how the tasks are
actually di
Hi Vignesh,
are you trying to achieve an even distribution of tasks for this one
operator that has the parallelism set to 16? Or do you observe the
described behavior also on a job level?
I'm adding Chesnay to the thread as he might have more insights on this
topic.
Best,
Matthias
On Mon, Mar 22,
Hello Everyone,
Can someone help me with a solution?
I have a flink job(2 task-managers) with a job parallelism of 64 and task
slot of 64.
I have a parallelism set for one of the operators as 16. This operator(16
parallelism) slots are not getting evenly distributed across two task
managers. It o
