Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

2024-06-06 Thread Kevin Lam
here. > > Best, > Zhanghao Chen > > From: Kevin Lam > Sent: Thursday, June 6, 2024 2:32 > To: dev@flink.apache.org > Subject: Poor Load Balancing across TaskManagers for Multiple Kafka Sources > > Hey all, > > I'm seeing an issue with poor load bala

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

2024-06-05 Thread Zhanghao Chen
From: Kevin Lam Sent: Thursday, June 6, 2024 2:32 To: dev@flink.apache.org Subject: Poor Load Balancing across TaskManagers for Multiple Kafka Sources Hey all, I'm seeing an issue with poor load balancing across TaskManagers for Kafka Sources using the Flink SQL API and wondering if FLIP

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

2024-06-05 Thread Kevin Lam
I've also just found https://issues.apache.org/jira/browse/FLINK-31762 which tracks the Kafka specific issue. On Wed, Jun 5, 2024 at 3:05 PM Kevin Lam wrote: > cc. panyuep...@apache.org as related to FLIP-370 > > On Wed, Jun 5, 2024 at 2:32 PM Kevin Lam wrote: > >> Hey all, >> >> I'm seeing an

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

2024-06-05 Thread Kevin Lam
cc. panyuep...@apache.org as related to FLIP-370 On Wed, Jun 5, 2024 at 2:32 PM Kevin Lam wrote: > Hey all, > > I'm seeing an issue with poor load balancing across TaskManagers for Kafka > Sources using the Flink SQL API and wondering if FLIP-370 will help with > it, or if not, interested in any

Poor Load Balancing across TaskManagers for Multiple Kafka Sources

2024-06-05 Thread Kevin Lam
Hey all, I'm seeing an issue with poor load balancing across TaskManagers for Kafka Sources using the Flink SQL API and wondering if FLIP-370 will help with it, or if not, interested in any ideas the community has to mitigate the issue. The Kafka SplitEnumerator uses the following logic to assign