Which version of Kafka are you using with your consumer? Is it Scala or Java consumers?
Guozhang On Wed, Nov 23, 2016 at 6:38 AM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > No, that is not the reason. Initially all the partitions were assigned the > messages and those were processed very fast and sit idle even other > partitions are having a lot of messages to be processed. > So I was under impression that rebalance should be triggered and messages > will be re-distributed equally again. > > Thanks > Achintya > > -----Original Message----- > From: Sharninder [mailto:sharnin...@gmail.com] > Sent: Wednesday, November 23, 2016 12:33 AM > To: users@kafka.apache.org > Cc: d...@kafka.apache.org > Subject: Re: Kafka consumers are not equally distributed > > Could it be because of the partition key ? > > On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < > achintya_gh...@comcast.com> wrote: > > > Hi there, > > > > We are doing the load test in Kafka with 25tps and first 9 hours it > > went fine almost 80K/hr messages were processed after that we see a > > lot of lags and we stopped the incoming load. > > > > Currently we see 15K/hr messages are processing. We have 40 consumer > > instances with concurrency 4 and 2 topics and both is having 160 > > partitions so each consumer with each partition. > > > > What we found that some of the partitions are sitting idle and some of > > are overloaded and its really slowing down the consumer message > processing. > > > > Why rebalancing is not happening and existing messages are not > > distributed equally among the instances? We tried to restart the app > > still the same pace. Any idea what could be the reason? > > > > Thanks > > Achintya > > > > > > > -- > -- > Sharninder > -- -- Guozhang
