The default partition assignment strategy is the RangePartitioner. Note it is per-topic, so if you use the default partitioner then in your case 160 partitions of each of the topic will be assigned to the first 160 consumer instances, each getting two partitions, one partition from each. So the consumer should be balanced on the consumer-instance basis.
I'm not sure what you meant by "some of the partitions are sitting idle and some of are overloaded", do you mean that some partitions does not have new data coming in and others keep getting high traffic producing to it that the consumer cannot keep up? In this case it is no the consumer's issue, but the producer not producing in a balanced manner. Guozhang On Thu, Nov 24, 2016 at 7:45 PM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > Java consumer. 0.9.1 > > Thanks > Achintya > > -----Original Message----- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: Thursday, November 24, 2016 8:28 PM > To: users@kafka.apache.org > Subject: Re: Kafka consumers are not equally distributed > > Which version of Kafka are you using with your consumer? Is it Scala or > Java consumers? > > > Guozhang > > > On Wed, Nov 23, 2016 at 6:38 AM, Ghosh, Achintya (Contractor) < > achintya_gh...@comcast.com> wrote: > > > No, that is not the reason. Initially all the partitions were assigned > > the messages and those were processed very fast and sit idle even > > other partitions are having a lot of messages to be processed. > > So I was under impression that rebalance should be triggered and > > messages will be re-distributed equally again. > > > > Thanks > > Achintya > > > > -----Original Message----- > > From: Sharninder [mailto:sharnin...@gmail.com] > > Sent: Wednesday, November 23, 2016 12:33 AM > > To: users@kafka.apache.org > > Cc: d...@kafka.apache.org > > Subject: Re: Kafka consumers are not equally distributed > > > > Could it be because of the partition key ? > > > > On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < > > achintya_gh...@comcast.com> wrote: > > > > > Hi there, > > > > > > We are doing the load test in Kafka with 25tps and first 9 hours it > > > went fine almost 80K/hr messages were processed after that we see a > > > lot of lags and we stopped the incoming load. > > > > > > Currently we see 15K/hr messages are processing. We have 40 consumer > > > instances with concurrency 4 and 2 topics and both is having 160 > > > partitions so each consumer with each partition. > > > > > > What we found that some of the partitions are sitting idle and some > > > of are overloaded and its really slowing down the consumer message > > processing. > > > > > > Why rebalancing is not happening and existing messages are not > > > distributed equally among the instances? We tried to restart the app > > > still the same pace. Any idea what could be the reason? > > > > > > Thanks > > > Achintya > > > > > > > > > > > > -- > > -- > > Sharninder > > > > > > -- > -- Guozhang > -- -- Guozhang
