RE: Kafka consumers are not equally distributed

2016-11-28 Thread Ghosh, Achintya (Contractor)
to speed up? > > Thanks > Acintya > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: Friday, November 25, 2016 12:09 PM > To: users@kafka.apache.org > Subject: Re: Kafka consumers are not equally distributed > > Note that consumer'

Re: Kafka consumers are not equally distributed

2016-11-25 Thread Guozhang Wang
? > > Thanks > Acintya > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: Friday, November 25, 2016 12:09 PM > To: users@kafka.apache.org > Subject: Re: Kafka consumers are not equally distributed > > Note that consumer's fetching p

RE: Kafka consumers are not equally distributed

2016-11-25 Thread Ghosh, Achintya (Contractor)
@kafka.apache.org Subject: Re: Kafka consumers are not equally distributed Note that consumer's fetching parallelism is per-partition, i.e., one partition is fetched by only a single consumer instance, so even if some partitions have heavy load other idle consumers will not come to share the messages. I

Re: Kafka consumers are not equally distributed

2016-11-25 Thread Guozhang Wang
buted. So we are concerned what should do this kind of situation and > make the consumers more speedy. > > Thanks > Achintya > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: Thursday, November 24, 2016 11:21 PM > To: users@kafka

RE: Kafka consumers are not equally distributed

2016-11-25 Thread Ghosh, Achintya (Contractor)
this kind of situation and make the consumers more speedy. Thanks Achintya -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Thursday, November 24, 2016 11:21 PM To: users@kafka.apache.org Subject: Re: Kafka consumers are not equally distributed The default

Re: Kafka consumers are not equally distributed

2016-11-24 Thread Guozhang Wang
gt; Achintya > > -Original Message- > From: Guozhang Wang [mailto:wangg...@gmail.com] > Sent: Thursday, November 24, 2016 8:28 PM > To: users@kafka.apache.org > Subject: Re: Kafka consumers are not equally distributed > > Which version of Kafka are you using with

RE: Kafka consumers are not equally distributed

2016-11-24 Thread Ghosh, Achintya (Contractor)
Java consumer. 0.9.1 Thanks Achintya -Original Message- From: Guozhang Wang [mailto:wangg...@gmail.com] Sent: Thursday, November 24, 2016 8:28 PM To: users@kafka.apache.org Subject: Re: Kafka consumers are not equally distributed Which version of Kafka are you using with your consumer

Re: Kafka consumers are not equally distributed

2016-11-24 Thread Guozhang Wang
inal Message- > From: Sharninder [mailto:sharnin...@gmail.com] > Sent: Wednesday, November 23, 2016 12:33 AM > To: users@kafka.apache.org > Cc: d...@kafka.apache.org > Subject: Re: Kafka consumers are not equally distributed > > Could it be because of the partition ke

RE: Kafka consumers are not equally distributed

2016-11-23 Thread Ghosh, Achintya (Contractor)
-distributed equally again. Thanks Achintya -Original Message- From: Sharninder [mailto:sharnin...@gmail.com] Sent: Wednesday, November 23, 2016 12:33 AM To: users@kafka.apache.org Cc: d...@kafka.apache.org Subject: Re: Kafka consumers are not equally distributed Could it be because of the

Re: Kafka consumers are not equally distributed

2016-11-22 Thread Sharninder
Could it be because of the partition key ? On Wed, Nov 23, 2016 at 12:33 AM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > Hi there, > > We are doing the load test in Kafka with 25tps and first 9 hours it went > fine almost 80K/hr messages were processed after that we see a

Kafka consumers are not equally distributed

2016-11-22 Thread Ghosh, Achintya (Contractor)
Hi there, We are doing the load test in Kafka with 25tps and first 9 hours it went fine almost 80K/hr messages were processed after that we see a lot of lags and we stopped the incoming load. Currently we see 15K/hr messages are processing. We have 40 consumer instances with concurrency 4 and