1. We’re using version 0.8.1.1. 2. No failures in the consumer logs 3. We’re using the ConsumerOffsetChecker to see what partitions are assigned to the consumer group and what their offsets are. 8 of the 12 process each have been assigned two partitions and they’re keeping up with the topic. The other 4 do not get assigned partitions and no consumers in the group are consuming those 8 partitions.
Thanks for your help, Dave On 4/28/15, 1:40 PM, "Aditya Auradkar" <aaurad...@linkedin.com.INVALID> wrote: >Couple of questions: >- What version of the consumer API are you using? >- Are you seeing any rebalance failures in the consumer logs? >- How do you determine that some partitions are unassigned? Just confirming >that you have partitions that are not being consumed from as opposed to >consumer threads that aren't assigned any partitions. > >Aditya > >________________________________________ >From: Dave Hamilton [dhamil...@nanigans.com] >Sent: Tuesday, April 28, 2015 10:19 AM >To: users@kafka.apache.org >Subject: Re: Unclaimed partitions > >I’m sorry, I forgot to specify that these processes are in the same consumer >group. > >Thanks, >Dave > > > > > >On 4/28/15, 1:15 PM, "Aditya Auradkar" <aaurad...@linkedin.com.INVALID> wrote: > >>Hi Dave, >> >>The simple consumer doesn't do any state management across consumer >>instances. So I'm not sure how you are assigning partitions in your >>application code. Did you mean to say that you are using the high level >>consumer API? >> >>Thanks, >>Aditya >> >>________________________________________ >>From: Dave Hamilton [dhamil...@nanigans.com] >>Sent: Tuesday, April 28, 2015 7:58 AM >>To: users@kafka.apache.org >>Subject: Unclaimed partitions >> >>Hi, I am trying to consume a 24-partition topic across 12 processes. Each >>process is using the simple consumer API, and each is being assigned two >>consumer threads. I have noticed when starting these processes that sometimes >>some of my processes are not being assigned any partitions, and no rebalance >>seems to ever be triggered, leaving some of the partitions unclaimed. >> >>When I first tried deploying this yesterday, I noticed 8 of the 24 >>partitions, for 4 of the consumer processes, went unclaimed. Redeploying >>shortly later corrected this issue. I tried deploying again today, and now I >>see a different set of 4 processes not getting assigned partitions. The >>processes otherwise appear to be running normally, they are currently running >>in production and we are working to get the consumers quietly running before >>enabling them to do any work. I’m not sure if we might be looking at some >>sort of timing issue. >> >>Does anyone know what might be causing the issues we’re observing? >> >>Thanks, >>Dave
