Couple of questions: - What version of the consumer API are you using? - Are you seeing any rebalance failures in the consumer logs? - How do you determine that some partitions are unassigned? Just confirming that you have partitions that are not being consumed from as opposed to consumer threads that aren't assigned any partitions.
Aditya ________________________________________ From: Dave Hamilton [dhamil...@nanigans.com] Sent: Tuesday, April 28, 2015 10:19 AM To: users@kafka.apache.org Subject: Re: Unclaimed partitions I’m sorry, I forgot to specify that these processes are in the same consumer group. Thanks, Dave On 4/28/15, 1:15 PM, "Aditya Auradkar" <aaurad...@linkedin.com.INVALID> wrote: >Hi Dave, > >The simple consumer doesn't do any state management across consumer instances. >So I'm not sure how you are assigning partitions in your application code. Did >you mean to say that you are using the high level consumer API? > >Thanks, >Aditya > >________________________________________ >From: Dave Hamilton [dhamil...@nanigans.com] >Sent: Tuesday, April 28, 2015 7:58 AM >To: users@kafka.apache.org >Subject: Unclaimed partitions > >Hi, I am trying to consume a 24-partition topic across 12 processes. Each >process is using the simple consumer API, and each is being assigned two >consumer threads. I have noticed when starting these processes that sometimes >some of my processes are not being assigned any partitions, and no rebalance >seems to ever be triggered, leaving some of the partitions unclaimed. > >When I first tried deploying this yesterday, I noticed 8 of the 24 partitions, >for 4 of the consumer processes, went unclaimed. Redeploying shortly later >corrected this issue. I tried deploying again today, and now I see a different >set of 4 processes not getting assigned partitions. The processes otherwise >appear to be running normally, they are currently running in production and we >are working to get the consumers quietly running before enabling them to do >any work. I’m not sure if we might be looking at some sort of timing issue. > >Does anyone know what might be causing the issues we’re observing? > >Thanks, >Dave
