Hi Vinay, This statement is very interesting.
"I noticed that in case where a consumer is marked dead or a rebalance is in progress, kafka throws CommitFailedException. A KafkaException is thrown only when something unknown has happened which is not yet categorized." I will test this out but was this with 0.9.01 or 0.10.1.0? If commitSync fails with CommitFailedException when the consumer is marked dead then that is the final piece to my jigsaw. If I get that exception I know I can safely abandon the current batch of records I am processing and return to my poll command knowing that the ones I haven’t committed will be picked up after the rebalance completes. Many thanks, Phil -----Original Message----- From: vinay sharma [mailto:vinsharma.t...@gmail.com] Sent: 28 April 2016 21:34 To: users@kafka.apache.org Subject: Re: Detecting rebalance while processing ConsumerRecords (0.9.0.1) Hi Phil, I tested my code and it is correct. I do see heartbeats getting missed sometimes and causing session time out for consumer where generation is marked dead. I see that there are long time windows where there is no heartbeat whereas i do commit in between these time windows and there is no reason to skip a heartbeat as the last hearbeat was way before configured heartbeat interval. I ran tests with kafka client jar build from trunk (0.10.1.0) and it works fine due to latest changes where each commitSync acts as a heartbeat. Calling commitSync after each record processing works fine because it triggers more number of heartbeat requests in a timeinterval making chances extremely small where heartbeats can be missed for more than session time out. You wrote in a previous mail that "I thought having commitSync throw an explicit 'rebalance in progress' type > exception rather than just a KafkaException would allow this to be > easily identified and handled." I noticed that in case where a consumer is marked dead or a rebalance is in progress, kafka throws CommitFailedException. A KafkaException is thrown only when something unknown has happened which is not yet categorized. Regards, Vinay On Wed, Apr 27, 2016 at 7:52 AM, vinay sharma <vinsharma.t...@gmail.com> wrote: > Hi Phil, > > This sounds great. Thanks for trying these serrings. This means > probably something wrong in my code or setup. I will check what is > causing this issue in my case. > > I have a 3 broker 1 zk cluster and my topic has 3 partitions with > replication factor 3. > > Regards, > Vinay Sharma >
