I saw this in code of 0.9.0.1. I am sure about this because i am caching this exception and executing a logic where i stop further record processing if autocommit is off else i keep processing records got in current poll even if commit fails. This is because kafka marks all records sent to user as committed just before a rebalance only when autocommit in on otherwise with autocommit off, in next poll kafka sends records from last commited offset by consumer. I have seen some cases where autocommit also fails but that happened rarely and i did not got a chance to investigate that.
Regards, Vinay On Apr 29, 2016 6:12 AM, "Phil Luckhurst" <phil.luckhu...@encycle.com> wrote: > Hi Vinay, > > This statement is very interesting. > > "I noticed that in case where a consumer is marked dead or a rebalance is > in progress, kafka throws CommitFailedException. A KafkaException is thrown > only when something unknown has happened which is not yet categorized." > > I will test this out but was this with 0.9.01 or 0.10.1.0? If commitSync > fails with CommitFailedException when the consumer is marked dead then that > is the final piece to my jigsaw. If I get that exception I know I can > safely abandon the current batch of records I am processing and return to > my poll command knowing that the ones I haven’t committed will be picked up > after the rebalance completes. > > Many thanks, > Phil > > -----Original Message----- > From: vinay sharma [mailto:vinsharma.t...@gmail.com] > Sent: 28 April 2016 21:34 > To: users@kafka.apache.org > Subject: Re: Detecting rebalance while processing ConsumerRecords (0.9.0.1) > > Hi Phil, > > I tested my code and it is correct. I do see heartbeats getting missed > sometimes and causing session time out for consumer where generation is > marked dead. I see that there are long time windows where there is no > heartbeat whereas i do commit in between these time windows and there is no > reason to skip a heartbeat as the last hearbeat was way before configured > heartbeat interval. I ran tests with kafka client jar build from trunk > (0.10.1.0) and it works fine due to latest changes where each commitSync > acts as a heartbeat. > > Calling commitSync after each record processing works fine because it > triggers more number of heartbeat requests in a timeinterval making chances > extremely small where heartbeats can be missed for more than session time > out. > > You wrote in a previous mail that > > "I thought having commitSync throw an explicit 'rebalance in progress' type > > exception rather than just a KafkaException would allow this to be > > easily identified and handled." > > > I noticed that in case where a consumer is marked dead or a rebalance is > in progress, kafka throws CommitFailedException. A KafkaException is thrown > only when something unknown has happened which is not yet categorized. > > > Regards, > Vinay > > On Wed, Apr 27, 2016 at 7:52 AM, vinay sharma <vinsharma.t...@gmail.com> > wrote: > > > Hi Phil, > > > > This sounds great. Thanks for trying these serrings. This means > > probably something wrong in my code or setup. I will check what is > > causing this issue in my case. > > > > I have a 3 broker 1 zk cluster and my topic has 3 partitions with > > replication factor 3. > > > > Regards, > > Vinay Sharma > > >
