Thanks Alex and John , it is very helpful. Regards, Sahitya Agrawal
On Wed, May 11, 2016 at 4:14 AM, Alex Loddengaard <a...@confluent.io> wrote: > Hi Sahitya, > > I wonder if your consumers are experiencing soft failures because they're > busy processing a large collection of messages and not calling poll() > within session.timeout.ms? In this scenario, the group coordinator (a > broker) would not receive a heartbeat within session.timeout.ms and would > consider the consumer failed. The coordinator would then reassign the > "failed" consumer's partitions to other consumers in the same group. If all > consumers are experiencing soft failures, you may observe them all > "freezing" their consumption. I suggest checking the logs to see if your > consumer group is being rebalanced frequently. > > If you are hitting the issue I've explained, checkout this KIP: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-41%3A+KafkaConsumer+Max+Records > > Or consider increasing session.timeout.ms. > > Hope this helps. > > Alex > > On Thu, May 5, 2016 at 8:22 AM, John Bickerstaff <j...@johnbickerstaff.com > > > wrote: > > > This may or may not help. I found it to be a clever workaround for some > of > > the limitations in the 8.x version of the high level consumer... I ended > > up writing code that "waited" a lot because I couldn't be sure how > quickly > > Kafka would respond... > > > > Nothing ever took minutes however... the waits were 30 seconds or so if > I > > recall... > > > > In case it helps... > > > > > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing- > > pieces/ > > < > http://ingest.tips/2014/10/12/kafka-high-level-consumer-frequently-missing-pieces/ > > > > > > On Thu, May 5, 2016 at 4:58 AM, sahitya agrawal <sahitya2...@gmail.com> > > wrote: > > > > > I am using high level consumer API ( Kafka API version 0.9.0.0 ) > > > > > > I am running consumers on a topic of 10 partitions. There are lot of > > unread > > > messages in that topic. Initially all of them , consume from the topic > > and > > > read messages. After sometime, all of them hangs and doesn't read any > > > message at all. > > > I have to manually restart them to make them consume again. > > > Surprisingly I do not see any exception or error in logs also. > > > > > > Has anybody ever faced this issue? > > > > > >
