Hi,
I have a custom Kafka consumer which reads messages from a topic, hands over
the processing of the messages to a different thread, and while the messages
are being processed, it pauses the topic and keeps polling the Kafka topic (to
maintain heartbeats) and also commits offsets using commitSync() once the
processing thread returns success.
This consumer is the only consumer in its group. Auto commit for offsets is set
to false.
The consumer also registers the onPartitionsAssigned() and
onPartitionsRevoked() listeners.
Recently I observed that the consumer frequently crashes (if consuming large
number of messages) with the following exception:
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot
be completed due to group rebalance at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:546)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle(ConsumerCoordinator.java:487)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:681)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:654)
at
org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
at
org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:350)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:288)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:303)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:197)
at
org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:187)
None of the rebalance listeners were called before this exception.
Could somebody suggest why this rebalancing is being triggered, always while
committing the offsets (or is the actual issue somewhere else?)
Regards,
Rahul Misra
This email message and any attachments are intended solely for the use of the
addressee. If you are not the intended recipient, you are prohibited from
reading, disclosing, reproducing, distributing, disseminating or otherwise
using this transmission. If you have received this message in error, please
promptly notify the sender by reply email and immediately delete this message
from your system. This message and any attachments may contain information that
is confidential, privileged or exempt from disclosure. Delivery of this message
to any person other than the intended recipient is not intended to waive any
right or privilege. Message transmission is not guaranteed to be secure or free
of software viruses.
***********************************************************************************************************************
