Since nothing was written to the __consumer_offsets topic for more than its configured retention period (offsets.retention.minutes, by default 1440 minutes, or one day), the offset info will be removed. Retention period is all about when the last offset was written, not the last time a Consumer looked at a topic.
You can increase the value of offsets.retention.minutes to ensure that offset info isn’t cleaned out before more messages are written to a topic and read by the Consumer (and hence the Consumer updates its offset info in __consumer_offsets). Ian. --- Ian Wrigley Director, Education Services Confluent, Inc > On Aug 20, 2016, at 11:36 AM, Misra, Rahul <rahul.mi...@altisource.com> wrote: > > Hi, > > I have observed the following scenario (the consumer here has > 'enable.auto.commit=false' and offsets are committed using commitSync() if > any messages are received): > > 1. Start a consumer (with a specific group.Id) and send some messages to > its subscribed topic. > > 2. The consumer consumes the messages and the group+consumer has an > entry in the __commit_offsets with the latest offsets for this group and > consumer. > > 3. The consumer will keep polling the topic but don't send any more > messages to the topic for a long time (longer than one day. The consumer > keeps polling the topic in a while loop). The default duration for which a > group's entry is retained in the offsets topic is 1 day. > > 4. Now stop the consumer. (there is no other consumer for this group) > > 5. Send some more messages to the topic. > > 6. Start the consumer (with the same group and consumer id as earlier). > > 7. The consumer does not pick up the new messages sent in step 5 as it > has lost the committed offsets and starts with the 'latest' offsets. > > Is this an expected behavior? Or do I have something wrong in the > configurations? > Is there a way to ensure that the offsets are retained even if there are no > messages flowing in? > I had assumed that if the consumer keeps polling the kafka topic, its > offsets will be retained even if no messages are received by the > corresponding topic. > > Regards, > Rahul > This email message and any attachments are intended solely for the use of the > addressee. If you are not the intended recipient, you are prohibited from > reading, disclosing, reproducing, distributing, disseminating or otherwise > using this transmission. If you have received this message in error, please > promptly notify the sender by reply email and immediately delete this message > from your system. This message and any attachments may contain information > that is confidential, privileged or exempt from disclosure. Delivery of this > message to any person other than the intended recipient is not intended to > waive any right or privilege. Message transmission is not guaranteed to be > secure or free of software viruses. > ***********************************************************************************************************************
