RE: Offsets getting lost if no messages sent for a long time

2016-08-20 Thread Misra, Rahul
Hi Ian, Thanks for the quick response. Your answer clears things up. I have some follow up questions though: 1. Given a limited set of groups and consumers, will increasing 'offsets.retention.minutes' to a high value (say 30 days) cause the __consumer_offsets topic to bloat unnecessarily or wil

Re: Offsets getting lost if no messages sent for a long time

2016-08-20 Thread Ian Wrigley
Since nothing was written to the __consumer_offsets topic for more than its configured retention period (offsets.retention.minutes, by default 1440 minutes, or one day), the offset info will be removed. Retention period is all about when the last offset was written, not the last time a Consumer

Offsets getting lost if no messages sent for a long time

2016-08-20 Thread Misra, Rahul
Hi, I have observed the following scenario (the consumer here has 'enable.auto.commit=false' and offsets are committed using commitSync() if any messages are received): 1. Start a consumer (with a specific group.Id) and send some messages to its subscribed topic. 2. The consumer con