Re: kafka 0.9 offset unknown after cleanup

Thanks Tom, Thats a lot clearer now. Michael On Wed, May 4, 2016 at 3:27 PM, Tom Crayford wrote: > Michael, > > Increasing the offset retention period means that old consumers will use up > space in the __consumer_offsets topic and in the cache the brokers hold on > that to

Re: kafka 0.9 offset unknown after cleanup

Michael, Increasing the offset retention period means that old consumers will use up space in the __consumer_offsets topic and in the cache the brokers hold on that topic. The cache is in memory, so if you churn consumer groups a lot that could be problematic. The other issue is that booting new b

Re: kafka 0.9 offset unknown after cleanup

Hey Tom, Are there any details on the negative side effects of increasing the offset retention period? I'd like to increase it but want to be aware of the risks. Thanks Michael > On 4 May 2016, at 05:06, Tom Crayford wrote: > > Jun, > > Yep, you got it. If there are no offs

Re: kafka 0.9 offset unknown after cleanup

Jun, Yep, you got it. If there are no offset commits for 24h, then your offset will disappear and be cleaned up. I think this behaviour is at least a little suboptimal and surprising, but it does prevent a lot of issues that arise when you rapidly add and remove consumer groups. I hope that helps

Re: kafka 0.9 offset unknown after cleanup

I think I figured that out. Based on the explanation on http://www.slideshare.net/jjkoshy/offset-management-in-kafka , offsets.retention.minutes is used for clean up dead consumer group. It means if one consumer group hasn’t commit a

Re: kafka 0.9 offset unknown after cleanup

Thanks for your reply. I checked the offset topic and the cleanup policy is actually compact. Topic:__consumer_offsetsPartitionCount:50 ReplicationFactor:3 Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=uncompressed And I’m using 0.9.0.1 so the default con

Re: kafka 0.9 offset unknown after cleanup

Looks like it, you need to be sure the offset topic is using compaction, and the broker is set to enable compaction. On Tue, May 3, 2016 at 9:56 AM Jun MA wrote: > Hi, > > I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up > all old log segments(reach delete.retention time)

kafka 0.9 offset unknown after cleanup

Hi, I’m using 0.9.0.1 new-consumer api. I noticed that after kafka cleans up all old log segments(reach delete.retention time), I got unknown offset. bin/kafka-consumer-groups.sh --bootstrap-server server:9092 --new-consumer --group testGroup --describe GROUP, TOPIC, PARTITION, CURRENT OFFSET,