Hi Nick,
First, I have the same question about removing offset groups as you do. I've seen answers like "you don't need to" on StackOverflow, but that's obviously not helpful. So I'll hope that part gets picked up by another user on the group. But what I can help answer is about offset storage. Using offsets.storage=kafka in your broker configuration means that the offsets get stored into an internal compacted topic. So, no they don't go away after restart. The next part of your question: why aren't they being replicated to zookeeper? This is another broker configuration property to explore for your cluster, dual.commit.enabled=true. With this setting and offsets.storage=kafka, the brokers will also back up to zookeeper, but this will not allow you to delete those offsets, because changes in zookeeper won't be applied down to kafka, it's the other way around. I see the other response that came in about expiration, and that tells how to get offsets to expire, which will solve your "junk offset groups" problem, as the default is they don't. But I still want to know if there's a tool to forcibly delete a group's offsets so we can discard that group's offset information and start fresh with the same group id, which would be useful in a handful of use cases we've run across in switching to 0.9. Seeing as how it is a compacted topic, I expect message(s) with the right keys and a null message is the way to remove the offsets from the topic. But is that advisable, and is there a tool to do it? David Krieg | Enterprise Software Engineer Early Warning david.kr...@earlywarning.com From: Cuneo, Nicholas [mailto:ncu...@tycoint.com] Sent: Tuesday, October 04, 2016 12:03 PM To: users@kafka.apache.org Subject: Delete Consumer Group Information We are using Kafka .9 and are playing around with the consumer group feature. We have a lot of junk and stale consumer group information in the consumer groups and want to get rid of it. What's the best way to do that? Using Kafka Tool, I see that all the consumer groups are stored in 'Kafka' and not 'Zookeeper'. When I go to zookeeper the consumers node is empty. I tried using the ConsumerGroupCommand to delete it but since zookeeper shows no consumers this does nothing. If the consumer groups show only being stored in Kafka, are they considered in memory and completely restarting all Kafka nodes remove them? If so - why aren't they being replicated to zookeeper for persistence? Thanks, Nick Cuneo / Software Engineer, Cloud / Enterprise Software Tel: +1 949 517 4802 / Mobile: +1 949 243 4952 3 Ada / Irvine, CA 92618 / USA <mailto:ncu...@tycoint.com> ncu...@tycoint.com / <http://www.tyco.com> www.tyco.com This email (including any attachments) may contain information that is private or business confidential. If you received this email in error, please delete it from your system without copying it and notify sender by reply email so that our records can be corrected _____ This e-mail contains privileged and confidential information intended for the use of the addressees named above. If you are not the intended recipient of this e-mail, you are hereby notified that you must not disseminate, copy or take any action in respect of any information contained in it. If you have received this e-mail in error, please notify the sender immediately by e-mail and immediately destroy this e-mail and its attachments.
smime.p7s
Description: S/MIME cryptographic signature
