Bhavesh, All consumer groups with at least one alive consumer will have a registration node for that consumer instance under /consumers/<group>/ids. Any consumer group that is long gone will have no registered consumer instance in zookeeper. The right way to clean up defunct consumer groups is to scan consumer groups that have no children under /consumers/<group>/ids for every <group>. For each such group, recursively delete the top level zk path /consumers/<group>. It goes without saying that while you clean up the zk path, someone could try to bring up a consumer instance for the group and *might* throw some kind of exception. But this is fairly easy to fix.
This is some wasted work since we are trying to move away from the zookeeper based consumer, but if you or someone else is up for writing this script, I can shepherd it through a commit. Thanks, Neha On Thu, Oct 9, 2014 at 11:09 PM, Bhavesh Mistry <mistry.p.bhav...@gmail.com> wrote: > We just want to clean-up old configuration from ZK. We can check from the > offset API so we can delete based on offset .. is that right ? there is no > date last associated with Consumer Group ? Is that right in ZK > configuration ? > > Thanks, > > Bhavesh > > On Thu, Oct 9, 2014 at 9:23 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > The problem with Kafka is that we never know when a consumer is > > "truly" inactive. > > > > But - if you decide to define inactive as consumer who's last offset > > is lower than anything available on the log (or perhaps lagging by > > over X messages?), its fairly easy to write a script to detect and > > clean them directly on ZK. > > > > BTW. Why do you need to clean them? What issue do you see with just > > letting them hang around? > > > > Gwen > > > > On Thu, Oct 9, 2014 at 9:18 PM, Bhavesh Mistry > > <mistry.p.bhav...@gmail.com> wrote: > > > Hi Kafka, > > > > > > We have lots of lingering console consumer group people have created > for > > > testing or debugging purpose for one time use via > > > bin/kafka-console-consumer.sh. Is there auto purging that clean script > > > that Kafka provide ? Is three any API to find out inactive Consumer > > group > > > and delete consumer group configuration. > > > > > > Thanks, > > > > > > Bhavesh > > >
