There is a JIRA open for exactly this issue with patch available: KAFKA-559 <https://issues.apache.org/jira/browse/KAFKA-559>
On Fri, Oct 10, 2014 at 9:33 AM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > Bhavesh, > > All consumer groups with at least one alive consumer will have a > registration node for that consumer instance under /consumers/<group>/ids. > Any consumer group that is long gone will have no registered consumer > instance in zookeeper. The right way to clean up defunct consumer groups is > to scan consumer groups that have no children under /consumers/<group>/ids > for every <group>. For each such group, recursively delete the top level zk > path /consumers/<group>. It goes without saying that while you clean up the > zk path, someone could try to bring up a consumer instance for the group > and *might* throw some kind of exception. But this is fairly easy to fix. > > This is some wasted work since we are trying to move away from the > zookeeper based consumer, but if you or someone else is up for writing this > script, I can shepherd it through a commit. > > Thanks, > Neha > > On Thu, Oct 9, 2014 at 11:09 PM, Bhavesh Mistry < > mistry.p.bhav...@gmail.com> > wrote: > > > We just want to clean-up old configuration from ZK. We can check from > the > > offset API so we can delete based on offset .. is that right ? there is > no > > date last associated with Consumer Group ? Is that right in ZK > > configuration ? > > > > Thanks, > > > > Bhavesh > > > > On Thu, Oct 9, 2014 at 9:23 PM, Gwen Shapira <gshap...@cloudera.com> > > wrote: > > > > > The problem with Kafka is that we never know when a consumer is > > > "truly" inactive. > > > > > > But - if you decide to define inactive as consumer who's last offset > > > is lower than anything available on the log (or perhaps lagging by > > > over X messages?), its fairly easy to write a script to detect and > > > clean them directly on ZK. > > > > > > BTW. Why do you need to clean them? What issue do you see with just > > > letting them hang around? > > > > > > Gwen > > > > > > On Thu, Oct 9, 2014 at 9:18 PM, Bhavesh Mistry > > > <mistry.p.bhav...@gmail.com> wrote: > > > > Hi Kafka, > > > > > > > > We have lots of lingering console consumer group people have created > > for > > > > testing or debugging purpose for one time use via > > > > bin/kafka-console-consumer.sh. Is there auto purging that clean > script > > > > that Kafka provide ? Is three any API to find out inactive Consumer > > > group > > > > and delete consumer group configuration. > > > > > > > > Thanks, > > > > > > > > Bhavesh > > > > > > -- -- Guozhang
