Jason, This is mainly a problem that we don't have a formal way of deleting a topic in 0.7, which we are trying to fix in 0.8.
The extra watchers on those deleted topics are likely registered by the producers. They should be gone once /brokers/topics/deletedtopic are removed from ZK. You probably don't need to clean /consumers/<appname>/owners. Thanks, Jun On Sun, Mar 17, 2013 at 11:11 PM, Jason Rosenberg <j...@squareup.com> wrote: > Jun, > > There are indeed no nodes under /brokers/topics/deletedtopic > > Also, do I need to remove the deleted apps from the > /consumers/<appname>/owners path? > > So, should this be considered a bug, or not? Essentially, each broker > knows what topics it has, but the info seems to have not propagated to zk > at all? Is that right? > > Although from the telnet interface, the removed topics don't show up under > a 'dump' command, but they do under wchc, etc. What's that about? > > Should the original procedure have been to: (1) stop all consumers, then > (2) remove the topic files on the brokers, then (3) restart the brokers, > then (4) restart the consumers? > > The consumers have been up and down since the topic files were removed, but > just not ever at the same time the brokers were down.... > > Jason > > > On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <jun...@gmail.com> wrote: > > > Jason, > > > > Use zkCli.sh to see if there is any child node in ZK under > > /brokers/topics/deletedtopic. You should see none since you already > removed > > those topics from brokers' log. If so, you can manually remove those > paths > > in ZK using zkCli.sh (stop the consumers first). After the consumers > > restart, you should not see the the deleted topics in the log of the > > consumers any more. > > > > consumer.ZookeeperConsumerConnector - > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to > > consume = List(<a bazillion topics, most of which have been deleted > listed > > here>) > > > > Thanks, > > > > Jun > > > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > Hi Jun, yes, we're using zk-based producers. Producers have all been > > > restarted (they get cycled many times a day as well). > > > > > > The specific producers that generated all those bogus topics have > > > definitely been restarted, etc. > > > > > > Using zkCli.sh, I do see all the deleted topics, e.g.: > > > > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics > > > [<1000's of deleted topics>,<also all the remaining valid topics>] > > > > > > I see also all the deleted topics under /consumers/<appname>/owners/ > > > > > > Jason > > > > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > Are you using ZK-based producer? If so, those watches could be left > by > > > the > > > > producers if they haven't been restarted. Could you also use zkCli.sh > > to > > > > see if deleted topics are there in ZK? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <j...@squareup.com> > > > wrote: > > > > > > > > > Jun, > > > > > > > > > > So, I connected to zookeeper just using telnet, and using the 4 > > letter > > > > > commands. > > > > > > > > > > If I do a dump: > > > > > > > > > > I do not see anything but valid topics, and valid consumer/owners > > > > mappings. > > > > > > > > > > If I check watches, I see all the 1000's of bogus topics, e.c.: > > > > > > > > > > wchc: > > > > > .... > > > > > /brokers/topics/<obsolete-topic> > > > > > .... > > > > > > > > > > or > > > > > > > > > > wchp > > > > > .... > > > > > /brokers/topics/<obsolete-topic>. > > > > > 0x383d6fe1b83f0008 > > > > > 0x383d6fe1b83f0009 > > > > > 0x7c3d6f9020d00041 > > > > > 0x833d6fe2826e0000 > > > > > 0x383d6fe1b83f000a > > > > > .... > > > > > > > > > > The kafka and zookeeper servers, as well as the consumer apps, are > > > rolled > > > > > several times a day, automatically. > > > > > > > > > > I'm not sure how to remove the watches, is that what I need to do? > > > > > > > > > > Since this is a highly available system, can I bring down each zk > > > server > > > > 1 > > > > > at a time, and clear the bogus topics (or watches)? How do I do > > this? > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > > > > > Could you check if the following path for a deleted topic exists > in > > > ZK? > > > > > It > > > > > > should have no children. > > > > > > > > > > > > /brokers/topics/[topic] > > > > > > > > > > > > If this is the case, try manually removing those paths from ZK > > (when > > > > the > > > > > > brokers and the consumers are down). > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg < > j...@squareup.com > > > > > > > > wrote: > > > > > > > > > > > > > Hi Neha, > > > > > > > > > > > > > > So I did this, but I still see the full list of topics (most of > > > which > > > > > > have > > > > > > > been deleted), in the consumer logs, e.g.: > > > > > > > > > > > > > > consumer.ZookeeperConsumerConnector - > > > > > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb > > > > Topics > > > > > to > > > > > > > consume = List(<a bazillion topics, most of which have been > > deleted > > > > > > listed > > > > > > > here>) > > > > > > > > > > > > > > I select topics using the white list topic selector (using a > > > > wild-card > > > > > > > pattern, etc.). > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > Does zookeeper keep separate info for the consumer, that needs > to > > > be > > > > > > > removed? > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede < > > > > > neha.narkh...@gmail.com > > > > > > > >wrote: > > > > > > > > > > > > > > > In Kafka 0.7.2, data log directory delete and then rolling > > > restart > > > > > will > > > > > > > > work to get rid of the topic completely from zookeeper as > well > > as > > > > the > > > > > > > > broker. > > > > > > > > This is because the broker registers the topics and > partitions > > > from > > > > > its > > > > > > > > data directory into zookeeper on startup. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Neha > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg < > > > j...@squareup.com> > > > > > > > wrote: > > > > > > > > > > > > > > > > > Thanks Neha, > > > > > > > > > > > > > > > > > > So are you saying that on 0.7.2, to delete a topic I need > > only > > > > > remove > > > > > > > > it's > > > > > > > > > data log directory from each broker, and the restart the > > > brokers? > > > > > Is > > > > > > > it > > > > > > > > ok > > > > > > > > > if it's a rolling restart? > > > > > > > > > > > > > > > > > > For some reason I thought I also had to do something on > > > zookeeper > > > > > > > servers > > > > > > > > > too, to remove the topics. Is that not true? > > > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede < > > > > > > neha.narkh...@gmail.com > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > >> The topics get automagically created if they don't > > exist. > > > > > Will > > > > > > > > > > that still exist in 0.8? > > > > > > > > > > > > > > > > > > > > In 0.8, this is controlled by the > auto.create.topics.enable > > > > > config > > > > > > on > > > > > > > > the > > > > > > > > > > brokers. If this is set to true, topics will be created > > when > > > a > > > > > > topic > > > > > > > > > > metadata request is sent for a new topic. This feature is > > > > > provided > > > > > > to > > > > > > > > aid > > > > > > > > > > with the 0.7->0.8 migration tool and the mirror maker > > tools. > > > > > These > > > > > > > > tools > > > > > > > > > > copy data for almost all topics from one cluster to the > > > other, > > > > so > > > > > > > auto > > > > > > > > > > creating topics makes it much easier to migrate data > > between > > > > > > > clusters. > > > > > > > > > > > > > > > > > > > > >> What's the procedure for deleting topics in kafka > 0.7.2? > > > > > > > > > > > > > > > > > > > > There is no official way to delete a topic. You have to > > > delete > > > > > the > > > > > > > data > > > > > > > > > > directory and bounce the brokers to take note of that. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Neha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg < > > > > > j...@squareup.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Thanks Neha, > > > > > > > > > > > > > > > > > > > > > > I added a comment to that ticket. > > > > > > > > > > > > > > > > > > > > > > Another question, I'm not sure if the behavior is > > different > > > > in > > > > > > 0.8 > > > > > > > > from > > > > > > > > > > > 0.7. But currently, I'm not predeclaring topics, > before > > > > > sending > > > > > > > > > messages > > > > > > > > > > > to topics. The topics get automagically created if > they > > > > don't > > > > > > > exist. > > > > > > > > > > Will > > > > > > > > > > > that still exist in 0.8? If so, it should also be > > > automatic > > > > > > > (using a > > > > > > > > > > > default expiration for all such topics), that they > > > > > automagically > > > > > > go > > > > > > > > > away > > > > > > > > > > > after no longer being in use. > > > > > > > > > > > > > > > > > > > > > > What's the procedure for deleting topics in kafka > 0.7.2? > > > > > > > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede < > > > > > > > > neha.narkh...@gmail.com > > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > > > We plan to include support for deleting topics before > > the > > > > > > > official > > > > > > > > > > > release. > > > > > > > > > > > > The JIRA tracking this feature is > > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-330 > > > > > > > > > > > > > > > > > > > > > > > > This is mainly an admin tool and is proposed to be > run > > > > > > manually. > > > > > > > I > > > > > > > > > > think > > > > > > > > > > > > your suggestion is to set some sort of expiration > time > > on > > > > > > topics > > > > > > > so > > > > > > > > > > when > > > > > > > > > > > > all data gets garbage collected and the expiration > time > > > > > lapses, > > > > > > > the > > > > > > > > > > topic > > > > > > > > > > > > is automatically deleted. > > > > > > > > > > > > > > > > > > > > > > > > For this expiration to work, we need some mechanism > of > > > > > knowing > > > > > > > the > > > > > > > > > size > > > > > > > > > > > of > > > > > > > > > > > > the data logs for that topic on all brokers. And only > > > when > > > > > its > > > > > > > > empty > > > > > > > > > > for > > > > > > > > > > > > all partitions of that topic, we delete it. This > > requires > > > > > some > > > > > > > API > > > > > > > > > that > > > > > > > > > > > > will expose the data size of a topic on a broker, > which > > > we > > > > > > don't > > > > > > > > have > > > > > > > > > > > right > > > > > > > > > > > > now. It might be worth thinking about it. Do you mind > > > > posting > > > > > > > your > > > > > > > > > > > > suggestions for this tool on that JIRA ? > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Neha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg < > > > > > > > j...@squareup.com> > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > I suspect this is not currently supported, but it > > seems > > > > to > > > > > be > > > > > > > for > > > > > > > > > us > > > > > > > > > > a > > > > > > > > > > > > real > > > > > > > > > > > > > use case. > > > > > > > > > > > > > > > > > > > > > > > > > > If we have a topic that is no longer receiving > > > messages, > > > > > and > > > > > > > all > > > > > > > > > > > messages > > > > > > > > > > > > > have been removed from the brokers, after the > > > > > > > log_retention_hours > > > > > > > > > has > > > > > > > > > > > > > expired, I'd love to have it then automatically > > remove > > > > the > > > > > > > topic > > > > > > > > > from > > > > > > > > > > > > > kafka. > > > > > > > > > > > > > > > > > > > > > > > > > > If subsequently a new message arrives for that > topic, > > > it > > > > > > could > > > > > > > > just > > > > > > > > > > > > > recreate the topic as originally, and continue on. > > > > > > > > > > > > > > > > > > > > > > > > > > Is this easy to implement? > > > > > > > > > > > > > Has it been requested? > > > > > > > > > > > > > Does it make sense? > > > > > > > > > > > > > Should I file a jira? > > > > > > > > > > > > > > > > > > > > > > > > > > Currently, we had a test client that inadvertently > > > > created > > > > > > > 1000's > > > > > > > > > of > > > > > > > > > > > > > topics. The job of deleting all these appears to > not > > > be > > > > a > > > > > > > > > cumbersome > > > > > > > > > > > > > manual process (are there any quick links to the > > > > > recommended > > > > > > > way > > > > > > > > > for > > > > > > > > > > > > doing > > > > > > > > > > > > > this?). > > > > > > > > > > > > > > > > > > > > > > > > > > I am using 0.7.2, but will eventually upgrade to > 0.8, > > > > once > > > > > > it's > > > > > > > > > > > released > > > > > > > > > > > > to > > > > > > > > > > > > > beta.... > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
