Jun, There are indeed no nodes under /brokers/topics/deletedtopic
Also, do I need to remove the deleted apps from the /consumers/<appname>/owners path? So, should this be considered a bug, or not? Essentially, each broker knows what topics it has, but the info seems to have not propagated to zk at all? Is that right? Although from the telnet interface, the removed topics don't show up under a 'dump' command, but they do under wchc, etc. What's that about? Should the original procedure have been to: (1) stop all consumers, then (2) remove the topic files on the brokers, then (3) restart the brokers, then (4) restart the consumers? The consumers have been up and down since the topic files were removed, but just not ever at the same time the brokers were down.... Jason On Sun, Mar 17, 2013 at 9:08 AM, Jun Rao <jun...@gmail.com> wrote: > Jason, > > Use zkCli.sh to see if there is any child node in ZK under > /brokers/topics/deletedtopic. You should see none since you already removed > those topics from brokers' log. If so, you can manually remove those paths > in ZK using zkCli.sh (stop the consumers first). After the consumers > restart, you should not see the the deleted topics in the log of the > consumers any more. > > consumer.ZookeeperConsumerConnector - > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb Topics to > consume = List(<a bazillion topics, most of which have been deleted listed > here>) > > Thanks, > > Jun > > On Sat, Mar 16, 2013 at 11:08 PM, Jason Rosenberg <j...@squareup.com> > wrote: > > > Hi Jun, yes, we're using zk-based producers. Producers have all been > > restarted (they get cycled many times a day as well). > > > > The specific producers that generated all those bogus topics have > > definitely been restarted, etc. > > > > Using zkCli.sh, I do see all the deleted topics, e.g.: > > > > [zk: localhost:26101(CONNECTED) 32] ls /brokers/topics > > [<1000's of deleted topics>,<also all the remaining valid topics>] > > > > I see also all the deleted topics under /consumers/<appname>/owners/ > > > > Jason > > > > On Fri, Mar 15, 2013 at 9:48 PM, Jun Rao <jun...@gmail.com> wrote: > > > > > Are you using ZK-based producer? If so, those watches could be left by > > the > > > producers if they haven't been restarted. Could you also use zkCli.sh > to > > > see if deleted topics are there in ZK? > > > > > > Thanks, > > > > > > Jun > > > > > > On Fri, Mar 15, 2013 at 2:19 PM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > > > Jun, > > > > > > > > So, I connected to zookeeper just using telnet, and using the 4 > letter > > > > commands. > > > > > > > > If I do a dump: > > > > > > > > I do not see anything but valid topics, and valid consumer/owners > > > mappings. > > > > > > > > If I check watches, I see all the 1000's of bogus topics, e.c.: > > > > > > > > wchc: > > > > .... > > > > /brokers/topics/<obsolete-topic> > > > > .... > > > > > > > > or > > > > > > > > wchp > > > > .... > > > > /brokers/topics/<obsolete-topic>. > > > > 0x383d6fe1b83f0008 > > > > 0x383d6fe1b83f0009 > > > > 0x7c3d6f9020d00041 > > > > 0x833d6fe2826e0000 > > > > 0x383d6fe1b83f000a > > > > .... > > > > > > > > The kafka and zookeeper servers, as well as the consumer apps, are > > rolled > > > > several times a day, automatically. > > > > > > > > I'm not sure how to remove the watches, is that what I need to do? > > > > > > > > Since this is a highly available system, can I bring down each zk > > server > > > 1 > > > > at a time, and clear the bogus topics (or watches)? How do I do > this? > > > > > > > > Jason > > > > > > > > > > > > > > > > On Fri, Mar 15, 2013 at 8:17 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > > > Could you check if the following path for a deleted topic exists in > > ZK? > > > > It > > > > > should have no children. > > > > > > > > > > /brokers/topics/[topic] > > > > > > > > > > If this is the case, try manually removing those paths from ZK > (when > > > the > > > > > brokers and the consumers are down). > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Thu, Mar 14, 2013 at 2:03 PM, Jason Rosenberg <j...@squareup.com > > > > > > wrote: > > > > > > > > > > > Hi Neha, > > > > > > > > > > > > So I did this, but I still see the full list of topics (most of > > which > > > > > have > > > > > > been deleted), in the consumer logs, e.g.: > > > > > > > > > > > > consumer.ZookeeperConsumerConnector - > > > > > > samsa-consumer-graphite_alg2.sjc1.square-1363290849309-2816c1cb > > > Topics > > > > to > > > > > > consume = List(<a bazillion topics, most of which have been > deleted > > > > > listed > > > > > > here>) > > > > > > > > > > > > I select topics using the white list topic selector (using a > > > wild-card > > > > > > pattern, etc.). > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > Does zookeeper keep separate info for the consumer, that needs to > > be > > > > > > removed? > > > > > > > > > > > > Jason > > > > > > > > > > > > On Thu, Mar 7, 2013 at 10:20 PM, Neha Narkhede < > > > > neha.narkh...@gmail.com > > > > > > >wrote: > > > > > > > > > > > > > In Kafka 0.7.2, data log directory delete and then rolling > > restart > > > > will > > > > > > > work to get rid of the topic completely from zookeeper as well > as > > > the > > > > > > > broker. > > > > > > > This is because the broker registers the topics and partitions > > from > > > > its > > > > > > > data directory into zookeeper on startup. > > > > > > > > > > > > > > Thanks, > > > > > > > Neha > > > > > > > > > > > > > > > > > > > > > On Thu, Mar 7, 2013 at 9:13 PM, Jason Rosenberg < > > j...@squareup.com> > > > > > > wrote: > > > > > > > > > > > > > > > Thanks Neha, > > > > > > > > > > > > > > > > So are you saying that on 0.7.2, to delete a topic I need > only > > > > remove > > > > > > > it's > > > > > > > > data log directory from each broker, and the restart the > > brokers? > > > > Is > > > > > > it > > > > > > > ok > > > > > > > > if it's a rolling restart? > > > > > > > > > > > > > > > > For some reason I thought I also had to do something on > > zookeeper > > > > > > servers > > > > > > > > too, to remove the topics. Is that not true? > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 6:07 PM, Neha Narkhede < > > > > > neha.narkh...@gmail.com > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > >> The topics get automagically created if they don't > exist. > > > > Will > > > > > > > > > that still exist in 0.8? > > > > > > > > > > > > > > > > > > In 0.8, this is controlled by the auto.create.topics.enable > > > > config > > > > > on > > > > > > > the > > > > > > > > > brokers. If this is set to true, topics will be created > when > > a > > > > > topic > > > > > > > > > metadata request is sent for a new topic. This feature is > > > > provided > > > > > to > > > > > > > aid > > > > > > > > > with the 0.7->0.8 migration tool and the mirror maker > tools. > > > > These > > > > > > > tools > > > > > > > > > copy data for almost all topics from one cluster to the > > other, > > > so > > > > > > auto > > > > > > > > > creating topics makes it much easier to migrate data > between > > > > > > clusters. > > > > > > > > > > > > > > > > > > >> What's the procedure for deleting topics in kafka 0.7.2? > > > > > > > > > > > > > > > > > > There is no official way to delete a topic. You have to > > delete > > > > the > > > > > > data > > > > > > > > > directory and bounce the brokers to take note of that. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Neha > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 4:20 PM, Jason Rosenberg < > > > > j...@squareup.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Thanks Neha, > > > > > > > > > > > > > > > > > > > > I added a comment to that ticket. > > > > > > > > > > > > > > > > > > > > Another question, I'm not sure if the behavior is > different > > > in > > > > > 0.8 > > > > > > > from > > > > > > > > > > 0.7. But currently, I'm not predeclaring topics, before > > > > sending > > > > > > > > messages > > > > > > > > > > to topics. The topics get automagically created if they > > > don't > > > > > > exist. > > > > > > > > > Will > > > > > > > > > > that still exist in 0.8? If so, it should also be > > automatic > > > > > > (using a > > > > > > > > > > default expiration for all such topics), that they > > > > automagically > > > > > go > > > > > > > > away > > > > > > > > > > after no longer being in use. > > > > > > > > > > > > > > > > > > > > What's the procedure for deleting topics in kafka 0.7.2? > > > > > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 3:38 PM, Neha Narkhede < > > > > > > > neha.narkh...@gmail.com > > > > > > > > > > >wrote: > > > > > > > > > > > > > > > > > > > > > We plan to include support for deleting topics before > the > > > > > > official > > > > > > > > > > release. > > > > > > > > > > > The JIRA tracking this feature is > > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-330 > > > > > > > > > > > > > > > > > > > > > > This is mainly an admin tool and is proposed to be run > > > > > manually. > > > > > > I > > > > > > > > > think > > > > > > > > > > > your suggestion is to set some sort of expiration time > on > > > > > topics > > > > > > so > > > > > > > > > when > > > > > > > > > > > all data gets garbage collected and the expiration time > > > > lapses, > > > > > > the > > > > > > > > > topic > > > > > > > > > > > is automatically deleted. > > > > > > > > > > > > > > > > > > > > > > For this expiration to work, we need some mechanism of > > > > knowing > > > > > > the > > > > > > > > size > > > > > > > > > > of > > > > > > > > > > > the data logs for that topic on all brokers. And only > > when > > > > its > > > > > > > empty > > > > > > > > > for > > > > > > > > > > > all partitions of that topic, we delete it. This > requires > > > > some > > > > > > API > > > > > > > > that > > > > > > > > > > > will expose the data size of a topic on a broker, which > > we > > > > > don't > > > > > > > have > > > > > > > > > > right > > > > > > > > > > > now. It might be worth thinking about it. Do you mind > > > posting > > > > > > your > > > > > > > > > > > suggestions for this tool on that JIRA ? > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > Neha > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Mar 6, 2013 at 3:15 PM, Jason Rosenberg < > > > > > > j...@squareup.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > I suspect this is not currently supported, but it > seems > > > to > > > > be > > > > > > for > > > > > > > > us > > > > > > > > > a > > > > > > > > > > > real > > > > > > > > > > > > use case. > > > > > > > > > > > > > > > > > > > > > > > > If we have a topic that is no longer receiving > > messages, > > > > and > > > > > > all > > > > > > > > > > messages > > > > > > > > > > > > have been removed from the brokers, after the > > > > > > log_retention_hours > > > > > > > > has > > > > > > > > > > > > expired, I'd love to have it then automatically > remove > > > the > > > > > > topic > > > > > > > > from > > > > > > > > > > > > kafka. > > > > > > > > > > > > > > > > > > > > > > > > If subsequently a new message arrives for that topic, > > it > > > > > could > > > > > > > just > > > > > > > > > > > > recreate the topic as originally, and continue on. > > > > > > > > > > > > > > > > > > > > > > > > Is this easy to implement? > > > > > > > > > > > > Has it been requested? > > > > > > > > > > > > Does it make sense? > > > > > > > > > > > > Should I file a jira? > > > > > > > > > > > > > > > > > > > > > > > > Currently, we had a test client that inadvertently > > > created > > > > > > 1000's > > > > > > > > of > > > > > > > > > > > > topics. The job of deleting all these appears to not > > be > > > a > > > > > > > > cumbersome > > > > > > > > > > > > manual process (are there any quick links to the > > > > recommended > > > > > > way > > > > > > > > for > > > > > > > > > > > doing > > > > > > > > > > > > this?). > > > > > > > > > > > > > > > > > > > > > > > > I am using 0.7.2, but will eventually upgrade to 0.8, > > > once > > > > > it's > > > > > > > > > > released > > > > > > > > > > > to > > > > > > > > > > > > beta.... > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > Jason > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
