@Carl, There is nothing under /admin/delete_topics other than
[] And nothing under /admin other than delete_topics :) The topics DO exist, however, under /config/topics! We may be on to something. I will remove them here and see if that clears it up. Thanks so much for all the help! Chris On Thu, Jul 20, 2017 at 10:37 PM, Chris Neal <cwn...@gmail.com> wrote: > Thanks again for the replies. VERY much appreciated. I'll check both > /admin/delete_topics and /config/topics. > > Chris > > On Thu, Jul 20, 2017 at 9:22 PM, Carl Haferd <chaf...@groupon.com.invalid> > wrote: > >> If delete normally works, there would hopefully be some log entries when >> it >> fails. Are there any unusual zookeeper entries in the >> /admin/delete_topics >> path or in the other /admin folders? >> >> Does the topic name still exist in zookeeper under /config/topics? If so, >> that should probably deleted as well. >> >> Carl >> >> On Thu, Jul 20, 2017 at 6:42 PM, Chris Neal <cwn...@gmail.com> wrote: >> >> > Delete is definitely there. The delete worked fine, based on the fact >> that >> > there is nothing in Zookeeper, and that the controller reported that the >> > delete was successful, it's just something seems to have gotten out of >> > sync. >> > >> > delete.topic.enabled is true. I've successfully deleted topics in the >> > past, so I know it *should* work. :) >> > >> > I also had already checked in Zookeeper, and there is no directory for >> the >> > topics under /brokers/topics.... Very strange indeed. >> > >> > If I just remove the log directories from the filesystem, is that >> enough to >> > get the broker to stop asking about the topics? I would guess there >> would >> > need to be more than just that, but I could be wrong. >> > >> > Thanks guys for the suggestions though! >> > >> > On Thu, Jul 20, 2017 at 8:19 PM, Stephen Powis <spo...@salesforce.com> >> > wrote: >> > >> > > I could be totally wrong, but I seem to recall that delete wasn't >> fully >> > > implemented in 0.8.x? >> > > >> > > On Fri, Jul 21, 2017 at 10:10 AM, Carl Haferd >> > <chaf...@groupon.com.invalid >> > > > >> > > wrote: >> > > >> > > > Chris, >> > > > >> > > > You could first check to make sure that delete.topic.enable is true >> and >> > > try >> > > > deleting again if not. If that doesn't work with 0.8.1.1 you might >> > need >> > > to >> > > > manually remove the topic's log files from the configured log.dirs >> > folder >> > > > on each broker in addition to removing the topic's zookeeper path. >> > > > >> > > > Carl >> > > > >> > > > On Thu, Jul 20, 2017 at 10:06 AM, Chris Neal <cwn...@gmail.com> >> wrote: >> > > > >> > > > > Hi all, >> > > > > >> > > > > I have a weird situation here. I have deleted a few topics on my >> > > 0.8.1.1 >> > > > > cluster (old, I know...). The deletes succeeded according to the >> > > > > controller.log: >> > > > > >> > > > > [2017-07-20 16:40:31,175] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: >> > > > > [Set(perf_doorway-supplier-adapter-uat_raw)], new partition >> replica >> > > > > assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > [2017-07-20 16:40:33,507] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: >> > > > > [Set(perf_doorway-supplier-scheduler-uat_raw)], new partition >> > replica >> > > > > assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > [2017-07-20 16:40:36,504] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: [Set(perf_gocontent-uat_raw)], >> new >> > > > > partition replica assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > [2017-07-20 16:40:38,290] INFO [TopicChangeListener on Controller >> 1]: >> > > New >> > > > > topics: [Set()], deleted topics: [Set(perf_goplatform-uat_raw)], >> new >> > > > > partition replica assignment [Map()] >> > > > > (kafka.controller.PartitionStateMachine$TopicChangeListener) >> > > > > >> > > > > I query Zookeeper and the path is not there under /brokers/topics >> as >> > > > well. >> > > > > >> > > > > But, one of the nodes in my cluster continues to try and use them: >> > > > > >> > > > > [2017-07-20 17:04:36,723] ERROR Conditional update of path >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ >> > > > partitions/3/state >> > > > > with data >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ >> > > > > epoch":2,"isr":[1,0]} >> > > > > and expected version 69 failed due to >> > > > > org.apache.zookeeper.KeeperException$NoNodeException: >> > KeeperErrorCode >> > > = >> > > > > NoNode for >> > > > > /brokers/topics/perf_doorway-supplier-scheduler-uat_raw/ >> > > > partitions/3/state >> > > > > (kafka.utils.ZkUtils$) >> > > > > [2017-07-20 17:04:36,723] INFO Partition >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached >> > > > zkVersion >> > > > > [69] not equal to that in zookeeper, skip updating ISR >> > > > > (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,723] INFO Partition >> > > > > [perf_doorway-supplier-scheduler-uat_raw,3] on broker 1: Cached >> > > > zkVersion >> > > > > [69] not equal to that in zookeeper, skip updating ISR >> > > > > (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,764] INFO Partition >> [perf_goplatform-uat_raw,2] >> > on >> > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] >> > from >> > > > 1,0 >> > > > > to 1 (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,764] INFO Partition >> [perf_goplatform-uat_raw,2] >> > on >> > > > > broker 1: Shrinking ISR for partition [perf_goplatform-uat_raw,2] >> > from >> > > > 1,0 >> > > > > to 1 (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with >> data >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ >> > > > epoch":2,"isr":[1]} >> > > > > and expected version 70 failed due to >> > > > > org.apache.zookeeper.KeeperException$NoNodeException: >> > KeeperErrorCode >> > > = >> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/ >> > 2/state >> > > > > (kafka.utils.ZkUtils$) >> > > > > [2017-07-20 17:04:36,765] ERROR Conditional update of path >> > > > > /brokers/topics/perf_goplatform-uat_raw/partitions/2/state with >> data >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ >> > > > epoch":2,"isr":[1]} >> > > > > and expected version 70 failed due to >> > > > > org.apache.zookeeper.KeeperException$NoNodeException: >> > KeeperErrorCode >> > > = >> > > > > NoNode for /brokers/topics/perf_goplatform-uat_raw/partitions/ >> > 2/state >> > > > > (kafka.utils.ZkUtils$) >> > > > > [2017-07-20 17:04:36,765] INFO Partition >> [perf_goplatform-uat_raw,2] >> > on >> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, >> skip >> > > > > updating ISR (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,765] INFO Partition >> [perf_goplatform-uat_raw,2] >> > on >> > > > > broker 1: Cached zkVersion [70] not equal to that in zookeeper, >> skip >> > > > > updating ISR (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,981] INFO Partition >> [perf_gocontent-uat_raw,1] >> > on >> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] >> from >> > > 1,0 >> > > > > to 1 (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,981] INFO Partition >> [perf_gocontent-uat_raw,1] >> > on >> > > > > broker 1: Shrinking ISR for partition [perf_gocontent-uat_raw,1] >> from >> > > 1,0 >> > > > > to 1 (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with >> data >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ >> > > > epoch":4,"isr":[1]} >> > > > > and expected version 90 failed due to >> > > > > org.apache.zookeeper.KeeperException$NoNodeException: >> > KeeperErrorCode >> > > = >> > > > > NoNode for /brokers/topics/perf_gocontent >> -uat_raw/partitions/1/state >> > > > > (kafka.utils.ZkUtils$) >> > > > > [2017-07-20 17:04:36,988] ERROR Conditional update of path >> > > > > /brokers/topics/perf_gocontent-uat_raw/partitions/1/state with >> data >> > > > > {"controller_epoch":34,"leader":1,"version":1,"leader_ >> > > > epoch":4,"isr":[1]} >> > > > > and expected version 90 failed due to >> > > > > org.apache.zookeeper.KeeperException$NoNodeException: >> > KeeperErrorCode >> > > = >> > > > > NoNode for /brokers/topics/perf_gocontent >> -uat_raw/partitions/1/state >> > > > > (kafka.utils.ZkUtils$) >> > > > > [2017-07-20 17:04:36,988] INFO Partition >> [perf_gocontent-uat_raw,1] >> > on >> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, >> skip >> > > > > updating ISR (kafka.cluster.Partition) >> > > > > [2017-07-20 17:04:36,988] INFO Partition >> [perf_gocontent-uat_raw,1] >> > on >> > > > > broker 1: Cached zkVersion [90] not equal to that in zookeeper, >> skip >> > > > > updating ISR (kafka.cluster.Partition) >> > > > > >> > > > > I've tried a rolling restart of the cluster to see if that fixed >> it, >> > > but >> > > > it >> > > > > did not. >> > > > > >> > > > > Can someone please help me out here? I'm not sure how I can get >> > things >> > > > > back in sync. >> > > > > >> > > > > Thank you so much for your time. >> > > > > Chris >> > > > > >> > > > >> > > >> > >> > >