Hi Simon, This is a known issue in kafka v3.3.1, and will be fixed in kafka v3.3.2 and v3.4.0. Here's the JIRA issue: https://issues.apache.org/jira/browse/KAFKA-14337
Thank you Luke On Tue, Dec 20, 2022 at 7:13 PM Simon Dahlbacka <simon.dahlba...@gmail.com> wrote: > I am running a single node cluster in KRaft mode on > confluentinc/cp-kafka:7.3.0-1-ubi8 > aka kafka 3.3.x > > I cannot recreate a topic 'base.m3.ocusma' (but I can create,delete,create > other topics just fine. > > the topic is deleted: > > ls /opt/kafka/data-0/logs/base.m3.ocusma > > ls: cannot access '/opt/kafka/data-0/logs/base.m3.ocusma': No such file or > directory > > > When I try to create the topic I see the following in the kafka log: > > [2022-12-20 09:38:15,882] WARN [Controller 0] createTopics: failed with > unknown server exception NoSuchElementException at epoch 93 in 1311 us. > Renouncing leadership and reverting to the last committed offset 2037058. > (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:15 am java.util.NoSuchElementException > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:167) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:139) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.timeline.TimelineHashSet$ValueIterator.next(TimelineHashSet.java:120) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.controller.ReplicationControlManager.validateNewTopicNames(ReplicationControlManager.java:799) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.controller.ReplicationControlManager.createTopics(ReplicationControlManager.java:567) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.controller.QuorumController.lambda$createTopics$7(QuorumController.java:1832) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:767) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200) > Tue, Dec 20 2022 11:38:15 am at > > org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173) > Tue, Dec 20 2022 11:38:15 am at > java.base/java.lang.Thread.run(Thread.java:829) > Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,884] INFO [RaftManager > nodeId=0] Received user request to resign from the current epoch 93 > (org.apache.kafka.raft.KafkaRaftClient) > Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,889] INFO [RaftManager > nodeId=0] Completed transition to ResignedState(localId=0, epoch=93, > voters=[0], electionTimeoutMs=1515, unackedVoters=[], > preferredSuccessors=[]) (org.apache.kafka.raft.QuorumState) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,837] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,892] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,924] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,926] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,927] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,978] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,021] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,021] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,022] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,072] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,109] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,110] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,111] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,164] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,183] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,184] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,184] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,235] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,252] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,253] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,254] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,303] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,324] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,325] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,326] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,378] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,399] ERROR [Controller 0] > processBrokerHeartbeat: unable to start processing because of > NotControllerException. (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,399] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested > disconnect from node 0 (org.apache.kafka.clients.NetworkClient) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,403] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,410] INFO [RaftManager > nodeId=0] Completed transition to CandidateState(localId=0, epoch=94, > retries=1, electionTimeoutMs=1303) (org.apache.kafka.raft.QuorumState) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,410] INFO [Controller 0] > In the new epoch 94, the leader is (none). > (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,418] INFO [RaftManager > nodeId=0] Completed transition to Leader(localId=0, epoch=94, > epochStartOffset=2037059, highWatermark=Optional.empty, > voterStates={0=ReplicaState(nodeId=0, endOffset=Optional.empty, > lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1, > hasAcknowledgedLeader=true)}) (org.apache.kafka.raft.QuorumState) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,419] INFO [Controller 0] > Becoming the active controller at epoch 94, committed offset 2037058, > committed epoch 93 (org.apache.kafka.controller.QuorumController) > Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,450] INFO > [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new > controller, from now on will use broker > kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null) > (kafka.server.BrokerToControllerRequestThread) > > > What has happened and is there a way of fixing? > > regards, > Simon >