Hi Simon,
This is a known issue in kafka v3.3.1, and will be fixed in kafka v3.3.2
and v3.4.0.
Here's the JIRA issue: https://issues.apache.org/jira/browse/KAFKA-14337
Thank you
Luke
On Tue, Dec 20, 2022 at 7:13 PM Simon Dahlbacka <simon.dahlba...@gmail.com>
wrote:
> I am running a single node cluster in KRaft mode on
> confluentinc/cp-kafka:7.3.0-1-ubi8
> aka kafka 3.3.x
>
> I cannot recreate a topic 'base.m3.ocusma' (but I can create,delete,create
> other topics just fine.
>
> the topic is deleted:
>
> ls /opt/kafka/data-0/logs/base.m3.ocusma
>
> ls: cannot access '/opt/kafka/data-0/logs/base.m3.ocusma': No such file or
> directory
>
>
> When I try to create the topic I see the following in the kafka log:
>
> [2022-12-20 09:38:15,882] WARN [Controller 0] createTopics: failed with
> unknown server exception NoSuchElementException at epoch 93 in 1311 us.
> Renouncing leadership and reverting to the last committed offset 2037058.
> (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:15 am java.util.NoSuchElementException
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:167)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.timeline.SnapshottableHashTable$CurrentIterator.next(SnapshottableHashTable.java:139)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.timeline.TimelineHashSet$ValueIterator.next(TimelineHashSet.java:120)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.ReplicationControlManager.validateNewTopicNames(ReplicationControlManager.java:799)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.ReplicationControlManager.createTopics(ReplicationControlManager.java:567)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.QuorumController.lambda$createTopics$7(QuorumController.java:1832)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.controller.QuorumController$ControllerWriteEvent.run(QuorumController.java:767)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.queue.KafkaEventQueue$EventContext.run(KafkaEventQueue.java:121)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.handleEvents(KafkaEventQueue.java:200)
> Tue, Dec 20 2022 11:38:15 am at
>
> org.apache.kafka.queue.KafkaEventQueue$EventHandler.run(KafkaEventQueue.java:173)
> Tue, Dec 20 2022 11:38:15 am at
> java.base/java.lang.Thread.run(Thread.java:829)
> Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,884] INFO [RaftManager
> nodeId=0] Received user request to resign from the current epoch 93
> (org.apache.kafka.raft.KafkaRaftClient)
> Tue, Dec 20 2022 11:38:15 am [2022-12-20 09:38:15,889] INFO [RaftManager
> nodeId=0] Completed transition to ResignedState(localId=0, epoch=93,
> voters=[0], electionTimeoutMs=1515, unackedVoters=[],
> preferredSuccessors=[]) (org.apache.kafka.raft.QuorumState)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,837] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,840] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,892] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,924] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,926] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,927] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:16 am [2022-12-20 09:38:16,978] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,021] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,021] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,022] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,072] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,109] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,110] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,111] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,164] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,183] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,184] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,184] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,235] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,252] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,253] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,254] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,303] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,324] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,325] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,326] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,378] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,399] ERROR [Controller 0]
> processBrokerHeartbeat: unable to start processing because of
> NotControllerException. (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,399] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat] Client requested
> disconnect from node 0 (org.apache.kafka.clients.NetworkClient)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,403] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,410] INFO [RaftManager
> nodeId=0] Completed transition to CandidateState(localId=0, epoch=94,
> retries=1, electionTimeoutMs=1303) (org.apache.kafka.raft.QuorumState)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,410] INFO [Controller 0]
> In the new epoch 94, the leader is (none).
> (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,418] INFO [RaftManager
> nodeId=0] Completed transition to Leader(localId=0, epoch=94,
> epochStartOffset=2037059, highWatermark=Optional.empty,
> voterStates={0=ReplicaState(nodeId=0, endOffset=Optional.empty,
> lastFetchTimestamp=-1, lastCaughtUpTimestamp=-1,
> hasAcknowledgedLeader=true)}) (org.apache.kafka.raft.QuorumState)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,419] INFO [Controller 0]
> Becoming the active controller at epoch 94, committed offset 2037058,
> committed epoch 93 (org.apache.kafka.controller.QuorumController)
> Tue, Dec 20 2022 11:38:17 am [2022-12-20 09:38:17,450] INFO
> [BrokerToControllerChannelManager broker=0 name=heartbeat]: Recorded new
> controller, from now on will use broker
> kafka-0.kafka-headless.core.svc.cluster.local:9093 (id: 0 rack: null)
> (kafka.server.BrokerToControllerRequestThread)
>
>
> What has happened and is there a way of fixing?
>
> regards,
> Simon
>
