Hi Francis, The rebalancing is your consumer group, it looks like your consumer is repeatedly entering and leaving when hitting that error. Do you have logs from the consumer side?
Cheers, Liam On Tue, 5 Apr 2022 at 11:30, Francis Chuang <francischu...@apache.org> wrote: > Not sure if this is due to the broker rebalancing multiple times every > second for the topic. Relevant logs for the 3 brokers below: > > Broker 1: > [2022-04-04 23:14:24,232] INFO Sent auto-creation request for > Set(test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704) to the > active controller. (kafka.server.DefaultAutoTopicCreationManager) > [2022-04-04 23:14:24,234] INFO [Controller 2] CreateTopics result(s): > CreatableTopic(name='test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704', > > numPartitions=1, replicationFactor=1, assignments=[], configs=[]): > SUCCESS (org.apache.kafka.controller.ReplicationControlManager) > [2022-04-04 23:14:24,234] INFO [Controller 2] Created topic > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 with topic > ID JujvHxuwSF65PjxapwU4KA. > (org.apache.kafka.controller.ReplicationControlManager) > [2022-04-04 23:14:24,235] INFO [Controller 2] Created partition > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704-0 with topic > ID JujvHxuwSF65PjxapwU4KA and PartitionRegistration(replicas=[3], > isr=[3], removingReplicas=[], addingReplicas=[], leader=3, > leaderEpoch=0, partitionEpoch=0). > (org.apache.kafka.controller.ReplicationControlManager) > > Broker 2 (there are a lot more rebalancing entries in the log, so this > is just a sample): > [2022-04-04 23:14:23,787] INFO [GroupCoordinator 3]: Dynamic Member with > unknown member id joins group > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 in Empty > state. Created a new member id > sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8 for this member and add to > the group. (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:23,799] INFO [GroupCoordinator 3]: Preparing to > rebalance group > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 in state > PreparingRebalance with old generation 0 (__consumer_offsets-15) > (reason: Adding new member sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8 > with group instance id None) (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:23,808] INFO [GroupCoordinator 3]: Stabilized group > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 generation 1 > (__consumer_offsets-15) with 1 members > (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:23,818] INFO [GroupCoordinator 3]: Assignment received > from leader sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8 for group > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 for > generation 1. The group has 1 members, 0 of which are static. > (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:24,030] INFO [Controller 3] Created topic > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 with topic > ID zVBVJQfOSrGmpjlYPEvv0w. > (org.apache.kafka.controller.ReplicationControlManager) > [2022-04-04 23:14:24,030] INFO [Controller 3] Created partition > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71-0 with topic > ID zVBVJQfOSrGmpjlYPEvv0w and PartitionRegistration(replicas=[2], > isr=[2], removingReplicas=[], addingReplicas=[], leader=2, > leaderEpoch=0, partitionEpoch=0). > (org.apache.kafka.controller.ReplicationControlManager) > [2022-04-04 23:14:24,208] INFO [GroupCoordinator 3]: Preparing to > rebalance group > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 in state > PreparingRebalance with old generation 1 (__consumer_offsets-15) > (reason: Removing member sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8 on > LeaveGroup) (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:24,210] INFO [GroupCoordinator 3]: Group > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 with > generation 2 is now empty (__consumer_offsets-15) > (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:24,216] INFO [GroupCoordinator 3]: Member > MemberMetadata(memberId=sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8, > groupInstanceId=None, clientId=sarama, clientHost=/172.21.0.6, > sessionTimeoutMs=10000, rebalanceTimeoutMs=60000, > supportedProtocols=List(range)) has left group > test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 through > explicit `LeaveGroup` request (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:24,494] INFO [GroupCoordinator 3]: Dynamic Member with > unknown member id joins group > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 in Empty > state. Created a new member id > sarama-9affaa64-3233-4deb-978e-771fc790cf34 for this member and add to > the group. (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:24,495] INFO [GroupCoordinator 3]: Preparing to > rebalance group > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 in state > PreparingRebalance with old generation 0 (__consumer_offsets-5) (reason: > Adding new member sarama-9affaa64-3233-4deb-978e-771fc790cf34 with group > instance id None) (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:24,499] INFO [GroupCoordinator 3]: Stabilized group > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 generation 1 > (__consumer_offsets-5) with 1 members > (kafka.coordinator.group.GroupCoordinator) > [2022-04-04 23:14:24,504] INFO [GroupCoordinator 3]: Assignment received > from leader sarama-9affaa64-3233-4deb-978e-771fc790cf34 for group > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 for > generation 1. The group has 1 members, 0 of which are static. > (kafka.coordinator.group.GroupCoordinator) > ... > > Broker 3: > [2022-04-04 23:14:24,772] INFO [Controller 1] Created topic > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 with topic > ID JujvHxuwSF65PjxapwU4KA. > (org.apache.kafka.controller.ReplicationControlManager) > [2022-04-04 23:14:24,772] INFO [Controller 1] Created partition > test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704-0 with topic > ID JujvHxuwSF65PjxapwU4KA and PartitionRegistration(replicas=[3], > isr=[3], removingReplicas=[], addingReplicas=[], leader=3, > leaderEpoch=0, partitionEpoch=0). > (org.apache.kafka.controller.ReplicationControlManager) > > On 2/04/2022 7:56 pm, Francis Chuang wrote: > > Hey Everyone, > > > > I have a Go library that publishes and consumes from Kafka using > > Shopify's sarama library. > > > > Initially, my test environment used Kafka and Zookeeper. Recently, I > > updated it to use Kafka in Kraft (no-zookeeper) mode. I noticed > > previously passing and stable tests started failing with the "kafka > > server: Request was for a topic or partition that does not exist on this > > broker." error message. > > > > Here's what I am doing: > > - Creating a Kafka consumer > > - Using the consumer or consumer group to subscribe to a topic that does > > not exist > > > > Note, in my project it is valid to subscribe to a topic that does not > > exist yet and receive messages from it when publishers write to it later. > > > > The error message does not reproduce consistently, so it takes a few > > tries for the error message to show up. However, the error will > > eventually show for Kafka in Kraft mode, but never in Zookeeper mode. > > > > I have a repository demonstrating this problem here: > > https://github.com/F21/kafka-topic-does-not-exist > > > > It basically creates a kafka consumer and consumers a topic that does > > not exist multiple times to trigger the error. > > > > To run it with Kafka and Zookeeper, use `docker compose -f > > docker-compose-zk.yml` > > > > To run it in Kraft mode, use `docker compose run test` > > > > After running each type of test, use `docker compose down` to tear down > > the environment completely. > > > > Can someone please shed some light on why this is happening? > > > > Thanks, > > Francis > > > > > >
