Not sure if this is due to the broker rebalancing multiple times every
second for the topic. Relevant logs for the 3 brokers below:
Broker 1:
[2022-04-04 23:14:24,232] INFO Sent auto-creation request for
Set(test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704) to the
active controller. (kafka.server.DefaultAutoTopicCreationManager)
[2022-04-04 23:14:24,234] INFO [Controller 2] CreateTopics result(s):
CreatableTopic(name='test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704',
numPartitions=1, replicationFactor=1, assignments=[], configs=[]):
SUCCESS (org.apache.kafka.controller.ReplicationControlManager)
[2022-04-04 23:14:24,234] INFO [Controller 2] Created topic
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 with topic
ID JujvHxuwSF65PjxapwU4KA.
(org.apache.kafka.controller.ReplicationControlManager)
[2022-04-04 23:14:24,235] INFO [Controller 2] Created partition
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704-0 with topic
ID JujvHxuwSF65PjxapwU4KA and PartitionRegistration(replicas=[3],
isr=[3], removingReplicas=[], addingReplicas=[], leader=3,
leaderEpoch=0, partitionEpoch=0).
(org.apache.kafka.controller.ReplicationControlManager)
Broker 2 (there are a lot more rebalancing entries in the log, so this
is just a sample):
[2022-04-04 23:14:23,787] INFO [GroupCoordinator 3]: Dynamic Member with
unknown member id joins group
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 in Empty
state. Created a new member id
sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8 for this member and add to
the group. (kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:23,799] INFO [GroupCoordinator 3]: Preparing to
rebalance group
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 in state
PreparingRebalance with old generation 0 (__consumer_offsets-15)
(reason: Adding new member sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8
with group instance id None) (kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:23,808] INFO [GroupCoordinator 3]: Stabilized group
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 generation 1
(__consumer_offsets-15) with 1 members
(kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:23,818] INFO [GroupCoordinator 3]: Assignment received
from leader sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8 for group
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 for
generation 1. The group has 1 members, 0 of which are static.
(kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:24,030] INFO [Controller 3] Created topic
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 with topic
ID zVBVJQfOSrGmpjlYPEvv0w.
(org.apache.kafka.controller.ReplicationControlManager)
[2022-04-04 23:14:24,030] INFO [Controller 3] Created partition
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71-0 with topic
ID zVBVJQfOSrGmpjlYPEvv0w and PartitionRegistration(replicas=[2],
isr=[2], removingReplicas=[], addingReplicas=[], leader=2,
leaderEpoch=0, partitionEpoch=0).
(org.apache.kafka.controller.ReplicationControlManager)
[2022-04-04 23:14:24,208] INFO [GroupCoordinator 3]: Preparing to
rebalance group
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 in state
PreparingRebalance with old generation 1 (__consumer_offsets-15)
(reason: Removing member sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8 on
LeaveGroup) (kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:24,210] INFO [GroupCoordinator 3]: Group
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 with
generation 2 is now empty (__consumer_offsets-15)
(kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:24,216] INFO [GroupCoordinator 3]: Member
MemberMetadata(memberId=sarama-f36324b2-ec03-4c44-a8dc-67188c1986c8,
groupInstanceId=None, clientId=sarama, clientHost=/172.21.0.6,
sessionTimeoutMs=10000, rebalanceTimeoutMs=60000,
supportedProtocols=List(range)) has left group
test-bounded-context-1-c42b0ec0-7646-4f9e-b82c-17f380e07f71 through
explicit `LeaveGroup` request (kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:24,494] INFO [GroupCoordinator 3]: Dynamic Member with
unknown member id joins group
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 in Empty
state. Created a new member id
sarama-9affaa64-3233-4deb-978e-771fc790cf34 for this member and add to
the group. (kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:24,495] INFO [GroupCoordinator 3]: Preparing to
rebalance group
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 in state
PreparingRebalance with old generation 0 (__consumer_offsets-5) (reason:
Adding new member sarama-9affaa64-3233-4deb-978e-771fc790cf34 with group
instance id None) (kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:24,499] INFO [GroupCoordinator 3]: Stabilized group
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 generation 1
(__consumer_offsets-5) with 1 members
(kafka.coordinator.group.GroupCoordinator)
[2022-04-04 23:14:24,504] INFO [GroupCoordinator 3]: Assignment received
from leader sarama-9affaa64-3233-4deb-978e-771fc790cf34 for group
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 for
generation 1. The group has 1 members, 0 of which are static.
(kafka.coordinator.group.GroupCoordinator)
...
Broker 3:
[2022-04-04 23:14:24,772] INFO [Controller 1] Created topic
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704 with topic
ID JujvHxuwSF65PjxapwU4KA.
(org.apache.kafka.controller.ReplicationControlManager)
[2022-04-04 23:14:24,772] INFO [Controller 1] Created partition
test-bounded-context-1-6cdaff21-6562-418a-92e8-1f2d8fa0c704-0 with topic
ID JujvHxuwSF65PjxapwU4KA and PartitionRegistration(replicas=[3],
isr=[3], removingReplicas=[], addingReplicas=[], leader=3,
leaderEpoch=0, partitionEpoch=0).
(org.apache.kafka.controller.ReplicationControlManager)
On 2/04/2022 7:56 pm, Francis Chuang wrote:
Hey Everyone,
I have a Go library that publishes and consumes from Kafka using
Shopify's sarama library.
Initially, my test environment used Kafka and Zookeeper. Recently, I
updated it to use Kafka in Kraft (no-zookeeper) mode. I noticed
previously passing and stable tests started failing with the "kafka
server: Request was for a topic or partition that does not exist on this
broker." error message.
Here's what I am doing:
- Creating a Kafka consumer
- Using the consumer or consumer group to subscribe to a topic that does
not exist
Note, in my project it is valid to subscribe to a topic that does not
exist yet and receive messages from it when publishers write to it later.
The error message does not reproduce consistently, so it takes a few
tries for the error message to show up. However, the error will
eventually show for Kafka in Kraft mode, but never in Zookeeper mode.
I have a repository demonstrating this problem here:
https://github.com/F21/kafka-topic-does-not-exist
It basically creates a kafka consumer and consumers a topic that does
not exist multiple times to trigger the error.
To run it with Kafka and Zookeeper, use `docker compose -f
docker-compose-zk.yml`
To run it in Kraft mode, use `docker compose run test`
After running each type of test, use `docker compose down` to tear down
the environment completely.
Can someone please shed some light on why this is happening?
Thanks,
Francis
