Hey Everyone,
I have a Go library that publishes and consumes from Kafka using
Shopify's sarama library.
Initially, my test environment used Kafka and Zookeeper. Recently, I
updated it to use Kafka in Kraft (no-zookeeper) mode. I noticed
previously passing and stable tests started failing with the "kafka
server: Request was for a topic or partition that does not exist on this
broker." error message.
Here's what I am doing:
- Creating a Kafka consumer
- Using the consumer or consumer group to subscribe to a topic that does
not exist
Note, in my project it is valid to subscribe to a topic that does not
exist yet and receive messages from it when publishers write to it later.
The error message does not reproduce consistently, so it takes a few
tries for the error message to show up. However, the error will
eventually show for Kafka in Kraft mode, but never in Zookeeper mode.
I have a repository demonstrating this problem here:
https://github.com/F21/kafka-topic-does-not-exist
It basically creates a kafka consumer and consumers a topic that does
not exist multiple times to trigger the error.
To run it with Kafka and Zookeeper, use `docker compose -f
docker-compose-zk.yml`
To run it in Kraft mode, use `docker compose run test`
After running each type of test, use `docker compose down` to tear down
the environment completely.
Can someone please shed some light on why this is happening?
Thanks,
Francis
