I have two Kafka consumers, subscribed to different topics and belonging to the same consumer group, having different consumer ids, running in the same thread. They are executing poll sequentially but after the first is done second seems to be stuck in poll. I tried using proper Kafka broker and embedded one for testing and nothing changes.I tried associating them with different consumer groups and that seems to be working but unfortunately, that is not a viable solution for me.
I found this in "Kafka: The Definitive Guide": "You can’t have multiple consumers that belong to the same group in one thread and you can’t have multiple threads safely use the same consumer. One consumer per thread is the rule." That quote directs me towards some form of thread cooperation due to the specific order of message processing I need to do. Can someone provide an explanation of why is it necessary to run different consumers belonging to the same consumer group, subscribed to different topics in separate threads? Test code with embedded kafka: @Test public void pollTest() { kafkaAdmin.createTopics(topics1); kafkaAdmin.createTopics(topics2); Consumer<String, String> consumer1 = createConsumer(topics1); Consumer<String, String> consumer2 = createConsumer(topics2); consumer1.poll(200); consumer2.poll(200); } Version of kafka - 2.3.0 Thank you.