Hi all,
He is my usecase:
I have three message producers that submits batch of messages to Kafka
from time to time. Let's assume now that one of them just submitted 1k
messages, second one submitted some number of messages after that and
third one also submitted some messages.
I would like to make sure that, when the consumer will start to work,
messages from all producers will be processed (more or less) together.
In other words: That messages from second producer will not have to wait
for all that 1k messages to be processed first.
I have two ideas how to solve it:
1. Prepare three different Kafka topics. Each producer will write to its
dedicated topic, consumer will read from all topics. In this case
consumer will read messages in round-robin fashion (is it true?). So the
messages from second (and third) producer will not have to wait for all
messages from first producer (submitted earlier) to be processed by the
consumer.
2. Have one topic for all producers. Each producer will submit messages
only to some subset of partitions of given topic. For example we will
have 10 partitions in our topic and producers will write only to two
(or three) partitions.
And the questions are:
1. Which solution is the best?
2. Maybe there is another (even better) solution that you can recommend?
Regards,
Pawel
