I'm using the Python API in my clients (Conda kafka-python package, v
1.3.3). I have one client which is kind of a catch-all, subscribing to
several topics. There is just one instance, but I pass the group_id
arg to the constructor because I want to control commit points. As I
understand it, use of group_id means some partition assignment
strategy comes into play. I'm also subscribing to a list of topics
using the subscribe method. Setup looks like this:
consumer = kafka.KafkaConsumer(bootstrap_servers="mybroker:9092",
group_id="mygroup")
consumer.subscribe(topics=["topic1", "topic2", "topic3", ...])
Sometimes it seems to receive messages from all partitions (0, 1, 2).
At other times, it seems to start up only listening to just one
partition. How do I unambiguously tell it, "listen to all partitions
for each topic"? How can I confirm that in my client? When I try
logging consumer.assignment(), it's always the empty set.
I happen to know I have three partitions, so I think I could create
TopicPartition instances for each topic/partition pair. Still, I don't
think I should rely on that knowledge. Looking in kafka.partitioner, I
didn't see a "GreedyPartitioner" object, or anything else which might
obviously be used to grab everything.
Thanks,
Skip Montanaro
