I'm running kafka_2.11-1.0.0 on my (test) server and version 1.3.5 of the kafka-python client library (available from PyPI). Subscribing using patterns seems not to work. I always have to enumerate a set of topics, and given that the topics are created dynamically as different kinds of content arrive at my virtual front door, I'm guaranteed to eventually miss something.
I don't know enough about the ins and outs of Kafka to know where to look for clues. I tried tailing server.log.2018-01-03-11 for a bit, but saw no obvious indication that it logs anything. (Maybe I need to crank up the verbosity?) As an example, this works: import kafka consumer = kafka.KafkaConsumer(bootstrap_servers="server:9092", group_id="skip-test") consumer.subscribe(topics=["topic1", "topic2", ...]) consumer.next() but this fails, the next() call hanging indefinitely: import kafka consumer = kafka.KafkaConsumer(bootstrap_servers="server:9092", group_id="skip-test") consumer.subscribe(pattern="^topic") consumer.next() The only way I got pattern subscription to work is by anchoring the tail end of the pattern. This wasn't sufficient: ^topic.* I needed to add the $ as well so the pattern was anchored at both ends: ^topic.*$ I don't remember seeing any sort of pattern restrictions in my reading of the docs. Shouldn't any old regular expression do the trick? Any clues/documentation pointers appreciated. Skip Montanaro
