Using `PartitionGrouper` is correct. As you mentioned correctly, Stream scales via "max number of partitions" and thus, be default only create one task for this case.
Another way would be, to deploy multiple streams applications each processing a different topic. Of course, for this you will need know all topics names (not sure if this is the case for you). -Matthias On 7/26/17 7:38 AM, Anish Mashankar wrote: > Hello All, > I have more than 100 topics in Kafka with one partition each. These 100 > topics are configured through a regex. When running the application, I > found that there is only one task that is being spawned as the default > partition grouper in Kafka spawns as many tasks as the maximum number of > partitions in a group of topics. > Hence, I created a PartitionGrouper that creates a group having only one > partition and returns the set of such groups. The TaskID is created > something like this: *TaskId(topic.hashCode(), partition). * > Is there a better way through which I can guarantee that every task listens > only to one partition? >
signature.asc
Description: OpenPGP digital signature
