Yes, topic names cannot be known before hand and they will keep growing in number. Thanks. The partition grouper appears to be working.
On Fri, Jul 28, 2017, 3:28 AM Matthias J. Sax <matth...@confluent.io> wrote: > Using `PartitionGrouper` is correct. > > As you mentioned correctly, Stream scales via "max number of partitions" > and thus, be default only create one task for this case. > > Another way would be, to deploy multiple streams applications each > processing a different topic. Of course, for this you will need know all > topics names (not sure if this is the case for you). > > -Matthias > > On 7/26/17 7:38 AM, Anish Mashankar wrote: > > Hello All, > > I have more than 100 topics in Kafka with one partition each. These 100 > > topics are configured through a regex. When running the application, I > > found that there is only one task that is being spawned as the default > > partition grouper in Kafka spawns as many tasks as the maximum number of > > partitions in a group of topics. > > Hence, I created a PartitionGrouper that creates a group having only one > > partition and returns the set of such groups. The TaskID is created > > something like this: *TaskId(topic.hashCode(), partition). * > > Is there a better way through which I can guarantee that every task > listens > > only to one partition? > > > > -- Regards, Anish Samir Mashankar R&D Engineer System Insights +91-9789870733
