Hey Mark, you could use a consumer group (check the consumer #subscribe API) to consume from 50 topics in a dynamic fashion, as long as the data processing function is the same for all the records. Consumer group could provide basic guarantees for balancing the number of partitions for each consumer if you use default assignors. I couldn't think of any cons for managing one consumer group vs multiple of them.
As for auto commit, I don't think I understand your question. You could turn it on by mutating consumer config. Boyang On Tue, Feb 25, 2020 at 11:05 PM Mark Zang <deepnight...@gmail.com> wrote: > Hi, > > I have a 20 brokers kafka cluster and there are about 50 topics to consume. > > Between creating a consumer for each topic and creating a consumer for all > 50 topics, what is the pros and cons? > > What would be the suggested way if I enable auto commit for each 10 > seconds? > > Kafka client version is 0.10.2 > > > Thanks! > > Mark >
