Hi, I have some doubt regarding using kafka consumer API in kafka version 0.10.0.1.
Consider I am having a three node kafka cluster in which there are 1000 topics with single partition. Each topic will be consumed by multiple consumer groups, say 100 in parallel. Therefore totally there can be 1000*100 consumer groups consuming from kafka in parallel. My concern is whether this would have any performance impact in kafka cluster on a larger scale ? Also, my model will be starting consumer for each user request and will fetch single data from queue and consumer will be closed on request end. For the next request from same user, consumer will be started for same group again and it will be closed. This will be repeated for all the users. Whether creating and tearing down consumer in such a fashion is a proper one? If not, is there any other proper way to handle this use case? Regards, Thangaram Senthamaraikannan
