I have a 16 broker kafka cluster. There is a topic with 32 partitions containing real time data and on the other side, I have 32 boxes w/ 1 consumer reading from these partitions.
Today our deployment strategy is stop, deploy and start the processes on all the 32 consumers. This triggers re-balancing and takes a long period of time (> 1hr). Such a long pause isn't good for real time processing. I was thinking of rolling deploy but I think that will still cause re-balancing b/c we will still have consumers go down and come up. How do you deploy to consumers without triggering re-balancing (or triggering one that doesn't affect your SLA) when doing real time processing? Thanks, Praveen
