Hey Sachin, your observation is correct, unfortunately Kafka Streams doesn't support adding partitions online. The rebalance could not guarantee the same key routing to the same partition when the input topic partition changes, as this is the upstream producer's responsibility to consistently route the same key data, which is not resolved today.
Boyang On Wed, Apr 15, 2020 at 7:23 AM Sachin Mittal <sjmit...@gmail.com> wrote: > Hi, > We have a kafka streams application which runs multiple instances and > consumes from a source topic. > Producers produces keyed messages to this source topic. > Keyed messages are events from different sources and each source has a > unique key. > > So what essentially happens is that messages from particular source always > gets added to a particular partition. > Hence we can run multiple instances of streams application with a > particular instance processing messages for certain partitions. > We will never get into a case where messages for a source are processed by > different instances of streams application simultaneously. > > So far so good. > > Now over time new sources are added. It may so happen that we reach a > saturation point and have no option but to increase number of partitions. > > So what is the best practice to increase number of partitions. > Is there a way to ensure that existing key's messages continue to get > published on same partition as before. > And only new source's keys gets their messages published on the new > partition we add. > > If this is not possible then does kafka's re-partition mechanism ensure > that during re-balance all the previous messages of a particular key gets > moved to same partition. > I guess under this approach we would have to stop our streaming application > till re-balance is over otherwise messages for same key may get processed > by different instances of the application. > > Anyway just wanted to know how such a problem is tackled on live systems > real time, or how some of you have approached the same. > > Thanks > Sachin >
