I don't believe this is a safe approach to do rebalancing whilst producing into the topic. This creates mismatches in partition/replication process, unless this has been handled in the upcoming .11 release?
Kindest Regards, On 14 June 2017 at 02:52, karan alang <karan.al...@gmail.com> wrote: > Some more details about my use-case > > > When i add the new Kafka Broker, it will not be included in the ISR of the > Kafka Topic automatically. > We'll need to do Kafka rebalancing - so the load gets balanced across all > the Brokers, including the newly added Broker. > > Can this be done while the Producer is pushing data into the Kafka Topic ? > What are the implications of this ? > > Pls. advise. > > > On Tue, Jun 13, 2017 at 4:29 PM, karan alang <karan.al...@gmail.com> > wrote: > > > > > Hello, clarifying the question -> > > > > Within the same Kafka cluster, if i want to add Broker & rebalance topic > > (say topic t1, so it starts using newly added Broker) - can i do that w/o > > stopping the existing load on topic t1 ? > > > > I guess - if i can do that, i should be able to use MirrorMaker to > > replicate Topic t1 on cluster c1 => Topic t1 on Cluster t2 > > > > Is that the best way of doing it ? > > > > > > > > > > On Tue, Jun 13, 2017 at 3:14 PM, Mohammed Manna <manme...@gmail.com> > > wrote: > > > >> You should plan this ahead and make sure that your topic partition and > >> replication factors are updated correctly using kafka topics admin or a > >> batch job setup for Kafka Mirror Maker. Apache wiki explains how Mirror > >> maker can guarantee a failsafe mirroring. I cannot quite remember how > deep > >> the mirroring goes e.g. Consumer offsets etc. But my understanding is > that > >> it should do it correctly. > >> > >> Regards, > >> > >> On Tue, 13 Jun 2017 at 10:58 pm, karan alang <karan.al...@gmail.com> > >> wrote: > >> > >> > Hello All, > >> > > >> > I've streaming data flowing into Kafka topics. > >> > What are the best practices wrt. adding or removing Kafka Brokers > while > >> > data is flowing into Kafka ? > >> > > >> > for eg. Can I add Broker & re-balance the topics while the data flows > >> into > >> > Kafka topic ? > >> > > >> > Pls. advise. > >> > > >> > Thanks! > >> > > >> > > > > >
