Hello Alexis, Could you share your findings about the command line tool? We can try to resolve if there's any issues.
Guozhang On Fri, Mar 4, 2016 at 3:13 PM, Alexis Midon < alexis.mi...@airbnb.com.invalid> wrote: > The command line tool that ships with Kafka is error prone. > > Our standard procedure is: > 1. spin up the new broker > 2. use `kafkat drain <old broker id> [--brokers <new broker id>] > 3. shut down old broker > > The `drain` command will generate and submit a partition assignment plan > where the new broker id replaces the old one. It's pretty much a "gsub(old, > new)". > > We do it regularly. It's almost a mundane operation. The only challenge is > the volume of data being transferred over the network. Since there is no > throttling mechanism, the network is sometime saturated which might impact > other consumers/producers > > See https://github.com/airbnb/kafkat > > > > > > On Fri, Mar 4, 2016 at 7:28 AM Todd Palino <tpal...@gmail.com> wrote: > > > To answer your questions… > > > > 1 - Not in the way you want it to. There is a setting for automatic > leader > > election (which I do not recommend anyone use at this time), but all that > > does is pick which of the currently assigned replicas should be the > leader. > > It does not reassign partitions from one broker to another. Kafka does > not > > have a facility for doing this automatically. > > > > 2 - No. The most you can do is move all the partitions off and then > > immediately shut down the broker process. Any broker that is live in the > > cluster can, and will, get partitions assigned to it by the controller. > > > > For what you want to do, you need you use the partition reassignment > > command line tool that ships with Kafka to reassign partitions from the > old > > broker to the new one. Once that is complete, you can double check that > the > > old broker has no partitions left and shut it down. I have a tool that we > > use internally to make this a lot easier, and I’m in the process of > getting > > a repository set up to make it available via open source. It allows for > > more easily removing and adding brokers, and rebalancing partitions in a > > cluster without having to craft the reassignments by hand. > > > > -Todd > > > > > > On Fri, Mar 4, 2016 at 5:07 AM, Muqtafi Akhmad <muqt...@traveloka.com> > > wrote: > > > > > dear Kafka users, > > > > > > I have some questions regarding decommissioning kafka broker node and > > > replacing it with the new one. Lets say that we have three broker nodes > > and > > > each topic in Kafka has replication factor = 3, we upgrade one node > with > > > the following steps : > > > 1. add one broker node to cluster > > > 2. shutdown old broker node > > > > > > My questions are > > > 1. When we add one new broker to the cluster will it trigger Kafka > topic > > / > > > group leadership rebalance? > > > 2. Is there any way to disable the to-be-decommissioned node to hold no > > > topic/group leadership (acting as passive copy) so that it can be > > > decommissioned with minimal effect to Kafka clients? > > > > > > Thank you, > > > > > > -- > > > Muqtafi Akhmad > > > Software Engineer > > > Traveloka > > > > > > > > > > > -- > > *—-* > > *Todd Palino* > > Staff Site Reliability Engineer > > Data Infrastructure Streaming > > > > > > > > linkedin.com/in/toddpalino > > > -- -- Guozhang
