also: . the command line tool packaging is not ideal, you have to download the (big) kafka zip, set paths, etc.
. the details of the zk cluster and kafka cluster must be repeated on the command line. with kafkat, you can use `gem install`, and a store the cluster details in a config file. On Sun, Mar 6, 2016 at 10:12 AM Alexis Midon <alexis.mi...@airbnb.com> wrote: > My recollection is that you have to come up with the partition assignment > yourself, and pass the json file as an argument. > This is quite error prone, especially during an outage. > > we quickly wrote kafkat to have a simple commands that would let us > express needs without having to deal with the assignment plan: "move topic > X on these brokers", "retire broker X", "set replication factor of topic X > to R", etc > > > > > On Sat, Mar 5, 2016 at 3:33 PM Guozhang Wang <wangg...@gmail.com> wrote: > >> Hello Alexis, >> >> Could you share your findings about the command line tool? We can try to >> resolve if there's any issues. >> >> Guozhang >> >> On Fri, Mar 4, 2016 at 3:13 PM, Alexis Midon < >> alexis.mi...@airbnb.com.invalid> wrote: >> >> > The command line tool that ships with Kafka is error prone. >> > >> > Our standard procedure is: >> > 1. spin up the new broker >> > 2. use `kafkat drain <old broker id> [--brokers <new broker id>] >> > 3. shut down old broker >> > >> > The `drain` command will generate and submit a partition assignment plan >> > where the new broker id replaces the old one. It's pretty much a >> "gsub(old, >> > new)". >> > >> > We do it regularly. It's almost a mundane operation. The only challenge >> is >> > the volume of data being transferred over the network. Since there is no >> > throttling mechanism, the network is sometime saturated which might >> impact >> > other consumers/producers >> > >> > See https://github.com/airbnb/kafkat >> > >> > >> > >> > >> > >> > On Fri, Mar 4, 2016 at 7:28 AM Todd Palino <tpal...@gmail.com> wrote: >> > >> > > To answer your questions… >> > > >> > > 1 - Not in the way you want it to. There is a setting for automatic >> > leader >> > > election (which I do not recommend anyone use at this time), but all >> that >> > > does is pick which of the currently assigned replicas should be the >> > leader. >> > > It does not reassign partitions from one broker to another. Kafka does >> > not >> > > have a facility for doing this automatically. >> > > >> > > 2 - No. The most you can do is move all the partitions off and then >> > > immediately shut down the broker process. Any broker that is live in >> the >> > > cluster can, and will, get partitions assigned to it by the >> controller. >> > > >> > > For what you want to do, you need you use the partition reassignment >> > > command line tool that ships with Kafka to reassign partitions from >> the >> > old >> > > broker to the new one. Once that is complete, you can double check >> that >> > the >> > > old broker has no partitions left and shut it down. I have a tool >> that we >> > > use internally to make this a lot easier, and I’m in the process of >> > getting >> > > a repository set up to make it available via open source. It allows >> for >> > > more easily removing and adding brokers, and rebalancing partitions >> in a >> > > cluster without having to craft the reassignments by hand. >> > > >> > > -Todd >> > > >> > > >> > > On Fri, Mar 4, 2016 at 5:07 AM, Muqtafi Akhmad <muqt...@traveloka.com >> > >> > > wrote: >> > > >> > > > dear Kafka users, >> > > > >> > > > I have some questions regarding decommissioning kafka broker node >> and >> > > > replacing it with the new one. Lets say that we have three broker >> nodes >> > > and >> > > > each topic in Kafka has replication factor = 3, we upgrade one node >> > with >> > > > the following steps : >> > > > 1. add one broker node to cluster >> > > > 2. shutdown old broker node >> > > > >> > > > My questions are >> > > > 1. When we add one new broker to the cluster will it trigger Kafka >> > topic >> > > / >> > > > group leadership rebalance? >> > > > 2. Is there any way to disable the to-be-decommissioned node to >> hold no >> > > > topic/group leadership (acting as passive copy) so that it can be >> > > > decommissioned with minimal effect to Kafka clients? >> > > > >> > > > Thank you, >> > > > >> > > > -- >> > > > Muqtafi Akhmad >> > > > Software Engineer >> > > > Traveloka >> > > > >> > > >> > > >> > > >> > > -- >> > > *—-* >> > > *Todd Palino* >> > > Staff Site Reliability Engineer >> > > Data Infrastructure Streaming >> > > >> > > >> > > >> > > linkedin.com/in/toddpalino >> > > >> > >> >> >> >> -- >> -- Guozhang >> >
