I might have misunderstood what you're asking for, but my understanding is that you were looking a way to have Kafka automatically remove a failed Kafka broker for you from the cluster. Doing so it would need to reassign partitions on that failed Kafka broker to the other brokers in your cluster.
What I was trying to reference specifically in the Apache Kafka documentation link I provided was the following paragraph, which would imply that this can't be automated for you: "*The partition reassignment tool does not have the ability to automatically generate a reassignment plan for decommissioning brokers yet*. As such, the admin has to come up with a reassignment plan to move the replica for all partitions hosted on the broker to be decommissioned, to the rest of the brokers. This can be relatively tedious as the reassignment needs to ensure that all the replicas are not moved from the decommissioned broker to only one other broker. To make this process effortless, *we plan to add tooling support for decommissioning brokers in the future*." On Fri, May 26, 2017 at 6:28 PM, Roman Naumenko <ro...@sproutling.com> wrote: > Thanks Waleed, I did read those guides and basically it was the reason I've > asked how Kafka is supposed to be managed. > > I believe managing small-ish cluster with 3-5, maybe dozen nodes is doable > with scripts. But what happens on the scale betoubd that? > > -- > Roman > > On Fri, May 26, 2017 at 4:19 PM Waleed Fateem <waleed.fat...@gmail.com> > wrote: > > > Hi Roman, > > > > I have not heard of an automated way to do this. You have to manually > > reassign partitions from the Kafka broker you're planning on removing > from > > the cluster. Have a look at the section "decommissioning brokers" in the > > documentation: > > https://kafka.apache.org/documentation/#basic_ops_ > decommissioning_brokers > > > > On Fri, May 26, 2017 at 2:15 PM, Roman Naumenko <ro...@sproutling.com> > > wrote: > > > > > Hi, > > > > > > We’re running Kafka in AWS with replication factor 2. There is a > > > requirement to rotate servers periodically (or add new ones). > > > Is there a way to make Kafka remove “failed” instances from cluster, > > > rebalance automatically whatever it needs to rebalance and continue to > > work > > > as usual? > > > > > > I’ve looked up few guides and it seems like there are tons of manual > > steps > > > required to make it work. > > > > > > —R > > >
