Is there a roadmap to have this feature? It’s pretty basic requirement these days to run stuff in the cloud.
— Roman > On May 26, 2017, at 8:57 PM, Waleed Fateem <waleed.fat...@gmail.com> wrote: > > I might have misunderstood what you're asking for, but my understanding is > that you were looking a way to have Kafka automatically remove a failed > Kafka broker for you from the cluster. Doing so it would need to reassign > partitions on that failed Kafka broker to the other brokers in your > cluster. > > What I was trying to reference specifically in the Apache Kafka > documentation link I provided was the following paragraph, which would > imply that this can't be automated for you: > > "*The partition reassignment tool does not have the ability to > automatically generate a reassignment plan for decommissioning brokers yet*. > As such, the admin has to come up with a reassignment plan to move the > replica for all partitions hosted on the broker to be decommissioned, to > the rest of the brokers. This can be relatively tedious as the reassignment > needs to ensure that all the replicas are not moved from the decommissioned > broker to only one other broker. To make this process effortless, *we plan > to add tooling support for decommissioning brokers in the future*." > > > On Fri, May 26, 2017 at 6:28 PM, Roman Naumenko <ro...@sproutling.com> > wrote: > >> Thanks Waleed, I did read those guides and basically it was the reason I've >> asked how Kafka is supposed to be managed. >> >> I believe managing small-ish cluster with 3-5, maybe dozen nodes is doable >> with scripts. But what happens on the scale betoubd that? >> >> -- >> Roman >> >> On Fri, May 26, 2017 at 4:19 PM Waleed Fateem <waleed.fat...@gmail.com> >> wrote: >> >>> Hi Roman, >>> >>> I have not heard of an automated way to do this. You have to manually >>> reassign partitions from the Kafka broker you're planning on removing >> from >>> the cluster. Have a look at the section "decommissioning brokers" in the >>> documentation: >>> https://kafka.apache.org/documentation/#basic_ops_ >> decommissioning_brokers >>> >>> On Fri, May 26, 2017 at 2:15 PM, Roman Naumenko <ro...@sproutling.com> >>> wrote: >>> >>>> Hi, >>>> >>>> We’re running Kafka in AWS with replication factor 2. There is a >>>> requirement to rotate servers periodically (or add new ones). >>>> Is there a way to make Kafka remove “failed” instances from cluster, >>>> rebalance automatically whatever it needs to rebalance and continue to >>> work >>>> as usual? >>>> >>>> I’ve looked up few guides and it seems like there are tons of manual >>> steps >>>> required to make it work. >>>> >>>> —R >>> >>
