Re: Kafka cluster management lifecycle

2017-05-29 Thread Roman Naumenko
Is there a roadmap to have this feature? It’s pretty basic requirement these days to run stuff in the cloud. — Roman > On May 26, 2017, at 8:57 PM, Waleed Fateem wrote: > > I might have misunderstood what you're asking for, but my understanding is > that you were looking a way to have Kafka a

Re: Kafka cluster management lifecycle

2017-05-26 Thread Waleed Fateem
I might have misunderstood what you're asking for, but my understanding is that you were looking a way to have Kafka automatically remove a failed Kafka broker for you from the cluster. Doing so it would need to reassign partitions on that failed Kafka broker to the other brokers in your cluster.

Re: Kafka cluster management lifecycle

2017-05-26 Thread Roman Naumenko
Thanks Waleed, I did read those guides and basically it was the reason I've asked how Kafka is supposed to be managed. I believe managing small-ish cluster with 3-5, maybe dozen nodes is doable with scripts. But what happens on the scale betoubd that? -- Roman On Fri, May 26, 2017 at 4:19 PM Wal

Re: Kafka cluster management lifecycle

2017-05-26 Thread Waleed Fateem
Hi Roman, I have not heard of an automated way to do this. You have to manually reassign partitions from the Kafka broker you're planning on removing from the cluster. Have a look at the section "decommissioning brokers" in the documentation: https://kafka.apache.org/documentation/#basic_ops_decom

Kafka cluster management lifecycle

2017-05-26 Thread Roman Naumenko
Hi, We’re running Kafka in AWS with replication factor 2. There is a requirement to rotate servers periodically (or add new ones). Is there a way to make Kafka remove “failed” instances from cluster, rebalance automatically whatever it needs to rebalance and continue to work as usual? I’ve look