Re: Kafka cluster management lifecycle

Is there a roadmap to have this feature? It’s pretty basic requirement these days to run stuff in the cloud. — Roman > On May 26, 2017, at 8:57 PM, Waleed Fateem wrote: > > I might have misunderstood what you're asking for, but my understanding is > that you were looking a way to have Kafka a

Re: Kafka cluster management lifecycle

I might have misunderstood what you're asking for, but my understanding is that you were looking a way to have Kafka automatically remove a failed Kafka broker for you from the cluster. Doing so it would need to reassign partitions on that failed Kafka broker to the other brokers in your cluster.

Re: Kafka cluster management lifecycle

Thanks Waleed, I did read those guides and basically it was the reason I've asked how Kafka is supposed to be managed. I believe managing small-ish cluster with 3-5, maybe dozen nodes is doable with scripts. But what happens on the scale betoubd that? -- Roman On Fri, May 26, 2017 at 4:19 PM Wal

Re: Kafka cluster management lifecycle

Hi Roman, I have not heard of an automated way to do this. You have to manually reassign partitions from the Kafka broker you're planning on removing from the cluster. Have a look at the section "decommissioning brokers" in the documentation: https://kafka.apache.org/documentation/#basic_ops_decom

Kafka cluster management lifecycle

Hi, We’re running Kafka in AWS with replication factor 2. There is a requirement to rotate servers periodically (or add new ones). Is there a way to make Kafka remove “failed” instances from cluster, rebalance automatically whatever it needs to rebalance and continue to work as usual? I’ve look

Re: Kafka Cluster management

New to Kafka here. > >> > >> Also, if you don't mind sharing what do you use as load balancer in this > >> scenario? > >> > >> Thanks > >> > >> --- Original Message --- > >> > >> From: "Todd Palino" > >

Re: Kafka Cluster management

t do you use as load balancer in this >> scenario? >> >> Thanks >> >> --- Original Message --- >> >> From: "Todd Palino" >> Sent: 12 March 2014 07:11 >> To: users@kafka.apache.org >> Subject: Re: Kafka Cluster management >

Re: Kafka Cluster management

his > scenario? > > Thanks > > --- Original Message --- > > From: "Todd Palino" > Sent: 12 March 2014 07:11 > To: users@kafka.apache.org > Subject: Re: Kafka Cluster management > > Both the producer and consumer clients will take care of rebalancing if &

Re: Kafka Cluster management

ssage --- From: "Todd Palino" Sent: 12 March 2014 07:11 To: users@kafka.apache.org Subject: Re: Kafka Cluster management Both the producer and consumer clients will take care of rebalancing if there is a broker failure. The consumer handles this entirely in Zookeeper. For the producer si

Re: Kafka Cluster management

Both the producer and consumer clients will take care of rebalancing if there is a broker failure. The consumer handles this entirely in Zookeeper. For the producer side, you do need some sort of front end to handle the metadata requests, or else you would constantly have to change your producer c

Kafka Cluster management

Hi, Is there documentation on how connection management, connection pooling works in Kafka for multiple brokers? Do I need to take care of my own connection management for cluster? e.g. while publishing messages, if one broker stops responding, I need to switch to the other. Does Kafka client take