Re: Kafka with Zookeeper behind AWS ELB

2017-07-20 Thread Luigi Tagliamonte
Hello Pradeep, thank you for sharing your experience, will certainly consider it. On Thu, Jul 20, 2017 at 9:29 AM, Pradeep Gollakota wrote: > Luigi, > > I strongly urge you to consider a 5 node ZK deployment. I've always done > that in the past for resiliency during maintenance. In a 3 node clus

Re: Kafka with Zookeeper behind AWS ELB

2017-07-20 Thread Pradeep Gollakota
Luigi, I strongly urge you to consider a 5 node ZK deployment. I've always done that in the past for resiliency during maintenance. In a 3 node cluster, you can only tolerate one "failure", so if you bring one node down for maintenance and another node crashes during said maintenance, your ZK clus

Re: Kafka with Zookeeper behind AWS ELB

2017-07-20 Thread Luigi Tagliamonte
Yes Andrey, you can use an ENI without EIP on AWS if you only want a private address. After some consideration, I think that growing the zookeeper cluster more than 3 nodes is really unlikely so I think that I will attach 3 ENI to 3 servers in autoscaling and I will configure Kafka in using this 3

Re: Kafka with Zookeeper behind AWS ELB

2017-07-19 Thread Andrey Dyachkov
The problem with EIP it is a public ip. Another option is to have the secondary interface attached to the instance on start(or a bit later) with the private static ip, but we are investigating the possibility. On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte < luigi.tagliamont...@gmail.com> wrote:

Re: Kafka with Zookeeper behind AWS ELB

2017-07-19 Thread Luigi Tagliamonte
Hello Andrey, I see that the ELB is not going to help directly with the bug, but introduces a nice layer that makes zookeeper DNS management easier. Introducing and ELB I don't have to deal with keep DNS in sync for all the servers in the zk ensemble. For the moment I can use an HAproxy with EIP an

Re: Kafka with Zookeeper behind AWS ELB

2017-07-19 Thread Andrey Dyachkov
Hi, I have just posted almost the same question in dev list. Zookeeper client resolves address only once, on start, introducing ELB won't really help here (ELBs can be replaced, which involved ip change), but I am eager to know if there is a solution for that. On Wed, 19 Jul 2017 at 23:08 Luigi Ta

Kafka with Zookeeper behind AWS ELB

2017-07-19 Thread Luigi Tagliamonte
Hello, Users! I'm designing a Kafka deployment on AWS and it's my first time working with Kafka and Zookeeper so I've collected a lot of info so far but also some questions that I would love to submit to a much expert audience like you. I have been experimenting with exhibitor and zookeeper in aut