Re: Kafka DNS Caching in AWS

2016-08-03 Thread Zuber Saiyed
Thank you all for your responses. So here is what I tried and it worked since EIP is not an option for me - 1) Created an ENI with a dedicated IP 2) Associated that IP with A Type address 3) Assigned that ENI to the EC2 instance 4) Created EBS volume to keep ZK data As EBS volume and ENI are boun

Re: Kafka DNS Caching in AWS

2016-08-03 Thread Alexis Midon
Hi Gwen, I have explored and tested this approach in the past. It does not work for 2 reasons: A. the first one relates to the ZKClient implementation, B. the second is the JVM behavior. A. The ZKConnection [1] managed by ZKClient uses a legacy constructor of org,apache.Zookeeper [2]. The crea

Re: Kafka DNS Caching in AWS

2016-08-03 Thread Joe Lawson
In the past on classic EC2 with an autoscaling group of zookeeper instances, I've used elastic IPs for my list. There we subscribed an SQS queue to the autoscaling SNS topic and when a new instances was brought online one of the spare IPs was allocated to the instance. It has to try over and over s

Re: Kafka DNS Caching in AWS

2016-08-03 Thread Gian Merlino
Hey Zuber, Our AWS ZK deployment involves a subnet that is not used for other things, fixed private IP addresses, and EBS volumes for ZK data. That way, if a ZK instance fails, it can be replaced with another instance with the same IP and data volume. On Wed, Aug 3, 2016 at 7:22 AM, Zuber wrote:

Re: Kafka DNS Caching in AWS

2016-08-03 Thread Gwen Shapira
Can you define a DNS name that round-robins to multiple IP addresses? This way ZKClient will cache the name and you can rotate IPs behind the scenes with no issues? On Wed, Aug 3, 2016 at 7:22 AM, Zuber wrote: > Hello – > > We are planning to use Kafka as Event Store in a system which is being

Kafka DNS Caching in AWS

2016-08-03 Thread Zuber
Hello – We are planning to use Kafka as Event Store in a system which is being built using event sourcing design approach. Here is how we deployed the cluster in AWS to verify HA in the cloud (in our POC we only had 1 topic with 1 partition and 3 replication factor) - 1)3 ZK servers running