We have a separate daemon process that assigns EIPs to servers when they startup in an autoscaling group based off of an autoscaling message. So for a cluster of 3 we have 3 EIPs. Then we inject the EIPs into startup script for Kafka which checks to see if it has one of the EIPs and assigns itself the index of that IP so in the list: 10.0.0.1 10.0.0.2 10.0.0.3
1 is broker 0, 2 is broker 1 and 3 is broker 2. All this is injected via cloudformation and then we have a mod value so if we want to spin brokers in the same group we do mod 1,2 and get brokers mod * 3 + index to determine which is in the group. (the EIPs are different as it is a different cloudformation) For redundancy make sure you run at least two that have full replicas of all other partitions. We run replication factor of 3 with three instances so if any goes down the other two bring it back in sync once a fresh server spins in the autoscaling group. ________________________________________ From: Dillian Murphey <crackshotm...@gmail.com> Sent: Wednesday, January 14, 2015 2:42 PM To: users@kafka.apache.org Subject: kafka cluster on aws I can't seem to find much information to help me (being green to kafka) on setting up a cluster on aws. Does anyone have any sources? The question I have off the bat is, what methods have already been explored to generate a unique broker id? If I spin up a new server, do I just need to maintain my own broker-id list somewhere so I don't re-use an already allocated broker id? Also, I read an article about a broker going down and requiring a new broker be spun up with the same id. Is this also something I need to implement? I want to setup a kafka auto-scaling group on AWS, so I can add brokers at well or based on load. It doesn't seem too complicated, or maybe I'm too green to see it, but I don't want to re-invent everything myself. I know Loggly uses AWS/Kafka, so I'm hunting for more details on that too. Thanks for any help
