We have an open source framework you can use to spin up Kafka (any version
or even any build you want) clusters (and Zookeeper) with CloudFormation on
AWS https://github.com/stealthly/minotaur

It is very nice/handy you basically specify your instance types, counts,
versions of code, etc and hit a <enter>
https://github.com/stealthly/minotaur/tree/master/labs/kafka e.g.

./minotaur.py lab deploy kafka -e bdoss-dev -d testing -r us-east-1 -z
us-east-1a -k http://example.com/kafka.tar.gz -n 3 -i m1.small

There is some setup for the bastion host (
https://github.com/stealthly/minotaur/tree/master/infrastructure/aws/bastion)
and supervisor (https://github.com/stealthly/minotaur/tree/master/supervisor)
and after that it is really nice and easy.

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/

On Wed, Jan 14, 2015 at 2:54 PM, Joseph Lawson <jlaw...@roomkey.com> wrote:

> We have a separate daemon process that assigns EIPs to servers when they
> startup in an autoscaling group based off of an autoscaling message.  So
> for a cluster of 3 we have 3 EIPs. Then we inject the EIPs into startup
> script for Kafka which checks to see if it has one of the EIPs and assigns
> itself the index of that IP so in the list:
> 10.0.0.1 10.0.0.2 10.0.0.3
>
> 1 is broker 0, 2 is broker 1 and 3 is broker 2.  All this is injected via
> cloudformation and then we have a mod value so if we want to spin brokers
> in the same group we do mod 1,2 and get brokers mod * 3 + index to
> determine which is in the group. (the EIPs are different as it is a
> different cloudformation)
>
> For redundancy make sure you run at least two that have full replicas of
> all other partitions.  We run replication factor of 3 with three instances
> so if any goes down the other two bring it back in sync once a fresh server
> spins in the autoscaling group.
>
> ________________________________________
> From: Dillian Murphey <crackshotm...@gmail.com>
> Sent: Wednesday, January 14, 2015 2:42 PM
> To: users@kafka.apache.org
> Subject: kafka cluster on aws
>
> I can't seem to find much information to help me (being green to kafka) on
> setting up a cluster on aws. Does anyone have any sources?
>
> The question I have off the bat is, what methods have already been explored
> to generate a unique broker id? If I spin up a new server, do I just need
> to maintain my own broker-id list somewhere so I don't re-use an already
> allocated broker id?
>
> Also, I read an article about a broker going down and requiring a new
> broker be spun up with the same id. Is this also something I need to
> implement?
>
> I want to setup a kafka auto-scaling group on AWS, so I can add brokers at
> well or based on load. It doesn't seem too complicated, or maybe I'm too
> green to see it, but I don't want to re-invent everything myself.
>
> I know Loggly uses AWS/Kafka, so I'm hunting for more details on that too.
>
> Thanks for any help
>

Reply via email to