Dillian, On Mon, May 4, 2015 at 1:52 PM, Dillian Murphey <crackshotm...@gmail.com> wrote: > > I'm interested in this topic as well. If you put kafka brokers inside an > autoscaling group, then AWS will automatically add brokers if demand > increases, and the ELB will automatically round-robin across all of your > kafka instances. So in your config files and code, you only need to > provide a single DNS name (the load balancer). You don't need to specify > all your kafka brokers inside your config file. If a broker dies, the ELB > will only route to healthy nodes. > > So you get a lot of robustness, scalability, and fault-tolerance by using > the AWS services. Kafka Brokers will automatically load balance, but the > question is whether it is ok to put all your brokers behind an ELB and > expect the system to work properly. > You should not expect it to work properly. Broker nodes are data bearing which means that any operation to scale down would need to be aware of the data distribution. The client connects to specific nodes to send them data so even the Level 4 load balancing wouldn't work.
> What alternatives are there to dynamic/scalable broker clusters? I don't > want to have to modify my config files or code if I add more brokers, and I > want to be able to handle a broker going down. So these are the reasons AWS > questions like this come up. > The clients already give you options for specifying only a subset of brokers https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example one of which must be alive to discover the rest of the cluster. The main clients handle node failures (you'll still have some operational work). Kafka and other data storage systems do not work the same as an HTTP driven web application. While it can certainly be scaled, and automation could be done to do so in response to load it's going to be more complicated. AWS's off the shelf solution/low operations offering for some (definitely not all) of Kafka's use cases is Kinesis, Azure's is EventHubs. Before using Kakfa or any system in production you'll want to be sure you understand the operational aspects of it. Christian > > Thanks for any comments too. :) > > > > > On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat < gharatmayures...@gmail.com> > wrote: > > > Ok. You can deploy kafka in AWS. You can have brokers on AWS servers. > > Kafka is not a push system. So you will need someone writing to kafka and > > consuming from kafka. It will work. My suggestion will be to try it out on > > a smaller instance in AWS and see the effects. > > > > As I do not know the actual use case about why you want to use kafka for, I > > cannot comment on whether it will work for you personalized use case. > > > > Thanks, > > > > Mayuresh > > > > On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar < > > shekhar.kote...@gmail.com> wrote: > > > > > I am sorry but I cannot reveal those details due to confidentiality > > issues. > > > I hope you understand. > > > > > > > > > Regards, > > > Chandrash3khar Kotekar > > > Mobile - +91 8600011455 > > > > > > On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat < > > > gharatmayures...@gmail.com> > > > wrote: > > > > > > > Hi Chandrashekar, > > > > > > > > Can you please elaborate the use case for Kafka here, like how you are > > > > planning to use it. > > > > > > > > > > > > Thanks, > > > > > > > > Mayuresh > > > > > > > > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar < > > > > shekhar.kote...@gmail.com> wrote: > > > > > > > > > Hi, > > > > > > > > > > I am new to Apache Kafka. I have played with it on my laptop. > > > > > > > > > > I want to use Kafka in AWS. Currently we have tomcat web servers > > based > > > > REST > > > > > API. We want to replace REST API with Apache Kafka, web servers are > > > > behind > > > > > ELB. > > > > > > > > > > I would like to know if we can keep Kafka brokers behind ELB? Will it > > > > work? > > > > > > > > > > Regards, > > > > > Chandrash3khar Kotekar > > > > > Mobile - +91 8600011455 > > > > > > > > > > > > > > > > > > > > > -- > > > > -Regards, > > > > Mayuresh R. Gharat > > > > (862) 250-7125 > > > > > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > >
