Hi Dillian, Yeah, ELB + ASG will is pretty popular however might need further tricks to use them for Kafka brokers.
As I replied to Chandrashekhar in another email. You can use ELB as bootstrap.servers/metadata.broker.list to serve as client bootstrap use case. But all the producing/consuming traffic still need to go to the brokers directly. Another thing about ASG is that when new Kafka broker comes up, it might need to copy a lot of data if its log is empty. That might take a while. This might not be the case if you are using EBS. I think Netflix has some practical experience of running Kafka on AWS with dynamically changing number of brokers, but they had some tricks to manage the metadata(broker id, etc) and log files. Anyway, I would be surprised if ELB + ASG can be easily used for dynamic provision of Kafka brokers. But I think it probably is a very helpful improvement if we can do that. Jiangjie (Becket) Qin On 5/4/15, 1:52 PM, "Dillian Murphey" <crackshotm...@gmail.com> wrote: >I'm interested in this topic as well. If you put kafka brokers inside an >autoscaling group, then AWS will automatically add brokers if demand >increases, and the ELB will automatically round-robin across all of your >kafka instances. So in your config files and code, you only need to >provide a single DNS name (the load balancer). You don't need to specify >all your kafka brokers inside your config file. If a broker dies, the ELB >will only route to healthy nodes. > >So you get a lot of robustness, scalability, and fault-tolerance by using >the AWS services. Kafka Brokers will automatically load balance, but the >question is whether it is ok to put all your brokers behind an ELB and >expect the system to work properly. > >What alternatives are there to dynamic/scalable broker clusters? I don't >want to have to modify my config files or code if I add more brokers, and >I >want to be able to handle a broker going down. So these are the reasons >AWS >questions like this come up. > >Thanks for any comments too. :) > > > > >On Mon, May 4, 2015 at 9:03 AM, Mayuresh Gharat ><gharatmayures...@gmail.com> >wrote: > >> Ok. You can deploy kafka in AWS. You can have brokers on AWS servers. >> Kafka is not a push system. So you will need someone writing to kafka >>and >> consuming from kafka. It will work. My suggestion will be to try it out >>on >> a smaller instance in AWS and see the effects. >> >> As I do not know the actual use case about why you want to use kafka >>for, I >> cannot comment on whether it will work for you personalized use case. >> >> Thanks, >> >> Mayuresh >> >> On Mon, May 4, 2015 at 8:55 AM, Chandrashekhar Kotekar < >> shekhar.kote...@gmail.com> wrote: >> >> > I am sorry but I cannot reveal those details due to confidentiality >> issues. >> > I hope you understand. >> > >> > >> > Regards, >> > Chandrash3khar Kotekar >> > Mobile - +91 8600011455 >> > >> > On Mon, May 4, 2015 at 9:18 PM, Mayuresh Gharat < >> > gharatmayures...@gmail.com> >> > wrote: >> > >> > > Hi Chandrashekar, >> > > >> > > Can you please elaborate the use case for Kafka here, like how you >>are >> > > planning to use it. >> > > >> > > >> > > Thanks, >> > > >> > > Mayuresh >> > > >> > > On Sat, May 2, 2015 at 9:08 PM, Chandrashekhar Kotekar < >> > > shekhar.kote...@gmail.com> wrote: >> > > >> > > > Hi, >> > > > >> > > > I am new to Apache Kafka. I have played with it on my laptop. >> > > > >> > > > I want to use Kafka in AWS. Currently we have tomcat web servers >> based >> > > REST >> > > > API. We want to replace REST API with Apache Kafka, web servers >>are >> > > behind >> > > > ELB. >> > > > >> > > > I would like to know if we can keep Kafka brokers behind ELB? >>Will it >> > > work? >> > > > >> > > > Regards, >> > > > Chandrash3khar Kotekar >> > > > Mobile - +91 8600011455 >> > > > >> > > >> > > >> > > >> > > -- >> > > -Regards, >> > > Mayuresh R. Gharat >> > > (862) 250-7125 >> > > >> > >> >> >> >> -- >> -Regards, >> Mayuresh R. Gharat >> (862) 250-7125 >>
