Thank you Dillian and Guozhang for the responses. Yes, Dillian you are understanding my issue correctly. I am not sure what the best approach to this is...I'm not sure if there's a way to whitelist certain IPs, create a VPC, use the cluster launcher as the kafka zookeeper/broker. I guess this is more of an AWS question, but I thought this is a problem some Kafka users must have solved already.
Edit: I just tried using the cluster launcher as an intermediate. I started Zookeeper/Kafka Server on my Cluster launcher and then created a topic/produced messages. I set up a kafka consumer on one of my private EC2 instances, but I got a No Route to host error. I pinged the cluster launcher <-> private instance and it works fine. I was hoping I could use this is as a temporary solution...any suggestions on this issue would also be greatly appreciated. Thanks! Best, Su On Wed, Jan 28, 2015 at 9:11 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Su, > > Does this help for your case? > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ > > Guozhang > > On Wed, Jan 28, 2015 at 3:36 PM, Dillian Murphey <crackshotm...@gmail.com> > wrote: > > > Am I understanding your question correctly... You're asking how do you > > establish connectivity to an instance in a private subnet from the > outside > > world? Are you thinking in terms of zookeeper or just general aws > network > > connectivity? > > > > On Wed, Jan 28, 2015 at 11:03 AM, Su She <suhsheka...@gmail.com> wrote: > > > > > Hello All, > > > > > > I have set up a cluster of EC2 instances using this method: > > > > > > > > > > > > http://blogs.aws.amazon.com/bigdata/post/Tx2D0J7QOVRJBRX/Deploying-Cloudera-s-Enterprise-Data-Hub-on-AWS > > > > > > As you can see the instances are w/in a private subnet. I was wondering > > if > > > anyone has any advice on how I can set up a Kafka zookeeper/server on > an > > > instance that receives messages from a Kafka Producer outside of the > > > private subnet. I have tried using the cluster launcher, but I feel > like > > it > > > is not a best practice and only a temporary situation. > > > > > > Thank you for the help! > > > > > > Best, > > > > > > Su > > > > > > > > > -- > -- Guozhang >
