Re: Consuming Kafka Messages Inside of EC2 Instances

Thanks Jonathon! Ahh got it, I connected to a different port. Also it just happened to be that the CM zookeeper was running on the one of 10 instances that I could have manually started zookeeper on -_-. Thanks for the help! On Thu, Jan 29, 2015 at 12:51 PM, Jonathan Natkins wrote: > Responses

Re: Consuming Kafka Messages Inside of EC2 Instances

Responses inline Jonathan "Natty" Natkins StreamSets | Customer Engagement Engineer mobile: 609.577.1600 | linkedin On Thu, Jan 29, 2015 at 12:47 PM, Su She wrote: > Thanks Jonathon! > > 1) How can I get the whole command string? What i c/p was the whole o

Re: Consuming Kafka Messages Inside of EC2 Instances

Thanks Jonathon! 1) How can I get the whole command string? What i c/p was the whole output 2) I do have Zookeeper as a service from Cloudera Manager, except I used to have this before as well and was never a problem. Is there a way I can keep the Zookeeper from CM as well as manually start it up

Re: Consuming Kafka Messages Inside of EC2 Instances

Hey Su, It's hard to say, because you didn't copy the entire command string, but it looks like it's probably a process that's being managed by Cloudera Manager. Do you have a Cloudera Manager instance that is running a Zookeeper quorum? If so, that's where the conflict is. You can either reconfigu

Re: Consuming Kafka Messages Inside of EC2 Instances

Thanks Jonathan! This was the result, would it be okay for me to kill 3544? Or do I try to access a new port?: *sudo netstat -tulpn | grep 2181* tcp0 0 0.0.0.0:21810.0.0.0:* LISTEN 3544/java *ps 3544* PID TTY STAT TIME COMMAND 3544 ?

Re: Consuming Kafka Messages Inside of EC2 Instances

Hey Su, That exception indicates that there's something else already running on the port that you're trying to start up Zookeeper on. The quickest way to figure out what's causing the conflict is to run netstat and look for the port: [root@ip-10-0-0-45 ~]# *netstat -tulpn | grep 2181* tcp

Re: Consuming Kafka Messages Inside of EC2 Instances

I tried a new approach and have started ec2 instances in a public subnet rather than a private subnet. However, now when I try to start zookeeper I get this error. How can I go about solving this issue? Thank you. ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeepe

Re: Consuming Kafka Messages Inside of EC2 Instances

Thank you Guozhang for an updated link. I read the answer for "Why can't my consumers/producers connect to the brokers?". I am confused on a couple things. I can ping the private IP of A (Cluster Launcher) from B (EC2 instance). I can't ping the public ip of A (or google.com) from B which makes s

Re: Consuming Kafka Messages Inside of EC2 Instances

Sorry my previous link was not complete: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers ? On Wed, Jan 28, 2015 at 10:56 PM, Su She wrote: > Thank you Dillian and Guozhang for the responses. > > Yes, Dillian you are understanding my

Re: Consuming Kafka Messages Inside of EC2 Instances

Thank you Dillian and Guozhang for the responses. Yes, Dillian you are understanding my issue correctly. I am not sure what the best approach to this is...I'm not sure if there's a way to whitelist certain IPs, create a VPC, use the cluster launcher as the kafka zookeeper/broker. I guess this is m

Re: Consuming Kafka Messages Inside of EC2 Instances

Su, Does this help for your case? https://cwiki.apache.org/confluence/display/KAFKA/FAQ Guozhang On Wed, Jan 28, 2015 at 3:36 PM, Dillian Murphey wrote: > Am I understanding your question correctly... You're asking how do you > establish connectivity to an instance in a private subnet from th

Re: Consuming Kafka Messages Inside of EC2 Instances

Am I understanding your question correctly... You're asking how do you establish connectivity to an instance in a private subnet from the outside world? Are you thinking in terms of zookeeper or just general aws network connectivity? On Wed, Jan 28, 2015 at 11:03 AM, Su She wrote: > Hello All,

Consuming Kafka Messages Inside of EC2 Instances

Hello All, I have set up a cluster of EC2 instances using this method: http://blogs.aws.amazon.com/bigdata/post/Tx2D0J7QOVRJBRX/Deploying-Cloudera-s-Enterprise-Data-Hub-on-AWS As you can see the instances are w/in a private subnet. I was wondering if anyone has any advice on how I can set up a K