Responses inline Jonathan "Natty" Natkins StreamSets | Customer Engagement Engineer mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice>
On Thu, Jan 29, 2015 at 12:47 PM, Su She <suhsheka...@gmail.com> wrote: > Thanks Jonathon! > > 1) How can I get the whole command string? What i c/p was the whole output > Might be something to do with your screen size, or something. Probably if you did a `ps aux` and grepped for the process, you might end up finding it. > > 2) I do have Zookeeper as a service from Cloudera Manager, except I used to > have this before as well and was never a problem. Is there a way I can keep > the Zookeeper from CM as well as manually start it up? I don't want to > potentially cause other services depending on the CM zookeeper to fail. Or > maybe I can connect kafka producer to the zookeeper on CM? > Sure, makes sense. Just configure your manually-started Zookeeper with different ports. 2181 is the obvious problem port, but there's probably other web UI ports or something. Not sure off the top of my head. > Thanks a lot, really appreciate the help. > > On Thu, Jan 29, 2015 at 12:42 PM, Jonathan Natkins <na...@streamsets.com> > wrote: > > > Hey Su, > > > > It's hard to say, because you didn't copy the entire command string, but > it > > looks like it's probably a process that's being managed by Cloudera > > Manager. Do you have a Cloudera Manager instance that is running a > > Zookeeper quorum? If so, that's where the conflict is. You can either > > reconfigure your Zookeeper in CM, or stop the service, and start up a > > Zookeeper manually to try to bind to the public IPs. > > > > Thanks, > > Natty > > > > Jonathan "Natty" Natkins > > StreamSets | Customer Engagement Engineer > > mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice> > > > > > > On Thu, Jan 29, 2015 at 12:36 PM, Su She <suhsheka...@gmail.com> wrote: > > > > > Thanks Jonathan! > > > > > > This was the result, would it be okay for me to kill 3544? Or do I try > to > > > access a new port?: > > > > > > *sudo netstat -tulpn | grep 2181* > > > > > > tcp 0 0 0.0.0.0:2181 0.0.0.0:* > > > LISTEN 3544/java > > > > > > > > > *ps 3544* > > > > > > PID TTY STAT TIME COMMAND > > > 3544 ? Sl 0:19 /usr/java/jdk1.7.0_67-cloudera/bin/java - > > > > > > > > > > > > On Thu, Jan 29, 2015 at 12:31 PM, Jonathan Natkins < > na...@streamsets.com > > > > > > wrote: > > > > > > > Hey Su, > > > > > > > > That exception indicates that there's something else already running > on > > > the > > > > port that you're trying to start up Zookeeper on. The quickest way to > > > > figure out what's causing the conflict is to run netstat and look for > > the > > > > port: > > > > > > > > [root@ip-10-0-0-45 ~]# *netstat -tulpn | grep 2181* > > > > tcp 0 0 0.0.0.0:2181 0.0.0.0:* > > > > LISTEN 9814/java > > > > [root@ip-10-0-0-45 ~]# *ps 9814* > > > > PID TTY STAT TIME COMMAND > > > > 9814 ? Sl 8:11 /usr/java/jdk1.7.0_67-cloudera/bin/java > -cp > > > > > > > > > > > > > > /var/run/cloudera-scm-agent/process/98-zookeeper-server:/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/lib/z > > > > > > > > Jonathan "Natty" Natkins > > > > StreamSets | Customer Engagement Engineer > > > > mobile: 609.577.1600 | linkedin <http://www.linkedin.com/in/nattyice > > > > > > > > > > > > > > On Thu, Jan 29, 2015 at 11:47 AM, Su She <suhsheka...@gmail.com> > > wrote: > > > > > > > > > I tried a new approach and have started ec2 instances in a public > > > subnet > > > > > rather than a private subnet. However, now when I try to start > > > zookeeper > > > > I > > > > > get this error. How can I go about solving this issue? Thank you. > > > > > > > > > > ERROR Unexpected exception, exiting abnormally > > > > > (org.apache.zookeeper.server.ZooKeeperServerMain) > > > > > java.net.BindException: Address already in use > > > > > at sun.nio.ch.Net.bind0(Native Method) > > > > > at sun.nio.ch.Net.bind(Net.java:444) > > > > > at sun.nio.ch.Net.bind(Net.java:436) > > > > > at > > > > > > > > > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > > > > > at > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > > > > > at > > > > sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:95) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116) > > > > > at > > > > > > > > > > > > > > > > > > > > org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78) > > > > > > > > > > > > > > > > > > > > On Thu, Jan 29, 2015 at 9:44 AM, Su She <suhsheka...@gmail.com> > > wrote: > > > > > > > > > > > Thank you Guozhang for an updated link. > > > > > > > > > > > > I read the answer for "Why can't my consumers/producers connect > to > > > the > > > > > > brokers?". I am confused on a couple things. I can ping the > private > > > IP > > > > of > > > > > > A (Cluster Launcher) from B (EC2 instance). I can't ping the > > public > > > ip > > > > > of > > > > > > A (or google.com) from B which makes sense as B is in a private > > > > subnet. > > > > > > But when I try to connect to the zookeeper at private.ip.A:2181 I > > > get a > > > > > No > > > > > > Route to Host error. Is there a reason why I can ping > private.ip.A > > > but > > > > > not > > > > > > connect to private.ip.A:2181? > > > > > > > > > > > > I have not changed any of my server or producer properties. I > have > > > > tried > > > > > > to change advertised host name, but no luck. > > > > > > > > > > > > Thanks for the help! > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Jan 29, 2015 at 8:12 AM, Guozhang Wang < > wangg...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > >> Sorry my previous link was not complete: > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers > > > > > >> ? > > > > > >> > > > > > >> > > > > > >> > > > > > >> On Wed, Jan 28, 2015 at 10:56 PM, Su She <suhsheka...@gmail.com > > > > > > wrote: > > > > > >> > > > > > >> > Thank you Dillian and Guozhang for the responses. > > > > > >> > > > > > > >> > Yes, Dillian you are understanding my issue correctly. I am > not > > > sure > > > > > >> what > > > > > >> > the best approach to this is...I'm not sure if there's a way > to > > > > > >> whitelist > > > > > >> > certain IPs, create a VPC, use the cluster launcher as the > kafka > > > > > >> > zookeeper/broker. I guess this is more of an AWS question, > but I > > > > > thought > > > > > >> > this is a problem some Kafka users must have solved already. > > > > > >> > > > > > > >> > Edit: I just tried using the cluster launcher as an > > intermediate. > > > I > > > > > >> started > > > > > >> > Zookeeper/Kafka Server on my Cluster launcher and then > created a > > > > > >> > topic/produced messages. I set up a kafka consumer on one of > my > > > > > private > > > > > >> EC2 > > > > > >> > instances, but I got a No Route to host error. I pinged the > > > cluster > > > > > >> > launcher <-> private instance and it works fine. I was hoping > I > > > > could > > > > > >> use > > > > > >> > this is as a temporary solution...any suggestions on this > issue > > > > would > > > > > >> also > > > > > >> > be greatly appreciated. Thanks! > > > > > >> > > > > > > >> > Best, > > > > > >> > > > > > > >> > Su > > > > > >> > > > > > > >> > > > > > > >> > On Wed, Jan 28, 2015 at 9:11 PM, Guozhang Wang < > > > wangg...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > > >> > > Su, > > > > > >> > > > > > > > >> > > Does this help for your case? > > > > > >> > > > > > > > >> > > https://cwiki.apache.org/confluence/display/KAFKA/FAQ > > > > > >> > > > > > > > >> > > Guozhang > > > > > >> > > > > > > > >> > > On Wed, Jan 28, 2015 at 3:36 PM, Dillian Murphey < > > > > > >> > crackshotm...@gmail.com> > > > > > >> > > wrote: > > > > > >> > > > > > > > >> > > > Am I understanding your question correctly... You're > asking > > > how > > > > do > > > > > >> you > > > > > >> > > > establish connectivity to an instance in a private subnet > > from > > > > the > > > > > >> > > outside > > > > > >> > > > world? Are you thinking in terms of zookeeper or just > > general > > > > aws > > > > > >> > > network > > > > > >> > > > connectivity? > > > > > >> > > > > > > > > >> > > > On Wed, Jan 28, 2015 at 11:03 AM, Su She < > > > suhsheka...@gmail.com > > > > > > > > > > >> > wrote: > > > > > >> > > > > > > > > >> > > > > Hello All, > > > > > >> > > > > > > > > > >> > > > > I have set up a cluster of EC2 instances using this > > method: > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > > > > > > > > > > http://blogs.aws.amazon.com/bigdata/post/Tx2D0J7QOVRJBRX/Deploying-Cloudera-s-Enterprise-Data-Hub-on-AWS > > > > > >> > > > > > > > > > >> > > > > As you can see the instances are w/in a private subnet. > I > > > was > > > > > >> > wondering > > > > > >> > > > if > > > > > >> > > > > anyone has any advice on how I can set up a Kafka > > > > > >> zookeeper/server on > > > > > >> > > an > > > > > >> > > > > instance that receives messages from a Kafka Producer > > > outside > > > > of > > > > > >> the > > > > > >> > > > > private subnet. I have tried using the cluster launcher, > > > but I > > > > > >> feel > > > > > >> > > like > > > > > >> > > > it > > > > > >> > > > > is not a best practice and only a temporary situation. > > > > > >> > > > > > > > > > >> > > > > Thank you for the help! > > > > > >> > > > > > > > > > >> > > > > Best, > > > > > >> > > > > > > > > > >> > > > > Su > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > -- > > > > > >> > > -- Guozhang > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> -- Guozhang > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > >
