Hi Jaikiran, Thanks for the reply!
1) I started Kafka server on instance A by simply downloading Kafka_2.10-0.8.2-beta.tgz from the kafka website, and using the scripts mentioned here: http://kafka.apache.org/documentation.html#introduction. This is the same way I downloaded Kafka on B, except I only started the producer, not the ZK or Server. 2) How do I check to see if the services are bound to non localhost IP? 3) I believe it is firewall because I started 2 instances (C and D) using the AWS Quick Start ( https://s3.amazonaws.com/quickstart-reference/cloudera/hadoop/latest/doc/Cloudera_EDH_on_AWS.pdf). I was able to download Kafka on C and D and pass messages using Kafka Producer/Consumer from D to C using the private/public ip address of C (which was the instance I was running the consumer/ZK/Server on). Since I used AWS Quick Start both of these were in a private subnet. Edit: Woops, I saw I said B was an ec2-instance, B is a gateway, and I downloaded Kafka same way I downloaded it on EC2 and I started a producer on the gateway hoping to send it to my EC2 instance. I can ping A (in a private subnet) from my gateway fine. I have pretty much changed all security settings on A so that it essentially has none, but still in a private subnet ( so can't ssh into it still from putty, can only ssh via the cluster launcher). I can't lessen the security anymore on instance A. Hope that wasn't too confusing, really appreciate the help. Thank you! Best, Su On Mon, Jan 19, 2015 at 7:58 PM, Jaikiran Pai <jai.forums2...@gmail.com> wrote: > Hi Su, > > How exactly did you start the Kafka server on instance "A"? Are you sure > the services on it are bound to non localhost IP? What does the following > command result from instance B: > > telnet public.ip.of.A 9092 > > > -Jaikiran > > On Tuesday 20 January 2015 07:16 AM, Su She wrote: > >> Hello Everyone, >> >> Thank you for the help! >> >> Preface: I've created producers/consumers before and they have worked. I >> have also made consumers/producers using java programs, but they have all >> been locally. >> >> 1) I have a Zookeeper/Kafka Server running on an EC2 instance called "A" >> >> 2) I started the Zookeeper/Kafka Server on A and created a topic "test" >> like it says on the kafka documentation >> >> 3) I then started a console consumer on A like the documentation states >> for >> test. >> >> 4) I then downloaded Kafka on EC2 instance called "B" and created a >> console >> producer with broker with broker-list public.ip.of.A:9092 and topic test. >> >> 5) Once I start publishing messages this is the message I get (what's >> worse >> is that I can't cntrl C to stop and have to exit putty): >> >> WARN Fetching topic metadata with correlation id 16 for topics [Set(test)] >> from broker [id:0,host:54.183.40.224,port:9092] failed >> (kafka.client.ClientUtils$) >> >> java.nio.channels.ClosedChannelException >> >> at kafka.network.BlockingChannel.send(BlockingChannel.scala:97) >> >> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer. >> scala:74) >> >> at >> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$ >> doSend(SyncProducer.scala:73) >> >> at kafka.producer.SyncProducer.send(SyncProducer.scala:114) >> >> at >> kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) >> >> at >> kafka.producer.BrokerPartitionInfo.updateInfo( >> BrokerPartitionInfo.scala:82) >> >> at >> kafka.producer.async.DefaultEventHandler$$anonfun$handle$2.apply$mcV$sp( >> DefaultEventHandler.scala:78) >> >> at kafka.utils.Utils$.swallow(Utils.scala:172) >> >> at kafka.utils.Logging$class.swallowError(Logging.scala:106) >> >> at kafka.utils.Utils$.swallowError(Utils.scala:45) >> >> at >> kafka.producer.async.DefaultEventHandler.handle( >> DefaultEventHandler.scala:78) >> >> at >> kafka.producer.async.ProducerSendThread.tryToHandle( >> ProducerSendThread.scala:104) >> >> at >> kafka.producer.async.ProducerSendThread.processEvents( >> ProducerSendThread.scala:93) >> >> at >> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) >> >> [2015-01-20 01:29:28,289] ERROR fetching topic metadata for topics >> [Set(test)] from broker [ArrayBuffer(id:0,host: >> >> Meanwhile in ZK...not sure if this happened as soon as I started >> publishing >> messages: >> >> Error:KeeperErrorCode = NoNode for /consumers/console-consumer- >> 2615/offsets >> (org.apache.zookeeper.server.PrepRequestProcessor) >> >> Any tips/suggestions are greatly appreciated! >> >> >