Re: Cannot connect to Kafka from outside of EC2

2014-11-11 Thread Sameer Yami
Is it ok, if I send you directly? On Tue, Nov 11, 2014 at 10:17 AM, Guozhang Wang wrote: > Hi Sameer, > > I think apache mailing list has blocked your attachment. If it is too long > to include in the email body could you paste it somewhere and give me the > link? > > Guozhang > > On Tue, Nov 11

Re: Cannot connect to Kafka from outside of EC2

2014-11-11 Thread Guozhang Wang
Hi Sameer, I think apache mailing list has blocked your attachment. If it is too long to include in the email body could you paste it somewhere and give me the link? Guozhang On Tue, Nov 11, 2014 at 10:01 AM, Sameer Yami wrote: > Hi Guozhang, > > I was wondering if you found anything wrong in

Re: Cannot connect to Kafka from outside of EC2

2014-11-11 Thread Sameer Yami
Hi Guozhang, I was wondering if you found anything wrong in the logs/ thanks On Fri, Nov 7, 2014 at 4:19 PM, Sameer Yami wrote: > Hi Guozhang, > > Attached are the two logs with debug enabled. > > Thanks! > > On Fri, Nov 7, 2014 at 2:09 PM, Sameer Yami wrote: > >> The version is kafka_2.10-0

Re: Cannot connect to Kafka from outside of EC2

2014-11-07 Thread Sameer Yami
The version is kafka_2.10-0.8.1.1. It is not the latest trunk. Will try enabling debug version. thanks On Thu, Nov 6, 2014 at 9:37 PM, Guozhang Wang wrote: > Sameer, > > The server logs do not contain any non-INFO logs, which is a bit wired. Did > you deploy the current trunk of Kafka? Also co

Re: Cannot connect to Kafka from outside of EC2

2014-11-06 Thread Guozhang Wang
Sameer, The server logs do not contain any non-INFO logs, which is a bit wired. Did you deploy the current trunk of Kafka? Also could you enable DEBUG level logging on Kafka brokers? Guozhang On Wed, Nov 5, 2014 at 3:50 PM, Sameer Yami wrote: > The server.log was taken separately. > We ran the

Re: Cannot connect to Kafka from outside of EC2

2014-11-05 Thread Sameer Yami
The server.log was taken separately. We ran the test again and the server and producer logs are below (to get same timings). Thanks! Producer Logs - 2014-11-05 23:38:58,693 Thread-3-SendThread(ip-172-31-25-

Re: Cannot connect to Kafka from outside of EC2

2014-11-05 Thread Guozhang Wang
Sameer, Yes, this is the server log. But there seems no abnormal entries in it, and it does not cover the same time range as the producer client throwing LeaderNotAvailableException (it was 10/24, 14:30). The reason that I want to check the server log at that same reason is that LeaderNotAvailabl

Re: Cannot connect to Kafka from outside of EC2

2014-11-04 Thread Sameer Yami
Hi Guozhang, This is the server.log - [2014-11-04 20:21:57,510] INFO Verifying properties (kafka.utils.VerifiableProperties) [2014-11-04 20:21:57,545] INFO Property advertised.host.name is overridden to x.x.x.x (kafka.utils.VerifiableProperties) [2014-11-04 20:21:57,545] INFO Property broker.id i

Re: Cannot connect to Kafka from outside of EC2

2014-10-31 Thread Guozhang Wang
The server.log should be the broker logs. Guozhang On Thu, Oct 30, 2014 at 11:32 AM, Sameer Yami wrote: > I do not see any other logs besides these. > I checked the log4j.properties and where the logs are getting stored. > The only other logs (apart from the controller logs) in the directory ar

Re: Cannot connect to Kafka from outside of EC2

2014-10-30 Thread Sameer Yami
I do not see any other logs besides these. I checked the log4j.properties and where the logs are getting stored. The only other logs (apart from the controller logs) in the directory are zookeeper-gc.log,kafkaServer-gc.log, server.log and state-change.log Thanks On Wed, Oct 29, 2014 at 4:13 PM, G

Re: Cannot connect to Kafka from outside of EC2

2014-10-29 Thread Guozhang Wang
Sorry for getting late on this Sameer. The logs you showed above are from controller logs, do you see brokers logs besides that? Guozhang On Wed, Oct 29, 2014 at 3:40 PM, Sameer Yami wrote: > Hi Guozhang, > > Is there any clue in the above data as to what is wrong? > > thanks > > > On Tue, Oct

Re: Cannot connect to Kafka from outside of EC2

2014-10-29 Thread Sameer Yami
Hi Guozhang, Is there any clue in the above data as to what is wrong? thanks On Tue, Oct 28, 2014 at 3:45 PM, Sameer Yami wrote: > This is the one from the file specified in log4j.properties - > > [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on Controller 0]: > Topic change listener f

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Sameer Yami
This is the one from the file specified in log4j.properties - [2014-10-28 22:38:36,082] DEBUG [TopicChangeListener on Controller 0]: Topic change listener fired for path /brokers/topics with children Test (kafka.controller.PartitionStateMachine$TopicChangeListener) [2014-10-28 22:38:36,094] INFO [

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Guozhang Wang
Sameer, the one you sent me is from the producer logs, not the broker logs. And /tmp/kafka-logs actually store the message log (i.e. the "data") instead of the server logs. Their locations should be specified in your log4j.properties file. On Tue, Oct 28, 2014 at 2:51 PM, Sameer Yami wrote: > Is

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Sameer Yami
Is this what you are looking for - 2014-10-28 21:46:24,146 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Connected to 172.31.16.31:9092 for producing 2014-10-28 21:46:24,149 pool-13-thread-1 INFO kafka.producer.SyncProducer-68: Disconnecting from 172.31.16.31:9092 2014-10-28 21:46:24,1

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Guozhang Wang
Sameer, could you check broker logs and see if it has similar LeaderNotAvailable exceptions? Guozhang On Tue, Oct 28, 2014 at 10:32 AM, Sameer Yami wrote: > Yes. I can telnet from outside using the public ip of the instance and port > of 9092 > > On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Sameer Yami
Yes. I can telnet from outside using the public ip of the instance and port of 9092 On Tue, Oct 28, 2014 at 10:08 AM, Roger Hoover wrote: > Just a guess but could it be a firewall issue? Did you enable connections > to port 9092 from outside EC2 in a security group? Can you telnet to each > br

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Roger Hoover
Just a guess but could it be a firewall issue? Did you enable connections to port 9092 from outside EC2 in a security group? Can you telnet to each broker IP and port? On Tue, Oct 28, 2014 at 10:01 AM, Sameer Yami wrote: > There was a typo earlier. > > This is the output - > > Topic:Test Parti

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Sameer Yami
There was a typo earlier. This is the output - Topic:Test PartitionCount:2ReplicationFactor:1Configs: Topic: TestPartition: 0Leader: 0Replicas: 0Isr: 0 Topic: TestPartition: 1Leader: 0Replicas: 0Isr: 0 thanks Sameer On Tue, Oct 28, 2014

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Guozhang Wang
Sameer, The topic name is not consistent from your describe outputs. Normally they should be either Topic:*Test*PartitionCount:2ReplicationFactor:1Configs: Topic: *Test*Partition: 0Leader: 0Replicas: 0Isr: 0 Topic: *Test*Partition: 1Leader: 0

Re: Cannot connect to Kafka from outside of EC2

2014-10-28 Thread Sameer Yami
Hi Guozhang, I was wondering if there is anything wrong in the above --describe data. thanks On Mon, Oct 27, 2014 at 2:56 PM, Sameer Yami wrote: > This is what I get - > bin/kafka-topics.sh --zookeeper localhost:2181 --describe > Topic:HeartbeatPartitionCount:2ReplicationFactor:1

Re: Cannot connect to Kafka from outside of EC2

2014-10-27 Thread Sameer Yami
This is what I get - bin/kafka-topics.sh --zookeeper localhost:2181 --describe Topic:HeartbeatPartitionCount:2ReplicationFactor:1Configs: Topic: TestPartition: 0Leader: 0Replicas: 0Isr: 0 Topic: TestPartition: 1Leader: 0Replicas: 0Isr: 0

Re: Cannot connect to Kafka from outside of EC2

2014-10-26 Thread Guozhang Wang
Could you use the topic tools "--describe" to check if the topic Test's partitions leaders are available? http://kafka.apache.org/documentation.html#basic_ops_add_topic Guozhang On Fri, Oct 24, 2014 at 2:43 PM, Sameer Yami wrote: > That is not the case. > This detailed error is below. Due to t

Re: Cannot connect to Kafka from outside of EC2

2014-10-24 Thread Sameer Yami
That is not the case. This detailed error is below. Due to this error , the producer is not able to send messages 2014-10-24 14:31:47,209 main DEBUG kafka.producer.BrokerPartitionInfo-52: Getting broker partition info for topic Test 2014-10-24 14:31:47,209 main INFO kafka.client.ClientUtils$-68

Re: Cannot connect to Kafka from outside of EC2

2014-10-24 Thread Guozhang Wang
Could you see if this is the case: https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whycan%27tmyconsumers/producersconnecttothebrokers ? On Fri, Oct 24, 2014 at 11:35 AM, Sameer Yami wrote: > Hi, > > We have a Kafka / Zookeeper test setup on a single EC2 machine. The > host.name is the

Cannot connect to Kafka from outside of EC2

2014-10-24 Thread Sameer Yami
Hi, We have a Kafka / Zookeeper test setup on a single EC2 machine. The host.name is the private ip address and the advertised.host.name is the public address in server.properties. We are trying to publish from outside EC2 using the public ip of the EC2 machine but get this error on the client si