Hi, All:
Thanks so much. Finally I fixed this bug.
1. set advertised.host.name at config/server.properites as AWS *private IP*
address (not public *DNS*)
2. comment host.name at config/server.properites
3. In remote java producer:
props.put("metadata.broker.list", borkerPrivateIp+ ":9092");
The value brokerPrivateIp is same as advertised.host.name at
config/server.properites
This bug blocked me long time...
Sincerely,
Selina
On Mon, Aug 17, 2015 at 5:09 PM, Hawin Jiang <hawin.ji...@gmail.com> wrote:
> if you want to connect remote kafka producer. Maybe you can try to use IP
> to start your producer. Making sure your producer and consumer are working
> fine. Then trying advertised host name to connect it.
>
> props.put("*metadata.broker.list", "I <http://52.16.17.181:9092/>P to
> start your producer")
>
>
>
> Best regards
> Hawin
>
>
> On Mon, Aug 17, 2015 at 2:50 PM, Job-Selina Wu <swucaree...@gmail.com>
> wrote:
>
> > Hi, All:
> >
> > When I tried find the bug in previous email, I tried to produce
> message
> > on Kafka broker server on AWS instance by command line below:
> > *$** bin/kafka-console-producer.sh --broker-list localhost:9092 --topic
> > temp1*
> > [2015-08-17 21:42:59,468] WARN Property topic is not valid
> > (kafka.utils.VerifiableProperties)
> > hi 1
> >
> >
> > *The error shown:*
> > [2015-08-17 21:43:06,610] WARN Error while fetching metadata
> > [{TopicMetadata for topic temp1 ->
> > No partition metadata for topic temp1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [temp1]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2015-08-17 21:43:06,615] WARN Error while fetching metadata
> > [{TopicMetadata for topic temp1 ->
> > No partition metadata for topic temp1 due to
> > kafka.common.LeaderNotAvailableException}] for topic [temp1]: class
> > kafka.common.LeaderNotAvailableException
> > (kafka.producer.BrokerPartitionInfo)
> > [2015-08-17 21:43:06,615] ERROR Failed to collate messages by topic,
> > partition due to: Failed to fetch topic metadata for topic: temp1
> > (kafka.producer.async.DefaultEventHandler)
> > .......
> > [2015-08-17 21:43:07,039] ERROR Error in handling batch of 1 events
> > (kafka.producer.async.ProducerSendThread)
> > kafka.common.FailedToSendMessageException: Failed to send messages after
> 3
> > tries.
> > at
> >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:105)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:88)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:68)
> > at scala.collection.immutable.Stream.foreach(Stream.scala:547)
> > at
> >
> >
> kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:67)
> > at
> kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:45)
> >
> > Does any one can give me some suggestion on it?
> >
> > Sincerely,
> > Selina
> >
> > On Mon, Aug 17, 2015 at 2:25 PM, Job-Selina Wu <swucaree...@gmail.com>
> > wrote:
> >
> > > Dear All:
> > > I am looking for experts on Kafka to help me on remote Kafka java
> > > producer configuration .
> > >
> > > My Kafka java broker and producer are at different *AWS*
> instances.
> > > How should I set "metadata.broker.list" value. According to
> > > https://kafka.apache.org/08/configuration.html, The format of "
> > > metadata.broker.list"is host1:port1,host2:port2, and the list can be a
> > > subset of brokers or a VIP pointing to a subset of brokers.
> > > I am wondering what is the value of "*VIP pointing to a subset of
> > > brokers"*, what is the correct value of metadata.broker.list
> > >
> > > My Kafka Broker server public ip address is 52.16.17.181
> > > My Kafka Broker server public DNS is *
> > ec2-51-16-17-181.us-west-1.compute.amazonaws.com
> > > <http://ec2-51-18-21-235.us-west-1.compute.amazonaws.com/>*
> > >
> > > Is My producer configuration below right? Do I miss anything?
> > >
> > > //I think the value of *metadata.broker.list *is not right, but I don
> not
> > > know what is the right value
> > > props.put("*metadata.broker.list", "52.16.17.181:9092
> > > <http://52.16.17.181:9092>"*);
> > > props.put("serializer.class",
> "kafka.serializer.StringEncoder");
> > > props.put("request.required.acks", "0");
> > >
> > > My Kafka Broker Sever and Error at Kafka Producer java client side are
> > > list below.
> > >
> > > This bug is blocking me a few days. Your help are highly appreciated.
> > >
> > > Sincerely,
> > > Selina
> > >
> > > --------*The configs/server.properties at Kafka Broker Server at
> > AWS*-----
> > >
> > > zookeeper.connect=localhost:2181
> > > zookeeper.connection.timeout.ms=6000
> > >
> > > delete.topic.enable=true
> > >
> > > broker.id=0
> > > port=9092
> > > host.name=localhost
> > >
> > > advertised.host.name=ec2-51-16-17-181.us-west-1.compute.amazonaws.com
> > >
> > >
> > > # below is same as default
> > > #advertised.port=<port accessible by clients>
> > > #advertised.port=<port accessible by clients>
> > > num.network.threads=3
> > > num.io.threads=8
> > > socket.send.buffer.bytes=102400
> > > socket.receive.buffer.bytes=102400
> > > socket.request.max.bytes=104857600
> > > log.dirs=/tmp/kafka-logs
> > > num.partitions=1
> > > num.recovery.threads.per.data.dir=1
> > > #log.flush.interval.messages=10000
> > > #log.flush.interval.ms=1000
> > > log.retention.hours=168
> > > #log.retention.bytes=1073741824
> > > log.segment.bytes=1073741824
> > > log.retention.check.interval.ms=300000
> > > log.cleaner.enable=false
> > >
> > > -------Error at Kafka Producer java client side -----
> > >
> > > kafka.common.FailedToSendMessageException: Failed to send messages
> after
> > 3
> > > tries.
> > > kafka.common.FailedToSendMessageException: Failed to send messages
> after
> > 3
> > > tries.
> > > at
> > >
> >
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90)
> > > at kafka.producer.Producer.send(Producer.scala:77)
> > > at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> > > at com.cinarra.kafka.Main.main(Main.java:21)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:606)
> > > at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:293)
> > > at java.lang.Thread.run(Thread.java:745)
> > >
> >
>
