Hi, all. Sorry for my late reply.
I tried advertised.host.name/advertised.port. But producer failed fetching metadata from kafka broker. Configuration is below. - Kafka cluster in DataCenter(each machine has Grobal IP Address). Firewall allows only ssh port(22). ex) 192.168.100.100 - Kafka producer exists out of DataCenter. ex) 172.16.0.100 Broker additional setting is below. advertised.host.name=localhost advertised.port=19092 Kafka producer operation is below. - SSH port fowarding ======================== # ssh root@192.168.100.100 -L 19092:192.168.100.100:9092 -L 12181: 192.168.100.100:2181 // Kafka broker and zookeeper ssh port fowarding. ======================== - Producer data put(failed) ======================== # ./kafka-topics.sh --create --zookeeper localhost:12181 --replication-factor 1 --partitions 3 --topic TestTopic Created topic "TestTopic". // Succeed create topic over ssh port fowarding. # ./kafka-topics.sh --list --zookeeper localhost:12181 TestTopic // Succeed get topic list over ssh port fowarding. # ./kafka-console-producer.sh --broker-list localhost:19092 --topic TestTopic SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Test [2014-04-06 20:42:01,352] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,364] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,365] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: TestTopic (kafka.producer.async.DefaultEventHandler) [2014-04-06 20:42:01,476] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,481] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,481] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: TestTopic (kafka.producer.async.DefaultEventHandler) [2014-04-06 20:42:01,589] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,595] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,595] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: TestTopic (kafka.producer.async.DefaultEventHandler) [2014-04-06 20:42:01,709] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartit ionInfo) [2014-04-06 20:42:01,719] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,719] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: TestTopic (kafka.producer.async.DefaultEventHandler) [2014-04-06 20:42:01,826] WARN Error while fetching metadata [{TopicMetadata for topic TestTopic ->No partition metadata for topic TestTopic due to kafka.common.LeaderNotAvailableException}] for topic [TestTopic]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2014-04-06 20:42:01,829] ERROR Failed to send requests for topics TestTopic with correlation ids in [0,8] (kafka.producer.async.DefaultEventHandler) [2014-04-06 20:42:01,830] ERROR Error in handling batch of 2 events (kafka.producer.async.ProducerSendThread) kafka.common.FailedToSendMessageException: Failed to send messages after 3 tries. at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:90) at kafka.producer.async.ProducerSendThread.tryToHandle(ProducerSendThread.scala:104) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:87) at kafka.producer.async.ProducerSendThread$$anonfun$processEvents$3.apply(ProducerSendThread.scala:67) at scala.collection.immutable.Stream.foreach(Stream.scala:526) at kafka.producer.async.ProducerSendThread.processEvents(ProducerSendThread.scala:66) at kafka.producer.async.ProducerSendThread.run(ProducerSendThread.scala:44) ======================== If advertised setting is not defined, producer connection is refused.(below error) So I think advertised setting is working. ======================== # ./kafka-console-producer.sh --broker-list localhost:19092 --topic TestTopic SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. Test [2014-04-08 21:11:44,231] ERROR Producer connection to localhost:9092 unsuccessful (kafka.producer.SyncProducer) java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect0(Native Method) at sun.nio.ch.Net.connect(Net.java:465) at sun.nio.ch.Net.connect(Net.java:457) ======================== Is there anyone you find the cause of error. regards. 2014-03-20 8:00 GMT+09:00 Sotaro Kimura <rfbrin...@gmail.com>: > Thanks for quick reply. > > > advertised.host.name/advertised.port setting is this. > I confirmed only producer setting, so I couldn't find these properties. > > I will try it. Thank you. > > > 2014-03-20 7:50 GMT+09:00 Joe Stein <joe.st...@stealth.ly>: > > With 0.8.1 update these two properties in the server.properties config of >> the broker to be 172.16.0.100 and 19092 respectively >> >> # Hostname the broker will advertise to producers and consumers. If not >> set, it uses the >> # value for "host.name" if configured. Otherwise, it will use the value >> returned from >> # java.net.InetAddress.getCanonicalHostName(). >> #advertised.host.name=<hostname routable by clients> >> >> # The port to publish to ZooKeeper for clients to use. If this is not set, >> # it will publish the same port that the broker binds to. >> #advertised.port=<port accessible by clients> >> >> /******************************************* >> Joe Stein >> Founder, Principal Consultant >> Big Data Open Source Security LLC >> http://www.stealth.ly >> Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> >> ********************************************/ >> >> >> On Wed, Mar 19, 2014 at 6:39 PM, Sotaro Kimura <rfbrin...@gmail.com> >> wrote: >> >> > Hi, all. >> > >> > I'm trying Log produce to Remote Kafka Cluster over SSH Tunneling. >> > >> > Configuration is below. >> > - Kafka cluster in DataCenter(each machine has Grobal IP Address). >> > Firewall allows only ssh port(22). >> > ex) 192.168.100.100 >> > - Kafka producer exists out of DataCenter. >> > ex) 172.16.0.100 >> > >> > I tried produce over SSH Tunneling. >> > Producer successed getting meta data from brokers. >> > But meta data is Address which producer can't access directry. >> > >> > Detail is below. >> > 1. Ssh connect producer machine to DataCenter. >> > 172.16.0.100 > 192.168.100.100:22 >> > 2. Create SSH Tunnel over ssh connection. >> > 172.16.0.100's localhost:19092 to 192.168.100.100:9092(Kafka receive >> > port). >> > 3. Producer connect to localhost:19092. >> > Producer get metadata from brokers(192.168.100.100:9092). >> > 4. Producer failed to connect brokers(192.168.100.100:9092). >> > >> > In such situations, how I can produce to Remote Kafka Cluster? >> > >> > -- >> > ################################# >> > 木村 宗太郎 Sotaro Kimura >> > <rfbrin...@gmail.com> >> > ################################# >> > >> > > > > -- > ################################# > 木村 宗太郎 Sotaro Kimura > <rfbrin...@gmail.com> > ################################# > -- ################################# 木村 宗太郎 Sotaro Kimura <rfbrin...@gmail.com> #################################
