I donot think you need public hostname.I have a similarsetup and its perfectly fine. What I would suggest you to change the hostname,make it persistent,and use FQDN everywhere with /etc/hosts entry locally and on AWS machines.Your problem will get fixed.
On 6/1/16, 8:54 PM, "Marco B." <marcu...@gmail.com> wrote: >Hi Ben, > >Thanks for your answer. What if the instance does not have a public DNS >hostname? >These are all private nodes without public/elastic IP, therefore I don't >know what to set. > >Marco > >2016-06-01 15:09 GMT+02:00 Ben Davison <ben.davi...@7digital.com>: > >> Hi Marco, >> >> We use the public DNS hostname that you can get from the AWS metadata >> service. >> >> Thanks, >> >> Ben >> >> On Wed, Jun 1, 2016 at 1:54 PM, Marco B. <marcu...@gmail.com> wrote: >> >> > Hello everyone, >> > >> > I am trying to setup a MirrorMaker between my company's local cluster and >> > another cluster in AWS to have replication over clusters. We have setup a >> > VPN between these two clusters, and as far as I can see, everything works >> > correctly, meaning that I can ping the nodes and telnet into them without >> > any issues. >> > >> > Now, when I run the following command in the local cluster to use a >> > Zookeeper instance located in AWS (10.1.83.6:2181), in order to read a >> > topic "test" >> > >> > ~/kafka_2.11-0.8.2.2$ ./bin/kafka-console-consumer.sh --zookeeper >> > 10.1.83.6:2181 --topic test --from-beginning >> > >> > A bunch of errors comes up: >> > >> > WARN Fetching topic metadata with correlation id 1 for topics [Set(test)] >> > from broker [id:2,host:ip-10-1-83-5.ec2.internal,port:9092] failed >> > (kafka.client.ClientUtils$) >> > java.nio.channels.ClosedChannelException >> > at kafka.network.BlockingChannel.send(BlockingChannel.scala:100) >> > at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73) >> > at >> > >> > >> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72) >> > at kafka.producer.SyncProducer.send(SyncProducer.scala:113) >> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) >> > at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:93) >> > at >> > >> > >> kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:66) >> > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) >> > WARN Fetching topic metadata with correlation id 1 for topics [Set(test)] >> > from broker [id:3,host:ip-10-1-83-6.ec2.internal,port:9092] failed >> > (kafka.client.ClientUtils$) >> > >> > As far as I know, this is due to the fact that Zookeeper has registered >> an >> > IP/Port for each Kafka instance and these need to be consistent with the >> > producer configuration, as described here ( >> > https://cwiki.apache.org/confluence/display/KAFKA/FAQ). >> > >> > I tried to search on the web, and some people were recommending to change >> > the setting "advertised.host.name" to be either the public IP address >> > coming from AWS (we cannot) or a specific hostname. Now, considering that >> > we have a VPN between the clusters, the only choice left seems to be the >> > one setting the hostname. >> > >> > What should this value be? Is there anything else I need to know for this >> > kind of setup? Any suggestions? >> > >> > Thanks in advance. >> > >> > Kind regards, >> > Marco >> > >> >> -- >> >> >> This email, including attachments, is private and confidential. If you have >> received this email in error please notify the sender and delete it from >> your system. Emails are not secure and may contain viruses. No liability >> can be accepted for viruses that might be transferred by this email or any >> attachment. Any unauthorised copying of this message or unauthorised >> distribution and publication of the information contained herein are >> prohibited. >> >> 7digital Limited. Registered office: 69 Wilson Street, London EC2A 2BB. >> Registered in England and Wales. Registered No. 04843573. >>
