I have Kafka Cluster setup on AWS Public Subnet with all brokers having elastic IPs My producers are on private subnet and not able to produce to the kafka on public subnet. Both subnets are in same VPC
I added the private ip/cidr of producer ec2 instance to Public Kafka's security group. (I can telnet from private ec2 instance to brokers private ip on 9092 port) >From the ec2 instance on private subnet, I can list the topics using ZK's private ip [ec2-user@ip-x-x-x-x kafka_2.10-0.9.0.1]$ bin/kafka-topics.sh --zookeeper <zk_private_ip>:2181 --list test When I try to produce from private ec2 instance using broker's private IP, I get following error [ec2-user@ip-x-x-x-x kafka_2.10-0.9.0.1]$ bin/kafka-console-producer.sh --broker-list <broker_private_ip>:9092 --topic test [2016-06-29 18:47:38,328] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: Batch Expired (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) When I try to produce from private ec2 instance using broker's public IP, I get following error. [2016-06-29 18:53:15,918] ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: Failed to update metadata after 60000 ms. (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) Few settings from server.properties # The id of the broker. This must be set to a unique integer for each broker. broker.id=0 ############################# Socket Server Settings ############################# listeners=PLAINTEXT://:9092 # The port the socket server listens on #port=9092 # Hostname the broker will bind to. If not set, the server will bind to all interfaces host.name=<Public IP> # Hostname the broker will advertise to producers and consumers. If not set, it uses the # value for "host.name" if configured. Otherwise, it will use the value returned from # java.net.InetAddress.getCanonicalHostName(). advertised.host.name=<Public IP> Please let me know if I am doing something wrong. Thank you Vivek
