Hi, I would like to know the right way to setup a Kafka cluster with Nginx in front of it as a reverse proxy. Let's say I have 2 Kafka brokers running on 2 different hosts; and an Nginx server running on another host. Nginx will listen on 2 different ports, and each will forward to one Kafka broker. Producers will connect to one of the 2 ports on the Nginx host. Nginx-Host: listens on 9000 ssl (forward to <kafka-host-0>:9092 in plain text); 9001 ssl (forward to <kafka-host-1>:9092 in plain text);
Kafka-Host-0: listeners=PLAINTEXT://<kafka-host-0-ip>:9092; advertised.listeners=PLAINTEXT://<nginx-host-ip>:9000Kafka-Host-1: listeners=PLAINTEXT://<kafka-host-1-ip>:9092; advertised.listeners=PLAINTEXT://<nginx-host-ip>:9001 Ports on Nginx will have SSL enabled so that messages sent from producers to Nginx will be encrypted; Traffic between Nginx and Kafka are in plain text since it's on the internal network. Why have producers go through Nginx? The main reason is that producers will only need to open their firewall to a single IP so that even later on when I add another Kafka broker, I don't need to modify the firewall of all the producers. My problem is that I can't make the above setup work. Brokers are unable to talk to one another. :( So, what's the right way to do this? Anyone has experience setting up something similar? Or any recommendations for a different setup that will not require changes on the producer's side when new Kafka brokers are added? Thanks!Yongtao PS. The producers in question are Filebeats (https://www.elastic.co/products/beats/filebeat).
