Hello Emanuel, It does not work because Kafka advertises the IP it is configured with (the ones of the broker) in the metadata response. I think you have two options:
1. You configure the default listener of each broker to advertise the IP address of their proxy instance. It means that the inter-broker traffic will also go trough the proxies; or 2. You keep the default listener with the internal IP and add another one in the config which advertises the external IP (the proxy one). There is a good explanation available here: https://rmoff.net/2018/08/02/kafka-listeners-explained/ Best, David On Mon, Aug 12, 2019 at 5:32 PM Emanuel Muhr <emanuel.m...@mhp.com> wrote: > Hi, > > we are using Kafka in an environment where we have restricted access to > the Kafka brokers. > The access needs to happen via TCP proxies. So, in my setup I have 3 Kafka > brokers (broker1-3) for which I created 3 proxy instances and I have set > these proxy instances as the bootstrap servers: > bootstrap.servers = [ssl://proxy1:9092, ssl://proxy2:9092, > ssl://proxy3:9092] > > Upon startup of a Kafka client I see that there is a successful metadata > update and the client receive the original kafka broker names: > [2019-08-12 16:09:56,921] DEBUG Updated cluster metadata version 2 to > Cluster(id = lFFAv0okS12Vg2rvVbsKdg, nodes = [broker1:9092 (id: 1 rack: > r1), broker3:9092 (id: 3 rack: r3), broker2:9092 (id: 2 rack: r2)], > partitions = [Partition(topic = mytopic, partition = 0, leader = 2, > replicas = [2], isr = [2], offlineReplicas = [])], controller = > broker1:9092 (id: 1 rack: r1)) (org.apache.kafka.clients.Metadata) > > So, the initial connection via the proxy instances does work correctly. > The Kafka documentation states that the bootstrap.servers are only used for > bootstrapping so I assume that if the hostnames do not match the ones from > the brokers they will be replaced. > Additionally I found this C++ library documentation ( > https://github.com/edenhill/librdkafka/wiki/FAQ#number-of-broker-tcp-connections) > which states the following: > The initial bootstrap broker connections will only be used for Metadata > queries, unless the hostname and port of a bootstrap broker exactly matches > the hostname and port of a broker returned in the Metadata response (this > is the advertised.listeners broker configuration property), in which case > the bootstrap broker connection will be associated with that broker's > broker id and used for the full protocol set (such as producing or > consuming). > > We do not use this library but Spring Kafka Java library instead. However, > the documentation makes sense with what I'm observing. > > I'm asking if there is any way to disable this behavior and let Kafka use > the hostnames from the bootstrap broker list for producing and consuming as > well. > I do not see any other way so that we can use Kafka in this restricted > scenario unless we introduce some DNS rerouting. > > I would appreciate any insights on this topic if I'm missing information > or if there is another way to achieve the successful connection under these > circumstances. > > Thanks > Emanuel > > > > >
