I'm looking for advice on running Kafka 0.8 behind VIPs. The goal is to
support SSL traffic, with encryption and decryption being performed by
back-to-back VIPs at the client and in front of the broker. That is:
Kafka client
--> vip1a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 1b (SSL decrypt)
--> Kafka broker 1
--> vip2a.myco.com:8080 (SSL encrypt) --- WAN ---> VIP 2b (SSL decrypt)
--> Kafka broker 2
Note there is a separate VIP pipeline per broker, no multiplexing involved.
We don't care about ordering of messages within partitions or elsewhere.
I've configured each broker to register in ZK and advertise itself as being
the VIP endpoint:
broker 1:
advertised.host.name=vip1a.myco.com
advertised.port=8080
broker 2:
advertised.host.name=vip2a.myco.com
advertised.port=8080
Should this work? Any issues?
And since it *isn't* working for me yet, any idea why I might consistently
see a LeaderNotAvailableException when trying to publish or consume?
One possible issue: the brokers need to talk directly to each other,
broker-to-broker, right? And they will try to talk to each other via the
VIP endpoints (vip1a, vip2a)? Seems like I will need to either make the VIP
address routeable from the Kafka broker hosts to allow the broker-to-broker
comms, or else hack the Kafka code so the brokers present the advertised
host/port in response to metadata requests from clients - but talk to each
other on their 'real' addresses.
Many thanks
Reiner
