Hello, How do you consume a kafka topic from a remote location without a dedicated connection? How do you protect the server?
The setup: data streams into our datacenter. We process it, and publish it to a kafka cluster. The consumer is located in a different datacenter with no direct connection. The most efficient scenario would be to setup a point-to-point link but that idea has no traction with our executives. We can setup a VPN; While functional, our IT department assures us that it won't be able to scale. What we're currently planning is to expose the kafka cluster IP addresses to the internet, and only allow access via firewall. Each message will be encrypted with a shared private key, so we're not worried about messages being intercepted. What we are worried about is this: how brokers refer to each other-- when a broker directs the consumer to the server that is in charge of a particular region, does it use the host name (that could be externally mapped to the public IP) or does it use the detected/private IP address. What solution would you use to consume a remote cluster? --Tom
