Running inside a Docker container introduced a slight wrinkle. It’s easily resolvable. This line from the controller log file was the tip off:
java.io.IOException: Connection to 64174aa85d04:9092 (id: 2 rack: null) failed 64174aa85d04 was a container ID of one of the other brokers. All the brokers were advertising the container ID as their hostname and the controller was not able to resolve any of those names. The advertised hostname:port is set via the "advertised.listeners” configuration variable in server.properties and defaults to the hostname. But inside a Docker container, the hostname defaults to the container ID. This name is not resolvable anywhere but inside the container. So, If you’re running Kafka in a container and expect other systems to connect to it via the IP address of the Docker host (the host running your container), there are a couple ways to achieve this: Technique #1 (First solution found, works fine) 1. Leave the “listeners” parameter unconfigured, so that Kafka can bind to the private network address that Docker assigns the container. Docker will take care of routing packets from outside systems to the container’s Kafka process. 2. Set “advertised.listeners" to the hostname of the Docker host, so that the broker can advertise a resolvable name for other systems to contact. Technique #2 (Discovered a little later, preferable due to less configuration) 1. Simply leave both “listeners” and “advertised.listeners" parameter unconfigured, but run the Docker container with the "—hostname” parameter, specifying the DNS name of the Docker host on which the container is running. This works because “advertised.listeners" will default to the specified name (which is resolvable). Also, Docker will add an entry to “/etc/hosts” that maps this name to the private network address of the container’s interface. So when “listeners” defaults to the same hostname, that name resolves locally to an address in the container and Kafka can successfully bind to this address on startup. Also, thanks for the link to the controller internals doc. I wanted some more general info on how the system works, so that was informative. Chuck > On Apr 30, 2017, at 7:40 AM, Michal Borowiecki > <michal.borowie...@openbet.com> wrote: > > Hi Chuck, > > Are you running zookeepers in the same containers as Kafka brokers? > > Kafka brokers should be able to communicate with any of the zookeepers and, > more importantly, zookeepers need to be able to talk to each-other. > > Therefore, the zookeeper port should be exposed too (2181 by default), > otherwise you're not going to have a valid cluster. > > Docs on the controller are here: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals > <https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internals> > > Let me know if that helped. > >>
