[
https://issues.apache.org/jira/browse/KAFKA-2426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15113979#comment-15113979
]
Mikaël Cluseau commented on KAFKA-2426:
---------------------------------------
Hi Alex,
there are good chances that I was testing the docker case with a version before
they introduced libnetwork in docker (which added hairpin support as I
understand it). As of Kubernetes, the issue was found while working on the
iptables proxy mode, and solved by the hairpin support.
This means that any topology where the Kafka nodes advertise a public IP that
isn't their own will break if the "firewall" uses a standard Linux bridge that
doesn't have hairpin enabled. This also forces SNAT/MASQUERADE of the
nodeA-to-nodeA traffic, so that the node's IP is hidden from itself (otherwise
TCP will fail to reconcile the reply packets with the connection). There's also
this annoying useless roundtrip on the network but that's minor :-)
So, as of today, this issue cannot be reproduced anymore with Kubernetes
(userland proxy only in 1.0, iptables in 1.1). People doing manual/other things
may hit it, thought.
It would still be nice if we could avoid some extra latency+badnwidth by having
a distinction between the public IPs from where the clients come and the
cluster IPs used for inter-broker communications (like replication). I assume
here that the leader broker is responsible of sending the messages to its
followers, and that this will be done through the public IPs.
> A Kafka node tries to connect to itself through its advertised hostname
> -----------------------------------------------------------------------
>
> Key: KAFKA-2426
> URL: https://issues.apache.org/jira/browse/KAFKA-2426
> Project: Kafka
> Issue Type: Bug
> Components: network
> Affects Versions: 0.8.2.1
> Environment: Docker https://github.com/wurstmeister/kafka-docker,
> managed by a Kubernetes cluster, with an "iptables proxy".
> Reporter: Mikaël Cluseau
> Assignee: Jun Rao
>
> Hi,
> when used behind a firewall, Apache Kafka nodes are trying to connect to
> themselves using their advertised hostnames. This means that if you have a
> service IP managed by the docker's host using *only* iptables DNAT rules, the
> node's connection to "itself" times out.
> This is the case in any setup where a host will DNAT the service IP to the
> instance's IP, and send the packet back on the same interface other a Linux
> Bridge port not configured in "hairpin" mode. It's because of this:
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/net/bridge/br_forward.c#n30
> The specific part of the kubernetes issue is here:
> https://github.com/BenTheElder/kubernetes/issues/3#issuecomment-123925060 .
> The timeout involves that the even if partition's leader is elected, it then
> fails to accept writes from the other members, causing a write lock. and
> generating very heavy logs (as fast as Kafka usualy is, but through log4j
> this time ;)).
> This also means that the normal docker case work by going through the
> userspace-proxy, which necessarily impacts the performance.
> The workaround for us was to add a "127.0.0.2 advertised-hostname" to
> /etc/hosts in the container startup script.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
