Hi all I investigated a bit deeper and I came to the conclusion that it's probably expected behaviour, that a broker keeps running after loosing the ZK session and does not necessarily restart or reconnect automatically. https://cwiki-test.apache.org/confluence/display/KAFKA/KIP-217%3A+Expose+a+timeout+to+allow+an+expired+ZK+session+to+be+re-created and tickets related to that KIP reflect that it is probably better to keep it running even without ZK connectivity.
In the meantime, I also found a JMX property (kafka.server -> SessionExpireListener -> SessionState) which reflects exactly this state. We'll add that to our monitoring and restart a broker if the state is CLOSED, but Zookeeper is available again. CU, Joe On 3/18/19 3:29 PM, Joe Ammann wrote: > Hi all > > We're running several clusters (mostly with 3 brokers) with 2.1.1 > > We quite regularly see the pattern that one of the 3 brokers "detaches" from > ZK (the broker id is not registered anymore under /brokers/ids). We assume > that the root cause for this is that the brokers are running on VMs (due to > company policy, no alternative) and that the VM gets "stalled" for several > minutes due to missing resources on the VMware ESX host. > > This is not new behaviour with 2.1.1, we already saw it with 0.10.2.1 before. > > The sequence of events is normally something like the following > - cluster is running ok > - one broker "gets stalled", not pingable anymore > - partitions go to underreplicated > - failed broker comes back and reports that ZK session was expired [1] > - some of the brokers that were ok report leader election problems [2] > - the failed/revived broker logs errors continuosly about expired session [3] > > This goes on, until we restart the broker on the VM that had failed. Until we > do this restart, the failed broker seems to think it is working perfectly ok. > We're monitoring all brokers via JMX, and that one does not report any > problems. It claims in the JMX values to be the leader of a number of > partitions, and have 0 underreplicated partitions. Whilst the other brokers > rightfully report via JMX that they in turn do have underreplicate paritions. > This then causes alerts to go off about the brokers that still work in > degraded mode, while the one that is really broken appears green/ok. > > Is this in any way expected behaviour? That a Kafka broker gets its ZK > session expired but continues to run (just issues the errors in [3]). I would > have expected that the broker would shutdown itsself in a similar manner it > does when it's unable to register with ZK on startup. > > Any idea how I could best detect this situation in monitoring? I'm thinking > about after polling the broker via JMX, I also poll ZK to check if > /brokers/ids/<brokerid> node exists. If not, restart that broker. > > BTW: I do know that probably the best answer is: "just run your ZK/Kafka on > hardware, not VMs". We're working on that, but company policies seem to > prefer outages over spending a little money).
