Hello! We have a 3 broker Kafka cluster ( KRaft ) brokers and Kraft controllers on the same nodes
CPU: 16 RAM: 32GB We have 2241 topic and 107262 online partitions with 23652 client connections kafka version is 3.6.1 And yesterday we have trouble from 12:08 to 12:11 We have so many logs on all brokers indicated connection troubles ( inter node ) Here is the logs from 1st broker 438]: [2024-11-29 13:08:35,199] INFO [Partition coication.in-42 broker=0] Shrinking IS from 1,0,2 to 0,2. Leader: (highWatermariv 4381: [2024-11-29 13:08:35,207] INFO Partition cldkafka.out-24 broker=0] Shrinking [SR from 2,1,0 to 0. Leader: ChighWatermark: 655805, endOffset: t.sh[1457438]: [2024-11-29 13:08:45,244] INFO [Partition communication.notificationmanager.sendnotification.in-42 broker=0] IS updated to 0,2 and version updated to 57 t.sh[1457438]: [2024-11-29 13:08:45,273] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(communication.notificationmanager.sendnotification.in-t.sh[1457438]: [2024-11-29 13:08:45,273] INFO [Partition afka.cliaqtokafka.out-24 broker=0] IS updated to (c) (under-min-isr) and version updated to 59 (kafk t.sh[1457438]: [2024-11-29 13:08:45,460] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set(colvir.cliaqtokafka.cliaqtokafka.out-24) (kafka.servers t.sh[1457438]: [2024-11-29 13:08:53,352] INFO [GroupCoordinator 0]: Member consumer 143-52312bed-6b44-41de-b717-9ab7c86fdaa in group MIB3.0_PROD has failed, removing it fi t.sh[1457438]: [2024-11-29 13:08:53,352] INFO [GroupCoordinator 0]: Preparing to rebalance group MIB3.0_PROD in state PreparingRebalance with old generation 2845 (- _consum t.sh [1457438]: [2024-11-29 13:08:53,352] INFO [GroupCoordinator 0]: Group hi with generation 2846 is now empty (__consumer_offsets-5) (kafka.coordinator.group.Gro t.sh[1457438]: [2024-11-29 13:08:53,718] INFO [Partition __consumer_offsets-5 broker=0] Shrinking ISR from 0,2,1 to 0. Leader: (highWatermark: 32463865, endOffset: 3246386-t.sh[1457438]: [2024-11-29 13:08:53,747] INFO [Partition _consumer_offsets-5 broker=0] IS updated to (c) (under-min-isr) and version updated to 137 (kafka.cluster.Partition t.sh[1457438]: [2024-11-29 13:08:53,756] WARN [GroupCoordinator 0]: Failed to write empty metadata for group hi: The coordinator is not available. (kafka. coordina t.sh[1457438]: [2024-11-29 13:08:53,964] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set_ __consumer_offsets-5) (kafka.server. ReplicaFetcherMana t.sh[1457438]: [2024-11-29 13:08:53,964] INFO [GroupCoordinator 0]: Elected as the group coordinator for partition 5 in epoch 12 (kafka.coordinator.group.GroupCoordinator) t.sh[1457438]: [2024-11-29 13:08:53,964] INFO [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from _consumer_offsets-5 for epoch 12 (kas t.sh[1457438]: [2024-11-29 13:08:53,965] INFO [GroupMetadataManager brokerId=0] Already loading offsets and group metadata from __consumer. _offsets-5 (kafka.coordinator.gros 438]: [2024-11-29 13:08:53,718] INFO [Partition __consumer_offsets-5 broker=0] Shrinking IS from 0,2,1 to 0. Leader: (highWatermark: 32463865, endOffset: 32463866). out of 438]: [2024-11-29 13:08:53,747] INFO [Partition __consumer_offsets-5 broker=0] IS updated to 0 (under-min-isr) and version updated to 137 (kafka.cluster.Partition) 438]: [2024-11-29 13:08:53,756] WARN [GroupCoordinator 0]: Failed to write empty metadata for group hi: The coordinator is not available. (kafka.coordinator.groups 438]: [2024-11-29 13:08:53,964] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions Set_ _consumer_offsets-5) (kafka.server. ReplicaFetcherManager) 438]: [2024-11-29 13:08:53,964] INFO [GroupCoordinator 0]: Elected as the group coordinator for partition 5 in epoch 12 (kafka.coordinator.group.GroupCoordinator) 438]: [2024-11-29 13:08:53,964] INFO [GroupMetadataManager brokerId=0] Scheduling loading of offsets and group metadata from _consumer_offsets-5 for epoch 12 (kafka.coords 438]: [2024-11-29 13:08:53,965] INFO [GroupMetadataManager brokerId=0] Already loading offsets and group metadata from __consumer_offsets-5 (kafka. coordinator.group. Groups 438]: [2024-11-29 13:09:03,047] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Disconnecting from node 1 due to request timeout. (org.apache.kafka.clients.Netw 438]: [2024-11-29 13:09:03,047] INFO [ReplicaFetcher replicaId=, leaderId=1, fetcherId=0] Cancelled in-flight FETCH request with correlation id 34677641 due to node 1 beis 438]: [2024-11-29 13:09:03,047] INFO [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Client requested connection close from node 1 (org.apache.kafka.clients.NetworkC 4381: [2024-11-29 13:09:03, 048] INFO [ReplicaFetcher replicaId=o, leaderId-1, fetcherId-0] Error sending fetch request (sessionId=660052978, epoch=34677641) to node 1: Cora 438]: java.o.IOException: Connection to 1 was disconnected before the response was read 438]: at org. apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:99) 438]: at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockinqSender.scala:113) 438]: 438]: at kafka.server. RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316) 438]: at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130) 438]: at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129) 438]: at scala. Option.foreach(Option.scala:437) 438]: 438]: at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129) at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112) 4381: at kafka.server.ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98) 4381: at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:130) 438]: [2024-11-29 13:09:03,050] WARN [ReplicaFetcher replicaId=0, leaderId=1, fetcherId=0] Error in response for fetch request (type=FetchRequest, replicaId-0, maxWait=2005 438]: java.io.IOException: Connection to 1 was disconnected before the response was read 4381: at org. apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:99) 4381: at kafka.server.BrokerBlockingSender.sendRequest(BrokerBlockingSender.scala:113) 4381 at kafka.server.RemoteLeaderEndPoint.fetch(RemoteLeaderEndPoint.scala:79) at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:316) 438] at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3(AbstractFetcherThread.scala:130) 4381 at kafka.server.AbstractFetcherThread.$anonfun$maybeFetch$3$adapted(AbstractFetcherThread.scala:129) 438] 438] at scala.Option.foreach(Option.scala: 437) 438] at kafka.server.AbstractFetcherThread.maybeFetch(AbstractFetcherThread.scala:129) 438]: at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:112) at kafka.server. ReplicaFetcherThread.doWork(ReplicaFetcherThread.scala:98) 4381: at org. apache. kafka.server.util.ShutdownableThread.run(ShutdownableThread. java: 130) 438]: [2024-11-29 13:09:05,589] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId-0] Disconnecting from node 2 due to request timeout. Corg.apache.kafka.clients.Netw 438]: [2024-11-29 13:09:05,589] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Cancelled in-flight FETCH request with correlation id 38762087 due to node 2 beis 438]: [2024-11-29 13:09:05,589] INFO [ReplicaFetcher replicaId=0, leaderId-2, fetcherId=0] Client requested connection close from node 2 (org.apache. kafka.clients. NetworkC 438]: [2024-11-29 13:09:05,590] INFO [ReplicaFetcher replicaId=0, leaderId=2, fetcherId=0] Error sending fetch request (sessionId=1014113766, epoch=38762087) to node 2: (os 438]: lava.io. IOException: Connection to 2 was disconnected before the response was read 4381: at org.apache.kafka.clients.NetworkClientUtils.sendAndReceive(NetworkClientUtils.java:99) And this logs appears on remaining 2 brokers nodes, it seems that kafka brokers is lost connection between nodes, but ssh and other traffic to/from nodes was worked! And on the network side there is no problems, What else could have caused the loss of connection with all brokers? Kafka itself worked, we didn't reboot it and the problem solved itself there is no high load on brokers on CPU,RAM,I/O but the first broker have LA twice bigger that other 2 nodes ( first broker is not a KRaft leader! ) but i don't see any problems related to kafka/kraft in the logs, only connections issues ???????? ????????, ??? ??? ?????????????? ? ????????? ???? ???????????? ? ???? ???????????????? ?????? ?? ?????????? ?????? ?????, ??? ?????????? ????????? ???? ?? ?????????? ?????????????????? ? ?????? ????? ??????, ? ????? ????????????? ????? ?? ???????????????????? ????????? ?????????? ???????? ??????.
