Hi, We have a 22-node Kafka 3.3.1 cluster on K8s. All data is sent with null partitionId and null key from 20 Java producers, so it should be distributed evenly across partitions. All was good for days, but a couple hours ago, broker 21 started receiving about 2x the data of the other brokers for a few topics (but not all). These topics are all 1x replicated and the 96 partitions are distributed evenly across brokers (each broker has 4 or 5 partitions). This was detected in Grafana, but I can also see the offsets increasing much faster for the partitions owned by broker 21 in KafkaOffsetsShell. What could cause this? I didn't see anything unusual in the broker 21 logs or the controller logs.
Looking back, I noticed that broker 11 also becomes a bit unbalanced each day at the time when we are processing the most data, but it is only 10-15% higher than the others. All other brokers are quite even, including broker 21 until today. Any ideas on what I can check? Unfortunately we'll probably have to restart Kafka and/or the producers pretty soon. Thanks a lot! Meg
