Sudden imbalance between partitions

2023-03-24 Thread Margaret Figura
Hi, We have a 22-node Kafka 3.3.1 cluster on K8s. All data is sent with null partitionId and null key from 20 Java producers, so it should be distributed evenly across partitions. All was good for days, but a couple hours ago, broker 21 started receiving about 2x the data of the other brokers f

Re: Sudden imbalance between partitions

2023-03-24 Thread Greg Harris
Meg, What version are your clients, and what partitioner are you using for these records? If you're using the DefaultPartitioner from 2.4.0+, it has a known imbalance flaw that is described and addressed by this KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+St