I have a particular broker(version 0.8.2.1) in a cluster receiving about 15000 messages/second of around 100 bytes each (bytes-in / messages-in). This broker has bursts of really high log flush latency p95s. The latency sometimes goes to above 1.5 seconds from a steady state of < 20 ms.
Running top on the box shows bursts of wa (io wait) of over 10% during these high latency cycles. Running iostat during the same time shows that it was doing more than 16000 block writes/sec during these periods of high latency. Running iostat during a normal run shows around 3000 - 7000 block writes/second. So it does seem like during the latency cycle we are just trying to do too much IO. I have tried replacing the broker several times to see if that would help but it hasn't so far, so my guess is there is something peculiar about one or more of the partitions that are assigned to that broker. The other observation is that messages-in/sec and bytes-in/sec DO NOT go up higher when the broker is misbehaving. I have also noticed that every time I replace the broker it takes much longer for the new broker to bootstrap when compared to other brokers that I have replaced. For example it takes up to 20 minutes before the log flush latency on the newly bootstrapped broker (with the same ID as the one it replaced) to go to normal levels. Any hints on what else I can try (replacing broker does not seem to help? 15000 messages/second of around 100 bytes each on a dual SSD, 30 GB RAM and 16 core box would seem very doable. It doesn't bother the other brokers in the cluster either. Also any hints on how I can find the exact topic/partitions assigned to this broker? I know in ZK we can see the partition -> broker mapping, but I am looking for a broker -> partition mapping. I can't be sure if the load that is causing this problem is because of leader traffic or follower traffic. What is weird is that I rarely if ever see other brokers in the cluster have the same problem. With 3 way replication (leader + 2 replicas) I'd imagine that the same work load would cause problems on other brokers too. Thanks, Rajiv
