Re: Debugging periodic high kafka log flush times

2019-01-21 Thread Cremona, Guido M
Hi Rajiv, Did you ever find out what was causing this issue? I notice something similar on my Kafka Cluster only that the 95th percentile of the log flush goes above 10 or 20 Min, and then I start to see under replicated partitions. Besides the difference on time, the scenario is quite the sam

Re: Debugging periodic high kafka log flush times

2015-09-21 Thread Rajiv Kurian
Just an update. We moved all the partitions from the one topic that generated most of the 545 thousand messages/second to their own set of brokers. The old set of 9 brokers now only get 135 thousand messages/second or 15 thousand messages/sec/broker. We are still seeing the same log flush time iss

Debugging periodic high kafka log flush times

2015-09-17 Thread Rajiv Kurian
We have a 9 node cluster running 0.8.2.1 that does around 545 thousand messages(kafka-messages-in) per second. Each of our brokers has 30 G of memory and 16 cores. We give the brokers themselves 2G of heap. Each broker ranges from around 33 - 40% cpu utilization. The values for both kafka-bytes-in