Hi, All I've run bin/kafka-producer-perf-test.sh on our kafka-production cluster, I found the number of partitions really have huge impacts on the producer performance, see:
start.time, end.time, compression, message.size, batch.size, total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec 2014-12-22 19:53:27:392, 2014-12-22 19:54:25:581, 1, 3000, 200, 2861.02, 49.1678, 1000000, 17185.3787 2014-12-22 19:55:27:048, 2014-12-22 19:56:23:318, 1, 3000, 200, 2861.02, 50.8446, 1000000, 17771.4590 2014-12-22 19:58:09:466, 2014-12-22 19:59:05:068, 1, 3000, 200, 2861.02, 51.4554, 1000000, 17984.9646 2014-12-22 19:59:40:389, 2014-12-22 20:00:28:646, 1, 3000, 200, 2861.02, 59.2872, 1000000, 20722.3822 2014-12-22 20:02:41:993, 2014-12-22 20:03:22:481, 1, 3000, 200, 2861.02, 70.6635, 1000000, 24698.6762 2014-12-22 20:03:47:594, 2014-12-22 20:04:26:238, 1, 3000, 200, 2861.02, 74.0354, 1000000, 25877.2384 2014-12-22 20:11:49:492, 2014-12-22 20:12:25:843, 1, 3000, 200, 2861.02, 78.7055, 1000000, 27509.5596 2014-12-22 20:12:53:290, 2014-12-22 20:13:29:746, 1, 3000, 200, 2861.02, 78.4788, 1000000, 27430.3270 2014-12-22 20:13:53:194, 2014-12-22 20:14:29:470, 1, 3000, 200, 2861.02, 78.8682, 1000000, 27566.4351 2014-12-22 20:14:51:491, 2014-12-22 20:15:25:451, 1, 3000, 200, 2861.02, 84.2468, 1000000, 29446.4075 2014-12-22 20:16:51:369, 2014-12-22 20:17:27:452, 1, 3000, 200, 2861.02, 79.2901, 1000000, 27713.8819 2014-12-22 20:17:57:882, 2014-12-22 20:18:33:957, 1, 3000, 200, 2861.02, 79.3076, 1000000, 27720.0277 The number of partitions above are from 1 to 12, I wonder why it has such big difference? thanks -- Alec Li
