Hi, I have been testing Kafka in order to determine how does number of partitions affects performance. In order to do that, I have set up 2 Kafka nodes and 1 Zookeeper nodes, and used 8 producers running on different machines to send messages. First, my producers were sending messages to broker selecting topics in round-robin fashion - each message would go to different topic. In second scenario I created just as many topics, but all producers concentrated on sending to just one of them. In first case, with 1000-topic/cluster throughput was ~250 times smaller than with 1-topic/cluster scenario. In the second one it was more like 1.5-2 times slower - huge difference. Do you have any ideas what might be the cause? Two things I came up with were: under-utilization of producer batching and a lot of random IO on the brokers. Both are just wild guesses. I have been using stock configuration - can you think about any particular properties I should play with?
Details: Machines (Kafka, Zk): STRATOS S810-X52L (32 cores, 64GB RAM), data stored on single dedicated SATA drive Machines (producers): Openstacks VMs (4 vCPUs, 8GB, 40GBs on SSD) Partitions per topic: 2 Replication factor: 2 Cordially, Chris
