Re: Performance of producer sending to many vs to few topics

2016-07-20 Thread Krzysztof Nawara
Hello, No, in all test I used two partitions per topic, one per broker. The only variable in the test was number of topics. From what I've read what really affects performance is total number of partitions per broker (probably including replicas), so 1 topic with 1000 partitions should pretty much

Re: Performance of producer sending to many vs to few topics

2016-07-19 Thread R Krishna
We did similar testing recently, newbie here, assuming you did async publisher, did you also test with multiple partitions (1-1000) per topic as well. More topics, implies more metadata per topic exchanged every minute, more batches maintained and flushed per topic+partition per producer so higher

Performance of producer sending to many vs to few topics

2016-07-19 Thread Krzysztof Nawara
Hi, I have been testing Kafka in order to determine how does number of partitions affects performance. In order to do that, I have set up 2 Kafka nodes and 1 Zookeeper nodes, and used 8 producers running on different machines to send messages. First, my producers were sending messages to broker