Hey, I wanted to raise a performance issue that I encountered while testing on a large scale with Kafka. The issue pertains to low throughput with 20k topics (each topic with 1 partition) compared to good throughput when using 1 topic with 20k partitions.
Scenario: I have a setup of kafka with 20k partitions with replication factor of 3 and min.insync.replica of 2. We have 25 broker nodes and topics are evenly distributed across the brokers. Each broker has 8 cores, 32 GB memory and persistent HDD. I am testing with 1 producer node, writing a record of size 3KB randomly to one of the 20k topics. I have modified the kafka perf producer code to write randomly to different topic for each record. And have added additional print statements (as mentioned below). I did some analysis of kafka producer code and found the reason for bad performance. TLDR: poor performance is due to metadata fetch required for each topic. In case of 1 topic 20k partitions, it is only required once and is cached. In case of 20k topics, metadata is fetched for each topic. And thus takes a ramp up of about ~5 mins to start getting better throughput. Details: 1. I added timers in kafka producer code to identify where time was being spent. 2. Essentially, the producer calls producer.send() <https://github.com/apache/kafka/blob/trunk/tools/src/main/java/org/apache/kafka/tools/ProducerPerformance.java#L119>. Which in turn makes calls to fetch the metadata from cache. If not present, it calls the broker for metadata and waits for response inside awaitUpdate <https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L1176> 3. this awaitUpdate is causing the poor performance Test Runs - printing every one second, average, max and number of times method is called 1. 1 Topic 20k partitions I have no name!@kafka-producer-bitnami-74847c7dff-h5dbq:/opt/bitnami/kafka/bin$ ./kafka-producer-perf-test.sh --topic amit-1 --num-records 100000000 --record-size 3000 --throughput 0 --producer-props acks=all bootstrap.servers=test-cluster-kafka-bootstrap:9092 batch.size=3000 buffer.memory=100000000 linger.ms=50 | grep producer.send producer.send(): Avg - 0 ms, Max - 478 ms, num_records = 3838 producer.send(): Avg - 0 ms, Max - 44 ms, num_records = 3273 producer.send(): Avg - 0 ms, Max - 43 ms, num_records = 2574 producer.send(): Avg - 0 ms, Max - 45 ms, num_records = 2922 producer.send(): Avg - 0 ms, Max - 36 ms, num_records = 3086 producer.send(): Avg - 0 ms, Max - 37 ms, num_records = 3014 2. 20k topics I have no name!@kafka-producer-bitnami-74847c7dff-h5dbq:/opt/bitnami/kafka/bin$ ./kafka-producer-perf-test.sh --topic test-1 --num-records 100000000 --record-size 3000 --throughput 0 --producer-props acks=all bootstrap.servers=test-cluster-kafka-bootstrap:9092 batch.size=3000 buffer.memory=100000000 linger.ms=50 | grep producer.send producer.send(): Avg - 5 ms, Max - 236 ms, num_records = 188 producer.send(): Avg - 1 ms, Max - 20 ms, num_records = 498 producer.send(): Avg - 2 ms, Max - 23 ms, num_records = 438 producer.send(): Avg - 2 ms, Max - 16 ms, num_records = 425 producer.send(): Avg - 2 ms, Max - 17 ms, num_records = 343 producer.send(): Avg - 2 ms, Max - 15 ms, num_records = 334 producer.send(): Avg - 3 ms, Max - 15 ms, num_records = 295 producer.send(): Avg - 3 ms, Max - 20 ms, num_records = 268 Where: Avg - average time taken in 1 second Max - max time taken in the call within 1 sec num_records - number of times the send was called FYI - the average time keeps increasing to around 12 ms in case of 20k topics. In case of 1 topic, it is 0 due to cache return I wanted to check if there could be a capability to fetch metadata for all topics (or matching a pattern) at the very beginning. Just like it is done for 1 topic with 20k partitions. Thanks, Amit -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it.
