Hi Group, Recently I am trying to turn Kafka write performance to improve throughput. On my Kafka broker, there are 3 disks (7200 RPM). For one disk, the Kafka write throughput can reach 150MB/s. In my opinion, if I send message to topic test_p3 (which has 3 partitions located on different disk in the same server), the whole write throughput can reach 450 MB/s due to parallel writing disk. However the test result is still 150MB/s. Is there any reason that multiple disk doesn’t multiply the write throughput? And is there any bottleneck for the Kafka write throughput or I need some configuration to update?
I also try to test sending message to two different topic (whose partition on different disk of that server), and the total throughput only reach 200 MB/s instead of 300 MB/s as I expect. Below is my Kafka configuration and setting. ##Kafka producer setting ./kafka-run-class org.apache.kafka.tools.ProducerPerformance --topic test_p3 --num-records 50000000 --record-size 100 --throughput -1 --producer-props acks=0 bootstrap.servers=localhost:9092 buffer.memory=33554432 batch.size=16384 ##OS tuning setting net.core.rmem_default = 124928 net.core.rmem_max = 2048000 net.core.wmem_default = 124928 net.core.wmem_max = 2048000 net.ipv4.tcp_rmem = 4096 87380 4194304 net.ipv4.tcp_wmem = 4096 87380 4194304 net.ipv4.tcp_max_tw_buckets = 262144 net.ipv4.tcp_max_syn_backlog = 1024 vm.oom_kill_allocating_task = 1 vm.max_map_count = 200000 vm.swappiness = 1 vm.dirty_writeback_centisecs = 500 vm.dirty_expire_centisecs = 500 vm.dirty_ratio = 60 vm.dirty_background_ratio = 5 Thanks, Eric
