Hi,
Recently in our experiment, we find that even though no resource usage
achieve 80%, consumers will slow down producer (which we were not
expected), especially when there is no message in the topic.
*We wonder if we did something wrong (where?), or this is a Kafka's
characteristic(explanation**?lock?).*
Here are the details (single partition fan-out):
- Topic: has only 1 partition
- Producer: single thread, max.in.flight.requests.per.connection=1 (we
need in-order delivery), batch.size=400000
- Consumers: 0-50 single thread consumers all in different consumer groups
- Broker: 3
- CPU: 2 * E5 2690v4(2.6GHz 14C28T)
- RAM: 384GiB
- HD: 2 * RAID5(6+1, 1.2T), fio direct write throughput > 800MiB/s
- Network: 10GbE
Test case:
- replication factor: 3
- producer acks=all
- empty topic
- start producer and consumers at the same time
- Result (number of consumers: producer throughput in MiB/s)
- 0: 103
- 1: 88
- 2: 77
- 3: 73
- 4: 67
- 8: 50
- 16: 33
- 30: 20
- 50: 12
- Resource usage
- very low CPU Usage (<5%), No single core achieve 100%
- mid network bandwidth usage (50%-60%)
- low disk utility
We did some additional tests:
- if there are some data in the topic, the producer speed will slow down
but not as much. But after consumers catch up, we get the same result
- even if acks=1, the interferance is still there
- tune some system parameters like noatime, but nothing changed.
