[
https://issues.apache.org/jira/browse/KAFKA-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16955559#comment-16955559
]
Guozhang Wang commented on KAFKA-9063:
--------------------------------------
[~bansalp] I think one potential cost which is even more than linear to the
num.partitions today is the replication fetching, that means, even if 99% of
your topic partitions do not have any day, the replica fetcher still need to
pull from the leaders which incurs CPU costs (our current benchmark even shows
that this cost is more then linearly increasing as the num.partitions, details
in https://issues.apache.org/jira/browse/KAFKA-9039).
It would be great if you can help verifying that by turning replication.factor
to 1, and rerun the test and compare. At the mean time, the community is
actively working on further optimizing the replica fetching logic here:
https://issues.apache.org/jira/browse/KAFKA-9048
> KAfka producer throughput drops with number of topics even when producer is
> producing on one topic
> --------------------------------------------------------------------------------------------------
>
> Key: KAFKA-9063
> URL: https://issues.apache.org/jira/browse/KAFKA-9063
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 2.2.1
> Reporter: Pradeep Bansal
> Priority: Major
> Attachments: image-2019-10-18-10-22-40-372.png
>
>
> 5 broker cluster
> Topic partitions = 1
> Replication factor = 3
> Ac mode = all
> Send type = Asynchronous
> Message size = 100 bytes
> Log compaction = Enabled
> # We started throughput test with 1 topic and number of topics present in
> cluster at that time were 1 (excluding already existing topics 1000)
> # We left it to run for about 2 hours (The mean throughput we observed
> during this period was 54k msgs/sec)
> # After this, we started creating 10,000 topics one by one using a script
> # We noted throughput values after creating 100 topics and after
> 200,300,400…so on till 10,000 were created
> # After all 10,000 topics were created we left test to run for another 1 hr.
> During the entire duration, we were producing only on a single topic.
>
> What could be causing producer throughput drop by just adding number of
> topics?
> !image-2019-10-18-10-22-40-372.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
