[
https://issues.apache.org/jira/browse/KAFKA-9191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Pettitt updated KAFKA-9191:
---------------------------------
Description:
We have observed, using two entirely different tools, that a simple Kafka
application (read 1 topic and immediately produce to another) suffers
substantial throughput degradation when scaling up topics. Below is the output
of one of these tools, showing that going from 1 partition to 1000 partitions
yields a ~30% throughput decrease when messages are 100 bytes long.
Using the same two tools, we observed that increasing the message size to 512
bytes yields a throughput increase of ~20% going from 1 topic partition to 1000
topic partitions with all other variables held constant.
|Kafka Core Testing| | | | | | | | | | |
|Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval
(ms)|Num Records|Record Size (b)|Num Input Topics|1 Partition MB/s|1000
Partitions MB/s|MB/s delta|
|FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
|FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
was:
We have observed, using two entirely different tools, that a simple Kafka
application (read 1 topic and immediately produce to another) suffers
substantial performance degradation when scaling up topics. Below is the output
of one of these tools, showing that going from 1 partition to 1000 partitions
yields a ~30% throughput decrease when messages are 100 bytes long.
Using the same two tools, we observed that increasing the message size to 512
bytes yields a throughput increase of ~20% going from 1 topic partition to 1000
topic partitions with all other variables held constant.
|Kafka Core Testing| | | | | | | | | | |
|Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval
(ms)|Num Records|Record Size (b)|Num Input Topics|1 Partition MB/s|1000
Partitions MB/s|MB/s delta|
|FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
|FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
> Kafka throughput suffers substantially when scaling topic partitions with
> small messages
> ----------------------------------------------------------------------------------------
>
> Key: KAFKA-9191
> URL: https://issues.apache.org/jira/browse/KAFKA-9191
> Project: Kafka
> Issue Type: Bug
> Reporter: Chris Pettitt
> Priority: Major
>
> We have observed, using two entirely different tools, that a simple Kafka
> application (read 1 topic and immediately produce to another) suffers
> substantial throughput degradation when scaling up topics. Below is the
> output of one of these tools, showing that going from 1 partition to 1000
> partitions yields a ~30% throughput decrease when messages are 100 bytes long.
> Using the same two tools, we observed that increasing the message size to 512
> bytes yields a throughput increase of ~20% going from 1 topic partition to
> 1000 topic partitions with all other variables held constant.
>
> |Kafka Core Testing| | | | | | | | | | |
> |Enable Transaction|Batch Size (b)|Linger (ms)|Max Inflight|Commit Interval
> (ms)|Num Records|Record Size (b)|Num Input Topics|1 Partition MB/s|1000
> Partitions MB/s|MB/s delta|
> |FALSE|16384|100|5|1000|20000000|100|1|45.633625|31.482193|-31.01%|
> |FALSE|16384|100|5|1000|20000000|512|1|70.217902|85.319107|21.51%|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
