[
https://issues.apache.org/jira/browse/KAFKA-2486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14720040#comment-14720040
]
Guozhang Wang commented on KAFKA-2486:
--------------------------------------
The sleep is introduced in b6d326b0893e60b350608260fd1bd2542337cb5a as part of
KAFKA-2168. The motivation was that when there is no enough data on the broker
yet we should just treat it as "failed fetch request" and try to avoid
bombarding the servers with continuous fetch requests. I think there is still
value in backing off when there is no fetchable data, while the "retry.backoff"
config's default value 100ms could be OK in some real production environments,
for performance profiling it should really be 0.
> New consumer performance
> ------------------------
>
> Key: KAFKA-2486
> URL: https://issues.apache.org/jira/browse/KAFKA-2486
> Project: Kafka
> Issue Type: Sub-task
> Components: consumer
> Reporter: Ewen Cheslack-Postava
> Assignee: Jason Gustafson
> Fix For: 0.8.3
>
>
> The new consumer was previously reaching getting good performance. However, a
> recent report on the mailing list indicates it's dropped significantly. After
> evaluation, even with a local broker it seems to only be reaching a 2-10MB/s,
> compared to 600+MB/s previously. Before release, we should get the
> performance back on par.
> Some details about where the regression occurred from the mailing list
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201508.mbox/%3CCAAdKFaE8bPSeWZf%2BF9RuA-xZazRpBrZG6vo454QLVHBAk_VOJg%40mail.gmail.com%3E
> :
> bq. At 49026f11781181c38e9d5edb634be9d27245c961 (May 14th), we went from good
> performance -> an error due to broker apparently not accepting the partition
> assignment strategy. Since this commit seems to add heartbeats and the server
> side code for partition assignment strategies, I assume we were missing
> something on the client side and by filling in the server side, things
> stopped working.
> bq. On either 84636272422b6379d57d4c5ef68b156edc1c67f8 or
> a5b11886df8c7aad0548efd2c7c3dbc579232f03 (July 17th), I am able to run the
> perf test again, but it's slow -- ~10MB/s for me vs the 2MB/s Jay was seeing,
> but that's still far less than the 600MB/s I saw on the earlier commits.
> Ideally we would also at least have a system test in place for the new
> consumer, even if regressions weren't automatically detected. It would at
> least allow for manually checking for regressions. This should not be
> difficult since there are already old consumer performance tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
