[
https://issues.apache.org/jira/browse/KAFKA-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979260#comment-13979260
]
Jun Rao commented on KAFKA-1417:
--------------------------------
Interesting. The problem is that the leaderFinderThread uses the same
SimpleConsumer (used by the fetcher thread) when issuing the OffsetBefore
request. We could somehow let them use different SimpleConsumer instances. Not
sure if this is the best solution though.
Also, is there a particular reason that you use a 20s maxwait in the fetch
request?
> Very slow initial high-level consumer startup in low traffic/blocking fetch
> scenario
> ------------------------------------------------------------------------------------
>
> Key: KAFKA-1417
> URL: https://issues.apache.org/jira/browse/KAFKA-1417
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.0
> Reporter: Sam Meder
> Assignee: Neha Narkhede
>
> We're seeing very slow startup times when starting a high level consumer in a
> low traffic/blocking fetch type setup. The example we've come across has a
> consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout.
> What happens is that the leader finder thread adds partitions one by one and
> since the offset is not know this causes a call to figure out the offset.
> This call uses the fetcher threads simple consumer instance and locks around
> the call. Initially this is not a problem, but as soon as the fetcher thread
> has some partitions it will start fetching and since this is a low traffic
> situation the fetch will at least sometimes take up to 20s (again locking
> around the simple consumer). This leads to behavior like:
> # Finder thread adds a partition
> # Data thread notices it has partitions to fetch data for, locks the consumer
> for 20s
> # Finder thread tries to add a partition, tries to lock consumer and blocks
> for 20s
> # Rinse, repeat for each partition
--
This message was sent by Atlassian JIRA
(v6.2#6252)
