[
https://issues.apache.org/jira/browse/KAFKA-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980590#comment-13980590
]
Guozhang Wang commented on KAFKA-1417:
--------------------------------------
In 0.8.1, the leader finder thread would not add partition one-by-one but in
batches. Would this help your case?
> Very slow initial high-level consumer startup in low traffic/blocking fetch
> scenario
> ------------------------------------------------------------------------------------
>
> Key: KAFKA-1417
> URL: https://issues.apache.org/jira/browse/KAFKA-1417
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.0
> Reporter: Sam Meder
> Assignee: Neha Narkhede
>
> We're seeing very slow startup times when starting a high level consumer in a
> low traffic/blocking fetch type setup. The example we've come across has a
> consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout.
> What happens is that the leader finder thread adds partitions one by one and
> since the offset is not know this causes a call to figure out the offset.
> This call uses the fetcher threads simple consumer instance and locks around
> the call. Initially this is not a problem, but as soon as the fetcher thread
> has some partitions it will start fetching and since this is a low traffic
> situation the fetch will at least sometimes take up to 20s (again locking
> around the simple consumer). This leads to behavior like:
> # Finder thread adds a partition
> # Data thread notices it has partitions to fetch data for, locks the consumer
> for 20s
> # Finder thread tries to add a partition, tries to lock consumer and blocks
> for 20s
> # Rinse, repeat for each partition
--
This message was sent by Atlassian JIRA
(v6.2#6252)
