[
https://issues.apache.org/jira/browse/KAFKA-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13979267#comment-13979267
]
Sam Meder commented on KAFKA-1417:
----------------------------------
I think the timeout is somewhat arbitrary, but since we react to any data (1
byte requirement) we don't want to be be doing a whole bunch of unnecessary
fetches if there is not data. I'm going to implement the simple second consumer
approach and attach a patch.
> Very slow initial high-level consumer startup in low traffic/blocking fetch
> scenario
> ------------------------------------------------------------------------------------
>
> Key: KAFKA-1417
> URL: https://issues.apache.org/jira/browse/KAFKA-1417
> Project: Kafka
> Issue Type: Bug
> Components: consumer
> Affects Versions: 0.8.0
> Reporter: Sam Meder
> Assignee: Neha Narkhede
>
> We're seeing very slow startup times when starting a high level consumer in a
> low traffic/blocking fetch type setup. The example we've come across has a
> consumer that is set up to use 3 topics and uses a 20s/1 byte fetch timeout.
> What happens is that the leader finder thread adds partitions one by one and
> since the offset is not know this causes a call to figure out the offset.
> This call uses the fetcher threads simple consumer instance and locks around
> the call. Initially this is not a problem, but as soon as the fetcher thread
> has some partitions it will start fetching and since this is a low traffic
> situation the fetch will at least sometimes take up to 20s (again locking
> around the simple consumer). This leads to behavior like:
> # Finder thread adds a partition
> # Data thread notices it has partitions to fetch data for, locks the consumer
> for 20s
> # Finder thread tries to add a partition, tries to lock consumer and blocks
> for 20s
> # Rinse, repeat for each partition
--
This message was sent by Atlassian JIRA
(v6.2#6252)
