[
https://issues.apache.org/jira/browse/KAFKA-4007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15701234#comment-15701234
]
James Cheng commented on KAFKA-4007:
------------------------------------
If there is another fetch response on the socket, but the consumer doesn't read
it from the socket, does this have any impact on the broker? Will there be a
fetch response on the broker-side that is waiting to be written to the socket
or anything like that? Or will there be any other broker-side resource that is
held onto while waiting for the client to read the socket?
We had a situation that was resolved in
https://issues.apache.org/jira/browse/KAFKA-3996, where a consumer was slow in
reading from the socket, and therefore caused some sort of backup on the
broker. The scenario that Jun and Jason mentioned above about not reading from
the socket made me wonder if something similar might happen with this JIRA.
> Improve fetch pipelining for low values of max.poll.records
> -----------------------------------------------------------
>
> Key: KAFKA-4007
> URL: https://issues.apache.org/jira/browse/KAFKA-4007
> Project: Kafka
> Issue Type: Improvement
> Components: consumer
> Reporter: Jason Gustafson
> Assignee: Mickael Maison
>
> Currently the consumer will only send a prefetch for a partition after all
> the records from the previous fetch have been consumed. This can lead to
> suboptimal pipelining when max.poll.records is set very low since the
> processing latency for a small set of records may be small compared to the
> latency of a fetch. An improvement suggested by [~junrao] is to send the
> fetch anyway even if we have unprocessed data buffered, but delay reading it
> from the socket until that data has been consumed. Potentially the consumer
> can delay reading _any_ pending fetch until it is ready to be returned to the
> user, which may help control memory better.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
