Hi, Mickael, I agree with others that it's better to be able to control the bytes the consumer can read from sockets, instead of limiting the fetch requests. KIP-72 has a proposal to bound the memory size at the socket selector level. Perhaps that can be leveraged in this KIP too.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-72%3A+Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests Thanks, Jun On Thu, Oct 27, 2016 at 3:23 PM, Jay Kreps <j...@confluent.io> wrote: > This is a good observation on limiting total memory usage. If I understand > the proposal I think it is that the consumer client would stop sending > fetch requests once a certain number of in-flight fetch requests is met. I > think a better approach would be to always issue one fetch request to each > broker immediately, allow the server to process that request, and send data > back to the local machine where it would be stored in the socket buffer (up > to that buffer size). Instead of throttling the requests sent, the consumer > should ideally throttle the responses read from the socket buffer at any > given time. That is, in a single poll call, rather than reading from every > single socket it should just read until it has a given amount of memory > used then bail out early. It can come back and read more from the other > sockets after those messages are processed. > > The advantage of this approach is that you don't incur the additional > latency. > > -Jay > > On Mon, Oct 10, 2016 at 6:41 AM, Mickael Maison <mickael.mai...@gmail.com> > wrote: > > > Hi all, > > > > I would like to discuss the following KIP proposal: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 81%3A+Max+in-flight+fetches > > > > > > Feedback and comments are welcome. > > Thanks ! > > > > Mickael > > >
