Agreed with this approach. One detail to be wary of is that since we multiplex various other requests (e.g., heartbeats, offset commits, metadata, etc.) over the client that connects to the coordinator this could delay some of these critical requests. Realistically I don't think it will be an issue except in extreme scenarios where someone sets the memory limit to be unreasonably low.
Thanks, Joel On Sun, Oct 30, 2016 at 12:32 PM, Jun Rao <j...@confluent.io> wrote: > Hi, Mickael, > > I agree with others that it's better to be able to control the bytes the > consumer can read from sockets, instead of limiting the fetch requests. > KIP-72 has a proposal to bound the memory size at the socket selector > level. Perhaps that can be leveraged in this KIP too. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 72%3A+Allow+putting+a+bound+on+memory+consumed+by+Incoming+requests > > Thanks, > > Jun > > On Thu, Oct 27, 2016 at 3:23 PM, Jay Kreps <j...@confluent.io> wrote: > > > This is a good observation on limiting total memory usage. If I > understand > > the proposal I think it is that the consumer client would stop sending > > fetch requests once a certain number of in-flight fetch requests is met. > I > > think a better approach would be to always issue one fetch request to > each > > broker immediately, allow the server to process that request, and send > data > > back to the local machine where it would be stored in the socket buffer > (up > > to that buffer size). Instead of throttling the requests sent, the > consumer > > should ideally throttle the responses read from the socket buffer at any > > given time. That is, in a single poll call, rather than reading from > every > > single socket it should just read until it has a given amount of memory > > used then bail out early. It can come back and read more from the other > > sockets after those messages are processed. > > > > The advantage of this approach is that you don't incur the additional > > latency. > > > > -Jay > > > > On Mon, Oct 10, 2016 at 6:41 AM, Mickael Maison < > mickael.mai...@gmail.com> > > wrote: > > > > > Hi all, > > > > > > I would like to discuss the following KIP proposal: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > 81%3A+Max+in-flight+fetches > > > > > > > > > Feedback and comments are welcome. > > > Thanks ! > > > > > > Mickael > > > > > >
