Hi Mickael, I am +1 on the overall approach of this KIP, but have a couple of comments (sorry, should have brought them up on the discuss thread earlier):
1. Perhaps it would be better to do this after KAFKA-4137 <https://issues.apache.org/jira/browse/KAFKA-4137> is implemented? At the moment, coordinator shares the same NetworkClient (and hence the same Selector) with consumer connections used for fetching records. Since freeing of memory relies on consuming applications invoking poll() after processing previous records and potentially after committing offsets, it will be good to ensure that coordinator is not blocked for read by fetch responses. This may be simpler once coordinator has its own Selector. 2. The KIP says: *Once messages are returned to the user, messages are deleted from the MemoryPool so new messages can be stored.* Can you expand that a bit? I am assuming that partial buffers never get freed when some messages are returned to the user since the consumer is still holding a reference to the buffer. Would buffers be freed when fetches for all the partitions in a response are parsed, but perhaps not yet returned to the user (i.e., is the memory freed when a reference to the response buffer is no longer required)? It will be good to document the (approximate) maximum memory requirement for the non-compressed case. There is data read from the socket, cached in the Fetcher and (as Radai has pointed out), the records still with the user application. On Tue, Dec 6, 2016 at 2:04 AM, radai <radai.rosenbl...@gmail.com> wrote: > +1 (non-binding). > > small nit pick - just because you returned a response to user doesnt mean > the memory id no longer used. for some cases the actual "point of > termination" may be the deserializer (really impl-dependant), but > generally, wouldnt it be "nice" to have an explicit dispose() call on > responses (with the addition that getting the next batch of data from a > consumer automatically disposes the previous results) > > On Mon, Dec 5, 2016 at 6:53 AM, Edoardo Comar <eco...@uk.ibm.com> wrote: > > > +1 (non binding) > > -------------------------------------------------- > > Edoardo Comar > > IBM MessageHub > > eco...@uk.ibm.com > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > IBM United Kingdom Limited Registered in England and Wales with number > > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. > PO6 > > 3AU > > > > > > > > From: Mickael Maison <mickael.mai...@gmail.com> > > To: dev@kafka.apache.org > > Date: 05/12/2016 14:38 > > Subject: [VOTE] KIP-81: Bound Fetch memory usage in the consumer > > > > > > > > Hi all, > > > > I'd like to start the vote for KIP-81: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > 81%3A+Bound+Fetch+memory+usage+in+the+consumer > > > > > > Thank you > > > > > > > > > > Unless stated otherwise above: > > IBM United Kingdom Limited - Registered in England and Wales with number > > 741598. > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 > 3AU > > > -- Regards, Rajini
