Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

t; > > wrote: >> > > > > > Hi Mickael, >> > > > > > as discussed we could change the priority parameter to be an int >> > > rather >> > > > > > than a boolean. >> > > > > > >> > > > &g

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

he priority parameter to be an int > > > rather > > > > > > than a boolean. > > > > > > > > > > > > That's a bit more extensible > > > > > > ------------------ > > > >

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

gt; > > IBM MessageHub > > > > > eco...@uk.ibm.com > > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > > > > > IBM United Kingdom Limited Registered in England and Wales with > > number > > > > > 74159

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

> > > > IBM MessageHub > > > > eco...@uk.ibm.com > > > > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > > > > IBM United Kingdom Limited Registered in England and Wales with > number > > > > 741598 Registered office: PO Box 41

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

stered in England and Wales with number > > > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. > > PO6 > > > 3AU > > > > > > > > > > > > From: Guozhang Wang > > > To: "dev@kafka.apache.org"

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

> > > > IBM United Kingdom Limited Registered in England and Wales with number > > 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. > PO6 > > 3AU > > > > > > > > From: Guozhang Wang > > To: "dev@kafka.ap

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

ng Wang > To: "dev@kafka.apache.org" > Date: 28/03/2017 19:02 > Subject:Re: [VOTE] KIP-81: Bound Fetch memory usage in the > consumer > > > > 1) Makes sense. > 2) Makes sense. Thanks! > > On Tue, Mar 28, 2017 at 10:11 AM, Mickael Maison >

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

mited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU From: Guozhang Wang To: "dev@kafka.apache.org" Date: 28/03/2017 19:02 Subject: Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer 1)

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

1) Makes sense. 2) Makes sense. Thanks! On Tue, Mar 28, 2017 at 10:11 AM, Mickael Maison wrote: > Hi Guozhang, > > Thanks for the feedback. > > 1) By MemoryPool, I mean the implementation added in KIP-72. That will > most likely be SimpleMemoryPool, but the PR for KIP-72 has not been > merged ye

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Hi Guozhang, Thanks for the feedback. 1) By MemoryPool, I mean the implementation added in KIP-72. That will most likely be SimpleMemoryPool, but the PR for KIP-72 has not been merged yet. I've updated the KIP to make it more obvious. 2) I was thinking to pass in the priority when creating the C

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Mickael, Sorry for the late review of the KIP. I'm +1 on the proposed change as well. Just a few minor comments on the wiki itself: 1. By the "MemoryPool" are you referring to a new class impl or to reusing " org.apache.kafka.clients.producer.internals.BufferPool"? I assume it was the latter case

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Thanks Jason for the feedback! Yes it makes sense to always use the MemoryPool is we can. I've updated the KIP with the suggestion On Fri, Mar 10, 2017 at 1:18 AM, Jason Gustafson wrote: > Just a minor comment. The KIP suggests that coordinator responses are > always allocated outside of the memo

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Just a minor comment. The KIP suggests that coordinator responses are always allocated outside of the memory pool, but maybe we can reserve that capability for only when the pool does not have enough space? It seems a little nicer to use the pool if we can. If that seems reasonable, I'm +1 on the K

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Yes I agree, having a generic flag is more future proof. I'll update the KIP in the coming days. Thanks On Tue, Feb 28, 2017 at 5:08 AM, Jason Gustafson wrote: > Hey Mickael, > > The suggestion to add something to Node makes sense. I could imagine for > example adding a flag to indicate that the

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Hey Mickael, The suggestion to add something to Node makes sense. I could imagine for example adding a flag to indicate that the connection has a higher "priority," meaning that we can allocate outside of the memory pool if necessary. That would still be generic even if the only use case is the co

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Apologies for the late response. Thanks Jason for the suggestion. Yes you are right, the Coordinator connection is "tagged" with a different id, so we could retrieve it in NetworkReceive to make the distinction. However, currently the coordinator connection are made different by using: Integer.MAX

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Good point. The consumer does use a separate connection to the coordinator, so perhaps the connection itself could be tagged for normal heap allocation? -Jason On Mon, Jan 23, 2017 at 10:26 AM, Onur Karaman wrote: > I only did a quick scan but I wanted to point out what I think is an > incorrec

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

I only did a quick scan but I wanted to point out what I think is an incorrect assumption in the KIP's caveats: " There is a risk using the MemoryPool that, after we fill up the memory with fetch data, we can starve the coordinator's connection ... To alleviate this issue, only messages larger than

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

I've updated the KIP to address all the comments raised here and from the "DISCUSS" thread. See: https://cwiki.apache.org/confluence/display/KAFKA/KIP-81%3A+Bound+Fetch+memory+usage+in+the+consumer Now, I'd like to restart the vote. On Tue, Dec 6, 2016 at 9:02 AM, Rajini Sivaram wrote: > Hi Mic

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

Hi Mickael, I am +1 on the overall approach of this KIP, but have a couple of comments (sorry, should have brought them up on the discuss thread earlier): 1. Perhaps it would be better to do this after KAFKA-4137 is implemented? At the moment, co

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

+1 (non-binding). small nit pick - just because you returned a response to user doesnt mean the memory id no longer used. for some cases the actual "point of termination" may be the deserializer (really impl-dependant), but generally, wouldnt it be "nice" to have an explicit dispose() call on resp

Re: [VOTE] KIP-81: Bound Fetch memory usage in the consumer

+1 (non binding) -- Edoardo Comar IBM MessageHub eco...@uk.ibm.com IBM UK Ltd, Hursley Park, SO21 2JN IBM United Kingdom Limited Registered in England and Wales with number 741598 Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6