Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

2016-03-07 Thread Neha Narkhede
+1 On Mon, Mar 7, 2016 at 4:58 PM, Gwen Shapira wrote: > Hi Team, > > Since there is a fairly busy release coming up in 2 weeks, and since > partition-assignors are pluggable and don't need to be part of an > Apache Kafka release in order to be useful, can we delay this KIP to > release 0.10.1 o

Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

2016-03-07 Thread Gwen Shapira
Hi Team, Since there is a fairly busy release coming up in 2 weeks, and since partition-assignors are pluggable and don't need to be part of an Apache Kafka release in order to be useful, can we delay this KIP to release 0.10.1 or 0.11.0 (whichever is earlier)? This will give the community a chan

Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

2016-03-07 Thread Joel Koshy
Hey Andrew, Thanks for adding the example. No I don't think we have a jira open for that issue. I'm not sure if we need to fix it in roundrobin (now that it's already out there and some may be using it) vs. just going with your new "fair" strategy and maybe add a new explicit roundrobinv2. Thanks

Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

2016-02-29 Thread Olson,Andrew
Thanks for the feedback. I have added a concrete example to the document that I think illustrates the benefit relatively well. The observation about scaling the workload of individual consumers is certainly valid. I had not really considered this. Our primary concern is being able to gradually

Re: [DISCUSS] KIP-49: Fair Partition Assignment Strategy

2016-02-26 Thread Joel Koshy
Hi Andrew, Thanks for the wiki. Just a couple of comments: - The disruptive config change issue that you mentioned is pretty much a non-issue in the new consumer due to central assignment. - Optional: but it may be helpful to add a concrete example. - More of an orthogonal observation

[DISCUSS] KIP-49: Fair Partition Assignment Strategy

2016-02-26 Thread Olson,Andrew
Here is a proposal for a new partition assignment strategy, https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy This KIP corresponds to these two pending pull requests, https://github.com/apache/kafka/pull/146 https://github.com/apache/kafka/pull/979 than