+1 On Mon, Mar 7, 2016 at 4:58 PM, Gwen Shapira <g...@confluent.io> wrote:
> Hi Team, > > Since there is a fairly busy release coming up in 2 weeks, and since > partition-assignors are pluggable and don't need to be part of an > Apache Kafka release in order to be useful, can we delay this KIP to > release 0.10.1 or 0.11.0 (whichever is earlier)? > > This will give the community a chance to focus on features that are > critical for the next release. > > Gwen > > On Mon, Mar 7, 2016 at 10:30 AM, Joel Koshy <jjkosh...@gmail.com> wrote: > > Hey Andrew, > > > > Thanks for adding the example. No I don't think we have a jira open for > > that issue. I'm not sure if we need to fix it in roundrobin (now that > it's > > already out there and some may be using it) vs. just going with your new > > "fair" strategy and maybe add a new explicit roundrobinv2. > > > > Thanks > > > > Joel, > > > > On Mon, Feb 29, 2016 at 7:53 AM, Olson,Andrew <aols...@cerner.com> > wrote: > > > >> Thanks for the feedback. I have added a concrete example to the document > >> that I think illustrates the benefit relatively well. > >> > >> The observation about scaling the workload of individual consumers is > >> certainly valid. I had not really considered this. Our primary concern > is > >> being able to gradually roll out consumption configuration changes in a > >> minimally disruptive fashion, including load-balancing. If the round > robin > >> strategy can be enhanced to adequately handle that use case, we would be > >> happy. Is there a Jira open for the "flaw" that you mentioned? > >> > >> > >> > >> > >> On 2/26/16, 7:22 PM, "Joel Koshy" <jjkosh...@gmail.com> wrote: > >> > >> >Hi Andrew, > >> > > >> >Thanks for the wiki. Just a couple of comments: > >> > > >> > - The disruptive config change issue that you mentioned is pretty > much > >> a > >> > non-issue in the new consumer due to central assignment. > >> > - Optional: but it may be helpful to add a concrete example. > >> > - More of an orthogonal observation than a comment: with heavily > skewed > >> > subscriptions fairness is sort of moot. i.e., people would generally > >> scale > >> > up or down subscription counts with the express purpose of > >> > reducing/increasing load on those instances. > >> > - WRT roundrobin we later realized a significant flaw in the way we > lay > >> > out partitions: we originally wanted to randomize the partition > layout > >> to > >> > reduce the likelihood of most partitions of the same topic from > ending > >> up > >> > on a given consumer which is important if you have a few very large > >> topics. > >> > Unfortunately we used hashCode - which does a splendid job of > clumping > >> > partitions from the same topic together :( We can probably just > "fix" > >> that > >> > in the new consumer's roundrobin assignor. > >> > > >> >Thanks, > >> > > >> >Joel > >> > > >> > > >> >On Fri, Feb 26, 2016 at 2:32 PM, Olson,Andrew <aols...@cerner.com> > wrote: > >> > > >> >> Here is a proposal for a new partition assignment strategy, > >> >> > >> >> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-49+-+Fair+Partition+Assignment+Strategy > >> >> > >> >> This KIP corresponds to these two pending pull requests, > >> >> https://github.com/apache/kafka/pull/146 > >> >> https://github.com/apache/kafka/pull/979 > >> >> > >> >> thanks, > >> >> Andrew > >> >> > >> >> CONFIDENTIALITY NOTICE This message and any included attachments are > >> from > >> >> Cerner Corporation and are intended only for the addressee. The > >> information > >> >> contained in this message is confidential and may constitute inside > or > >> >> non-public information under international, federal, or state > securities > >> >> laws. Unauthorized forwarding, printing, copying, distribution, or > use > >> of > >> >> such information is strictly prohibited and may be unlawful. If you > are > >> not > >> >> the addressee, please promptly delete this message and notify the > >> sender of > >> >> the delivery error by e-mail or you may call Cerner's corporate > offices > >> in > >> >> Kansas City, Missouri, U.S.A at (+1) (816)221-1024. > >> >> > >> > -- Thanks, Neha
