Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-05-05 Thread Guozhang Wang
; > > > > > > > Thanks for the KIP and the patience during discussion! > > > > +1 binding from me. > > > > > > > > Luke > > > > > > > > On Thu, Mar 24, 2022 at 3:43 AM Ismael Juma > wrote: >

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-05-05 Thread Artem Livshits
e > > > > > > On Thu, Mar 24, 2022 at 3:43 AM Ismael Juma wrote: > > > > > > > Thanks for the KIP and for taking the time to address all the > feedback. > > > +1 > > > > (binding) > > > > > > &

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-05-05 Thread Guozhang Wang
el > > > > > > On Mon, Mar 21, 2022 at 4:52 PM Artem Livshits > > > wrote: > > > > > > > Hi all, > > > > > > > > I'd like to start a vote on > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > > > > . > > > > > > > > -Artem > > > > > > > > > > -- -- Guozhang

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-26 Thread Lucas Bradstreet
> > > > Hi all, > > > > > > I'd like to start a vote on > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > > > . > > > > > > -Artem > > > > > >

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-23 Thread Luke Chen
PM Artem Livshits > wrote: > > > Hi all, > > > > I'd like to start a vote on > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > > . > > > > -Artem > > >

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-23 Thread Ismael Juma
Thanks for the KIP and for taking the time to address all the feedback. +1 (binding) Ismael On Mon, Mar 21, 2022 at 4:52 PM Artem Livshits wrote: > Hi all, > > I'd like to start a vote on > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Unifor

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-23 Thread David Jacot
gt; > Hi all, > > > > I'd like to start a vote on > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > > . > > > > -Artem > >

Re: [VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-22 Thread Jun Rao
Hi, Artem, Thanks for the KIP. +1 from me. Jun On Mon, Mar 21, 2022 at 4:52 PM Artem Livshits wrote: > Hi all, > > I'd like to start a vote on > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > . > > -Artem >

[VOTE] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-21 Thread Artem Livshits
Hi all, I'd like to start a vote on https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner . -Artem

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-17 Thread Artem Livshits
s, I'll start voting in the next couple of days. -Artem On Mon, Mar 14, 2022 at 6:19 PM Artem Livshits wrote: > Hi Jun, > > 33. Sounds good. Updated the KIP. > > -Artem > > On Mon, Mar 14, 2022 at 5:45 PM Jun Rao wrote: > >> Hi, Artem, >> >> 33.

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-14 Thread Artem Livshits
Hi Jun, 33. Sounds good. Updated the KIP. -Artem On Mon, Mar 14, 2022 at 5:45 PM Jun Rao wrote: > Hi, Artem, > > 33. We introduced onNewBatch() primarily for the sticky partitioner. It > seems to be a very subtle API to explain and to use properly. If we can't > find

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-14 Thread Jun Rao
Hi, Artem, 33. We introduced onNewBatch() primarily for the sticky partitioner. It seems to be a very subtle API to explain and to use properly. If we can't find any convincing usage, it's probably better to deprecate it so that we could keep the API clean. Thanks, Jun On Mon, Mar 1

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-14 Thread Artem Livshits
Hi Jun, 33. That's an interesting point. Technically, onNewBatch is just a way to pass some signal to the partitioner, the sticky partitioner uses this signal that is suboptimal, in theory someone could use it for something else -Artem On Mon, Mar 14, 2022 at 9:11 AM Jun Rao wrote:

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-14 Thread Jun Rao
; > > > partition.availability.timeout.ms. Luke's an > > > Jun's > > > > > > > > questions > > > > > > > > > > are > > > > > > > > > > > > > > slightly > > > > &

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-10 Thread Artem Livshits
it tries to preserve general fairness > and > > > not > > > > > > > overreact > > > > > > > > > > on > > > > > > > > > > > > the > > > > > > > > > > > > > > broker's state at each moment in time. But > because > > > > it's > > > > > > > smooth, > > > > > > > > > it > > > > > > > > > > > may >

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-10 Thread Jun Rao
; > > > choppy), > > > > > > > > so > > > > > > > > > > > > > partition.availability.timeout.ms provides an > > > > opportunity > > > > > to > > > > > > > > tune > > > > > > > > > > > > > adaptiveness. > > > > > &g

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-10 Thread Artem Livshits
at > > > > > > > > > > > > proves that over time it's going to be generally > > fair), I > > > > > feel > > > > > > > that > > > > > > > > > the > > > > > > > > > > > > algorithm should try to be fair in general a

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-10 Thread Jun Rao
> > > fully > > > > > > > > > > > remove brokers, just make it less likely) and relative > > info > > > > > > rather > > > > > > > > > than a > > > > > > > > > > > threshold (so if all brokers are heavi

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-09 Thread Artem Livshits
ust not use brokers that are not ready, but again, I > think > > > that > > > > > > it's > > > > > > > > > good > > > > > > > > > > to try to be fair under normal circumstances, so if > > normal

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-04 Thread Luke Chen
ies. > The > > > > > value, > > > > > > of > > > > > > > > > course, would depend on the environment and app > requirements, > > > > hence > > > > > > > it's > > > > > > > > > configurable. > > > > > > > > > > > > > > > > > > 10. Added a statement at the beginning of the proposed

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-04 Thread Ismael Juma
gt; Thanks for the KIP. A few comments below. > > > > > > > > > > > > > > > > > > 1. I agree with Luke that having the partitioner returning > -1 > > > is > > > > > kind > > > &g

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-04 Thread David Jacot
existing batch.size? > > > > > > > > > > > > > > > > 4. I also agree with Luke that it's not clear why we need > > > > > > > > partition.availability.timeout.ms. The KIP says the broker > > > "would > > > &

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-03 Thread Artem Livshits
gt; > > > 10. Currently, partitioner.class defaults to > DefaultPartitioner, > > > > which > > > > > > uses > > > > > > > StickyPartitioner when the key is specified. Since this KIP > > > improv

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-03 Thread Jun Rao
> > > > > > Jun > > > > > > > > > > > > > > > > > > On Wed, Feb 16, 2022 at 7:30 PM Luke Chen > > wrote: > > > > > > > > > > > > > Hi Artem, > > > > > > > > &g

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-03-03 Thread Artem Livshits
> > On Wed, Feb 16, 2022 at 3:28 PM Luke Chen > > wrote: > > > > > > > > > > > > > Hi Artem, > > > > > > > > > > > > > > Thanks for the update. I have some questions about it: > > > > &g

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-28 Thread Jun Rao
;Configuration" part, you didn't explain it. And default to 0, I > > > guess > > > > > it's > > > > > > the same as current behavior for backward compatibility, right? > You > > > > > should > > > > > > menti

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-25 Thread Artem Livshits
gt; > new > > > > > batch to change to next partition, ex: partition2. But I think if > we > > > set > > > > a > > > > > threshold to 95% (for example), we can know previous 15.5KB already > > > > exceeds > > > > > the thr

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-23 Thread Jun Rao
> > 4. I think the improved queuing logic should be good enough. I can't > > get > > > > the benefit of having `partition.availability.timeout.ms` config. In > > > > short, you want to make the partitioner take the broker load into > > > > cons

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-18 Thread Artem Livshits
need it. > > > > > > Thank you. > > > Luke > > > > > > On Wed, Feb 16, 2022 at 8:57 AM Artem Livshits > > > wrote: > > > > > >> Hello, > > >> > > >> Please add your comments about the KIP. I

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-17 Thread Jun Rao
gt; > Luke > > > > On Wed, Feb 16, 2022 at 8:57 AM Artem Livshits > > wrote: > > > >> Hello, > >> > >> Please add your comments about the KIP. If there are no considerations, > >> I'll put it up for vote in the next few days. > >&

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-16 Thread Luke Chen
ns, >> I'll put it up for vote in the next few days. >> >> -Artem >> >> On Mon, Feb 7, 2022 at 6:01 PM Artem Livshits >> wrote: >> >> > Hello, >> > >> > After trying a few prototypes, I've made some changes to the pu

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-15 Thread Luke Chen
ve made some changes to the public > > interface. Please see the updated document > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > > . > > > > -Artem > > > > On Thu, Nov 4, 2021 at 10:37 AM Artem Livshit

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-15 Thread Artem Livshits
interface. Please see the updated document > https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner > . > > -Artem > > On Thu, Nov 4, 2021 at 10:37 AM Artem Livshits > wrote: > >> Hello, >> >> This is the discussion

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2022-02-07 Thread Artem Livshits
Hello, After trying a few prototypes, I've made some changes to the public interface. Please see the updated document https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner . -Artem On Thu, Nov 4, 2021 at 10:37 AM Artem Livshits wrote: &g

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2021-11-19 Thread Artem Livshits
ioner.partition method that takes a callback that can be used to estimate record size. I've updated the KIP correspondingly https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner -Artem On Mon, Nov 8, 2021 at 5:42 PM Artem Livshits wrote: > Hi Luke,

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2021-11-08 Thread Artem Livshits
ving the partition an opportunity to hit 5 in-flight and start accumulating sooner. KIP-782 will make this even better, so batches could also grow beyond 16KB if production rate is high. -Artem On Mon, Nov 8, 2021 at 11:56 AM Justine Olshan wrote: > Hi Artem, > Thanks for working on improving

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2021-11-08 Thread Justine Olshan
Hi Artem, Thanks for working on improving the Sticky Partitioner! I had a few questions about this portion: *The batching will continue until either an in-flight batch completes or we hit the N bytes and move to the next partition. This way it takes just 5 records to get to batching mode, not 5

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2021-11-08 Thread Luke Chen
Thanks Artem, It's much better now. I've got your idea. In KIP-480: Sticky Partitioner, we'll change partition (call partitioner) when either 1 of below condition match 1. the batch is full 2. when linger.ms is up But, you are changing the definition, into a "partitioner.sticky

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2021-11-05 Thread Artem Livshits
uggestion > (4)) > 2. In the "Proposed change" section, you take an example to use > "ClassicDefaultPartitioner", is that referring to the current default > sticky partitioner? I think it'd better you name your proposed partition > with a different name for

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2021-11-04 Thread Luke Chen
ease explain more in KIP (with an example will be better as suggestion (4)) 2. In the "Proposed change" section, you take an example to use "ClassicDefaultPartitioner", is that referring to the current default sticky partitioner? I think it'd better you name your proposed pa

[DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

2021-11-04 Thread Artem Livshits
Hello, This is the discussion thread for https://cwiki.apache.org/confluence/display/KAFKA/KIP-794%3A+Strictly+Uniform+Sticky+Partitioner . The proposal is a bug fix for https://issues.apache.org/jira/browse/KAFKA-10888, but it does include a client config change, therefore we have a KIP to

Re: Sticky Partitioner

2020-12-02 Thread Justine Olshan
Hi Evelyn, Thanks for taking a look at improving the sticky partitioner! These edge cases seem like they would cause quite a bit a trouble. I think the idea to check for max.in.flight.requests.per.connection is a good one, but one concern I have is how this information will be available to the

Sticky Partitioner

2020-11-30 Thread Eevee
Hi all, I've noticed a couple edge cases in the Sticky Partitioner and I'd like to discuss introducing a new KIP to fix it. Behavior 1. Low throughput producers The first edge case occurs when a broker becomes temporarily unavailable for a period less then replica.lag.time.max.

[jira] [Resolved] (KAFKA-8601) Producer Improvement: Sticky Partitioner

2019-08-24 Thread Justine Olshan (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-8601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justine Olshan resolved KAFKA-8601. --- Resolution: Fixed > Producer Improvement: Sticky Partitio

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-26 Thread Jun Rao
Hi, Justine, Thanks for the updated KIP. The new interface seems cleaner to me. +1 Jun On Fri, Jul 26, 2019 at 4:14 PM Justine Olshan wrote: > Hello all, > I've just added the proposed changes to the KIP page > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+S

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-26 Thread Justine Olshan
Hello all, I've just added the proposed changes to the KIP page https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner . The PR has been updated as well. https://github.com/apache/kafka/pull/6997. The idea is that there will just be a separate void method to chang

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-26 Thread Justine Olshan
t; > > > > > > > > > Thank you, > > > > > > Justine > > > > > > > > > > > > On Sun, Jul 14, 2019 at 3:07 AM M. Manna > > wrote: > > > > > > >

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-26 Thread Jun Rao
n Sun, Jul 14, 2019 at 3:07 AM M. Manna > wrote: > > > > > > > > > > > +1(na) > > > > > > > > > > > > On Sat, 13 Jul 2019 at 22:17, Stanislav Kozlovski < > > > > > stanis...@confluent.io

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-19 Thread Justine Olshan
; > > > > > > > On Sat, 13 Jul 2019 at 22:17, Stanislav Kozlovski < > > > > stanis...@confluent.io> > > > > > wrote: > > > > > > > > > > > +1 (non-binding) > > > > >

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-19 Thread Bill Bejeck
) > > > > > > > > > > Thanks! > > > > > > > > > > On Fri, Jul 12, 2019 at 6:02 PM Gwen Shapira > > > wrote: > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Thank yo

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-19 Thread Mickael Maison
> Thanks! > > > > > > > > On Fri, Jul 12, 2019 at 6:02 PM Gwen Shapira > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > Thank you for the KIP. This was long awaited. > > > > &

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-18 Thread David Arthur
> > > +1 (non-binding) > > > > > > Thanks! > > > > > > On Fri, Jul 12, 2019 at 6:02 PM Gwen Shapira > wrote: > > > > > > > +1 (binding) > > > > > > > > Thank you for the KIP. This was long awaited. >

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-17 Thread Justine Olshan
> > > On Tue, Jul 9, 2019 at 5:15 PM Justine Olshan > > > wrote: > > > > > > > > Hello all, > > > > > > > > I'd like to start the vote for KIP-480 : Sticky Partitioner. > > > > > > > > > > http

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-14 Thread M. Manna
Jul 9, 2019 at 5:15 PM Justine Olshan > > wrote: > > > > > > Hello all, > > > > > > I'd like to start the vote for KIP-480 : Sticky Partitioner. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitione

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-13 Thread Stanislav Kozlovski
+1 (non-binding) Thanks! On Fri, Jul 12, 2019 at 6:02 PM Gwen Shapira wrote: > +1 (binding) > > Thank you for the KIP. This was long awaited. > > On Tue, Jul 9, 2019 at 5:15 PM Justine Olshan > wrote: > > > > Hello all, > > > > I'd like to

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-13 Thread Jun Rao
ay in the first PR, of course.) It would be an > > option > > > for people who wanted to configure this behavior. > > > > > > best, > > > Colin > > > > > > > > > On Wed, Jul 10, 2019, at 08:48, Justine Olshan wrote: > > > &

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-12 Thread Justine Olshan
; > However, we could add a new, separate StickyRoundRobinPartitioner > class > > > to > > > > KIP-480 which just implemented the sticky behavior regardless of > whether > > > > the key was null. That seems pretty easy to add (and it wouldn't > have to &

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-12 Thread Colin McCabe
e an > > option > > > for people who wanted to configure this behavior. > > > > > > best, > > > Colin > > > > > > > > > On Wed, Jul 10, 2019, at 08:48, Justine Olshan wrote: > > > > Hi M, > > > > > > &

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-12 Thread Gwen Shapira
+1 (binding) Thank you for the KIP. This was long awaited. On Tue, Jul 9, 2019 at 5:15 PM Justine Olshan wrote: > > Hello all, > > I'd like to start the vote for KIP-480 : Sticky Partitioner. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Parti

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-12 Thread Justine Olshan
n't have to > > implemented right away in the first PR, of course.) It would be an > option > > for people who wanted to configure this behavior. > > > > best, > > Colin > > > > > > On Wed, Jul 10, 2019, at 08:48, Justine Olshan wrote: > > >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-11 Thread Jun Rao
fused by what you mean by extending the behavior on to > the > > RoundRobinPartitioner. > > The sticky partitioner plans to remove the round-robin behavior from > > records with no keys. Instead of sending them to each partition in order, > > it sends them all to the same p

Re: [VOTE] KIP-480 : Sticky Partitioner

2019-07-11 Thread Colin McCabe
: > Hello all, > > I'd like to start the vote for KIP-480 : Sticky Partitioner. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > > Thank you, > Justine Olshan >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-11 Thread Colin McCabe
at you mean by extending the behavior on to the > RoundRobinPartitioner. > The sticky partitioner plans to remove the round-robin behavior from > records with no keys. Instead of sending them to each partition in order, > it sends them all to the same partition until the batch is sent. >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-10 Thread Justine Olshan
Hi M, I'm a little confused by what you mean by extending the behavior on to the RoundRobinPartitioner. The sticky partitioner plans to remove the round-robin behavior from records with no keys. Instead of sending them to each partition in order, it sends them all to the same partition unti

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-10 Thread M. Manna
t; traffic, > > > the > > > >>> producer stops sending to it, which puts even more load on the > > > remaining > > > >>> partitions, which are even more likely to fail then, etc. It also > will > > > >>> create unbala

[VOTE] KIP-480 : Sticky Partitioner

2019-07-09 Thread Justine Olshan
Hello all, I'd like to start the vote for KIP-480 : Sticky Partitioner. https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner Thank you, Justine Olshan

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-09 Thread Colin McCabe
gt; partitions, which are even more likely to fail then, etc. It also will > > >>> create unbalanced load patterns on the consumers. > > >>> > > >>> > > > >>> > 2. If there's no measurable performance difference, I agree with >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-09 Thread M. Manna
unbalanced load patterns on the consumers. > >>> > >>> > > >>> > 2. If there's no measurable performance difference, I agree with > >>> Stanislav > >>> > that Optional would be better tha

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-08 Thread Justine Olshan
ld be better than Integer. >>> > >>> > 3. We should include the javadoc for the newly introduced method that >>> > specifies it and its parameters. In particular, it would good to >>> specify if >>> > it gets called when an explicit partition id has been provid

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-07-02 Thread Justine Olshan
meters. In particular, it would good to >> specify if >> > it gets called when an explicit partition id has been provided. >> >> Agreed. >> >> best, >> Colin >> >> > >> > Ismael >> > >> > On Mon, Jun 24, 2019, 2:04 PM Justine Olshan >> wrote: >> > >> > > Hello, >> > > This is the discussion thread for KIP-480: Sticky Partitioner. >> > > >> > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner >> > > >> > > Thank you, >> > > Justine Olshan >> > > >> > >> >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-28 Thread Justine Olshan
ify > if > > it gets called when an explicit partition id has been provided. > > Agreed. > > best, > Colin > > > > > Ismael > > > > On Mon, Jun 24, 2019, 2:04 PM Justine Olshan > wrote: > > > > > Hello, > > > This is t

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-27 Thread Colin McCabe
lled when an explicit partition id has been provided. Agreed. best, Colin > > Ismael > > On Mon, Jun 24, 2019, 2:04 PM Justine Olshan wrote: > > > Hello, > > This is the discussion thread for KIP-480: Sticky Partitioner. > > > > > > https://cwiki.

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-27 Thread Justine Olshan
partitions are safer in general. There could be >> some downsides too, so worth thinking about the trade-offs. >> >> Ismael >> >> On Thu, Jun 27, 2019, 10:24 AM Justine Olshan >> wrote: >> >> > Ismael, >> > >> > Thanks for the feed

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-27 Thread Justine Olshan
t; Ismael, > > > > Thanks for the feedback! > > > > For 1, currently the sticky partitioner favors "available partitions." > From > > my understanding, these are partitions that are not under-replicated. If > > that is not the same, please let me know. >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-27 Thread Ismael Juma
smael On Thu, Jun 27, 2019, 10:24 AM Justine Olshan wrote: > Ismael, > > Thanks for the feedback! > > For 1, currently the sticky partitioner favors "available partitions." From > my understanding, these are partitions that are not under-replicated. If > that is not th

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-27 Thread Justine Olshan
Ismael, Thanks for the feedback! For 1, currently the sticky partitioner favors "available partitions." From my understanding, these are partitions that are not under-replicated. If that is not the same, please let me know. As for 2, I've switched to Optional, and the few tests

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-27 Thread Ismael Juma
discussion thread for KIP-480: Sticky Partitioner. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > > Thank you, > Justine Olshan >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-26 Thread Justine Olshan
24, 2019 at 4:51 PM Colin McCabe < > > cmcc...@apache.org> > > > >> wrote: > > > >> > > > > > > >> > > > > Hi Justine, > > > >> > > > > > > > >> > > > &

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-26 Thread Stanislav Kozlovski
> > > > > > >> > > > > Thanks for the KIP. This looks great! > > >> > > > > > > >> > > > > In one place in the KIP, you write: "Remove > > >> > > > > testRoundRobinWithUnavai

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Colin McCabe
Partitions() and testRoundRobin() > >> since > >> > > the > >> > > > > round robin functionality of the partitioner has been removed." > >> You > >> > > can > >> > > > > skip this and similar lines. We don't need to describe changes to > >> > > internal > >> > > > > test classes in the KIP since they're not visible to users or > >> external > >> > > > > developers. > >> > > > > > >> > > > > It seems like maybe the performance tests should get their own > >> section. > >> > > > > Right now, the way the layout is makes it look like they are part > >> of > >> > > the > >> > > > > "Compatibility, Deprecation, and Migration Plan" > >> > > > > > >> > > > > best, > >> > > > > Colin > >> > > > > > >> > > > > > >> > > > > On Mon, Jun 24, 2019, at 14:04, Justine Olshan wrote: > >> > > > > > Hello, > >> > > > > > This is the discussion thread for KIP-480: Sticky Partitioner. > >> > > > > > > >> > > > > > > >> > > > > > >> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > >> > > > > > > >> > > > > > Thank you, > >> > > > > > Justine Olshan > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > > >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Justine Olshan
lity of the partitioner has been removed." >> You >> > > can >> > > > > skip this and similar lines. We don't need to describe changes to >> > > internal >> > > > > test classes in the KIP since they're not visible to users or >> external >> > > > > developers. >> > > > > >> > > > > It seems like maybe the performance tests should get their own >> section. >> > > > > Right now, the way the layout is makes it look like they are part >> of >> > > the >> > > > > "Compatibility, Deprecation, and Migration Plan" >> > > > > >> > > > > best, >> > > > > Colin >> > > > > >> > > > > >> > > > > On Mon, Jun 24, 2019, at 14:04, Justine Olshan wrote: >> > > > > > Hello, >> > > > > > This is the discussion thread for KIP-480: Sticky Partitioner. >> > > > > > >> > > > > > >> > > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner >> > > > > > >> > > > > > Thank you, >> > > > > > Justine Olshan >> > > > > > >> > > > > >> > > > >> > > >> > >> >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Justine Olshan
> > > test classes in the KIP since they're not visible to users or > external > > > > > developers. > > > > > > > > > > It seems like maybe the performance tests should get their own > section. > > > > > Right now, the way the layout is makes it look like they are part > of > > > the > > > > > "Compatibility, Deprecation, and Migration Plan" > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > > > > > > On Mon, Jun 24, 2019, at 14:04, Justine Olshan wrote: > > > > > > Hello, > > > > > > This is the discussion thread for KIP-480: Sticky Partitioner. > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > > > > > > > > > > > > Thank you, > > > > > > Justine Olshan > > > > > > > > > > > > > > > > > > > > >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Colin McCabe
ip this and similar lines. We don't need to describe changes to > > internal > > > > test classes in the KIP since they're not visible to users or external > > > > developers. > > > > > > > > It seems like maybe the perf

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Justine Olshan
hey are part of > the > > > "Compatibility, Deprecation, and Migration Plan" > > > > > > best, > > > Colin > > > > > > > > > On Mon, Jun 24, 2019, at 14:04, Justine Olshan wrote: > > > > Hello, > > > > This is the discussion thread for KIP-480: Sticky Partitioner. > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > > > > > > > > Thank you, > > > > Justine Olshan > > > > > > > > > >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Colin McCabe
t; > test classes in the KIP since they're not visible to users or external > > > > developers. > > > > > > > > It seems like maybe the performance tests should get their own section. > > > > Right now, the way the layout is makes it look like they are p

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Justine Olshan
t; > > > It seems like maybe the performance tests should get their own section. > > > Right now, the way the layout is makes it look like they are part of > the > > > "Compatibility, Deprecation, and Migration Plan" > > > > > > best, > >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-25 Thread Colin McCabe
the layout is makes it look like they are part of the > > "Compatibility, Deprecation, and Migration Plan" > > > > best, > > Colin > > > > > > On Mon, Jun 24, 2019, at 14:04, Justine Olshan wrote: > > > Hello, > > > This is the discussion thread for KIP-480: Sticky Partitioner. > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > > > > > > Thank you, > > > Justine Olshan > > > > > >

[jira] [Created] (KAFKA-8601) Producer Improvement: Sticky Partitioner

2019-06-25 Thread Justine Olshan (JIRA)
Justine Olshan created KAFKA-8601: - Summary: Producer Improvement: Sticky Partitioner Key: KAFKA-8601 URL: https://issues.apache.org/jira/browse/KAFKA-8601 Project: Kafka Issue Type

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-24 Thread Boyang Chen
, at 14:04, Justine Olshan wrote: > > Hello, > > This is the discussion thread for KIP-480: Sticky Partitioner. > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > > > > Thank you, > > Justine Olshan > > >

Re: [DISCUSS] KIP-480 : Sticky Partitioner

2019-06-24 Thread Colin McCabe
eprecation, and Migration Plan" best, Colin On Mon, Jun 24, 2019, at 14:04, Justine Olshan wrote: > Hello, > This is the discussion thread for KIP-480: Sticky Partitioner. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner > > Thank you, > Justine Olshan >

[DISCUSS] KIP-480 : Sticky Partitioner

2019-06-24 Thread Justine Olshan
Hello, This is the discussion thread for KIP-480: Sticky Partitioner. https://cwiki.apache.org/confluence/display/KAFKA/KIP-480%3A+Sticky+Partitioner Thank you, Justine Olshan