;>
> >>>>>>
> >>>>>>> 2. In "Consumer Coordinator Algorithm", 1c suggests to revoke
> all
> >>>>>>> partition upon heartbeat/commit fail. What's the gain here? Do we
> >> want
> >>&g
rdinator Algorithm", 1c suggests to revoke all
>>>>>>> partition upon heartbeat/commit fail. What's the gain here? Do we
>> want
>>>> to
>>>>>>> keep all partitions running at this moment, to be optimistic for the
>>>>
KIP
> >> is
> >>>>>> for
> >>>>>>> consumer coordinator itself, and the rest of the paragraph did not
> >>>>> talk
> >>>>>>> about Streams any more. If you feel it's a bit distracted I can
> >> remove
> >>
>>>>>>> cleanup assignment and re-join, and not letting assignor making any
>>>>>>> proactive changes. The idea is to keep logic simpler and not doing
>> any
>>>>>>> "split brain" stuff.
>>>>>>>
>>>>
.
> >>
> >>
> >>> 4. Put two "looking into the future" into a separate category from
> >>> migration session. It seems inconsistent for readers to see this
> before we
> >>> finished discussion for everything.
> >>>
>
t for readers to see this before we
>>> finished discussion for everything.
>>>
>>
>> Ack.
>>
>>
>>> 5. Have we discussed the concern on the serialization? Could the new
>>> metadata we are adding grow larger than the message size cap
ation session. It seems inconsistent for readers to see this before we
>> finished discussion for everything.
>>
>
> Ack.
>
>
>> 5. Have we discussed the concern on the serialization? Could the new
>> metadata we are adding grow larger than the message size c
verything.
>
Ack.
> 5. Have we discussed the concern on the serialization? Could the new
> metadata we are adding grow larger than the message size cap?
>
We're completing https://issues.apache.org/jira/browse/KAFKA-7149 which
should largely reduce the message size (will update
ld the new
metadata we are adding grow larger than the message size cap?
Boyang
________________
From: Guozhang Wang
Sent: Monday, April 15, 2019 9:20 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Hello Jason,
I agree with you that for range /
> > it is a rather fixed trade-off between
> > >
> > > task availability and resource shuffling. If we eventually trigger
> > > rebalance after rolling bounce, certain consumer
> > >
> > > setup is still faced with global shuffles, for example member.id
, no version probing would be needed since
> we
> >> are guaranteed the leader knows everyone's version -- again it is
> assuming
> >> that higher versioned protocol is always backward compatible -- and
> hence
> >> can successfully do the assignment at that round.
> >>
> >> 60). My b
that
>>> it is a rather fixed trade-off between
>>>
>>> task availability and resource shuffling. If we eventually trigger
>>> rebalance after rolling bounce, certain consumer
>>>
>>> setup is still faced with global shuffles, for example member.id ranki
will be assigned with new member.id which reorders the
> > assignment. So I think the primary goal of incremental
> >
> > rebalancing is still improving the cluster availability during rebalance,
> > because it didn't revoke any partition during this
> >
> >
evoke any partition during this
>
> process. Also, the perk is minimum configuration requirement :)
>
>
> Best,
>
> Boyang
>
> ____________________
> From: Matthias J. Sax
> Sent: Tuesday, April 9, 2019 7:47 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-
> > because it didn't revoke any partition during this
> >
> > process. Also, the perk is minimum configuration requirement :)
> >
> >
> > Best,
> >
> > Boyang
> >
> >
> > From: Matthias J. Sax
>
>
> Best,
>
> Boyang
>
>
> From: Matthias J. Sax
> Sent: Tuesday, April 9, 2019 7:47 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
>
> Thank for the KIP, Boyang and Guozhang!
>
>
&g
uring this
process. Also, the perk is minimum configuration requirement :)
Best,
Boyang
From: Matthias J. Sax
Sent: Tuesday, April 9, 2019 7:47 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Thank for the KIP, Boyang and Guo
e
>>> should breakdown our
>>> delivery into different stages. At very beginning, our goal is to trigger
>>> learner task assignment only on
>>> `new` hosts, where we shall leverage leader's knowledge of previous round
>>> of rebalance to figure o
ebalance to figure out. After
>> stage one, our goal is to have a smooth scaling up experience, but the
>> task balance problem is kind of orthogonal.
>> The load balance problem is a much broader topic than auto scaling, which
>> I figure worth discussing within
>> this
Sounds good Guozhang! I will kick another discussion thread for KIP-444.
Boyang
From: Guozhang Wang
Sent: Thursday, March 21, 2019 9:17 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Hello Boyang,
I've made another tho
aling support should be treated as `a helpful
> mechanism to reach load balance`, but not `an algorithm defining load
> balance`. It would be great if you could share some insights of the stream
> task balance, which eventually helps us to break out of the KIP-429's scope
> and even
Boyang
________________
From: Guozhang Wang
Sent: Saturday, March 2, 2019 6:00 AM
To: dev
Subject: Re: [DISCUSS] KIP-429 : Smooth Auto-Scaling for Kafka Streams
Hello Boyang,
I've just made a quick pass on the KIP and here are some thoughts.
Meta:
1. I'm still no
Hello Boyang,
I've just made a quick pass on the KIP and here are some thoughts.
Meta:
1. I'm still not sure if it's worthwhile to add a new type of "learner
task" in addition to "standby task": if the only difference is that for the
latter, we would consider workload balance while for the forme
Hey community friends,
I'm gladly inviting you to have a look at the proposal to add incremental
rebalancing to Kafka Streams, A.K.A auto-scaling support.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Smooth+Auto-Scaling+for+Kafka+Streams
Special thanks to Guozhang for giving gre
24 matches
Mail list logo
