bie workers when
> incremental cooperative rebalancing is used
>
>
> Key: KAFKA-9849
> URL: https://issues.apache.org/jira
Konstantine Karantasis created KAFKA-9849:
-
Summary: Fix issue with worker.unsync.backoff.ms creating zombie
workers when incremental cooperative rebalancing is used
Key: KAFKA-9849
URL: https
HeryLong created KAFKA-8931:
---
Summary: PR '#6850' fix issue 'KAFKA-8449', but lead to restart
all cached tasks, which conflict to the motivation of incremental cooperative
rebalancing.
Key: KAFKA-8931
afkaConnect
>Affects Versions: 2.3.0
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
>Priority: Blocker
> Fix For: 2.3.0
>
>
> Tasks that are already running and are not redistributed are not currently
>
tal cooperative rebalancing and
> enable them for both eager and incremental cooperative rebalancing
> --
>
> Key: KAFKA-8473
>
Konstantine Karantasis created KAFKA-8473:
-
Summary: Adjust Connect system tests for incremental cooperative
rebalancing and enable them for both eager and incremental cooperative
rebalancing
Key: KAFKA-8473
Konstantine Karantasis created KAFKA-8449:
-
Summary: Restart task on reconfiguration under incremental
cooperative rebalancing
Key: KAFKA-8449
URL: https://issues.apache.org/jira/browse/KAFKA-8449
upgrade confusion.
> > > > >
> > > > > Best,
> > > > > Jason
> > > > >
> > > > > On Fri, Mar 8, 2019 at 10:37 AM Robert Yokota
> > > > wrote:
> > > > >
> > > > > > T
, Mar 8, 2019 at 10:37 AM Robert Yokota
> > > wrote:
> > > >
> > > > > Thanks for the great KIP Konstantine!
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Robert
> >
> > > On Fri, Mar 8, 2019 at 10:37 AM Robert Yokota
> > wrote:
> > >
> > > > Thanks for the great KIP Konstantine!
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Robert
> > > >
> > &
gt; >
> > > Robert
> > >
> > > On Thu, Mar 7, 2019 at 2:56 PM Guozhang Wang
> wrote:
> > >
> > > > Thanks Konstantine, I've read the updated section on
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/co
t; +1 (non-binding)
> >
> > Robert
> >
> > On Thu, Mar 7, 2019 at 2:56 PM Guozhang Wang wrote:
> >
> > > Thanks Konstantine, I've read the updated section on
> > >
> > >
> >
> https://cwiki.apache.org/c
;
> +1 (non-binding)
>
> Robert
>
> On Thu, Mar 7, 2019 at 2:56 PM Guozhang Wang wrote:
>
> > Thanks Konstantine, I've read the updated section on
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+i
Thanks for the great KIP Konstantine!
+1 (non-binding)
Robert
On Thu, Mar 7, 2019 at 2:56 PM Guozhang Wang wrote:
> Thanks Konstantine, I've read the updated section on
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka
Thanks Konstantine, I've read the updated section on
https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect
and it lgtm.
I'm +1 on the KIP.
Guozhang
On Thu, Mar 7, 2019 at 2:35 PM Konstantine Karantasis <
konstant...
en they can
> trigger another rebalance based on V0 where everything will revoke the
> tasks before sending join group requests.
>
>
> Guozhang
>
> On Wed, Mar 6, 2019 at 2:28 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > I'd like t
equests.
Guozhang
On Wed, Mar 6, 2019 at 2:28 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:
> I'd like to open the vote on KIP-415: Incremental Cooperative Rebalancing
> in Kafka Connect
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+I
+1 (non-binding)
> On Mar 6, 2019, at 3:40 PM, Ryanne Dolan wrote:
>
> +1 (non-binding)
>
> Thanks!
> Ryanne
>
> On Wed, Mar 6, 2019, 4:28 PM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
>> I'd like to open the vote on KIP-415:
+1 (non-binding)
Thanks!
Ryanne
On Wed, Mar 6, 2019, 4:28 PM Konstantine Karantasis <
konstant...@confluent.io> wrote:
> I'd like to open the vote on KIP-415: Incremental Cooperative Rebalancing
> in Kafka Connect
>
>
> https://cwiki.apache.org/confluence/display/KA
I'd like to open the vote on KIP-415: Incremental Cooperative Rebalancing
in Kafka Connect
https://cwiki.apache.org/confluence/display/KAFKA/KIP-415%3A+Incremental+Cooperative+Rebalancing+in+Kafka+Connect
a proposal that will allow Kafka Connect to scale significantly the number
of conne
After almost two months since this discussion thread on KIP-415 was opened
and more than 4 months since the design doc on Incremental Cooperative
Rebalancing was posted,
I believe that the open items have been discussed and that KIP-415, in its
current form, reflects the outcome of our discussions
current form of rebalancing), is allowed because the barrier imposed by
rebalancing is implicit, it concerns the Connect workers and is not exposed
in Connect's API. Connect tasks run independently and when a task stops and
resumes progress it is not required that all tasks of a connector do th
the trade-offs for using a new serialization format for the
> protocol encoding. We probably could discuss a wider options and benchmark
> on the performance before reaching a final decision.
>
> Best,
> Boyang
> ________
> From: Konstantine Karantasis
&
From: Konstantine Karantasis
Sent: Tuesday, February 5, 2019 4:23 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka
Connect
Hi all,
Thank you for your comments so far.
Now that KIP freeze and feature freeze are behind
e arguments for such a transition make sense, but the pros are
probably not enough to outweigh the introduction of a dependency at this
point and justify changes in every client that will potentially use
incremental cooperative rebalancing in the future. The changes in the
rebalancing protocol hav
;
> konstant...@confluent.io wrote:
>
> > Hi all,
> >
> > I just published KIP-415: Incremental Cooperative Rebalancing in Kafka
> > Connect
> > on the wiki here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposal
g
updates with every single improvement in the protocol. However, the overall
Incremental Cooperative Rebalancing proposition has already three policies,
and some other flavors might come up in the near future, depending how
conservative we are with V1.
* We rely on the robustness of the current
>
> I just published KIP-415: Incremental Cooperative Rebalancing in Kafka
> Connect
> on the wiki here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> This is the first KIP to suggest an implementation of incremental and
> cooperat
d, that if this KIP deprecates a config property, we should note
> > that.
> > > > explicitly.
> > > >
> > >
> > >
> > > As I mentioned above, KIP-345 and KIP-415 conceptually share some
> common
> > > goals. They avoid unnecessar
KIP deprecates a config property, we should note
> that.
> > > explicitly.
> > >
> >
> >
> > As I mentioned above, KIP-345 and KIP-415 conceptually share some common
> > goals. They avoid unnecessary re-assignments of resources to the members
> of
> &
ups and KIP-415
> will be implemented for Connect worker groups. KIP-345 is more precise in
> re-assignment by requiring a unique id for each group member. KIP-415
> tolerates temporary imbalances and is more broad in the sense that
> addresses group resizes as well. Static membership could be
5 is targeting Consumer groups and KIP-415
will be implemented for Connect worker groups. KIP-345 is more precise in
re-assignment by requiring a unique id for each group member. KIP-415
tolerates temporary imbalances and is more broad in the sense that
addresses group resizes as well. Static members
> > connect protocols instead of 3. Because the user knows when all the
> > workers
> > > all upgraded to version 1, we could just use `compatible` for the first
> > > rolling bounce
> > > and 'cooperative' for the second bounce. Could you
here? I guess we won't do the
> > > > actual rebalance until the original scheduled delay d is reached
> right?
> > > >
> > > > 2. we are basically relying on the leader subscription to persist the
> > > > group assignment across the generation a
goal here is to avoid unnecessary rebalance due to leader
> bounces
> > > by specifying a field called JoinReason for broker to interpret. With
> > that
> > > change in mind, I think it's worth mentioning this potential dependency
> > > within KIP-415 so that we do
uation explanation?
> >
> > 3. cooperative cmeans -> means that only Incremental Cooperative Connect
> > protocol is enabled (version 1 or higher).
> >
> > 4. For the compatibility change, I'm wondering whether we could just use
> 2
> > connect protocols instead of 3. Because the user knows
Incremental Cooperative Rebalancing in
> Kafka Connect
>
> Hi Stanislav and Boyang. Thanks for your comments.
>
> You are both asking how KIP-345 affects this KIP, so, first, I'll address
> this point.
>
> KIP-345 has recently introduced an option that will allow Ka
; all upgraded to version 1, we could just use `compatible` for the first
> rolling bounce
> and 'cooperative' for the second bounce. Could you explain a bit why we
> need to start from `eager` stage?
>
> cc Mayuresh on this thread.
>
> Thanks,
> Boyang
>
>
_
From: Konstantine Karantasis
Sent: Friday, January 18, 2019 8:32 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka
Connect
Hi Stanislav and Boyang. Thanks for your comments.
You are both asking how KIP-345 affects this KIP, so,
of a member in the group. That way KIP-345 offers a remedy for the
cases of rolling bounces and replacement of nodes due to failures that can
happen quickly.
Without ruling out that policies of Incremental Cooperative Rebalancing may
use static membership eventually in order to better address suc
backs.
Thanks a lot!
Boyang
From: Stanislav Kozlovski
Sent: Monday, January 14, 2019 3:15 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka
Connect
Hey Konstantine,
This is a very exciting and fundam
<
konstant...@confluent.io> wrote:
> Hi all,
>
> I just published KIP-415: Incremental Cooperative Rebalancing in Kafka
> Connect
> on the wiki here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>
> This is the first KIP to sugges
Hi all,
I just published KIP-415: Incremental Cooperative Rebalancing in Kafka
Connect
on the wiki here:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
This is the first KIP to suggest an implementation of incremental and
cooperative rebalancing in the context of
eams
> should
> > leverage on the new protocol along with standby tasks to achieve the
> better
> > operational goals during rebalances.
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Reduce+multiple+con
r+rebalances+by+specifying+member+id
> [2]
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal#KafkaClient-sideAssignmentProposal-CoordinatorStateMachine
> [3]
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+D
ka0.9ConsumerRewriteDesign-Interestingscenariostoconsider
[4]
https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing+for+Streams
On Thu, Oct 4, 2018 at 12:16 PM, McCaig, Rhys
wrote:
> This is fantastic. Im really excited to see the work on this.
>
> > On O
a's wiki space:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
>
> It deals with the subject of Rebalancing of groups in Kafka and proposes
> basic infrastructure to support improvements on the current rebalancing
>
gt; published in Apache Kafka's wiki space:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
>
> It deals with the subject of Rebalancing of groups in Kafka and proposes
> basic infrastructure to support improve
org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
>
> It deals with the subject of Rebalancing of groups in Kafka and proposes
> basic infrastructure to support improvements on the current rebalancing
> protocol as well as a set of policies that can
Hey everyone,
I'd like to bring to your attention a general design document that was just
published in Apache Kafka's wiki space:
https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
It deals with the subject of Rebalancing of
50 matches
Mail list logo
