Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

rkerCoordinator and Connect's protocol > definitions. At the moment I don't see significant opportunity for code > sharing and abstraction with respect to incremental cooperative > rebalancing. Also, flatbuffers consideration has been postponed and might > not return soon. I

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

the great summary! +1 for having a separate KIP > > discussing the trade-offs for using a new serialization format for the > > protocol encoding. We probably could discuss a wider options and > benchmark > > on the performance before reaching a final decision. > > > >

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

the trade-offs for using a new serialization format for the > protocol encoding. We probably could discuss a wider options and benchmark > on the performance before reaching a final decision. > > Best, > Boyang > ____________ > From: Konstantine Karantasis &

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

From: Konstantine Karantasis Sent: Tuesday, February 5, 2019 4:23 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect Hi all, Thank you for your comments so far. Now that KIP freeze and feature freeze are behind

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hi all, Thank you for your comments so far. Now that KIP freeze and feature freeze are behind us for version 2.2, I'd like to bring this thread back at the top of the email stack, with the following suggestion: I'll be changing KIP-415's description to include a serialization format that extends

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hi Ismael, thanks for bringing up serialization in the discussion! Indeed, JSON was considered given it's the prevalent text-based serialization option. In comparison to flatbuffers, most generic pros and cons are valid in this context too. Higher perfomance during serde, small size, optional fie

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

> > happy to adjust or keep the plan open if we think this is > > > preferrable. > > > > > Just > > > > > > to note here, that my initial suggestion is for deprecation to be > > > > > triggered > > > > > > in the next maj

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Thanks for the KIP Konstantine. Quick question: introducing a new serialization format (ie flatbuffers) has major implications. Have we considered json? If so, why did we reject it? Ismael On Fri, Jan 11, 2019, 3:44 PM Konstantine Karantasis < konstant...@confluent.io wrote: > Hi all, > > I just

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

> > > > > > > > > > > On Fri, Jan 18, 2019 at 2:00 PM Boyang Chen > > > > > wrote: > > > > > > > > > > > > > Thanks a lot for the detailed explanation here Konstantine! I > > > > strong

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

the client. Also > > > > > > I fully understood your explanation on task shuffle minimum > impact > > in > > > > the > > > > > > workers scenario, because > > > > > > the local storage usage is very limited. > > > > &

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

computes and sends assignments: > > > > > W2(delay: d'', assigned: [], revoked: []) > > > > > W3(delay: d'', assigned: [BT1, AT1], revoked: []) > > > > > after we start d' round of delayed rebalance. Why does W3

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

e generation and leader rejoin to trigger > > > > necessary rebalance. This assumption could potentially be broken with > > > > future upgrades of > > > > broker as we are discussing > > > > https://issues.apache.org/jira/browse/KAFKA-7728. This JIRA w

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

he > > examples > > > easy to read. I'll change the KIP to show who's leader at the beginning > > as > > > well. > > > Boyang, I'll add a paragraph to highlight why local state is not the > most > > > pressing issue in Connect. > >

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

here? I guess we won't do the > > > > actual rebalance until the original scheduled delay d is reached > right? > > > > > > > > 2. we are basically relying on the leader subscription to persist the > > > > group assignment across the generation a

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

goal here is to avoid unnecessary rebalance due to leader > bounces > > > by specifying a field called JoinReason for broker to interpret. With > > that > > > change in mind, I think it's worth mentioning this potential dependency > > > within KIP-415 so that we do

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

uation explanation? > > > > 3. cooperative cmeans -> means that only Incremental Cooperative Connect > > protocol is enabled (version 1 or higher). > > > > 4. For the compatibility change, I'm wondering whether we could just use > 2 > > connect protocols instead of 3. Because the user knows

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

d be great if we could look at these two initiatives > > holistically to help reduce the common workload. > > 2. Since I never used Connect before, I do hope you could enlighten me > > on the potential effort involved in task transfer between workers. The > > reaso

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

; all upgraded to version 1, we could just use `compatible` for the first > rolling bounce > and 'cooperative' for the second bounce. Could you explain a bit why we > need to start from `eager` stage? > > cc Mayuresh on this thread. > > Thanks, > Boyang > >

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

_ From: Konstantine Karantasis Sent: Friday, January 18, 2019 8:32 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect Hi Stanislav and Boyang. Thanks for your comments. You are both asking how KIP-345 affects this KIP, so,

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

; would be good to also provide this background in the KIP motivation so that > people could understand better of the symptom and build constructive > feedbacks. > > Thanks a lot! > > Boyang > ________________ > From: Stanislav Kozlovski > Sent: Monday

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

backs. Thanks a lot! Boyang From: Stanislav Kozlovski Sent: Monday, January 14, 2019 3:15 PM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect Hey Konstantine, This is a very exciting and fundam

Re: [DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hey Konstantine, This is a very exciting and fundamental-improving KIP, thanks a lot for working on it! Have you seen KIP-345 ? I was wondering whether Connect would support the static group membership - potentially limiting the need to h

[DISCUSS] KIP-415: Incremental Cooperative Rebalancing in Kafka Connect

Hi all, I just published KIP-415: Incremental Cooperative Rebalancing in Kafka Connect on the wiki here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals This is the first KIP to suggest an implementation of incremental and cooperative rebalancing in the context of Ka