, June 14, 2019 8:42 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Thanks for the update. To what extent will KIP-345 be available in 2.3.0?
Mike
On 6/13/19, 5:36 PM, "Boyang Chen" wrote:
Hey al
, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey all,
there is a minor change to the stream side logic for static
membership<https://cwiki.apache.org/confluence/display/
any concerns.
Best,
Boyang
From: Boyang Chen
Sent: Friday, April 26, 2019 11:16 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey all,
there is a minor change to the stream side logic for
: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey all,
there is a minor change to the stream side logic for static
membership<https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalan
nsumer rebalances by
specifying member id
Hi Mike,
Yes that's the plan!
From: Mike Freyberger
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyan
Hi Mike,
Yes that's the plan!
From: Mike Freyberger
Sent: Saturday, March 9, 2019 10:04 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Is this work targeted for Kafka 2
Hi Boyang,
Is this work targeted for Kafka 2.3? I am eager to use this new feature.
Thanks,
Mike Freyberger
On 12/21/18, 1:21 PM, "Mayuresh Gharat" wrote:
Hi Boyang,
Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points t
Yep, that is correct!
Get Outlook for iOS<https://aka.ms/o0ukef>
From: Mayuresh Gharat
Sent: Wednesday, January 2, 2019 8:30 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi
at
> Sent: Saturday, December 22, 2018 2:21 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
>
> Regarding "However, we shall still attempt to remove the member static info
> if the g
ent: Saturday, December 22, 2018 2:21 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an
Hi Boyang,
Regarding "However, we shall still attempt to remove the member static info
if the given `member.id` points to an existing `group.instance.id` upon
LeaveGroupRequest, because I could think of the possibility that in long
term we could want to add static membership leave group logic for
7;s instance ids.
>
> I agree with the suggestion to make the leave group request change
> generic. So this new Stream API
> will be added on the rest layer to expose the necessary ids correct?
>
> Looking forward to your confirmation 😊
>
> Best,
> Boyang
>
> ---
leave group request change
> generic. So this new Stream API
> will be added on the rest layer to expose the necessary ids correct?
>
> Looking forward to your confirmation 😊
>
> Best,
> Boyang
>
>
> From: Guozhang Wang
> Sent: Saturday, D
eid prefix], or we could add two
> > fields: instanceid prefix, and list[instanceid]
> > for clarity purpose. As you know, two options are equivalent since full
> > name is subset of prefix.
> >
> > Let me know your thoughts!
> >
> > Boyang
> >
_
> From: Boyang Chen
> Sent: Thursday, November 29, 2018 3:39 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Thanks Guozhang for the new proposal here!
>
> So I'd li
ubject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Thanks Guozhang for the new proposal here!
So I'd like to propose a slightly modified version of LeaveGroupRequest:
instead of letting the static member consumer client themselves to send the
request (which
without client management tooling.
How does this workaround sound?
Boyang
From: Guozhang Wang
Sent: Thursday, November 29, 2018 2:38 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
I was thinking that with the optional stat
cking off rebalance since we are shutting them
> down already.
>
> 3) personally I favor "ids" over "names" :) Since we already have some
> "ids" and hence it sounds more consistent, plus on the producer side we
> have a `transactional.id` whose semantics is a bit similar to thi
o this one,
i.e. for unique distinguishment of a client which may comes and goes but
need to be persist over multiple "instance life-times".
Sure we have enough votes for ids 😊I will finalize the name to
`group.instance.id`, does that
sound good?
Best,
Boyang
From: Guozhang Wang
Sent
> I understand it, it is used for scaling down a consumer group and somehow
> > bypasses normal session timeout expiration. I am wondering how critical
> > this piece is and whether we can leave it for future work. If not, then
> it
> > would be helpful to elaborate on its implementation.
om/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-394%253A%2BRequire%2Bmember%2Bid%2Bfor%2Binitial%2Bjoin%2Bgroup%2Brequest&data=02%7C01%7C%7Cfa7f68b0cbc94390f04b08d653fa2832%7C84df9e7fe9f640afb435aaaa%7C1%7C0%7C636788731208857126&sdata=EDM7PmpOo2HenYh
n of uniqueness in the group. How about `group.instance.id`
to go along with `group.id`?
Yea, Dong and Stanislav also mentioned this naming. I personally buy in the
namespace idea, and
since we already use `member.name` in a lot of context, I decide to rename the
config to `group.member.name`
%7C%7Cdde139857e7a4a3a83dd08d651d9c93e%7C84df9e7fe9f640afb435aaaa%7C1%7C0%7C636786393153281080&sdata=%2BNasMFlJf9rEJc9iDfndcyxA4%2BGWieS1azSKbtdGRW4%3D&reserved=0
> >> .
> >> Added the script.
> >>
> >> 7) Would it be simpler to replace name &quo
ing application that consumes from Kafka
>> cluster
>> > > A and produces to Kafka cluster B.
>> > > Depending on the data and the Kafka cluster configuration, Kafka
>> service
>> > > providers can set a mirroring group saying that it will take, for
&g
ce the consumer rebalance method through admin
> client API.
>
> 8) It is not very clear how the newly added AdminClient API trigger
> rebalance. For example, does it send request? Can this be explained in the
> KIP?
>
> Sure, I will add more details to the API.
>
>
> Thank
will add more details to the API.
Thanks again for the helpful suggestions!
Best,
Boyang
________
From: Dong Lin
Sent: Saturday, November 24, 2018 2:54 PM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Boyang,
Thanks
ce
> APIs to set hard-coded
> client ids to the group and replace the dead host, or as you proposed to
> define spare host as
> what I understood as hot backup. I will put both Jason and your
> suggestions into a separate section
> called "Future works". Note
o solvable
through rebalance protocol
IMO.
Thank you again for the great feedback!
Boyang
____________
From: Boyang Chen
Sent: Thursday, November 22, 2018 3:39 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying memb
you for your
valuable feedback
to help me design the KIP better! (And I will try to address your feedbacks in
next round Mayuresh 😊)
Best,
Boyang
____________
From: Mayuresh Gharat
Sent: Wednesday, November 21, 2018 7:50 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-
that we don't need fencing in offset commit request,
> since
> > duplicate member.name issue could be handled by join group request. We
> > shall reject join group with known member name but no member id (which
> > means we already have an active member using this identity).
y have an active member using this identity).
> 6. I agree to remove that internal config once we move forward with
> static membership. And I already removed the entire section from the KIP.
>
> Let me know if you have other concerns.
>
> Best,
> Boyang
> ___
membership. And I already removed the entire section from the KIP.
>
> Let me know if you have other concerns.
>
> Best,
> Boyang
>
> From: Guozhang Wang
> Sent: Tuesday, November 20, 2018 4:21 PM
> To: dev
> Subject: Re: [DISCUSS]
1 PM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hello Boyang,
Thanks a lot for the KIP! It is a great write-up and I appreciate your
patience answering to the feedbacks from the community. I'd like to add my
2cents here:
1. By introd
Hello Boyang,
Thanks a lot for the KIP! It is a great write-up and I appreciate your
patience answering to the feedbacks from the community. I'd like to add my
2cents here:
1. By introducing another two timeout configs, registration_timeout and
expansion_timeout, we are effectively having four ti
Hey Boyang,
Thanks for the proposal! This is very useful. I have some comments below:
1) The motivation currently explicitly states that the goal is to improve
performance for heavy state application. It seems that the motivation can
be stronger with the following use-case. Currently for MirrorMa
but I'm not familiar with it. If we do have mechanism to handle
> > issues like I mentioned above for replication (auto healing during
> > mid-night if one broker is never back), we could continue discussing the
> > new approaches to have basic guarantee of consumer group li
tion (auto healing during
> > mid-night if one broker is never back), we could continue discussing the
> > new approaches to have basic guarantee of consumer group liveness.
> >
> >
> > The discussion so far is to make sure that all the design approaches we
> > ha
ld definitely propose better solution on top of it. I hope these
> discussions make sense. Thanks again for helping make the design solid!
>
>
> Boyang
>
>
> From: Jason Gustafson
> Sent: Thursday, November 15, 2018 9:54 AM
> To: dev
>
hese discussions make
sense. Thanks again for helping make the design solid!
Boyang
From: Jason Gustafson
Sent: Thursday, November 15, 2018 9:54 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
>
> I fee
; Best,
>
> Boyang
>
> ____________
> From: Jason Gustafson
> Sent: Wednesday, November 14, 2018 9:31 AM
> To: dev
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hey Boyang,
>
> Thanks for t
improve our documentation correspondingly in the
> follow-up KIPs.
>
>
> Boyang
>
>
> From: Mayuresh Gharat
> Sent: Sunday, November 11, 2018 1:06 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specif
-up KIPs.
>
>
> Boyang
>
>
> From: Mayuresh Gharat
> Sent: Sunday, November 11, 2018 1:06 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
> specifying member id
>
> Hi Boyang,
reuse
> consumer offsets topic. I will do more analysis and make a trade-off
> comparison. Nice catch!
>
>
> I hope the explanations make sense to you. I will keep polishing on the
> edge cases and details.
>
>
> Best,
>
> Boyang
>
> __
s topic. I will do more analysis and make a trade-off
> comparison. Nice catch!
>
>
> I hope the explanations make sense to you. I will keep polishing on the
> edge cases and details.
>
>
> Best,
>
> Boyang
>
> ____
> From: Mayuresh Gharat
st,
Boyang
From: Mayuresh Gharat
Sent: Saturday, November 10, 2018 10:25 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi Boyang,
Thanks for the KIP and sorry for being late to the party
ming up with
> this great client proposal. This is great complementation to KIP 345. In a
> high level, we are not having any collision on the path and both proposals
> are making sense here. Just need better sync to avoid duplicate effort :)
>
>
> Best,
>
> Boyang
>
&
built on top of the success of KIP-345.
________
From: Matthias J. Sax
Sent: Wednesday, November 7, 2018 7:02 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey,
there was quite a pause on
7:02 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey,
there was quite a pause on this KIP discussion and in the mean time, a
new design for incremental cooporative rebalance was suggested:
https://cwiki.apache.org/con
ntroduce admin API in this KIP or a separate one.
>
>
> Thanks again for the proposed ideas!
>
>
> Boyang
>
> ____________
> From: Mike Freyberger
> Sent: Monday, November 5, 2018 6:13 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS
that, but agree we could discuss whether we want to
introduce admin API in this KIP or a separate one.
Thanks again for the proposed ideas!
Boyang
From: Mike Freyberger
Sent: Monday, November 5, 2018 6:13 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS]
t;> When rolling restart/shutting down gracefully, the client will
> >> send a
> >>>>>> leave group request (static membership mode). In static membership,
> >>> we
> >>>>> will
> >>>>>
hanges to the KIP with our newly discussed
items and details. Really excited to see the design has become more solid.
Best,
Boyang
From: Jason Gustafson
Sent: Saturday, August 25, 2018 6:04 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebala
cited to see the design has become more solid.
Best,
Boyang
From: Jason Gustafson
Sent: Saturday, August 25, 2018 6:04 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Mike,
Yeah, that's a good
t;>>> be identified as a `certificated member`. If not, this means there
> >>> are
> >>>>>> duplicate consumer members with same member name and the request
> >> will
> >>>> be
> >>>>>> rejected to guarantee consum
ed by
>>>>> leader.
>>>>>> So we will wait for all the members to rejoin the group and do
>>> exactly
>>>>> one
>>>>>> rebalance since all members are expected to rejoin within timeout.
>> If
>>>>
> > And consider the switch between dynamic to static membership.
> > > > >
> > > > > 1. Dynamic to static: the first joiner shall revise the
> membership
> > > to
> > > > > static and wait for all the current members to restart,
a downgrade which should
> be
> > > > smooth: just erase the cached mapping, and wait for session timeout
> to
> > > > trigger rebalance should be sufficient. (Fallback to current
> behavior)
> > > > 3. Halfway switch: a corner case is like some clients ke
rever without progress because dynamic/static states are
> > > bouncing each other. This could guarantee that we will not make the
> > > consumer group work in a wrong state by having half static and half
> > dynamic.
> > >
> > > To guarantee corr
ssed back in the KIP.
> >
> >
> > Are there any concern for this high level proposal? Just want to
> reiterate
> > on the core idea of the KIP: "If the broker recognize this consumer as an
> > existing member, it shouldn't trigger rebalance".
> >
> >
el proposal? Just want to reiterate
> on the core idea of the KIP: "If the broker recognize this consumer as an
> existing member, it shouldn't trigger rebalance".
>
> Thanks a lot for everyone's input! I feel this proposal is much more
> robust than previous one!
CUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hi,
thanks for the detailed discussion. I learned a lot about internals again :)
I like the idea or a user config `member.name` and to keep `member.id`
internal. Also agree with Guozhang, that reusing `client.id` might not
b
Hi,
thanks for the detailed discussion. I learned a lot about internals again :)
I like the idea or a user config `member.name` and to keep `member.id`
internal. Also agree with Guozhang, that reusing `client.id` might not
be a good idea.
To clarify the algorithm, each time we generate a new `me
@James
What you described is true: the transition from dynamic to static
memberships are not thought through yet. But I do not think it is an
impossible problem: note that we indeed moved the offset commit from ZK to
kafka coordinator in 0.8.2 :) The migration plan is to first to
double-commits on
Guozhang, in a previous message, you proposed said this:
> On Jul 30, 2018, at 3:56 PM, Guozhang Wang wrote:
>
> 1. We bump up the JoinGroupRequest with additional fields:
>
> 1.a) a flag indicating "static" or "dynamic" membership protocols.
> 1.b) with "static" membership, we also add the p
oyang
From: Guozhang Wang
Sent: Monday, August 6, 2018 8:46 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Re: reuse of "client.id": good question. Although client.id could be unique
to guarantee fencing, by using it we are bin
;
> >
> > Also a minor comment is that our current codebase has two different
> > classes: the GroupCoordinator and ConsumerCoordinator. When talking about
> > term "coordinator", it would be great to distinguish between the two, or
> > explicitly nominate b
ation!
From: Guozhang Wang
Sent: Saturday, August 4, 2018 5:25 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
@Boyang,
* Yes registry id should be generated via a config, similar to what you
proposed in the KIP currently for m
coordinator", it would be great to distinguish between the two, or
> explicitly nominate broker coordinator as default "coordinator".
>
>
> Best,
>
> Boyang
>
> ________________
> From: Guozhang Wang
> Sent: Thursday, August 2, 2018 1:59 A
;coordinator", it would be great to distinguish between the two, or
> explicitly nominate broker coordinator as default "coordinator".
>
>
> Best,
>
> Boyang
>
> ________
> From: Guozhang Wang
> Sent: Thursday, August 2, 2018 1:59 AM
t: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Guozhang,
thanks for this detailed proposal! Quickly I want to clarify, where this
registry id is generated? My understanding is that the registry id is a unique
id provided by user correct? This way even if mul
___
From: Guozhang Wang
Sent: Thursday, August 2, 2018 1:59 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hello Boyang,
Thanks for the great summary.
Personally I think 1) is still doable as we are not necessarily have to
rely o
ined to
> 2 as 1 is intrinsically complex (no source of truth model). Hope we are on
> the same page now.
>
>
> Best,
>
> Boyang
>
>
> From: John Roesler
> Sent: Wednesday, August 1, 2018 5:28 AM
> To: dev@kafka.apache.org
> Subj
ined to 2 as
1 is intrinsically complex (no source of truth model). Hope we are on the same
page now.
Best,
Boyang
From: John Roesler
Sent: Wednesday, August 1, 2018 5:28 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer
t; >
> > > > > From: Jason Gustafson
> > > > > Sent: Saturday, July 28, 2018 6:50 AM
> > > > > To: dev
> > > > > Subject: R
tand that the identities of multiple consumers belong
> to
> > one
> > > > instance, where each Stream thread will be using one
> dedicated
> > main
> > > > consumer. So in a Stream use case, we could internally
> generate
> > &g
config, user may use member id config even before
> they fully
> > > understand the problem, and use the same set of initialization
> logic
> > cross
> > > multiple consumers on the same instance.
> > >
> > >
> >
to understand and fix. Although I still assume this
> is an
> > > advanced config, user may use member id config even before
> they fully
> > > understand the problem, and use the same set of initialization
> logic
> > cross
> > > multiple
e non-functional? Will just those instances be
> > > non-functional? Or will the group be functional, but the
> rebalancing be
> > > non-optimal and require more round-trips/data-transfer? (similar
> to the
> > > curre
y at first, but considering our
> original
> > motivation to leader-follower rebalance model, I feel that having
> broker to
> > create membership info and let client maintain would be less
> appealing and
> > fragile. Having client generate membership data could build up
e some sort of validation (host
> +
> > port) on the server side. To simplify the first version, we do not plan
> to
> > enforce validation. A good comparison would be the EOS producer which is
> in
> > charge of generating unique transaction id seque
ll break unless we enforce some sort of validation (host
> +
> > port) on the server side. To simplify the first version, we do not plan
> to
> > enforce validation. A good comparison would be the EOS producer which is
> in
> > charge of generating unique transactio
From: Jason Gustafson
Sent: Saturday, July 28, 2018 6:50 AM
To: dev
Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by
specifying member id
Hey Boyang,
Thanks for the KIP. I think my main question is in the same vein as James'.
The problem is that the
tion id sequence. IMO for broker logic,
> the tolerance of client side error is not unlimited.
> >
> >
> > Thank you!
> >
> >
> > ____________
> > From: James Cheng
> > Sent: Saturday, July 28, 2018 1:26 AM
> > To: dev@kafka.apache.org
> >
the
> tolerance of client side error is not unlimited.
>
>
> Thank you!
>
>
>
> From: James Cheng
> Sent: Saturday, July 28, 2018 1:26 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-345: Reduce multiple consum
-345: Reduce multiple consumer rebalances by
specifying member id
> On Jul 26, 2018, at 11:09 PM, Guozhang Wang wrote:
>
> Hi Boyang,
>
> Thanks for the proposed KIP. I made a pass over the wiki and here are some
> comments / questions:
>
> 1. In order to preserve broker c
> On Jul 26, 2018, at 11:09 PM, Guozhang Wang wrote:
>
> Hi Boyang,
>
> Thanks for the proposed KIP. I made a pass over the wiki and here are some
> comments / questions:
>
> 1. In order to preserve broker compatibility, we need to make sure the
> broker version discovery logic can be integra
Hi Boyang,
Thanks for the proposed KIP. I made a pass over the wiki and here are some
comments / questions:
1. In order to preserve broker compatibility, we need to make sure the
broker version discovery logic can be integrated with this new logic. I.e.
if a newer versioned consumer is talking to
Hey friends,
I would like to open a discussion thread on KIP-345:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Reduce+multiple+consumer+rebalances+by+specifying+member+id
This KIP is trying to resolve multiple rebalances by maintaining the consumer
member id across rebalance g
88 matches
Mail list logo
