Hi Guozhang,
Yes the clock will be reset not extended. Sorry incorrect wording in the
KIP. I'll update it.
Thanks,
Damian
On Wed, 29 Mar 2017 at 23:18 Guozhang Wang wrote:
> Made another pass over the KIP wiki, overall LGTM. One quick question on
> the described logic: "they will be added to t
Made another pass over the KIP wiki, overall LGTM. One quick question on
the described logic: "they will be added to the group and the delay will be
extended by min(remainingRebalanceTimeout, group.initial.rebalance.delay.ms)"
though:
>From your previous email I thought you are "resetting the cloc
Thanks Damian!
On Wed, Mar 29, 2017 at 1:27 AM, Damian Guy wrote:
> Thanks everyone for the discussion, very helpful. I've updated the KIP to
> make the delay such that it is extended as new members join the group and
> that it never exceeds the groups rebalance timeout.
>
> If everyone is ok wi
Thanks everyone for the discussion, very helpful. I've updated the KIP to
make the delay such that it is extended as new members join the group and
that it never exceeds the groups rebalance timeout.
If everyone is ok with this I'll kick off the voting thread.
Thanks again,
Damian
On Tue, 28 Mar
I think separating leave/join makes sense. The scenario I can think of for
delaying a rebalance on LeaveGroupRequest is rolling bounce of a service.
But that scenario could be tricky because there may be mixture of joining
and leaving. What happens if a consumer left the group right after another
c
Hey Damian,
Thanks for the KIP. I think the proposal makes sense as a workaround maybe
for some advanced users. However, I'm not sure we can depend on average
users knowing that the config exists, much less setting it to something
that makes sense. It's kind of a trend in streams that I'm not too
Just clarifying on "session.timeout.ms": today we already have a
rebalance.timeout value in the JoinGroupRequest protocol, which is used to
determine how long the coordinator will wait for each consumer to re-join
the group during prepare-rebalance phase; and I was thinking we can use
that value fo
I think it makes sense to reset the timer when group changes. This allows
broker to possibly wait longer if more consumers are restarted in parallel.
For example, user may start a large number of consumers in sequence using
script. The time for all consumers to send JoinGroupRequest will likely be
@Ismael - yeah sure we can reduce the default, though i'm not sure what the
"right" default would be.
On Tue, 28 Mar 2017 at 15:40 Ismael Juma wrote:
> Is 3 seconds the right default if the timer gets reset after each consumer
> joins? Maybe we can lower the default value given the new approach.
Is 3 seconds the right default if the timer gets reset after each consumer
joins? Maybe we can lower the default value given the new approach.
Ismael
On Tue, Mar 28, 2017 at 9:53 AM, Damian Guy wrote:
> All,
> I'd like to get this back to the original discussion about Delaying initial
> consume
I don't understand, why you want to split join/leave into two parts...
But it up to you I guess.
+1 for broker config plus "retriggering" delay
-Matthias
On 3/28/17 1:53 AM, Damian Guy wrote:
> All,
> I'd like to get this back to the original discussion about Delaying initial
> consumer group r
All,
I'd like to get this back to the original discussion about Delaying initial
consumer group rebalance.
I think i'm leaning towards sticking with the broker config and changing
the delay so that the timer starts again when a new consumer joins the
group. What are peoples thoughts on that?
Doing
Matthias,
Yes i know.
Thanks,
Damian
On Mon, 27 Mar 2017 at 18:17 Matthias J. Sax wrote:
> Damian,
>
> about "rebalance immediately" on timeout -- I guess, that's a different
> case as no LeaveGroupRequest will be sent. Thus, the broker should be
> able to distinguish both cases easily, and ap
Damian,
about "rebalance immediately" on timeout -- I guess, that's a different
case as no LeaveGroupRequest will be sent. Thus, the broker should be
able to distinguish both cases easily, and apply the delay only if it
received the LeaveGroupRequest but not if a consumer times out.
Does this mak
@Becket
Thanks for the feedback. Yes, i like the idea of extending the delay as
each new consumer joins the group. Though, i think this could be done with
either a consumer or broker side config. But i get your point that some
consumers in the group can be misconfigured.
@Matthias & @Eno - yes we
Hi Matthias,
Yes, that was what I was thinking. We will keep delay it until either
reaching the rebalance timeout or no new consumer joins in that small delay
which is configured on the broker side.
Thanks,
Jiangjie (Becket) Qin
On Fri, Mar 24, 2017 at 1:39 PM, Matthias J. Sax
wrote:
> @Becke
@Becket:
I am not sure, if I understand this correctly. Instead of applying a
fixed delay, that starts when the first consumer of an (empty) group
joins, you suggest to re-trigger/re-set the delay each time a new
consumer joins?
This sound like a good strategy to me, if the config is on the broke
Thanks Damian,
This KIP deals with the initial phase only. What about the cases when several
consumers leave a group? Won't there be several expensive rebalances then as
well? I'm wondering if it makes sense for the delay to hold anytime the "set"
of consumers in a group changes, be it addition
Thanks for the KIP, Damian.
My two cents on this. It seems there are two things worth thinking here:
1. Better rebalance timing. We will try to rebalance only when all the
consumers in a group have joined. The challenge would be someone has to
define what does ALL consumers mean, it could either
Thanks for the feedback.
Ewen: I'm happy to make it a client side config. Other than the protocol
bump i think the effort is almost the same. Personally i see no other
issues, but based on discussions with others this is what we came up with.
True, it can probably be tested easily via an integrat
I have the same initial response as Ismael re: broker vs consumer settings.
The global setting seems questionable.
Could we maybe summarize what the impact of making this a client config
would be? Protocol bump is obvious, but is there any other significant
issue? For the protocol bump in particul
Thanks for the KIP Damian!
My two cents:
- we should have an explicit parameter for this -- implicit setting are
always tricky (the "importance" of this parameter would be LOW)
- the config should be different for each consumer group:
* assume you have a stateless app, you want to rebalance
Thanks Gouzhang - i think another problem with this is that is overloading
session.timeout.ms to mean multiple things. I'm not sure that is a good
thing.
On Thu, 23 Mar 2017 at 17:14 Guozhang Wang wrote:
> The downside of it, though, is that although it "hides" this from most of
> the users need
The downside of it, though, is that although it "hides" this from most of
the users needing to be aware of it, by default session timeout i.e. the
rebalance timeout is 10 seconds which could arguably too long.
Guozhang
On Thu, Mar 23, 2017 at 10:12 AM, Guozhang Wang wrote:
> Just throwing anot
Just throwing another alternative idea here: we can consider using the
rebalance timeout value which is already included in the join request
protocol (and on the current Java client it is always written as the
session timeout value), that the first member joining will always force the
coordinator t
Hi Ismael,
Mostly to avoid the protocol bump.
I agree that it may be difficult to choose the right delay for all consumer
groups, but we wanted to make this something that most users don't really
need to think about, i.e., a small enough default delay that works in the
majority of cases. However
Thanks for the KIP, Damian. It makes sense to avoid multiple rebalances
during start-up. One issue with having this as a broker config is that it
may be difficult to choose the right delay for all consumer groups. Can you
elaborate a little more on why the first alternative (add a consumer
config)
Hi All,
I've prepared a KIP to add a configurable delay to the initial consumer
group rebalance.
Please have look here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance
Thanks,
Damian
BTW, i apologize if this appears twice. Seems the first one
28 matches
Mail list logo
