. Sax"
To: dev@kafka.apache.org
Date: 04/04/2018 06:28 PM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
I was just reading the whole KIP for the first time. Nice work!
One minor comment. In the table of the standalone consumer, the fi
I was just reading the whole KIP for the first time. Nice work!
One minor comment. In the table of the standalone consumer, the first
line, first column says:
> = Empty
> (protocolType = Some("consumer"))
I think this should be
> = Empty
> (protocolType != None)
Note, that for example KafkaStr
Hi all,
Thanks for the feedback on this KIP so far.
If there is no additional feedback, I'll start a vote on Wed.
Thanks.
--Vahid
coordinator
change case.
Thanks!
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 03/17/2018 11:50 AM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Hey Vahid,
Sorry for the delay. I've read through the current KIP
:)
>
> Thanks.
> --Vahid
>
>
>
> From: Jason Gustafson
> To: dev@kafka.apache.org
> Date: 03/05/2018 03:42 PM
> Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
> Consumer Group Offsets
>
>
>
> Hey Vahid,
>
>
time in case I've
missed something :)
Thanks.
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 03/05/2018 03:42 PM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Hey Vahid,
On point #1 below: since the expiration
s subscribed to). This can be done as part of the offset cleanup
> scheduled task that by default does not run very frequently. Were you
> thinking of a different method for capturing the subscription change?
>
> Thanks.
> --Vahid
>
>
>
>
> From: Jason Gustafson
&g
apturing the subscription change?
Thanks.
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 02/18/2018 01:16 PM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Hey Vahid,
Sorry for the late response. The KIP looks good. A few
n the update is
ready for review.
Apologies for the confusion.
--Vahid
From: "Vahid S Hashemian"
To: dev@kafka.apache.org
Date: 03/01/2018 11:43 AM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Hi Jason,
Thanks for your f
supported with '--delete'.
Regarding the other JIRA you referred to, sure, I'll add that in the KIP.
Thanks.
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 02/28/2018 12:10 PM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Gro
'll mention it in the KIP
> so we have it documented. Do you think my suggestion of manually removing
> topic offsets from group (as an interim solution) is worth additional
> discussion / implementation?
>
> I'll wait for your feedback and clarification on the above it
before
updating the KIP.
Thanks.
--Vahid
From: Jason Gustafson
To: dev@kafka.apache.org
Date: 02/18/2018 01:16 PM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Hey Vahid,
Sorry for the late response. The KIP looks good. A few comments:
; --Vahid
>
>
>
> From: James Cheng
> To: dev@kafka.apache.org
> Date: 02/01/2018 12:37 AM
> Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
> Consumer Group Offsets
>
>
>
> Vahid,
>
> Under rejected alternatives, we had decided
02/01/2018 12:37 AM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Vahid,
Under rejected alternatives, we had decided that we did NOT want to do
per-partition expiration, and instead we wait until the entire group is
empty and then (after the righ
Vahid,
Under rejected alternatives, we had decided that we did NOT want to do
per-partition expiration, and instead we wait until the entire group is empty
and then (after the right time has passed) expire the entire group at once.
I thought of one scenario that might benefit from per-partition
Bumping this as I'd like to see it land...
It's one of the "features" that tends to catch Kafka n00bs unawares and
typically results in message skippage/loss, vs the proposed solution is
much more intuitive behavior.
Plus it's more wire efficient because consumers no longer need to commit
offsets
There has been no further discussion on this KIP for about two months.
So I thought I'd provide the scoop hoping it would spark additional
feedback and move the KIP forward.
The KIP proposes a method to preserve group offsets as long as the group
is not in Empty state (even when offsets are comm
: long
--Vahid
From: James Cheng
To: dev@kafka.apache.org
Date: 11/16/2017 12:01 AM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
How fast does the in-memory cache grow?
As a random datapoint...
10 months ago we set our
t to hear feedback from others (and committers) on this.
>
> --Vahid
>
>
>
>
> From: Jeff Widman
> To: dev@kafka.apache.org
> Date: 11/15/2017 01:04 PM
> Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
> Consumer Group Offsets
>
@kafka.apache.org
Date: 11/15/2017 01:04 PM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
I thought about this scenario as well.
However, my conclusion was that because __consumer_offsets is a compacted
topic, this extra clutter from short-lived consumer
I thought about this scenario as well.
However, my conclusion was that because __consumer_offsets is a compacted
topic, this extra clutter from short-lived consumer groups is negligible.
The disk size is the product of the number of consumer groups and the
number of partitions in the group's subs
I'm forwarding this feedback from John to the mailing list, and responding
at the same time:
John, thanks for the feedback. I agree that the scenario you described
could lead to unnecessary long offset retention for other consumer groups.
If we want to address that in this KIP we could either ke
Thanks Jeff.
I'll wait until EOD tomorrow (Wednesday), and then I'll start a vote.
--Vahid
From: Jeff Widman
To: dev@kafka.apache.org
Date: 11/14/2017 11:35 AM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Any other inp
Any other input on this?
Otherwise Vahid what do you think about moving this to a vote?
On Tue, Nov 7, 2017 at 2:34 PM, Jeff Widman wrote:
> Any other feedback from folks on KIP-211?
>
> A prime benefit of this KIP is that it removes the need for the consumer
> to commit offsets for partitions
Any other feedback from folks on KIP-211?
A prime benefit of this KIP is that it removes the need for the consumer to
commit offsets for partitions where the offset hasn't changed. Right now,
if the consumer doesn't commit those offsets, they will be deleted, so the
consumer keeps blindly (re)comm
I support this as the proposed change seems both more intuitive and safer.
Right now we've essentially hacked this at my day job by bumping the offset
retention period really high, but this is a much cleaner solution.
I don't have any use-cases that require custom retention periods on a
per-group
Bump!
From: Vahid S Hashemian/Silicon Valley/IBM
To: dev
Date: 10/18/2017 04:45 PM
Subject:[DISCUSS] KIP-211: Revise Expiration Semantics of Consumer
Group Offsets
Hi all,
I created a KIP to address the group offset expiration issue reported in
KAFKA-4682:
https://cwiki.apa
Thanks Ted. I filled out that section.
--Vahid
From: Ted Yu
To: dev@kafka.apache.org
Date: 10/18/2017 04:59 PM
Subject:Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
Please fill out 'Rejected Alternatives' section.
Thanks
On W
Thanks Thomas for the suggestion.
I updated the KIP to explicitly describe that situation.
--Vahid
From: Thomas Becker
To: "dev@kafka.apache.org"
Date: 10/19/2017 08:23 AM
Subject: Re: [DISCUSS] KIP-211: Revise Expiration Semantics of
Consumer Group Offsets
I think it would be helpful to clarify what happens if consumers rejoin an
empty group. I would presume that the expiration timer is stopped and reset
back to offsets.retention.minutes when it is empty again but the KIP doesn't
say.
On Wed, 2017-10-18 at 16:45 -0700, Vahid S Hashemian wrote:
H
Please fill out 'Rejected Alternatives' section.
Thanks
On Wed, Oct 18, 2017 at 4:45 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:
> Hi all,
>
> I created a KIP to address the group offset expiration issue reported in
> KAFKA-4682:
> https://cwiki.apache.org/confluence/display/KAFKA/
31 matches
Mail list logo
