Sure Guozhang! Working on it and will post it when it is ready...
Thanks,
Senthil
-Original Message-
From: Guozhang Wang
Sent: Sunday, December 8, 2019 6:47 PM
To: dev
Subject: [EXTERNAL] Re: [VOTE] KIP-280: Enhanced log compaction
Thanks for the updated KIP, recasting my vote +1 on
> Thanks Matthias!
> >
> > Received 2 +1 binding... looking for one more +1 binding !
> >
> > Regards,
> > Senthil
> >
> > -Original Message-----
> > From: Matthias J. Sax
> > Sent: Wednesday, November 6, 2019 12:10 AM
> > To: dev
RNAL] Re: [VOTE] KIP-280: Enhanced log compaction
Hi, Senthil,
Thanks for the updated KIP. +1 from me.
Could you also add in the recommendation section what the users should do when
consuming the compacted topic with the new strategies (e.g. have to be more
careful with what records to keep)?
Senthilnathan Muthusamy
wrote:
> Jun,
>
> If the updated KIP looks good, can you please vote for it.
>
> Thanks,
> Senthil
>
> -Original Message-
> From: Jun Rao
> Sent: Thursday, November 7, 2019 4:33 PM
> To: dev
> Subject: Re: [VOTE] KIP-280: Enhanced log co
Jun,
If the updated KIP looks good, can you please vote for it.
Thanks,
Senthil
-Original Message-
From: Jun Rao
Sent: Thursday, November 7, 2019 4:33 PM
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi, Senthil,
Thanks for the KIP. Added a few more comments on the
inal Message-
> From: Matthias J. Sax
> Sent: Wednesday, November 6, 2019 12:10 AM
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
> +1 (binding)
>
> On 11/5/19 11:44 AM, Senthilnathan Muthusamy wrote:
> > Thanks Gouzhang and I
Thanks Matthias!
Received 2 +1 binding... looking for one more +1 binding !
Regards,
Senthil
-Original Message-
From: Matthias J. Sax
Sent: Wednesday, November 6, 2019 12:10 AM
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
+1 (binding)
On 11/5/19 11
--
> From: Guozhang Wang
> Sent: Monday, November 4, 2019 11:01 AM
> To: dev
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
> I only have one minor comment on the DISCUSS thread, otherwise I'm +1
> (binding).
>
> On Mon, Nov 4, 2019 at 9:53 AM Senthilnath
Thanks Gouzhang and I have made a note in the JIRA item to update the wiki.
Till now got 1 +1 binding... waiting for 2 more +1 binding... thnx!
Regards,
Senthil
-Original Message-
From: Guozhang Wang
Sent: Monday, November 4, 2019 11:01 AM
To: dev
Subject: Re: [VOTE] KIP-280
I only have one minor comment on the DISCUSS thread, otherwise I'm +1
(binding).
On Mon, Nov 4, 2019 at 9:53 AM Senthilnathan Muthusamy
wrote:
> Hi all,
>
> I would like to start the vote on the updated KIP-280: Enhanced log
> compaction. Thanks to Guozhang, Matthias & Tom for the valuable feedb
have an absolute sequence.
-Original Message-
From: Luís Cabral
Sent: Thursday, August 16, 2018 10:32 AM
To: dev@kafka.apache.org
Subject: RE: [VOTE] KIP-280: Enhanced log compaction
Hi,
@Guozhang & @Jun:
There were some left over comments from when the topic was still fresh (you
tion.
As for the suggestion, that is indeed a wonderful idea. I suggest that you
create a KIP and address the topic with the rest of the community.
Kind Regards,
Luis
From: Jason Gustafson
Sent: 14 August 2018 01:25
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hey Luis,
Thanks
added to the KIP as the result of the
> discussions, where it was seen as a potential benefit for other clients who
> may prefer that.
>
> Cheers,
> Luís
>
> From: Jason Gustafson
> Sent: 10 August 2018 22:38
> To: dev
> Subject: Re: [VOTE] KIP-280: Enhanced log compacti
2018 22:38
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi Luis,
It's still not very clear to me why we need the header-based strategy. Can
you elaborate why having the timestamp-based approach alone is not
sufficient? The use case in the motivation just describes a "m
find the willpower to read a 200-page
> > > essay
> > > > > about each one), so that influences me to avoid writing about every
> > > > > micro-impact that may exist, and simply leave it inferred (as you
> > have
> > > > just
> > >
careful
> with
> > > > removing the last message in a partition (which is possible now). We
> > need
> > > > to preserve the offset of the last message so that we don't reuse the
> > > > offset for a different message. One way to simply never
> -: 3. With the new cleaning strategy, we want to be a bit careful
> with
> > > > removing the last message in a partition (which is possible now). We
> > need
> > > > to preserve the offset of the last message so that we don't reuse the
> > &
ed by Jason) is to create an empty message
> > > batch.
> > >
> > > That is a good point, but isn’t the last message always kept
> regardless?
> > > In all of my tests with this approach, never have I seen it being
> > removed.
> > > This is not b
Thanks for the KIP, Luis. A brief comment below.
On Wed, Jul 4, 2018 at 11:11 AM Luís Cabral
wrote:
> As a reader, I tend to prefer brief documentation on new features (they
> tend to be too many for me to find the willpower to read a 200-page essay
> about each one), so that influences me to av
is is not because I made it so while changing the code, it was simply
> > like this before, which made me smile!
> > Given these results, I just *assumed* (oops) that these scenarios
> > represented the reality, so the compaction would only affected the
> “tail”,
> > whi
e compaction would only affected the “tail”,
> while the “head” remained untouched. Now that you say its possible that the
> last message actually gets overwritten somehow, I guess a new bullet point
> will have to be added to the KIP for this (after I’ve found the time to
> review the po
llet point will have to be
added to the KIP for this (after I’ve found the time to review the portion of
the code that enacts this behaviour).
Kind Regards,
Luís Cabral
From: Jun Rao
Sent: 03 July 2018 23:58
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi, Luis,
Thanks f
To: dev
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Sorry to join the discussion late. Can you you add to the motivation the
use cases for header-based compaction. This seems not very clear to me.
Thanks,
Jason
On Mon, Jul 2, 2018 at 11:00 AM, Guozhang Wang wrote:
> Hi Luis,
&g
Hi, Luis,
Thanks for the KIP. Overall, this seems a useful KIP. A few comments below.
1. I guess both new configurations will be at the topic level?
2. Since the log cleaner now needs to keep both the offset and another long
(say timestamp) in the de-dup map, it reduces the number of keys that we
Sorry to join the discussion late. Can you you add to the motivation the
use cases for header-based compaction. This seems not very clear to me.
Thanks,
Jason
On Mon, Jul 2, 2018 at 11:00 AM, Guozhang Wang wrote:
> Hi Luis,
>
> I believe that compaction property is indeed overridable at per-top
Hi Luis,
I believe that compaction property is indeed overridable at per-topic
level, as in
https://github.com/apache/kafka/blob/0cacbcf30e0a90ab9fad7bc310e5477cf959f1fd/clients/src/main/java/org/apache/kafka/common/config/TopicConfig.java#L116
And also documented in https://kafka.apache.org/doc
Hi Guozhang,
You are right that it is not straightforward to add a dependent property
validation.
Though it is possible to re-design it to allow for this, that effort would be
better placed under its own KIP, if it really becomes useful for other
properties as well.
Given this, the fallback-t
Hi Guozhang,
At the moment the KIP has your vote, Matthias' and Ted's.
Should I ask someone else to have a look?
Cheers,
Luis
On Monday, July 2, 2018, 12:16:48 PM GMT+2, Mickael Maison
wrote:
+1 (non binding). Thanks for the KIP!
On Sat, Jun 30, 2018 at 12:26 AM, Guozhang Wang wrot
+1 (non binding). Thanks for the KIP!
On Sat, Jun 30, 2018 at 12:26 AM, Guozhang Wang wrote:
> Hi Luis,
>
> Regarding the minor suggest, I agree it would be better to make it as
> mandatory, but it might be a bit tricky because it is a conditional
> mandatory one depending on the other config's v
Hi Luis,
Regarding the minor suggest, I agree it would be better to make it as
mandatory, but it might be a bit tricky because it is a conditional
mandatory one depending on the other config's value. Would like to see your
updated PR.
Regarding the KIP itself, both Matthias and myself can recast
Hi,
Thank you all for having a look!
The KIP is now updated with the result of these late discussions, though I did
take some liberty with this part:
- If the "compaction.strategy.header" configuration is not set (or is
blank), then the compaction strategy will fallback to "offset";
+1
On Thu, Jun 28, 2018 at 4:56 AM, Luís Cabral
wrote:
> Hi Ted,
> Can I also get your input on this?
>
> bq. +1 from my side for using `compaction.strategy` with values
> "offset","timestamp" and "header" and `compaction.strategy.header`
> -Matthias
>
> bq. +1 from me as well.
> -Guozhang
>
>
>
Hi Ted,
Can I also get your input on this?
bq. +1 from my side for using `compaction.strategy` with values
"offset","timestamp" and "header" and `compaction.strategy.header`
-Matthias
bq. +1 from me as well.
-Guozhang
Cheers,
Luis
t` or
> >> `timestamp` either. The issue is, that we cannot introduce a new system
> >> supported compaction strategy in the future without potentially breaking
> >> something, as we basically assign the whole space of Strings (minus
> >> `offset`, `timestamp`) as v
nfigs to enable header based compaction.
>>
>> Personally, I prefer either adding a config or going with
>> `header=`. Using `_timestamp_`, `_offset_`, and `` might be
>> good enough (even if this is the solution I like least)---for this case,
>> we should state explicitly, t
t, I
> would also add a check for the config that only allows for `_offset_`
> and `_timestamp_` and throws an exception for all other `_*_` configs.
>
>
> -Matthias
>
>
> On 6/18/18 2:03 PM, Luís Cabral wrote:
> > I’m ok with that...
> >
> > Ted / Matthias?
> &g
nd users are not allowed to set those for header compaction. In fact, I
> would also add a check for the config that only allows for `_offset_`
> and `_timestamp_` and throws an exception for all other `_*_` configs.
>
>
> -Matthias
>
>
> On 6/18/18 2:03 PM, Luís Cabral wrote:
>
amp_` and throws an exception for all other `_*_` configs.
-Matthias
On 6/18/18 2:03 PM, Luís Cabral wrote:
> I’m ok with that...
>
> Ted / Matthias?
>
>
> From: Guozhang Wang
> Sent: 18 June 2018 22:49
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-280: Enhanced
I’m ok with that...
Ted / Matthias?
From: Guozhang Wang
Sent: 18 June 2018 22:49
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
How about make the preserved values to be "_offset_" and "_timestamp_"
then? Currently in the KIP they are rese
Luís
>
>
> From: Guozhang Wang
> Sent: 18 June 2018 19:35
> To: dev@kafka.apache.org
> Subject: Re: [VOTE] KIP-280: Enhanced log compaction
>
> Hello Luís,
>
> I agree that having an expression evaluation as a config value is not the
> best approach; if there are better
,
Luís
From: Guozhang Wang
Sent: 18 June 2018 19:35
To: dev@kafka.apache.org
Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hello Luís,
I agree that having an expression evaluation as a config value is not the
best approach; if there are better ideas to allow users to specify the
header key
sure what you mean by "Shouldn't the selection in header
> >>>> have higher precedence over the configuration"? What selection do you
> >>>> mean? And want configuration?
> >>>>
> >>>>
> >>>> About the first point
ction do you
>>>> mean? And want configuration?
>>>>
>>>>
>>>> About the first point, I think this is actually a valid concern: To
>>>> address this issue, it seems that we would need to change the accepted
>>>> format of the conf
;>> address this issue, it seems that we would need to change the accepted
>>>> format of the config. Instead of "offset", "timestamp", "",
>>>> we could replace the last one with "header=".
>>>>
>>>> WDYT?
gt; About the first point, I think this is actually a valid concern: To
> > >> address this issue, it seems that we would need to change the accepted
> > >> format of the config. Instead of "offset", "timestamp",
> "",
> > >> we could replac
> About the first point, I think this is actually a valid concern: To
> >> address this issue, it seems that we would need to change the accepted
> >> format of the config. Instead of "offset", "timestamp", "",
> >> we could replace the las
/18 3:06 AM, Ted Yu wrote:
>>> If selection exists in header, the selection should override the config
>> value.
>>> Cheers
>>> Original message From: Luis Cabral
>> Date: 6/15/18 1:40 AM (GMT-08:00) To:
>> dev@kafka.apache.org Subject: R
erride the config
> value.
> > Cheers
> > ---- Original message From: Luis Cabral
> Date: 6/15/18 1:40 AM (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [VOTE] KIP-280: Enhanced log compaction
> > Hi,
> >
> > bq. Can the value be determin
ide the config value.
> Cheers
> Original message From: Luis Cabral
> Date: 6/15/18 1:40 AM (GMT-08:00) To:
> dev@kafka.apache.org Subject: Re: [VOTE] KIP-280: Enhanced log compaction
> Hi,
>
> bq. Can the value be determined now ? My thinking is that wha
If selection exists in header, the selection should override the config value.
Cheers
Original message From: Luis Cabral
Date: 6/15/18 1:40 AM (GMT-08:00) To:
dev@kafka.apache.org Subject: Re: [VOTE] KIP-280: Enhanced log compaction
Hi,
bq. Can the value be determined now
Hi,
bq. Can the value be determined now ? My thinking is that what if there is a
third compaction strategy proposed in the future ? We should guard against user
unknowingly choosing the 'future' strategy.
The idea is that the header name to use is flexible, which protects current
clients that
bq. When this configuration is set to anything other than "*offset*" or "
*timestamp*", then the record headers are scanned for a key matching this
value.
Can the value be determined now ? My thinking is that what if there is a
third compaction strategy proposed in the future ? We should guard aga
+1 (binding)
-Matthias
On 6/9/18 12:39 AM, Luís Cabral wrote:
> Hi all,
>
> Any takers on having a look at this KIP and voting on it?
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction
>
> Cheers,
> Luis
>
signature.asc
Description: OpenPGP digital sig
Thanks Luís for working on this KIP. I'm +1.
One note is that there is another KIP, KIP-228 going on for allowing
negative timestamp (I'm cc'ing the contributor of that KIP as well). One
related issue from KIP-228 is the handling of "-1": we will treat all other
negative values normally as some ti
54 matches
Mail list logo
