>> only to receive the non-null message only in step (3) is this correct?
>>
>> Cheers
>> Mike
>>
>>
>>
>> Sent using OWA for iPhone
>> ____________
>> From: Mayuresh Ghar
only in step (3) is this correct?
>>
>> Cheers
>> Mike
>>
>>
>>
>> Sent using OWA for iPhone
>> ________________
>> From: Mayuresh Gharat
>> Sent: Thursday, November 17, 2016 5:18:41 PM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS]
n step (3) is this correct?
>
> Cheers
> Mike
>
>
>
> Sent using OWA for iPhone
>
> From: Mayuresh Gharat
> Sent: Thursday, November 17, 2016 5:18:41 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - A
Phone
From: Mayuresh Gharat
Sent: Thursday, November 17, 2016 5:18:41 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
Hi Ismael,
Thanks for the explanation.
Specially I like this part where in you mentioned we can get rid of the
older
tombstone + non-null value
> * non-tombstone + non-null value
>
> Then we can discuss once KIP-87a is completed options later and how we
> support the second part KIP-87b to deliver:
> * non-tombstone + null value
>
> Cheers
> Mike
>
>
>
> _________________
: Thursday, November 17, 2016 1:43 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
Renu, Mayuresh and I had an offline discussion, and following is a brief
summary.
1. We agreed that not bumping up magic value may result in losing zero copy
during migration.
2.
Renu, Mayuresh and I had an offline discussion, and following is a brief
summary.
1. We agreed that not bumping up magic value may result in losing zero copy
during migration.
2. Given that bumping up magic value is almost free and has benefit of
avoiding potential performance issue. It is probabl
Hi Mayuresh,
Thanks for describing your plan in detail. I understand the concern about
having to support both old and new way for a long time. In my opinion, we
don't have a choice in that matter. We can't change semantics of the
message format without having a long transition period. And we can't
Hi Ismael,
This is something I can think of for migration plan:
So the migration plan can look something like this, with up conversion :
1) Currently lets say we have Broker at version x.
2) Currently we have clients at version x.
3) a) We move the version to Broker(x+1) : supports both tombstone
Hi Ismael,
That's a very good point which I might have not considered earlier.
Here is a plan that I can think of:
Stage 1) The broker from now on, up converts the message to have the
tombstone marker. The log compaction thread does log compaction based on
both null and tombstone marker. This is
One comment below.
On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat wrote:
>- If we don't bump up the magic byte, on the broker side, the broker
>will always have to look at both tombstone bit and the value when do the
>compaction. Assuming we do not bump up the magic byte,
>imag
n, Nov 14, 2016 at 1:43 PM, Michael Pearce >
> > > wrote:
> > >
> > > > I like the idea of up converting and then just having the logic to
> look
> > > > for tombstones. It makes that quite clean in nature.
> > > >
> > > > It
> > >
> > > It's quite late here in the UK, so I fully understand / confirm I
> > > understand what you propose could you write it on the kip wiki or fully
> > > describe exactly how you see it working, so uk morning I could read
> > through?
> > &g
the input on this it is appreciated.
> >
> >
> > Sent using OWA for iPhone
> > ____________
> > From: Mayuresh Gharat
> > Sent: Monday, November 14, 2016 9:28:16 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS]
_
> From: Mayuresh Gharat
> Sent: Monday, November 14, 2016 9:28:16 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> Hi Michael,
>
> Just another thing that came up during my discussion with Renu and I wanted
> to share it.
>
; Kip-87a that delivers phase 1
> > >
> > > And then
> > > Kip-87b in release after that delivers phase 2 but has a dependency on
> > > support of deprecating old api versions
> > >
> > > How does this approach sound?
> > >
> > >
we have
> > Kip-87a that delivers phase 1
> >
> > And then
> > Kip-87b in release after that delivers phase 2 but has a dependency on
> > support of deprecating old api versions
> >
> > How does this approach sound?
> >
> >
> > ___
And the old apis
> > would not be allowed to be called as the broker now expects all clients
> to
> > be using latest api only.
> >
> > At this second phase stage only also we can support a null value in a
> > compacted topic without it being treated as deletion. W
ld api versions
How does this approach sound?
From: Becket Qin
Sent: Monday, November 14, 2016 6:14:44 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
Hey Michael and Mayuresh,
If I understand correctly, you
c without it being treated as deletion. We solely base on the
> tombstone marker.
>
>
>
> From: Mayuresh Gharat
> Sent: Saturday, November 12, 2016 2:18:19 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> I think "In
n. We solely base on the tombstone
marker.
From: Mayuresh Gharat
Sent: Saturday, November 12, 2016 2:18:19 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
I think "In second stage we move on to supportin
> > >
> > > The two stage approach is because we want to end up just supporting
> > > tombstone marker (not both)
> > >
> > > But we accept we need to allow organisations and systems a period of
> > > transition (this is what stage 1 prov
ion (this is what stage 1 provides)
> >
> >
> >
> >
> >
> > From: Mayuresh Gharat
> > Sent: Thursday, November 10, 2016 8:57 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tom
m: Mayuresh Gharat
> Sent: Thursday, November 10, 2016 8:57 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> I am not sure if we are bumping magicByte value as the KIP does not mention
> it (If I didn't miss anything).
>
> @Nac
ition
(this is what stage 1 provides)
From: Mayuresh Gharat
Sent: Thursday, November 10, 2016 8:57 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
I am not sure if we are bumping magicByte value as the KIP does
n can be done in 2 stages :
> > > > > > >
> > > > > > > 1) In first stage the broker should understand the
> attribute
> > > flag
> > > > > as
> > > > > > > well
> > > > > > > as N
gt; > > > > > for log
> > > > > > compaction.
> > > > > >
> > > > > > I agree with Becket that for older clients (consumers) the
> > broker
> > > > > might
> > > > > > have to down co
ull value. But this should be in
> first
> > > > stage.
> > > > > Once all the clients have upgraded (clients start recognizing
> the
> > > > > attribute
> > > > > flag), we can move the broker to stage 2.
> > > > >
>
gt; compacting but has a non null value. But this should
> be in first
> > > > stage.
> > > > > Once all the clients have upgraded (clients start
> recognizing the
> > > > > attribute
> > > > > flag), we can move the broker to stage 2.
> > > >
t; michael.pea...@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Also we can add further guidance:
> > > > >
> > > > > To avoid the below caveat to org
On Tue, Nov 8, 2016 at 12:06 AM, Michael Pearce <
> > > michael.pea...@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Also we can add further guidance:
> > > > >
> > &g
ks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Tue, Nov 8, 2016 at 12:06 AM, Michael Pearce <
> > > michael.pea...@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > >
t; > > On Tue, Nov 8, 2016 at 12:06 AM, Michael Pearce <
> > michael.pea...@ig.com
> > > >
> > > wrote:
> > >
> > > > Also we can add further guidance:
> > > >
> > > > To avoid the below caveat to organisations b
; consumers first before relying on producing tombstone messages with
> > data
> > >
> > > Sent using OWA for iPhone
> > >
> > > From: Michael Pearce
> > > Sent: Tuesday, November 8, 2016 8:03:3
________________
> > From: Michael Pearce
> > Sent: Tuesday, November 8, 2016 8:03:32 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> >
> > Thanks Jun
t; Sent using OWA for iPhone
>
> From: Michael Pearce
> Sent: Tuesday, November 8, 2016 8:03:32 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> Thanks Jun on the feedback, I think I understand the is
ing OWA for iPhone
>
> From: Michael Pearce
> Sent: Tuesday, November 8, 2016 8:03:32 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> Thanks Jun on the feedback, I think I understand the issue/point now.
, 2016 8:03:32 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
Thanks Jun on the feedback, I think I understand the issue/point now.
We def can add that on older client version if tombstone marker make the value
null to preserve behaviour.
There is one
esday, November 8, 2016 12:34:41 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
For the use case, one potential use case is for schema registration. For
example, in Avro, a null value corresponds to a Null schema. So, if you
want to be able to keep the schema id
For the use case, one potential use case is for schema registration. For
example, in Avro, a null value corresponds to a Null schema. So, if you
want to be able to keep the schema id in a delete message, the value can't
be null. We could get around this issue by specializing null value during
schem
Hi Michael,
Yes, changing the logic in the log cleaner makes sense. There could be some
other thing worth thinking (e.g. the message size change after conversion),
though.
The scenario I was thinking is the following:
Imagine a distributed caching system built on top of Kafka. A user is
consuming
Hi Becket,
We were thinking more about having the logic that’s in the method
shouldRetainMessage configurable via
http://kafka.apache.org/documentation.html#brokerconfigs at a broker/topic
level. And then scrap auto converting the message, and allow organisations to
manage the rollout of enab
Hi Michael,
Do you mean using a new configuration it is just the exiting
message.format.version config? It seems the message.format.version config
is enough in this case. And the default value would always be the latest
version.
> Message version migration would be handled as like in KIP-32
Also
Hi Joel , et al.
Any comments on the below idea to handle roll out / compatibility of this
feature, using a configuration?
Does it make sense/clear?
Does it add value?
Do we want to enforce flag by default, or value by default, or both?
Cheers
Mike
On 10/27/16, 4:47 PM, "Michael Pearce" wro
; Mike
>
> From: Jay Kreps
> Sent: Thursday, October 27, 2016 10:54:45 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
>
> I kind of agree with James that it i
On a side note just a use case that came to my mind, if we ever plan to add
headers to kafka, we can provide this functionality of compaction based on
headers and make the compaction policy pluggable on the broker side. This
might give flexibility for doing compaction in future.
Thanks,
Mayuresh
My two cents on the magic value bumping.
Although the broker can infer from the request version to determine whether
it should read the new attribute bit or not, currently the request version
is not propagated to the replica manager.
Bumping up the magic value does not seem a big overhead in this
Xavier,
Yeah I think that post KIP-58 it is possible to depend on the delivery of
messages in compacted topics, if you override the default compaction time.
Prior to that it was true that you could control the delete retention, but
any message including the tombstone could be compacted away prior
>
> I kind of agree with James that it is a bit questionable how valuable any
> data in a delete marker can be since it will be deleted somewhat
> nondeterministically.
>
One could argue that even in normal topics, assuming a time-based log
retention policy is in place, any message will be deleted
where we are discussing the section about comparability
> and rollout.
>
>
>
> Rgds
> Mike
> ____
> From: Jay Kreps
> Sent: Thursday, October 27, 2016 10:54:45 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
*compatability
From: Michael Pearce
Sent: Friday, October 28, 2016 5:17:07 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
Hi Jay,
I think use case that is the issue that Konstantin mentioned in the kip-82
llout.
Rgds
Mike
From: Jay Kreps
Sent: Thursday, October 27, 2016 10:54:45 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
I kind of agree with James that it is a bit questionable how valuable any
data in a delete marke
I kind of agree with James that it is a bit questionable how valuable any
data in a delete marker can be since it will be deleted somewhat
nondeterministically.
Let's definitely ensure the change is worth the resulting pain and
additional complexity in the data model.
I think the two things we ma
Thanks, James, I think this is a really good addition to the KIP details,
please feel free to amend the wiki/add the use cases, also if any others you
think of. I definitely think its worthwhile documenting. If you can’t let me
know ill add them next week (just leaving for a long weekend off)
R
This KIP would definitely address a gap in the current functionality, where you
currently can't have a tombstone with any associated content.
That said, I'd like to talk about use cases, to make sure that this is in fact
useful. The KIP should be updated with whatever use cases we come up with.
+1 @Joel.
I think a clear migration plan of upgrading and downgrading of server and
clients along with handling of issues that Joel mentioned, on the KIP would
be really great.
Thanks,
Mayuresh
On Wed, Oct 26, 2016 at 3:31 PM, Joel Koshy wrote:
> I'm not sure why it would be useful, but it sho
I'm not sure why it would be useful, but it should be theoretically
possible if the attribute bit alone is enough to mark a tombstone. OTOH, we
could consider that as invalid if we wish. These are relevant details that
I think should be added to the KIP.
Also, in the few odd scenarios that I menti
Does this mean that starting with V4 requests we would allow storing null
messages in compacted topics? The KIP should probably clarify the behavior
for null messages where the tombstone flag is not net.
On Wed, Oct 26, 2016 at 1:32 AM Magnus Edenhill wrote:
> 2016-10-25 21:36 GMT+02:00 Nacho So
A magic byte bump would be required for the addition of new fields; or
removal of existing fields. Changing the interpretation of an existing
field (e.g., switching from absolute to relative offsets) almost always
needs a magic byte bump as well.
One concern Nacho alluded to (I think) is if a newe
I agree with Magnus about deciding rules on when to update.
If we can handle such changes by request versioning, it would be good to
understand the use cases for bumping up magic byte or vice versa.
Unless I am not understanding it correctly, having two versioning schemes
seems little weird.
Thank
Hi Renu,
that is not completely true, the LZ4 compression codec was added without a
MagicByte bump.
(LZ4 might be a bad example though since this feature was added without
updating the protocol docs..)
Unless the broker needs the MagicByte internally (for translating old logs
on disk or whatever)
+1 @Magnus.
It is also in line with traditional use of the magic field to indicate a
change in the format of the message. Thus a change in the magic field
indicates a different "schema" which in this case would reflect adding a
new field, removing a field or changing the type of fields etc.
The ve
I see the reasoning Magnus described.
If you check the docs https://kafka.apache.org/documentation#messageformat,
it says :
1 byte "magic" identifier to allow format changes, value is 0 or 1
Moreover as per comments in the code :
/**
* The "magic" value
* When magic value is 0, the message
2016-10-25 21:36 GMT+02:00 Nacho Solis :
> I think you probably require a MagicByte bump if you expect correct
> behavior of the system as a whole.
>
> From a client perspective you want to make sure that when you deliver a
> message that the broker supports the feature you're expecting
> (compact
5, 2016 8:36:24 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
I think you probably require a MagicByte bump if you expect correct
behavior of the system as a whole.
From a client perspective you want to make sure that when you deliver a
message that the
I think you probably require a MagicByte bump if you expect correct
behavior of the system as a whole.
>From a client perspective you want to make sure that when you deliver a
message that the broker supports the feature you're expecting
(compaction). So, depending on the behavior of the broker o
It is safe to assume that a previously undefined attributes bit will be
unset in protocol requests from existing clients, if not, such a client is
already violating the protocol and needs to be fixed.
So I dont see a need for a MagicByte bump, both broker and client has the
information it needs to
Hi Magnus,
I was wondering if I even needed to change those also, as technically we’re
just making use of a non used attribute bit, but im not 100% that it be always
false currently.
If someone can say 100% it will already be set false with current and historic
bit wise masking techniques used
Hi Michael,
With the version bumps for Produce and Fetch requests, do you really need
to bump MagicByte too?
Regards,
Magnus
2016-10-25 18:09 GMT+02:00 Michael Pearce :
> Hi All,
>
> I would like to discuss the following KIP proposal:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>
Hi All,
I would like to discuss the following KIP proposal:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-87+-+Add+Compaction+Tombstone+Flag
This is off the back of the discussion on KIP-82 / KIP meeting where it was
agreed to separate this issue and feature. See:
http://mail-archives.a
70 matches
Mail list logo
