Sorry for sending this late.
Please see the replies/comments inline :
- In practice, the up-conversion may not be used until all the clients
are upgraded.
- I am not sure what you meant by this. Up conversion is actually
done for old producer clients producing to the new broker only, right? So
we don't have to wait for all clients to be upgraded.
- When rolling out the new server code, we will keep the message format
at 0.10.1. We cannot use message format version 0.10.2 yet because if we do
that the brokers will suddenly lose zero copy.
- I am not sure if this is entirely true. There is no notion of zero
copy on the produce side, I think (and as per Joel). I am not
100% familiar
with this part but from the first look at the code, it does not seem to
have a notion of zero copy.
- If we bump up the message version to 0.10.2, the broker will have to
look at the message to see if down conversion is needed).
- I think Renu was talking about the request version and not the message
version. The down conversion is anyways needed for the new broker talking
to old consumer anyways.
- Later on when most of the consumers have upgraded, we will then bump
up the message format version to 0.10.2.
- I am not sure what you mean by most of the consumers. The way I
look at it is, either all consumers have upgraded then we don't have to
down convert or else we will have to down convert.
- If we don't bump up the magic byte, on the broker side, the broker
will always have to look at both tombstone bit and the value when do the
compaction. Assuming we do not bump up the magic byte,
imagine the broker sees a message which does not have a tombstone bit
set. The broker does not know when the message was produced (i.e. whether
the message has been up converted or not), it has to take a further look at
the value to see if it is null or not in order to determine if it is a
tombstone. The same logic has to be put on the consumer as well because the
consumer does not know if the message has been up converted or not.
- If we upconvert while appending, this is not the case, right? The
consumer will only look at the tombstone marker. Actually the consumers
don't care about the value is null or tombstone marker is set, they just
consume whatever is there on the broker right? It is the
applications that
are using the consumer that will care. And since in the second stage, the
applications will be aware of the behavior, I think it should be
fine. Also
the broker does not need to know when the message was produced
since at the
time of appending it knows what the request version was and since it has
already upconverted the message, if it gets a fetch request from old
consumer it will down convert (as you have suggested earlier) but for the
fetch new request version it will not do anything. During log compaction,
it will only take a look at the tombstone marker.
- As per the documentation of when to up the magic byte, it should be
done only during message format change. This is not ideally a message
format change right, since we are just using the reserved bit? If that is
not the case, then we should change the description of when to
up the magic
byte to something like any changes made to existing message format
(addition of new fields or using existing reserved fields)
Thanks,
Mayuresh
On Tue, Nov 15, 2016 at 3:35 AM, Becket Qin <becket....@gmail.com> wrote:
> Hi Renu,
>
> Technically speaking we may not need to bump up the magic value. A
> tombstone is a message either with tombstone bit set OR with a null
> value.(once all the clients has been upgraded, it automatically becomes
> only based on tombstone) However, this leads to a few subtle issues.
>
> 1. Migration. In practice, the up-conversion may not be used until all the
> clients are upgraded. Kafka requires the client version to be lower than
> the server version for compatibility. That means the new server code is
> always deployed when clients are still running old code. When rolling out
> the new server code, we will keep the message format at 0.10.1. We cannot
> use message format version 0.10.2 yet because if we do that the brokers
> will suddenly lose zero copy This is because all the clients are still
> running old code. If we bump up the message version to 0.10.2, the broker
> will have to look at the message to see if down conversion is needed).
> Later on when most of the consumers have upgraded, we will then bump up the
> message format version to 0.10.2. So the broker cannot always up convert
> and depend on the tombstone even with the new code.
>
> 2. Long term efficiency. If we don't bump up the magic byte, on the broker
> side, the broker will always have to look at both tombstone bit and the
> value when do the compaction. Assuming we do not bump up the magic byte,
> imagine the broker sees a message which does not have a tombstone bit set.
> The broker does not know when the message was produced (i.e. whether the
> message has been up converted or not), it has to take a further look at the
> value to see if it is null or not in order to determine if it is a
> tombstone. The same logic has to be put on the consumer as well because the
> consumer does not know if the message has been up converted or not. With a
> magic value bump, the broker/consumer knows for sure there is no need to
> look at the value anymore if the tombstone bit is not set. So if we want to
> eventually only use tombstone bit, a magic value bump is necessary.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Mon, Nov 14, 2016 at 11:39 PM, Renu Tewari <tewa...@gmail.com> wrote:
>
> > Is upping the magic byte to 2 needed?
> >
> > In your example say
> > For broker api version at or above 0.10.2 the tombstone bit will be used
> > for log compaction deletion.
> > If the producerequest version is less than 0.10.2 and the message is
> null,
> > the broker will up convert to set the tombstone bit on
> > If the producerequest version is at or above 0.10.2 then the tombstone
> bit
> > value is unchanged
> > Either way the new version of broker only uses the tombstone bit
> > internally.
> >
> > thanks
> > Renu
> >
> > On Mon, Nov 14, 2016 at 8:31 PM, Becket Qin <becket....@gmail.com>
> wrote:
> >
> > > If we follow the current way of doing this format change, it would work
> > the
> > > following way:
> > >
> > > 0. Bump up the magic byte to 2 to indicate the tombstone bit is used.
> > >
> > > 1. On receiving a ProduceRequest, broker always convert the messages to
> > the
> > > configured message.format.version.
> > > 1.1 If the message version does not match the configured
> > > message.format.version, the broker will either up convert or down
> convert
> > > the message. In that case, users pay the performance cost of
> > re-compression
> > > if needed.
> > > 1.2 If the message version matches the configured
> message.format.version,
> > > the broker will not do the format conversion and user may save the
> > > re-compression cost if the message.format.version is on or above
> 0.10.0.
> > >
> > > 2. On receiving a FetchRequest, the broker check the FetchRequest
> version
> > > to see if the consumer supports the configured message.format.version
> or
> > > not. If the consumer does not support it, down conversion is required
> and
> > > zero copy is lost. Otherwise zero copy is used to return the
> > FetchResponse.
> > >
> > > Notice that setting message.format.version to 0.10.2 is equivalent to
> > > always up convert if needed, but that also means to always down convert
> > if
> > > there is an old consumer.
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > >
> > >
> > > On Mon, Nov 14, 2016 at 1:43 PM, Michael Pearce <michael.pea...@ig.com
> >
> > > wrote:
> > >
> > > > I like the idea of up converting and then just having the logic to
> look
> > > > for tombstones. It makes that quite clean in nature.
> > > >
> > > > It's quite late here in the UK, so I fully understand / confirm I
> > > > understand what you propose could you write it on the kip wiki or
> fully
> > > > describe exactly how you see it working, so uk morning I could read
> > > through?
> > > >
> > > > Thanks all for the input on this it is appreciated.
> > > >
> > > >
> > > > Sent using OWA for iPhone
> > > > ________________________________________
> > > > From: Mayuresh Gharat <gharatmayures...@gmail.com>
> > > > Sent: Monday, November 14, 2016 9:28:16 PM
> > > > To: dev@kafka.apache.org
> > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> > > >
> > > > Hi Michael,
> > > >
> > > > Just another thing that came up during my discussion with Renu and I
> > > wanted
> > > > to share it.
> > > >
> > > > Other thing we can do to handle a mixture of old and new clients is
> > when
> > > > once the new broker with this KIP is deployed, the new code should
> > check
> > > > the request version from older producer we can up convert it with a
> > > > tombstone marker when appending the message to the log. This is
> similar
> > > to
> > > > down converting messages for older clients.
> > > >
> > > > If this is possible then the broker in this case has to rely only on
> > the
> > > > tombstone marker for log compaction. Using this approach we preserve
> > the
> > > > description of when to update the magic byte as described here :
> > > > https://kafka.apache.org/documentation#messageformat (1 byte "magic"
> > > > identifier to allow format changes).
> > > >
> > > > In stage 2, if we don't want open source kafka to make the decision
> of
> > > > deprecation of null value for log compaction (which is the approach I
> > > would
> > > > prefer as an end state) since there are some concerns on this,
> > individual
> > > > companies/orgs can then have a horizontal initiative at their
> > convenience
> > > > to move to stage 2 by asking all app users to upgrade to new kafka
> > > clients.
> > > >
> > > > Thanks,
> > > >
> > > > Mayuresh
> > > >
> > > > On Mon, Nov 14, 2016 at 11:50 AM, Becket Qin <becket....@gmail.com>
> > > wrote:
> > > >
> > > > > Michael,
> > > > >
> > > > > Yes, I am OK with stage 1. We can discuss about stage 2 later but
> > this
> > > > > sounds really an organization specific decision to deprecate an
> API.
> > It
> > > > > does not seem a general need in open source Kafka to only support
> > > > tombstone
> > > > > bit , which is a bad thing for people who are still running old
> > clients
> > > > in
> > > > > the community. This is exactly why we want to have magic byte bump
> -
> > > > there
> > > > > should be only one definitive way to interpret a message with a
> given
> > > > magic
> > > > > byte. We should avoid re-define the interpretation of a message
> with
> > an
> > > > > existing magic byte. The interpretation of an old message format
> > should
> > > > not
> > > > > be changed and should always be supported until deprecated.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jiangjie (Becket) Qin
> > > > >
> > > > > On Mon, Nov 14, 2016 at 11:28 AM, Mayuresh Gharat <
> > > > > gharatmayures...@gmail.com> wrote:
> > > > >
> > > > > > I am not sure about "If I understand correctly, you want to let
> the
> > > > > broker
> > > > > > to reject requests
> > > > > > from old clients to ensure everyone in an organization has
> > upgraded,
> > > > > > right?"
> > > > > >
> > > > > > I don't think we will be rejecting requests. What phase2 (stage
> 2)
> > > > meant
> > > > > > was we will only do log compaction based on tombstone marker and
> > > > nothing
> > > > > > else.
> > > > > > I am not sure about making this configurable, as a configuration
> > > makes
> > > > it
> > > > > > sound like the broker has the capability to support null and
> > > tombstone
> > > > > > marker bit, for log compaction, for life long.
> > > > > >
> > > > > > I agree with Michael's idea of delivering phase 1 (stage1) right
> > now.
> > > > It
> > > > > > will also be good to put a note saying that we will be
> deprecating
> > > the
> > > > > > older way of log compaction in next release.
> > > > > > Phase 2 (stage 2) will actually only rely on tombstone marker for
> > > doing
> > > > > log
> > > > > > compaction.
> > > > > > So this actually meets our end state of having tombstone marker
> for
> > > log
> > > > > > compaction support.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Mayuresh
> > > > > >
> > > > > >
> > > > > > On Mon, Nov 14, 2016 at 11:09 AM, Michael Pearce <
> > > > michael.pea...@ig.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I'm ok with this, but I'd like to at least get phase 1 in, for
> > the
> > > > next
> > > > > > > release, this is what I'm very keen for,
> > > > > > >
> > > > > > > As such shall we say we have
> > > > > > > Kip-87a that delivers phase 1
> > > > > > >
> > > > > > > And then
> > > > > > > Kip-87b in release after that delivers phase 2 but has a
> > dependency
> > > > on
> > > > > > > support of deprecating old api versions
> > > > > > >
> > > > > > > How does this approach sound?
> > > > > > >
> > > > > > >
> > > > > > > ________________________________________
> > > > > > > From: Becket Qin <becket....@gmail.com>
> > > > > > > Sent: Monday, November 14, 2016 6:14:44 PM
> > > > > > > To: dev@kafka.apache.org
> > > > > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> > > > > > >
> > > > > > > Hey Michael and Mayuresh,
> > > > > > >
> > > > > > > If I understand correctly, you want to let the broker to reject
> > > > > requests
> > > > > > > from old clients to ensure everyone in an organization has
> > > upgraded,
> > > > > > right?
> > > > > > > This is essentially deprecating an old protocol. I agree it is
> > > useful
> > > > > and
> > > > > > > that is why we have that baked in KIP-35. However, we haven't
> > > thought
> > > > > > > through how to deprecate an API so far. From broker's point of
> > > view.,
> > > > > it
> > > > > > > will always support all the old protocols according to Kafka's
> > > widely
> > > > > > known
> > > > > > > compatibility guarantee. If people decide to deprecate an API
> > > within
> > > > an
> > > > > > > organization, we can provide some additional configuration to
> let
> > > > > people
> > > > > > do
> > > > > > > this, which is not a bad idea but seems better to be discussed
> in
> > > > > another
> > > > > > > KIP.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jiangjie (Becket) Qin
> > > > > > >
> > > > > > > On Mon, Nov 14, 2016 at 8:52 AM, Michael Pearce <
> > > > michael.pea...@ig.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree with Mayuresh.
> > > > > > > >
> > > > > > > > I don't see how having a magic byte helps here.
> > > > > > > >
> > > > > > > > What we are saying is that on day 1 after an upgrade both
> > > tombstone
> > > > > > flag
> > > > > > > > or a null value will be treated as a marker to delete on
> > > compacted
> > > > > > topic.
> > > > > > > >
> > > > > > > > During this time we expect organisations to migrate
> themselves
> > > over
> > > > > > onto
> > > > > > > > the new producers and consumers that call the new Api,
> changing
> > > > their
> > > > > > > > application logic. during this point producers should start
> > > setting
> > > > > the
> > > > > > > > tombstone marker and consumers to start behaving based on the
> > > > > tombstone
> > > > > > > > marker.
> > > > > > > >
> > > > > > > > Only after all producers and consumers are upgraded then we
> > enter
> > > > > > phase 2
> > > > > > > > disabling the use of a null value as a delete marker. And the
> > old
> > > > > apis
> > > > > > > > would not be allowed to be called as the broker now expects
> all
> > > > > clients
> > > > > > > to
> > > > > > > > be using latest api only.
> > > > > > > >
> > > > > > > > At this second phase stage only also we can support a null
> > value
> > > > in a
> > > > > > > > compacted topic without it being treated as deletion. We
> solely
> > > > base
> > > > > on
> > > > > > > the
> > > > > > > > tombstone marker.
> > > > > > > >
> > > > > > > >
> > > > > > > > ________________________________________
> > > > > > > > From: Mayuresh Gharat <gharatmayures...@gmail.com>
> > > > > > > > Sent: Saturday, November 12, 2016 2:18:19 AM
> > > > > > > > To: dev@kafka.apache.org
> > > > > > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction Tombstone Flag
> > > > > > > >
> > > > > > > > I think "In second stage we move on to supporting only the
> > > > attribute
> > > > > > flag
> > > > > > > > for log
> > > > > > > > compaction." means that it will no longer support request
> from
> > > > older
> > > > > > > > clients.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > >
> > > > > > > > Mayuresh
> > > > > > > >
> > > > > > > > On Fri, Nov 11, 2016 at 10:02 AM, Becket Qin <
> > > becket....@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hey Michael,
> > > > > > > > >
> > > > > > > > > The way Kafka implements backwards compatibility is to let
> > the
> > > > > > brokers
> > > > > > > > > support old protocols. So the brokers have to support older
> > > > clients
> > > > > > > that
> > > > > > > > do
> > > > > > > > > not understand the new attribute bit. That means we will
> not
> > be
> > > > > able
> > > > > > to
> > > > > > > > get
> > > > > > > > > away with the null value as a tombstone. So we need a good
> > > > > definition
> > > > > > > of
> > > > > > > > > whether we should treat it as a tombstone or not. This is
> > why I
> > > > > think
> > > > > > > we
> > > > > > > > > need a magic value bump.
> > > > > > > > >
> > > > > > > > > My confusion on the second stage is exactly about "we want
> to
> > > end
> > > > > up
> > > > > > > just
> > > > > > > > > supporting tombstone marker (not both)", or in the original
> > > > > > statement:
> > > > > > > > "2)
> > > > > > > > > In second stage we move on to supporting only the attribute
> > > flag
> > > > > for
> > > > > > > log
> > > > > > > > > compaction." - We will always support the null value as a
> > > > tombstone
> > > > > > as
> > > > > > > > long
> > > > > > > > > as the message format version (i.e. magic byte) is less
> than
> > 2
> > > > for
> > > > > > the
> > > > > > > > > reason I mentioned above.
> > > > > > > > >
> > > > > > > > > Although the way to interpret the message bytes at
> producing
> > > time
> > > > > can
> > > > > > > be
> > > > > > > > > inferred from the the ProduceRequest version, this version
> > > > > > information
> > > > > > > > > won't be stored in the broker. So when an old consumer
> comes,
> > > we
> > > > > need
> > > > > > > to
> > > > > > > > > decide whether we have to do down conversion or not.. With
> a
> > > > clear
> > > > > > > > existing
> > > > > > > > > configuration of message version config (i.e. magic value),
> > we
> > > > know
> > > > > > for
> > > > > > > > > sure when to down convert the messages to adapt to older
> > > clients.
> > > > > > > > Otherwise
> > > > > > > > > we will have to always scan all the messages. It would
> > probably
> > > > > work
> > > > > > > but
> > > > > > > > > relies on guess or inference.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > >
> > > > > > > > > On Fri, Nov 11, 2016 at 8:42 AM, Mayuresh Gharat <
> > > > > > > > > gharatmayures...@gmail.com
> > > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Sounds good Michael.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > >
> > > > > > > > > > Mayuresh
> > > > > > > > > >
> > > > > > > > > > On Fri, Nov 11, 2016 at 12:48 AM, Michael Pearce <
> > > > > > > > michael.pea...@ig.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > @Mayuresh i don't think you've missed anything -
> > > > > > > > > > >
> > > > > > > > > > > as per earlier in the discussion.
> > > > > > > > > > >
> > > > > > > > > > > We're providing new api versions, but not planning on
> > > bumping
> > > > > the
> > > > > > > > magic
> > > > > > > > > > > number as there is no structural changes, we are simply
> > > using
> > > > > up
> > > > > > a
> > > > > > > > new
> > > > > > > > > > > attribute bit (as like adding new compression support
> > just
> > > > uses
> > > > > > up
> > > > > > > > > > > additional attribute bits)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > I think also difference between null vs 0 length byte
> > array
> > > > is
> > > > > > > > covered.
> > > > > > > > > > >
> > > > > > > > > > > @Becket,
> > > > > > > > > > >
> > > > > > > > > > > The two stage approach is because we want to end up
> just
> > > > > > supporting
> > > > > > > > > > > tombstone marker (not both)
> > > > > > > > > > >
> > > > > > > > > > > But we accept we need to allow organisations and
> systems
> > a
> > > > > period
> > > > > > > of
> > > > > > > > > > > transition (this is what stage 1 provides)
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > ________________________________________
> > > > > > > > > > > From: Mayuresh Gharat <gharatmayures...@gmail.com>
> > > > > > > > > > > Sent: Thursday, November 10, 2016 8:57 PM
> > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction
> Tombstone
> > > Flag
> > > > > > > > > > >
> > > > > > > > > > > I am not sure if we are bumping magicByte value as the
> > KIP
> > > > does
> > > > > > not
> > > > > > > > > > mention
> > > > > > > > > > > it (If I didn't miss anything).
> > > > > > > > > > >
> > > > > > > > > > > @Nacho : I did not understand what you meant by :
> > > Conversion
> > > > > from
> > > > > > > one
> > > > > > > > > to
> > > > > > > > > > > the other may be possible but I would rather not have
> to
> > do
> > > > it.
> > > > > > > > > > >
> > > > > > > > > > > You will have to do the conversion for the scenario
> that
> > > > Becket
> > > > > > has
> > > > > > > > > > > mentioned above where an old consumer talks to the new
> > > > broker,
> > > > > > > right?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > >
> > > > > > > > > > > Mayuresh
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Nov 10, 2016 at 11:54 AM, Becket Qin <
> > > > > > becket....@gmail.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Nacho,
> > > > > > > > > > > >
> > > > > > > > > > > > In Kafka protocol, a negative length is null, a zero
> > > length
> > > > > > means
> > > > > > > > > empty
> > > > > > > > > > > > byte array.
> > > > > > > > > > > >
> > > > > > > > > > > > I am still confused about the two stages. It seems
> that
> > > the
> > > > > > > broker
> > > > > > > > > only
> > > > > > > > > > > > needs to understand three versions of messages.
> > > > > > > > > > > > 1. MagicValue=0 - no timestamp, absolute offset, no
> > > > tombstone
> > > > > > > flag
> > > > > > > > > > > > 2. MagicValue=1 - with timestamp, relative offsets,
> no
> > > > > > tombstone
> > > > > > > > flag
> > > > > > > > > > > (null
> > > > > > > > > > > > value = tombstone)
> > > > > > > > > > > > 3. MagicValue=2 - with timestamp, relative offsets,
> > > > tombstone
> > > > > > > flag
> > > > > > > > > > > >
> > > > > > > > > > > > We are talking about two flavors for 3:
> > > > > > > > > > > > 3.1 tombstone flag set = tombstone (Allows a key with
> > > null
> > > > > > value
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > compacted topics)
> > > > > > > > > > > > 3.2 tombstone flag set OR null value = tombstone ( Do
> > not
> > > > > > allow a
> > > > > > > > key
> > > > > > > > > > > with
> > > > > > > > > > > > null value in the compacted topics)
> > > > > > > > > > > >
> > > > > > > > > > > > No matter which flavor we choose, we just need to
> stick
> > > to
> > > > > that
> > > > > > > way
> > > > > > > > > of
> > > > > > > > > > > > interpretation, right? Why would we need a second
> > stage?
> > > > > > > > > > > >
> > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Nov 10, 2016 at 10:37 AM, Ignacio Solis <
> > > > > > iso...@igso.net
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > A quick differentiation I would like to make is
> null
> > is
> > > > not
> > > > > > the
> > > > > > > > > same
> > > > > > > > > > as
> > > > > > > > > > > > > size 0.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Many times these are made equal, but they're not.
> > When
> > > > > > > > serializing
> > > > > > > > > > > data,
> > > > > > > > > > > > > we have to make a choice in null values and many
> > times
> > > > > these
> > > > > > > are
> > > > > > > > > > > > translated
> > > > > > > > > > > > > to zero length blobs. This is not really the same
> > > thing.
> > > > > > > > > > > > >
> > > > > > > > > > > > > From this perspective, what can Kafka represent and
> > > what
> > > > is
> > > > > > > > > > considered
> > > > > > > > > > > a
> > > > > > > > > > > > > valid value?
> > > > > > > > > > > > >
> > > > > > > > > > > > > If the user sends a byte array of length 0, or if
> the
> > > > > > > serializer
> > > > > > > > > > sends
> > > > > > > > > > > > > something of length 0, this should be a valid
> value.
> > It
> > > > is
> > > > > > not
> > > > > > > > > > kafka's
> > > > > > > > > > > > job
> > > > > > > > > > > > > to determine what the user is trying to send. For
> all
> > > we
> > > > > > know,
> > > > > > > > the
> > > > > > > > > > user
> > > > > > > > > > > > has
> > > > > > > > > > > > > a really good compression serializer that sends 0
> > bytes
> > > > > when
> > > > > > > > > nothing
> > > > > > > > > > > has
> > > > > > > > > > > > > changed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If the user is allowed to send null then some
> > behavior
> > > > > should
> > > > > > > be
> > > > > > > > > > > defined.
> > > > > > > > > > > > > However, this should semantically be different than
> > > > > sending a
> > > > > > > > > > command.
> > > > > > > > > > > It
> > > > > > > > > > > > > is possible for a null value could signify some
> form
> > of
> > > > > > delete,
> > > > > > > > > like
> > > > > > > > > > > > > "delete all messages with this key". However, if
> > kafka
> > > > has
> > > > > a
> > > > > > > goal
> > > > > > > > > to
> > > > > > > > > > > > write
> > > > > > > > > > > > > good, readable code, then this should not be
> allowed.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A delete or a purge is a call that can have certain
> > > side
> > > > > > > effects
> > > > > > > > or
> > > > > > > > > > > > > encounter errors that are unrelated to a send call.
> > > > > > > > > > > > >
> > > > > > > > > > > > > A couple of bullets from the Kafka style guide (
> > > > > > > > > > > > > http://kafka.apache.org/coding-guide.html ):
> > > > > > > > > > > > >
> > > > > > > > > > > > > - Clear code is preferable to comments. When
> possible
> > > > make
> > > > > > your
> > > > > > > > > > naming
> > > > > > > > > > > so
> > > > > > > > > > > > > good you don't need comments.
> > > > > > > > > > > > > - Logging, configuration, and public APIs are our
> > "UI".
> > > > > Make
> > > > > > > them
> > > > > > > > > > > pretty,
> > > > > > > > > > > > > consistent, and usable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > If you want sendTombstone() functionality make the
> > > > protocol
> > > > > > > > reflect
> > > > > > > > > > > that.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Right now we're struggling because we did not have
> a
> > > > clear
> > > > > > > > > separation
> > > > > > > > > > > of
> > > > > > > > > > > > > concerns.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I don't have a good solution for deployment. I'm in
> > > favor
> > > > > of
> > > > > > a
> > > > > > > > > harsh
> > > > > > > > > > > > road:
> > > > > > > > > > > > > - Add the flag/variable/header for tombstone
> > > > > > > > > > > > > - Add an API for tombstone behavior if needed
> > > > > > > > > > > > > - Error if somebody tries to API send null
> > > > > > > > > > > > > - Accept if somebody tries to API send byte[]
> length
> > 0
> > > > > > > > > > > > > - Rev broker version
> > > > > > > > > > > > > - broker accept flag/variable/header as tombstone
> > > > > > > > > > > > > - broker accept zero length values as normal
> messages
> > > > > > > > > > > > >
> > > > > > > > > > > > > The more gentle road would be:
> > > > > > > > > > > > > - Add the flag/variable/header for tombstone
> > > > > > > > > > > > > - Add an API for tombstone behavior if needed
> > > > > > > > > > > > > - warn/deprecate/etc if somebody sends null
> > > > > > > > > > > > > - broker accepts flags/variable/headers and zero
> > length
> > > > > > values
> > > > > > > as
> > > > > > > > > > > > > tombstones
> > > > > > > > > > > > >
> > > > > > > > > > > > > Conversion from one to the other may be possible
> > but I
> > > > > would
> > > > > > > > > rather
> > > > > > > > > > > not
> > > > > > > > > > > > > have to do it.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Nacho
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Wed, Nov 9, 2016 at 9:03 PM, radai <
> > > > > > > > radai.rosenbl...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > my personal opinion - a log compacted topic is
> > > > basically
> > > > > a
> > > > > > > > > > kv-store,
> > > > > > > > > > > > so a
> > > > > > > > > > > > > > map API.
> > > > > > > > > > > > > > map.put(key, null) is not the same as
> > > map.remove(key),
> > > > > > which
> > > > > > > to
> > > > > > > > > me
> > > > > > > > > > > > means
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > null value should not represent a delete. a
> delete
> > > > should
> > > > > > be
> > > > > > > > > > explicit
> > > > > > > > > > > > > > (meaning flag).
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Nov 9, 2016 at 11:01 AM, Mayuresh Gharat
> <
> > > > > > > > > > > > > > gharatmayures...@gmail.com
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I see the reasoning and might be inclined to
> > agree
> > > a
> > > > > bit
> > > > > > :
> > > > > > > > > > > > > > > If we go to stage 2, the only difference is
> that
> > we
> > > > can
> > > > > > > > > > > theoretically
> > > > > > > > > > > > > > > support a null value non-tombstone message in a
> > log
> > > > > > > compacted
> > > > > > > > > > > topic,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > am not sure if that has any use case.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > But as an end goal I see that kafka should
> > clearly
> > > > > > specify
> > > > > > > > what
> > > > > > > > > > it
> > > > > > > > > > > > > means
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > a tombstone : is it the attribute flag OR is it
> > the
> > > > > null
> > > > > > > > value.
> > > > > > > > > > If
> > > > > > > > > > > we
> > > > > > > > > > > > > > just
> > > > > > > > > > > > > > > do stage 1, I don't think we are defining the
> > > > end-goal
> > > > > > > > > > completely.
> > > > > > > > > > > > > > > Again this is more about semantics of
> correctness
> > > of
> > > > > end
> > > > > > > > state.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Mayuresh
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Wed, Nov 9, 2016 at 10:49 AM, Becket Qin <
> > > > > > > > > > becket....@gmail.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I am not sure if we need the second stage.
> > > Wouldn't
> > > > > it
> > > > > > be
> > > > > > > > > > enough
> > > > > > > > > > > to
> > > > > > > > > > > > > say
> > > > > > > > > > > > > > > > that a message is a tombstone if one of the
> > > > following
> > > > > > is
> > > > > > > > > true?
> > > > > > > > > > > > > > > > 1. tombstone flag is set.
> > > > > > > > > > > > > > > > 2. value is null.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > If we go to stage 2, the only difference is
> > that
> > > we
> > > > > can
> > > > > > > > > > > > theoretically
> > > > > > > > > > > > > > > > support a null value non-tombstone message
> in a
> > > log
> > > > > > > > compacted
> > > > > > > > > > > > topic,
> > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > I
> > > > > > > > > > > > > > > > am not sure if that has any use case.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Wed, Nov 9, 2016 at 9:23 AM, Mayuresh
> > Gharat <
> > > > > > > > > > > > > > > > gharatmayures...@gmail.com>
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think it will be a good idea. +1
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Mayuresh
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Wed, Nov 9, 2016 at 9:13 AM, Michael
> > Pearce
> > > <
> > > > > > > > > > > > > > michael.pea...@ig.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > +1 Mayuresh, I think this is a good
> > > > > > > solution/strategy.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Shall we update the KIP with this?
> > > > > Becket/Jun/Joel
> > > > > > > any
> > > > > > > > > > > comments
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > > > before we do?
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On 08/11/2016, 17:29, "Mayuresh Gharat" <
> > > > > > > > > > > > > > gharatmayures...@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I think the migration can be done in
> 2
> > > > > stages :
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 1) In first stage the broker should
> > > > > understand
> > > > > > > the
> > > > > > > > > > > > attribute
> > > > > > > > > > > > > > flag
> > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > well
> > > > > > > > > > > > > > > > > > as Null for the value for log
> > compaction.
> > > > > > > > > > > > > > > > > > 2) In second stage we move on to
> > > supporting
> > > > > > only
> > > > > > > > the
> > > > > > > > > > > > > attribute
> > > > > > > > > > > > > > > flag
> > > > > > > > > > > > > > > > > > for log
> > > > > > > > > > > > > > > > > > compaction.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > I agree with Becket that for older
> > > clients
> > > > > > > > > (consumers)
> > > > > > > > > > > the
> > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > have to down convert a message that
> has
> > > the
> > > > > > > > attribute
> > > > > > > > > > > flag
> > > > > > > > > > > > > set
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > compacting but has a non null value.
> > But
> > > > this
> > > > > > > > should
> > > > > > > > > be
> > > > > > > > > > > in
> > > > > > > > > > > > > > first
> > > > > > > > > > > > > > > > > stage.
> > > > > > > > > > > > > > > > > > Once all the clients have upgraded
> > > (clients
> > > > > > start
> > > > > > > > > > > > recognizing
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > attribute
> > > > > > > > > > > > > > > > > > flag), we can move the broker to
> stage
> > 2.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Mayuresh
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > On Tue, Nov 8, 2016 at 12:06 AM,
> > Michael
> > > > > > Pearce <
> > > > > > > > > > > > > > > > > michael.pea...@ig.com
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Also we can add further guidance:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > To avoid the below caveat to
> > > > organisations
> > > > > > by
> > > > > > > > > > > promoting
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > upgrading all
> > > > > > > > > > > > > > > > > > > consumers first before relying on
> > > > producing
> > > > > > > > > tombstone
> > > > > > > > > > > > > > messages
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Sent using OWA for iPhone
> > > > > > > > > > > > > > > > > > > ______________________________
> > > __________
> > > > > > > > > > > > > > > > > > > From: Michael Pearce
> > > > > > > > > > > > > > > > > > > Sent: Tuesday, November 8, 2016
> > 8:03:32
> > > > AM
> > > > > > > > > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-87 - Add
> > > > > > Compaction
> > > > > > > > > > > Tombstone
> > > > > > > > > > > > > Flag
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks Jun on the feedback, I
> think I
> > > > > > > understand
> > > > > > > > > the
> > > > > > > > > > > > > > > issue/point
> > > > > > > > > > > > > > > > > now.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > We def can add that on older client
> > > > version
> > > > > > if
> > > > > > > > > > > tombstone
> > > > > > > > > > > > > > marker
> > > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > value null to preserve behaviour.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > There is one caveats to this:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > * we have to be clear that data is
> > lost
> > > > if
> > > > > > > > reading
> > > > > > > > > > via
> > > > > > > > > > > > old
> > > > > > > > > > > > > > > > > > client/message
> > > > > > > > > > > > > > > > > > > format - I don't think this is a
> big
> > > > issue
> > > > > as
> > > > > > > > > mostly
> > > > > > > > > > > the
> > > > > > > > > > > > > > > idea/use
> > > > > > > > > > > > > > > > > > case is
> > > > > > > > > > > > > > > > > > > around meta data transport as such
> > > would
> > > > > only
> > > > > > > be
> > > > > > > > as
> > > > > > > > > > bad
> > > > > > > > > > > > as
> > > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > situation
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Re having configurable broker this
> > was
> > > to
> > > > > > > handle
> > > > > > > > > > cases
> > > > > > > > > > > > like
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > described
> > > > > > > > > > > > > > > > > > > but in another way by allowing
> > > > organisation
> > > > > > > > choose
> > > > > > > > > > the
> > > > > > > > > > > > > > > behaviour
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > compaction per broker or per topic
> so
> > > > they
> > > > > > > could
> > > > > > > > > > manage
> > > > > > > > > > > > > their
> > > > > > > > > > > > > > > > > > transition to
> > > > > > > > > > > > > > > > > > > using tombstone markers.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On hind sight it maybe easier to
> just
> > > > > upgrade
> > > > > > > and
> > > > > > > > > > > > downgrade
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > messages
> > > > > > > > > > > > > > > > > > > on version as you propose.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Sent using OWA for iPhone
> > > > > > > > > > > > > > > > > > > ______________________________
> > > __________
> > > > > > > > > > > > > > > > > > > From: Jun Rao <j...@confluent.io>
> > > > > > > > > > > > > > > > > > > Sent: Tuesday, November 8, 2016
> > > 12:34:41
> > > > AM
> > > > > > > > > > > > > > > > > > > To: dev@kafka.apache.org
> > > > > > > > > > > > > > > > > > > Subject: Re: [DISCUSS] KIP-87 - Add
> > > > > > Compaction
> > > > > > > > > > > Tombstone
> > > > > > > > > > > > > Flag
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > For the use case, one potential use
> > > case
> > > > is
> > > > > > for
> > > > > > > > > > schema
> > > > > > > > > > > > > > > > > registration.
> > > > > > > > > > > > > > > > > > For
> > > > > > > > > > > > > > > > > > > example, in Avro, a null value
> > > > corresponds
> > > > > > to a
> > > > > > > > > Null
> > > > > > > > > > > > > schema.
> > > > > > > > > > > > > > > So,
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > want to be able to keep the schema
> id
> > > in
> > > > a
> > > > > > > delete
> > > > > > > > > > > > message,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > can't
> > > > > > > > > > > > > > > > > > > be null. We could get around this
> > issue
> > > > by
> > > > > > > > > > specializing
> > > > > > > > > > > > > null
> > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > during
> > > > > > > > > > > > > > > > > > > schema registration though.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Now for the proposed changes. We
> > > probably
> > > > > > > should
> > > > > > > > > > > preserve
> > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > compatibility. If a client
> > application
> > > is
> > > > > > > > sending a
> > > > > > > > > > > null
> > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > to a
> > > > > > > > > > > > > > > > > > > compacted topic, ideally, it should
> > > work
> > > > > the
> > > > > > > same
> > > > > > > > > > after
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > upgrades.
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > I am not sure about making the
> > > tombstone
> > > > > > marker
> > > > > > > > > > > > > configurable,
> > > > > > > > > > > > > > > > > > especially at
> > > > > > > > > > > > > > > > > > > the topic level. Should we allow
> > users
> > > to
> > > > > > > change
> > > > > > > > > the
> > > > > > > > > > > > config
> > > > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > > > back and
> > > > > > > > > > > > > > > > > > > forth, and what would be the
> > > implication?
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > On Mon, Nov 7, 2016 at 10:48 AM,
> > Becket
> > > > > Qin <
> > > > > > > > > > > > > > > > becket....@gmail.com>
> > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Hi Michael,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Yes, changing the logic in the
> log
> > > > > cleaner
> > > > > > > > makes
> > > > > > > > > > > sense.
> > > > > > > > > > > > > > There
> > > > > > > > > > > > > > > > > > could be
> > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > other thing worth thinking (e.g.
> > the
> > > > > > message
> > > > > > > > size
> > > > > > > > > > > > change
> > > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > > > > conversion),
> > > > > > > > > > > > > > > > > > > > though.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > The scenario I was thinking is
> the
> > > > > > following:
> > > > > > > > > > > > > > > > > > > > Imagine a distributed caching
> > system
> > > > > built
> > > > > > on
> > > > > > > > top
> > > > > > > > > > of
> > > > > > > > > > > > > > Kafka. A
> > > > > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > consuming from a topic and it is
> > > > > guaranteed
> > > > > > > > that
> > > > > > > > > if
> > > > > > > > > > > the
> > > > > > > > > > > > > > user
> > > > > > > > > > > > > > > > > > consume to
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > log end it will get the latest
> > value
> > > > for
> > > > > > all
> > > > > > > > the
> > > > > > > > > > > keys.
> > > > > > > > > > > > > > > > Currently
> > > > > > > > > > > > > > > > > > if the
> > > > > > > > > > > > > > > > > > > > consumer sees a null value it
> knows
> > > the
> > > > > key
> > > > > > > has
> > > > > > > > > > been
> > > > > > > > > > > > > > removed.
> > > > > > > > > > > > > > > > Now
> > > > > > > > > > > > > > > > > > let's
> > > > > > > > > > > > > > > > > > > say
> > > > > > > > > > > > > > > > > > > > we rolled out this change. And
> the
> > > > > producer
> > > > > > > > > > applies a
> > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > tombstone flag set, but the value
> > was
> > > > not
> > > > > > > null.
> > > > > > > > > > When
> > > > > > > > > > > we
> > > > > > > > > > > > > > > append
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > to the log I suppose we will not
> do
> > > the
> > > > > > down
> > > > > > > > > > > conversion
> > > > > > > > > > > > > if
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > broker has
> > > > > > > > > > > > > > > > > > > > set the message.format.version to
> > the
> > > > > > latest.
> > > > > > > > > > Because
> > > > > > > > > > > > the
> > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > cleaner
> > > > > > > > > > > > > > > > > > > won't
> > > > > > > > > > > > > > > > > > > > touch the active log segment, so
> > that
> > > > > > message
> > > > > > > > > will
> > > > > > > > > > be
> > > > > > > > > > > > > > sitting
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > active
> > > > > > > > > > > > > > > > > > > > segment as is. Now when a
> consumer
> > > that
> > > > > > > hasn't
> > > > > > > > > > > upgraded
> > > > > > > > > > > > > yet
> > > > > > > > > > > > > > > > > > consumes that
> > > > > > > > > > > > > > > > > > > > tombstone message in the active
> > > > segment,
> > > > > it
> > > > > > > > seems
> > > > > > > > > > > that
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > broker
> > > > > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > > need
> > > > > > > > > > > > > > > > > > > > to down convert that message to
> > > remove
> > > > > the
> > > > > > > > value,
> > > > > > > > > > > > right?
> > > > > > > > > > > > > In
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > case, we
> > > > > > > > > > > > > > > > > > > > cannot wait for the log cleaner
> to
> > do
> > > > the
> > > > > > > down
> > > > > > > > > > > > conversion
> > > > > > > > > > > > > > > > because
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > message may have already been
> > > consumed
> > > > > > before
> > > > > > > > the
> > > > > > > > > > log
> > > > > > > > > > > > > > > > compaction
> > > > > > > > > > > > > > > > > > happens.
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > On Mon, Nov 7, 2016 at 9:59 AM,
> > > Michael
> > > > > > > Pearce
> > > > > > > > <
> > > > > > > > > > > > > > > > > > michael.pea...@ig.com>
> > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Becket,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > We were thinking more about
> > having
> > > > the
> > > > > > > logic
> > > > > > > > > > that’s
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > method
> > > > > > > > > > > > > > > > > > > > > shouldRetainMessage
> configurable
> > > via
> > > > > > > > > > > > > > > > http://kafka.apache.org/
> > > > > > > > > > > > > > > > > > > > > documentation.html#
> brokerconfigs
> > > > at a
> > > > > > > > > > > broker/topic
> > > > > > > > > > > > > > level.
> > > > > > > > > > > > > > > > And
> > > > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > > scrap
> > > > > > > > > > > > > > > > > > > > > auto converting the message,
> and
> > > > allow
> > > > > > > > > > > organisations
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > > manage
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > rollout
> > > > > > > > > > > > > > > > > > > > > of enabling of the feature.
> > > > > > > > > > > > > > > > > > > > > (this isn’t in documentation
> but
> > in
> > > > > > > response
> > > > > > > > to
> > > > > > > > > > the
> > > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > > thread
> > > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > an alternative approach to roll
> > out
> > > > the
> > > > > > > > > feature)
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Does this make any more sense?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks
> > > > > > > > > > > > > > > > > > > > > Mike
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On 11/3/16, 2:27 PM, "Becket
> > Qin" <
> > > > > > > > > > > > > becket....@gmail.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Hi Michael,
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Do you mean using a new
> > > > > configuration
> > > > > > > it
> > > > > > > > is
> > > > > > > > > > > just
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > exiting
> > > > > > > > > > > > > > > > > > > > > message.format.version
> > config?
> > > It
> > > > > > seems
> > > > > > > > the
> > > > > > > > > > > > > > > > > > message.format.version
> > > > > > > > > > > > > > > > > > > > > config
> > > > > > > > > > > > > > > > > > > > > is enough in this case. And
> > the
> > > > > > default
> > > > > > > > > value
> > > > > > > > > > > > would
> > > > > > > > > > > > > > > > always
> > > > > > > > > > > > > > > > > > be the
> > > > > > > > > > > > > > > > > > > > > latest
> > > > > > > > > > > > > > > > > > > > > version.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Message version migration
> > > would
> > > > > be
> > > > > > > > > handled
> > > > > > > > > > as
> > > > > > > > > > > > > like
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > KIP-32
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Also just want to confirm
> on
> > > > this.
> > > > > > > Today
> > > > > > > > if
> > > > > > > > > > an
> > > > > > > > > > > > old
> > > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > > > consumes
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > > > compacted topic and sees an
> > > empty
> > > > > > > value,
> > > > > > > > it
> > > > > > > > > > > knows
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > is a
> > > > > > > > > > > > > > > > > > > > tombstone.
> > > > > > > > > > > > > > > > > > > > > After we start to use the
> > > > attribute
> > > > > > > bit,
> > > > > > > > a
> > > > > > > > > > > > > tombstone
> > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > have a
> > > > > > > > > > > > > > > > > > > > > non-empty value. So by
> "like
> > in
> > > > > > KIP-32"
> > > > > > > > you
> > > > > > > > > > > mean
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > will
> > > > > > > > > > > > > > > > > > remove the
> > > > > > > > > > > > > > > > > > > > > value
> > > > > > > > > > > > > > > > > > > > > to down convert the message
> > if
> > > > the
> > > > > > > > consumer
> > > > > > > > > > > > version
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > old,
> > > > > > > > > > > > > > > > > > right?
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Thanks.
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > On Wed, Nov 2, 2016 at 1:37
> > AM,
> > > > > > Michael
> > > > > > > > > > Pearce
> > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > michael.pea...@ig.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Hi Joel , et al.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Any comments on the below
> > > idea
> > > > to
> > > > > > > > handle
> > > > > > > > > > roll
> > > > > > > > > > > > > out /
> > > > > > > > > > > > > > > > > > compatibility
> > > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > feature, using a
> > > configuration?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Does it make sense/clear?
> > > > > > > > > > > > > > > > > > > > > > Does it add value?
> > > > > > > > > > > > > > > > > > > > > > Do we want to enforce
> flag
> > by
> > > > > > > default,
> > > > > > > > or
> > > > > > > > > > > value
> > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > default, or
> > > > > > > > > > > > > > > > > > > > both?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > > > > > > > > > Mike
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On 10/27/16, 4:47 PM,
> > > "Michael
> > > > > > > Pearce"
> > > > > > > > <
> > > > > > > > > > > > > > > > > > michael.pea...@ig.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Thanks, James, I
> think
> > > this
> > > > > is
> > > > > > a
> > > > > > > > > really
> > > > > > > > > > > > good
> > > > > > > > > > > > > > > > addition
> > > > > > > > > > > > > > > > > > to the
> > > > > > > > > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > details, please feel free
> > to
> > > > > amend
> > > > > > > the
> > > > > > > > > > > wiki/add
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > cases,
> > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > if any
> > > > > > > > > > > > > > > > > > > > > > others you think of. I
> > > > definitely
> > > > > > > think
> > > > > > > > > its
> > > > > > > > > > > > > > > worthwhile
> > > > > > > > > > > > > > > > > > > documenting.
> > > > > > > > > > > > > > > > > > > > > If you
> > > > > > > > > > > > > > > > > > > > > > can’t let me know ill add
> > > them
> > > > > next
> > > > > > > > week
> > > > > > > > > > > (just
> > > > > > > > > > > > > > > leaving
> > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > a long
> > > > > > > > > > > > > > > > > > > > > weekend
> > > > > > > > > > > > > > > > > > > > > > off)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Re Joel and others
> > > comments
> > > > > > about
> > > > > > > > > > upgrade
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > compatibility.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Rather than trying to
> > > auto
> > > > > > manage
> > > > > > > > > this.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Actually maybe we
> make
> > a
> > > > > > > > > configuration
> > > > > > > > > > > > > option,
> > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > at
> > > > > > > > > > > > > > > > > > server
> > > > > > > > > > > > > > > > > > > > > and per
> > > > > > > > > > > > > > > > > > > > > > topic level to control
> the
> > > > > behavior
> > > > > > > of
> > > > > > > > > how
> > > > > > > > > > > the
> > > > > > > > > > > > > > server
> > > > > > > > > > > > > > > > > logic
> > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > work out
> > > > > > > > > > > > > > > > > > > > > > if the record, is a
> > tombstone
> > > > > > record
> > > > > > > .
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > e.g.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > key =
> > > > > > compation.tombstone.marker
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > value options:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > value (continues to
> > use
> > > > > null
> > > > > > > > value
> > > > > > > > > as
> > > > > > > > > > > > > > tombstone
> > > > > > > > > > > > > > > > > > marker)
> > > > > > > > > > > > > > > > > > > > > > flag (expects to use
> > the
> > > > > > > tombstone
> > > > > > > > > > flag)
> > > > > > > > > > > > > > > > > > > > > > value_or_flag (if
> > either
> > > is
> > > > > > true
> > > > > > > it
> > > > > > > > > > > treats
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > record
> > > > > > > > > > > > > > > > > > as a
> > > > > > > > > > > > > > > > > > > > > tombstone)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This way on upgrade
> > users
> > > > can
> > > > > > > keep
> > > > > > > > > > > current
> > > > > > > > > > > > > > > > behavior,
> > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > slowly
> > > > > > > > > > > > > > > > > > > > > > migrate to the new.
> Having
> > a
> > > > > > > transition
> > > > > > > > > > > period
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > using
> > > > > > > > > > > > > > > > > > > > > value_or_flag,
> > > > > > > > > > > > > > > > > > > > > > finally having flag only
> if
> > > an
> > > > > > > > > organization
> > > > > > > > > > > > > wishes
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > null
> > > > > > > > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > > > > > > > without it being treated
> > as a
> > > > > > > tombstone
> > > > > > > > > > > marker
> > > > > > > > > > > > > (use
> > > > > > > > > > > > > > > > case
> > > > > > > > > > > > > > > > > > noted
> > > > > > > > > > > > > > > > > > > > below)
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Having it both global
> > > > broker
> > > > > > > level
> > > > > > > > > and
> > > > > > > > > > > > topic
> > > > > > > > > > > > > > > > override
> > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > allows some
> > > > > > > > > > > > > > > > > > > > > > flexibility here.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Cheers
> > > > > > > > > > > > > > > > > > > > > > Mike
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > On 10/27/16, 8:03 AM,
> > > > "James
> > > > > > > > Cheng" <
> > > > > > > > > > > > > > > > > > wushuja...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > This KIP would
> > > > definitely
> > > > > > > > > address a
> > > > > > > > > > > gap
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > > > > functionality, where you
> > > > > currently
> > > > > > > > can't
> > > > > > > > > > > have a
> > > > > > > > > > > > > > > > tombstone
> > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > any
> > > > > > > > > > > > > > > > > > > > > > associated content.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > That said, I'd
> like
> > > to
> > > > > talk
> > > > > > > > about
> > > > > > > > > > use
> > > > > > > > > > > > > > cases,
> > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > make sure
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > this is in fact useful.
> The
> > > KIP
> > > > > > > should
> > > > > > > > be
> > > > > > > > > > > > updated
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > whatever
> > > > > > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > > > > > cases
> > > > > > > > > > > > > > > > > > > > > > we come up with.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > First of all, an
> > > > > > observation:
> > > > > > > > > When
> > > > > > > > > > we
> > > > > > > > > > > > > speak
> > > > > > > > > > > > > > > > about
> > > > > > > > > > > > > > > > > > log
> > > > > > > > > > > > > > > > > > > > > compaction,
> > > > > > > > > > > > > > > > > > > > > > we typically think of
> "the
> > > > latest
> > > > > > > > message
> > > > > > > > > > > for a
> > > > > > > > > > > > > key
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > retained".
> > > > > > > > > > > > > > > > > > > > In
> > > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > > respect, a delete
> tombstone
> > > > > (i.e. a
> > > > > > > > > message
> > > > > > > > > > > > with
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > null
> > > > > > > > > > > > > > > > > > payload)
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > treated
> > > > > > > > > > > > > > > > > > > > > > the same as any other
> Kafka
> > > > > > message:
> > > > > > > > the
> > > > > > > > > > > latest
> > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > retained.
> > > > > > > > > > > > > > > > > > > > > It
> > > > > > > > > > > > > > > > > > > > > > doesn't matter whether
> the
> > > > latest
> > > > > > > > message
> > > > > > > > > > is
> > > > > > > > > > > > > null,
> > > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > latest
> > > > > > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > > > has actual content. In
> all
> > > > cases,
> > > > > > the
> > > > > > > > > last
> > > > > > > > > > > > > message
> > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > retained.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > The only way a
> > delete
> > > > > > > tombstone
> > > > > > > > > is
> > > > > > > > > > > > > treated
> > > > > > > > > > > > > > > > > > differently
> > > > > > > > > > > > > > > > > > > from
> > > > > > > > > > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > > > Kafka messages is that it
> > > > > > > automatically
> > > > > > > > > > > > > disappears
> > > > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > while.
> > > > > > > > > > > > > > > > > > > > > The time
> > > > > > > > > > > > > > > > > > > > > > of deletion is specified
> > > using
> > > > > > > > > > > > > delete.retention.ms
> > > > > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > So what we're
> > really
> > > > > > talking
> > > > > > > > > about
> > > > > > > > > > > is,
> > > > > > > > > > > > do
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > support
> > > > > > > > > > > > > > > > > > > > > > messages in a
> log-compacted
> > > > topic
> > > > > > > that
> > > > > > > > > > > > > auto-delete
> > > > > > > > > > > > > > > > > > themselves
> > > > > > > > > > > > > > > > > > > after
> > > > > > > > > > > > > > > > > > > > > a while?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > In a thread from
> > > 2015,
> > > > > > there
> > > > > > > > was
> > > > > > > > > a
> > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > first-class
> > > > > > > > > > > > > > > > > > > > > > support of headers
> between
> > > > Roger
> > > > > > > > Hoover,
> > > > > > > > > > > Felix
> > > > > > > > > > > > > GV,
> > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > Rao, and
> > > > > > > > > > > > > > > > > > > I.
> > > > > > > > > > > > > > > > > > > > > See
> > > > > > > > > > > > > > > > > > > > > > thread at
> > > > > > > https://groups.google.com/d/
> > > > > > > > > > > > > > > > > > msg/confluent-platform/
> > > > > > > > > > > > > > > > > > > > > > 8xPbjyUE_7E/yQ1AeCufL_gJ
> <
> > > > > > > > > > > > > > > https://groups.google.com/d/
> > > > > > > > > > > > > > > > > > > > > > msg/confluent-platform/
> > > > > > > > > > > > 8xPbjyUE_7E/yQ1AeCufL_gJ>
> > > > > > > > > > > > > .
> > > > > > > > > > > > > > > In
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > thread,
> > > > > > > > > > > > > > > > > > > > > Jun
> > > > > > > > > > > > > > > > > > > > > > raised a good question
> > that I
> > > > > > didn't
> > > > > > > > > have a
> > > > > > > > > > > > good
> > > > > > > > > > > > > > > answer
> > > > > > > > > > > > > > > > > > for at
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > time: If
> > > > > > > > > > > > > > > > > > > > > > a message is going to
> > > > auto-delete
> > > > > > > > itself
> > > > > > > > > > > after
> > > > > > > > > > > > a
> > > > > > > > > > > > > > > while,
> > > > > > > > > > > > > > > > > how
> > > > > > > > > > > > > > > > > > > > > important was
> > > > > > > > > > > > > > > > > > > > > > the message? That is,
> what
> > > > > > > information
> > > > > > > > > did
> > > > > > > > > > > the
> > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > contain
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > > > > > important *for a while*
> but
> > > not
> > > > > so
> > > > > > > > > > important
> > > > > > > > > > > > that
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > needed to be
> > > > > > > > > > > > > > > > > > > > > kept
> > > > > > > > > > > > > > > > > > > > > > around forever?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Some use cases
> > that I
> > > > can
> > > > > > > think
> > > > > > > > > of:
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > 1) Tracability. I
> > > would
> > > > > > like
> > > > > > > to
> > > > > > > > > > know
> > > > > > > > > > > > who
> > > > > > > > > > > > > > > issued
> > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > delete
> > > > > > > > > > > > > > > > > > > > > > tombstone. It might
> include
> > > the
> > > > > > > > hostname,
> > > > > > > > > > IP
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > producer of
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > delete.
> > > > > > > > > > > > > > > > > > > > > > 2) Timestamps. I
> > > would
> > > > > like
> > > > > > > to
> > > > > > > > > know
> > > > > > > > > > > > when
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > delete was
> > > > > > > > > > > > > > > > > > > > > issued.
> > > > > > > > > > > > > > > > > > > > > > This use case is already
> > > > > addressed
> > > > > > by
> > > > > > > > the
> > > > > > > > > > > > > > > availability
> > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > per-message
> > > > > > > > > > > > > > > > > > > > > > timestamps that came in
> > > 0.10.0
> > > > > > > > > > > > > > > > > > > > > > 3) Data
> > provenance. I
> > > > > hope
> > > > > > > I'm
> > > > > > > > > > using
> > > > > > > > > > > > this
> > > > > > > > > > > > > > > > phrase
> > > > > > > > > > > > > > > > > > > correctly,
> > > > > > > > > > > > > > > > > > > > > but
> > > > > > > > > > > > > > > > > > > > > > what I mean is, where did
> > > this
> > > > > > delete
> > > > > > > > > come
> > > > > > > > > > > > from?
> > > > > > > > > > > > > > What
> > > > > > > > > > > > > > > > > > processing
> > > > > > > > > > > > > > > > > > > > job
> > > > > > > > > > > > > > > > > > > > > > emitted it? What input to
> > the
> > > > > > > > processing
> > > > > > > > > > job
> > > > > > > > > > > > > caused
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > delete
> > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > > > produced? For example,
> if a
> > > > > record
> > > > > > in
> > > > > > > > > > topic A
> > > > > > > > > > > > was
> > > > > > > > > > > > > > > > > > processed and
> > > > > > > > > > > > > > > > > > > > > caused a
> > > > > > > > > > > > > > > > > > > > > > delete tombstone to be
> > > emitted
> > > > to
> > > > > > > topic
> > > > > > > > > B,
> > > > > > > > > > I
> > > > > > > > > > > > > might
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > offset
> > > > > > > > > > > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > > > > > > topic A message to be
> > > attached
> > > > to
> > > > > > the
> > > > > > > > > > topic B
> > > > > > > > > > > > > > > message.
> > > > > > > > > > > > > > > > > > > > > > 4) Distributed
> > > tracing
> > > > > for
> > > > > > > > stream
> > > > > > > > > > > > > > topologies.
> > > > > > > > > > > > > > > > > This
> > > > > > > > > > > > > > > > > > might
> > > > > > > > > > > > > > > > > > > > be a
> > > > > > > > > > > > > > > > > > > > > > slight repeat of the
> above
> > > use
> > > > > > cases.
> > > > > > > > In
> > > > > > > > > > the
> > > > > > > > > > > > > > > > > microservices
> > > > > > > > > > > > > > > > > > world,
> > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > can
> > > > > > > > > > > > > > > > > > > > > > generate call-graphs of
> > > > > webservices
> > > > > > > > using
> > > > > > > > > > > tools
> > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > Zipkin/
> > > > > > > > > > > > > > > > > > > > > opentracing.io
> > > > > > > > > > > > > > > > > > > > > > <http://opentracing.io/
> >,
> > or
> > > > > > > something
> > > > > > > > > > > > homegrown
> > > > > > > > > > > > > > > like
> > > > > > > > > > > > > > > > > > > > > >
> > https://engineering.linkedin
> > > .
> > > > > > > > > > > > > > > > > com/distributed-service-call-
> > > > > > > > > > > > > > > > > > > > > >
> > graph/real-time-distributed-
> > > > > > > > > > > > > > > > tracing-website-performance-
> > > > > > > > > > > > > > > > > > > > and-efficiency
> > > > > > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > >
> > https://engineering.linkedin
> > > .
> > > > > > > > > > > > > > > > > com/distributed-service-call-
> > > > > > > > > > > > > > > > > > > > > >
> > graph/real-time-distributed-
> > > > > > > > > > > > > > > > tracing-website-performance-
> > > > > > > > > > > > > > > > > > > > > and-efficiency>.
> > > > > > > > > > > > > > > > > > > > > > I can imagine that you
> > might
> > > > want
> > > > > > to
> > > > > > > do
> > > > > > > > > > > > something
> > > > > > > > > > > > > > > > similar
> > > > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > > > > stream
> > > > > > > > > > > > > > > > > > > > > > processing topologies,
> > where
> > > > > stream
> > > > > > > > > > > processing
> > > > > > > > > > > > > jobs
> > > > > > > > > > > > > > > > carry
> > > > > > > > > > > > > > > > > > along
> > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > forward
> > > > > > > > > > > > > > > > > > > > > > along a globally unique
> > > > > identifier,
> > > > > > > > and a
> > > > > > > > > > > > > > distributed
> > > > > > > > > > > > > > > > > > topology
> > > > > > > > > > > > > > > > > > > > graph
> > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > generated.
> > > > > > > > > > > > > > > > > > > > > > 5) Cases where
> > > > > processing a
> > > > > > > > > delete
> > > > > > > > > > > > > requires
> > > > > > > > > > > > > > > > data
> > > > > > > > > > > > > > > > > > that is
> > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > available in the message
> > key.
> > > > I'm
> > > > > > not
> > > > > > > > > sure
> > > > > > > > > > I
> > > > > > > > > > > > > have a
> > > > > > > > > > > > > > > > good
> > > > > > > > > > > > > > > > > > example
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > this,
> > > > > > > > > > > > > > > > > > > > > > though. One hand-wavy
> > example
> > > > > might
> > > > > > > be
> > > > > > > > > > where
> > > > > > > > > > > I
> > > > > > > > > > > > am
> > > > > > > > > > > > > > > > > > publishing
> > > > > > > > > > > > > > > > > > > > > documents into
> > > > > > > > > > > > > > > > > > > > > > Kafka where the
> documentId
> > is
> > > > the
> > > > > > > > message
> > > > > > > > > > > key,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > text
> > > > > > > > > > > > > > > > > > > > contents
> > > > > > > > > > > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > > > > > > document are in the
> message
> > > > body.
> > > > > > > And I
> > > > > > > > > > have
> > > > > > > > > > > a
> > > > > > > > > > > > > > > > consuming
> > > > > > > > > > > > > > > > > > job that
> > > > > > > > > > > > > > > > > > > > > does some
> > > > > > > > > > > > > > > > > > > > > > analytics on the message
> > > body.
> > > > If
> > > > > > > that
> > > > > > > > > > > document
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > > > deleted,
> > > > > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > consuming job might need
> > the
> > > > > > original
> > > > > > > > > > message
> > > > > > > > > > > > > body
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > order to
> > > > > > > > > > > > > > > > > > > > > "delete"
> > > > > > > > > > > > > > > > > > > > > > that message's impact
> from
> > > the
> > > > > > > > analytics.
> > > > > > > > > > But
> > > > > > > > > > > > I'm
> > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > great
> > > > > > > > > > > > > > > > > > > > > > example. If the consumer
> > was
> > > > > > worried
> > > > > > > > > about
> > > > > > > > > > > > that,
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > consumer
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > probably keep the
> original
> > > > > message
> > > > > > > > > around,
> > > > > > > > > > > > stored
> > > > > > > > > > > > > > by
> > > > > > > > > > > > > > > > > > primary key.
> > > > > > > > > > > > > > > > > > > > > And then
> > > > > > > > > > > > > > > > > > > > > > all it would need from a
> > > delete
> > > > > > > message
> > > > > > > > > > would
> > > > > > > > > > > > be
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > primary key
> > > > > > > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > message.
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > Do people think
> > these
> > > > are
> > > > > > > valid
> > > > > > > > > use
> > > > > > > > > > > > > cases?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > What are other
> use
> > > > cases
> > > > > > that
> > > > > > > > > > people
> > > > > > > > > > > > can
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > > of?
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > -James
> > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Oct 26,
> 2016,
> > at
> > > > > 3:46
> > > > > > > PM,
> > > > > > > > > > > Mayuresh
> > > > > > > > > > > > > > > Gharat
> > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > >
> gharatmayures...@gmail.com
> > >
> > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > +1 @Joel.
> > > > > > > > > > > > > > > > > > > > > > > I think a clear
> > > > > migration
> > > > > > > > plan
> > > > > > > > > of
> > > > > > > > > > > > > > upgrading
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > downgrading of
> > > > > > > > > > > > > > > > > > > > > > server and
> > > > > > > > > > > > > > > > > > > > > > > clients along
> > with
> > > > > > handling
> > > > > > > > of
> > > > > > > > > > > issues
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > Joel
> > > > > > > > > > > > > > > > > > > > mentioned,
> > > > > > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > > > > > the KIP would
> > > > > > > > > > > > > > > > > > > > > > > be really
> great.
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > Mayuresh
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > > On Wed, Oct 26,
> > > 2016
> > > > at
> > > > > > > 3:31
> > > > > > > > > PM,
> > > > > > > > > > > Joel
> > > > > > > > > > > > > > > Koshy <
> > > > > > > > > > > > > > > > > > > > > jjkosh...@gmail.com>
> > > > > > > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > > > > > >> I'm not sure
> why
> > > it
> > > > > > would
> > > > > > > be
> > > > > > > > > > > useful,
> > > > > > > > > > > > > but
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > should be
> > > > > > > > > > > > > > > > > > > > > > theoretically
> > > > > > > > > > > > > > > > > > > > > > >> possible if
> the
> > > > > > attribute
> > > > > > > > bit
> > > > > > > > > > > alone
> > > > > > > > > > > > is
> > > > > > > > > > > > > > > > enough
> > > > > > > > > > > > > > > > > > to mark
> > > > > > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > > > > > > tombstone. OTOH, we
> > > > > > > > > > > > > > > > > > > > > > >> could consider
> > > that
> > > > as
> > > > > > > > invalid
> > > > > > > > > > if
> > > > > > > > > > > we
> > > > > > > > > > > > > > wish.
> > > > > > > > > > > > > > > > > > These are
> > > > > > > > > > > > > > > > > > > > > relevant
> > > > > > > > > > > > > > > > > > > > > > details that
> > > > > > > > > > > > > > > > > > > > > > >> I think should
> > be
> > > > > added
> > > > > > to
> > > > > > > > the
> > > > > > > > > > > KIP.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> Also, in the
> few
> > > odd
> > > > > > > > scenarios
> > > > > > > > > > > that
> > > > > > > > > > > > I
> > > > > > > > > > > > > > > > > mentioned
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > > > > > > consider
> > > > > > > > > > > > > > > > > > > > > > >> that fetches
> > could
> > > > be
> > > > > > > coming
> > > > > > > > > > from
> > > > > > > > > > > > > other
> > > > > > > > > > > > > > > > > > > > yet-to-be-upgraded
> > > > > > > > > > > > > > > > > > > > > > brokers in a
> > > > > > > > > > > > > > > > > > > > > > >> cluster that
> is
> > > > being
> > > > > > > > > upgraded.
> > > > > > > > > > So
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > want
> > > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > continue to
> > > > > > > > > > > > > > > > > > > > > > >> support nulls
> as
> > > > > > > tombstones
> > > > > > > > or
> > > > > > > > > > > > > > > down-convert
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > a way
> > > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > > > > are sure works
> > > > > > > > > > > > > > > > > > > > > > >> with least
> > > surprise
> > > > to
> > > > > > > > > fetchers.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> There is a
> > > slightly
> > > > > > vague
> > > > > > > > > > > statement
> > > > > > > > > > > > > > under
> > > > > > > > > > > > > > > > > > > > "Compatibility,
> > > > > > > > > > > > > > > > > > > > > > Deprecation, and
> > > > > > > > > > > > > > > > > > > > > > >> Migration
> Plan"
> > > that
> > > > > > could
> > > > > > > > > > benefit
> > > > > > > > > > > > > more
> > > > > > > > > > > > > > > > > details:
> > > > > > > > > > > > > > > > > > > *Logic
> > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > base on
> > > > > > > > > > > > > > > > > > > > > > >> current
> behavior
> > > of
> > > > > null
> > > > > > > > value
> > > > > > > > > > or
> > > > > > > > > > > if
> > > > > > > > > > > > > > > > tombstone
> > > > > > > > > > > > > > > > > > flag
> > > > > > > > > > > > > > > > > > > set
> > > > > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > > > > > true, as such
> > > > > > > > > > > > > > > > > > > > > > >> wouldn't
> impact
> > > any
> > > > > > > existing
> > > > > > > > > > flows
> > > > > > > > > > > > > > simply
> > > > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > new
> > > > > > > > > > > > > > > > > > > > > producers
> > > > > > > > > > > > > > > > > > > > > > to make use
> > > > > > > > > > > > > > > > > > > > > > >> of the
> feature*.
> > > It
> > > > is
> > > > > > > > unclear
> > > > > > > > > > to
> > > > > > > > > > > me
> > > > > > > > > > > > > > based
> > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > whether you
> > > > > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > > >> interpret null
> > as
> > > a
> > > > > > > > tombstone
> > > > > > > > > if
> > > > > > > > > > > the
> > > > > > > > > > > > > > > > tombstone
> > > > > > > > > > > > > > > > > > > attribute
> > > > > > > > > > > > > > > > > > > > > bit is
> > > > > > > > > > > > > > > > > > > > > > off.
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >> On Wed, Oct
> 26,
> > > 2016
> > > > > at
> > > > > > > 3:10
> > > > > > > > > PM,
> > > > > > > > > > > > > Xavier
> > > > > > > > > > > > > > > > > Léauté <
> > > > > > > > > > > > > > > > > > > > > > xav...@confluent.io>
> > > > > > > > > > > > > > > > > > > > > > >> wrote:
> > > > > > > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > > > > > > >>> Does this
> mean
> > > that
> > > > > > > > starting
> > > > > > > > > > with
> > > > > > > > > > > > V4
> > > > > > > > > > > > > > > > requests
> > > > > > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > > > > allow
> > > > > > > > > > > > > > > > > > > > > > storing null
> > > > > > > > > > > > > > > > > > > > > > >>> messages in
> > > > compacted
> > > > > > > > topics?
> > > > > > > > > > The
> > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > should
> > > > > > > > > > > > > > > > > > probably
> > > > > > > > > > > > > > > > > > > > > clarify
> > > > > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > > >> behavior
> > > > > > > > > > > > > > > > > > > > > > >>> for null
> > messages
> > > > > where
> > > > > > > the
> > > > > > > > > > > > tombstone
> > > > > > > > > > > > > > > flag
> > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > net.
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>> On Wed, Oct
> 26,
> > > > 2016
> > > > > at
> > > > > > > > 1:32
> > > > > > > > > AM
> > > > > > > > > > > > > Magnus
> > > > > > > > > > > > > > > > > > Edenhill <
> > > > > > > > > > > > > > > > > > > > > > mag...@edenhill.se>
> > > > > > > > > > > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > > > > > > >>>> 2016-10-25
> > 21:36
> > > > > > > GMT+02:00
> > > > > > > > > > Nacho
> > > > > > > > > > > > > Solis
> > > > > > > > > > > > > > > > > > > > > >
> > <nso...@linkedin.com.invalid
> > > >:
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>> I think you
> > > > > probably
> > > > > > > > > require
> > > > > > > > > > a
> > > > > > > > > > > > > > > MagicByte
> > > > > > > > > > > > > > > > > > bump if
> > > > > > > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > > > > > > > expect
> > > > > > > > > > > > > > > > > > > > > > correct
> > > > > > > > > > > > > > > > > > > > > > >>>>> behavior of
> > the
> > > > > > system
> > > > > > > > as a
> > > > > > > > > > > > whole.
> > > > > > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>> From a
> client
> > > > > > > perspective
> > > > > > > > > you
> > > > > > > > > > > > want
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > make
> > > > > > > > > > > > > > > > > > sure
> > > > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > > > > > when you
> > > > > > > > > > > > > > > > > > > > > > >> deliver a
> > > > > > > > > > > > > > > > > > > > > > >>>>> message
> that
> > > the
> > > > > > broker
> > > > > > > > > > > supports
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > feature
> > > > > > > > > > > > > > > > > > you're
> > > > > > > > > > > > > > > > > > > > > expecting
> > > > > > > > > > > > > > > > > > > > > > >>>>>
> (compaction).
> > > > So,
> > > > > > > > > depending
> > > > > > > > > > on
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > behavior
> > > > > > > > > > > > > > > > > > of the
> > > > > > > > > > > > > > > > > > > > > broker on
> > > > > > > > > > > > > > > > > > > > > > >>>> encountering
> > > > > > > > > > > > > > > > > > > > > > >>>>> a
> previously
> > > > > > undefined
> > > > > > > > bit
> > > > > > > > > > > flag I
> > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > suggest
> > > > > > > > > > > > > > > > > > > > making
> > > > > > > > > > > > > > > > > > > > > some
> > > > > > > > > > > > > > > > > > > > > > change to
> > > > > > > > > > > > > > > > > > > > > > >>>> make
> > > > > > > > > > > > > > > > > > > > > > >>>>> certain
> that
> > > > > > flag-based
> > > > > > > > > > > > compaction
> > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > supported.
> > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > going
> > > > > > > > > > > > > > > > > > > > > > to guess
> > > > > > > > > > > > > > > > > > > > > > >>> that
> > > > > > > > > > > > > > > > > > > > > > >>>>> the
> MagicByte
> > > > would
> > > > > > do
> > > > > > > > > this.
> > > > > > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>> I dont
> believe
> > > > this
> > > > > is
> > > > > > > > > needed
> > > > > > > > > > > > since
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > > > > > attributed
> > > > > > > > > > > > > > > > > > > > > > through
> > > > > > > > > > > > > > > > > > > > > > >> the
> > > > > > > > > > > > > > > > > > > > > > >>>> request's
> API
> > > > > version.
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>> Producer:
> > > > > > > > > > > > > > > > > > > > > > >>>> * if a
> client
> > > > sends
> > > > > > > > > > > ProduceRequest
> > > > > > > > > > > > > V4
> > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > > > attributes.bit5
> > > > > > > > > > > > > > > > > > > > > > indicates a
> > > > > > > > > > > > > > > > > > > > > > >>>> tombstone
> > > > > > > > > > > > > > > > > > > > > > >>>> * if a
> clients
> > > > sends
> > > > > > > > > > > > ProduceRequest
> > > > > > > > > > > > > > <V4
> > > > > > > > > > > > > > > > then
> > > > > > > > > > > > > > > > > > > > > attributes.bit5
> > > > > > > > > > > > > > > > > > > > > > is
> > > > > > > > > > > > > > > > > > > > > > >> ignored
> > > > > > > > > > > > > > > > > > > > > > >>>> and
> > value==null
> > > > > > > indicates
> > > > > > > > a
> > > > > > > > > > > > > tombstone
> > > > > > > > > > > > > > > > > > > > > > >>>> * in both
> > cases
> > > > the
> > > > > > > > on-disk
> > > > > > > > > > > > messages
> > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > stored with
> > > > > > > > > > > > > > > > > > > > > > attributes.bit5
> > > > > > > > > > > > > > > > > > > > > > >> (I
> > > > > > > > > > > > > > > > > > > > > > >>>> assume?)
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>> Consumer:
> > > > > > > > > > > > > > > > > > > > > > >>>> * if a
> clients
> > > > sends
> > > > > > > > > > > FetchRequest
> > > > > > > > > > > > V4
> > > > > > > > > > > > > > > > > messages
> > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > > sendfile():ed
> > > > > > > > > > > > > > > > > > > > > > >> directly
> > > > > > > > > > > > > > > > > > > > > > >>>> from disk
> > (with
> > > > > > > > > > attributes.bit5)
> > > > > > > > > > > > > > > > > > > > > > >>>> * if a
> client
> > > > sends
> > > > > > > > > > FetchRequest
> > > > > > > > > > > > <V4
> > > > > > > > > > > > > > > > > messages
> > > > > > > > > > > > > > > > > > are
> > > > > > > > > > > > > > > > > > > > > slowpathed
> > > > > > > > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > > > > > >>>> translated
> > from
> > > > > > > > > > attributes.bit5
> > > > > > > > > > > to
> > > > > > > > > > > > > > > > > value=null
> > > > > > > > > > > > > > > > > > as
> > > > > > > > > > > > > > > > > > > > > required.
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>> That's my
> > > > > > understanding
> > > > > > > > > > anyway,
> > > > > > > > > > > > > please
> > > > > > > > > > > > > > > > > > correct me if
> > > > > > > > > > > > > > > > > > > > I'm
> > > > > > > > > > > > > > > > > > > > > > wrong.
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>> /Magnus
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>> On Tue, Oct
> > 25,
> > > > > 2016
> > > > > > at
> > > > > > > > > 10:17
> > > > > > > > > > > AM,
> > > > > > > > > > > > > > > Magnus
> > > > > > > > > > > > > > > > > > Edenhill <
> > > > > > > > > > > > > > > > > > > > > > >>
> > > mag...@edenhill.se>
> > > > > > > > > > > > > > > > > > > > > > >>>>> wrote:
> > > > > > > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>> It is safe
> > to
> > > > > assume
> > > > > > > > that
> > > > > > > > > a
> > > > > > > > > > > > > > previously
> > > > > > > > > > > > > > > > > > undefined
> > > > > > > > > > > > > > > > > > > > > attributes
> > > > > > > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > > > > > > >> will
> > > > > > > > > > > > > > > > > > > > > > >>> be
> > > > > > > > > > > > > > > > > > > > > > >>>>>> unset in
> > > > protocol
> > > > > > > > requests
> > > > > > > > > > > from
> > > > > > > > > > > > > > > existing
> > > > > > > > > > > > > > > > > > clients,
> > > > > > > > > > > > > > > > > > > if
> > > > > > > > > > > > > > > > > > > > > not,
> > > > > > > > > > > > > > > > > > > > > > such a
> > > > > > > > > > > > > > > > > > > > > > >>> client
> > > > > > > > > > > > > > > > > > > > > > >>>>> is
> > > > > > > > > > > > > > > > > > > > > > >>>>>> already
> > > > violating
> > > > > > the
> > > > > > > > > > protocol
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > needs
> > > > > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > fixed.
> > > > > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>> So I dont
> > see
> > > a
> > > > > need
> > > > > > > > for a
> > > > > > > > > > > > > MagicByte
> > > > > > > > > > > > > > > > bump,
> > > > > > > > > > > > > > > > > > both
> > > > > > > > > > > > > > > > > > > > > broker and
> > > > > > > > > > > > > > > > > > > > > > client
> > > > > > > > > > > > > > > > > > > > > > >> has
> > > > > > > > > > > > > > > > > > > > > > >>>> the
> > > > > > > > > > > > > > > > > > > > > > >>>>>>
> information
> > it
> > > > > needs
> > > > > > > to
> > > > > > > > > > > > construct
> > > > > > > > > > > > > or
> > > > > > > > > > > > > > > > parse
> > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > message
> > > > > > > > > > > > > > > > > > > > > > according to
> > > > > > > > > > > > > > > > > > > > > > >>>>> request
> > > > > > > > > > > > > > > > > > > > > > >>>>>> version.
> > > > > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>> 2016-10-25
> > > 18:48
> > > > > > > > GMT+02:00
> > > > > > > > > > > > Michael
> > > > > > > > > > > > > > > > Pearce
> > > > > > > > > > > > > > > > > <
> > > > > > > > > > > > > > > > > > > > > > michael.pea...@ig.com>:
> > > > > > > > > > > > > > > > > > > > > > >>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> Hi
> Magnus,
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> I was
> > > wondering
> > > > > if
> > > > > > I
> > > > > > > > even
> > > > > > > > > > > > needed
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > change
> > > > > > > > > > > > > > > > > > those
> > > > > > > > > > > > > > > > > > > > > also, as
> > > > > > > > > > > > > > > > > > > > > > >>> technically
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> we’re
> just
> > > > making
> > > > > > use
> > > > > > > > of
> > > > > > > > > a
> > > > > > > > > > > non
> > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > attribute
> > > > > > > > > > > > > > > > > > > bit,
> > > > > > > > > > > > > > > > > > > > > but im
> > > > > > > > > > > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > > > > > > >> 100%
> > > > > > > > > > > > > > > > > > > > > > >>>> that
> > > > > > > > > > > > > > > > > > > > > > >>>>>> it
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> be always
> > > false
> > > > > > > > > currently.
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> If
> someone
> > > can
> > > > > say
> > > > > > > 100%
> > > > > > > > > it
> > > > > > > > > > > will
> > > > > > > > > > > > > > > already
> > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > set
> > > > > > > > > > > > > > > > > > > > false
> > > > > > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > > > > > > current
> > > > > > > > > > > > > > > > > > > > > > >>> and
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> historic
> > bit
> > > > wise
> > > > > > > > masking
> > > > > > > > > > > > > > techniques
> > > > > > > > > > > > > > > > used
> > > > > > > > > > > > > > > > > > over
> > > > > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > > > > > time,
> > > > > > > > > > > > > > > > > > > > > > we could
> > > > > > > > > > > > > > > > > > > > > > >>> do
> > > > > > > > > > > > > > > > > > > > > > >>>>> away
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> with
> both,
> > > and
> > > > > > simply
> > > > > > > > > just
> > > > > > > > > > > > start
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > use
> > > > > > > > > > > > > > > > > it.
> > > > > > > > > > > > > > > > > > > > > Unfortunately
> > > > > > > > > > > > > > > > > > > > > > I don’t
> > > > > > > > > > > > > > > > > > > > > > >>>> have
> > > > > > > > > > > > > > > > > > > > > > >>>>>> that
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> historic
> > > > > knowledge
> > > > > > so
> > > > > > > > was
> > > > > > > > > > > > hoping
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > would
> > > > > > > > > > > > > > > > > > be
> > > > > > > > > > > > > > > > > > > > flagged
> > > > > > > > > > > > > > > > > > > > > up in
> > > > > > > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > > > > > > >>>>>> discussion
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> thread ?
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> Cheers
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> Mike
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> On
> > 10/25/16,
> > > > 5:36
> > > > > > PM,
> > > > > > > > > > "Magnus
> > > > > > > > > > > > > > > > Edenhill" <
> > > > > > > > > > > > > > > > > > > > > > mag...@edenhill.se>
> > > > > > > > > > > > > > > > > > > > > > >>> wrote:
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> Hi
> > > Michael,
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> With
> the
> > > > > version
> > > > > > > > bumps
> > > > > > > > > > for
> > > > > > > > > > > > > > Produce
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > Fetch
> > > > > > > > > > > > > > > > > > > > > requests,
> > > > > > > > > > > > > > > > > > > > > > do you
> > > > > > > > > > > > > > > > > > > > > > >>>>> really
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> need
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> to
> bump
> > > > > > MagicByte
> > > > > > > > too?
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> Regards,
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> Magnus
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > 2016-10-25
> > > > > 18:09
> > > > > > > > > > GMT+02:00
> > > > > > > > > > > > > > Michael
> > > > > > > > > > > > > > > > > > Pearce <
> > > > > > > > > > > > > > > > > > > > > > >>>
> > > > > michael.pea...@ig.com
> > > > > > > > > > > > > > > > > > > > > > >>>>> :
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>> Hi All,
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>> I would
> > like
> > > > to
> > > > > > > > discuss
> > > > > > > > > > the
> > > > > > > > > > > > > > > following
> > > > > > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > proposal:
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > https://cwiki.apache.org/
> > > > > > > > > > > > > > > > > > > > > confluence/display/KAFKA/KIP-
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > 87+-+Add+Compaction+Tombstone+
> > > > > > > > > > > > > > Flag
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>> This is
> > off
> > > > the
> > > > > > back
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > > > discussion
> > > > > > > > > > > > > > > > > on
> > > > > > > > > > > > > > > > > > KIP-82
> > > > > > > > > > > > > > > > > > > > /
> > > > > > > > > > > > > > > > > > > > > KIP
> > > > > > > > > > > > > > > > > > > > > > >>> meeting
> > > > > > > > > > > > > > > > > > > > > > >>>>>>> where it
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>> was
> agreed
> > > to
> > > > > > > separate
> > > > > > > > > > this
> > > > > > > > > > > > > issue
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > > feature.
> > > > > > > > > > > > > > > > > > > > See:
> > > > > > > > > > > > > > > > > > > > > > >>>>>>>>
> > > > > > > > > > http://mail-archives.apache
> > > > > > > > > > > .
> > > > > > > > > > > > > > > > > > > > > org/mod_mbox/kafka-dev/201610
> > > > > > > > > > > > > > >
> ...
>
> [Message clipped]
--
-Regards,
Mayuresh R. Gharat
(862) 250-7125
