+1 Michael. Thanks,
Mayuresh On Thu, Nov 10, 2016 at 4:03 AM, Michael Pearce <michael.pea...@ig.com> wrote: > I have finally got around to updating the KIP from some of the bits so far > discussed in this discussion thread. > Please re-read. > > I have added the extra uses cases over and above those already detailed > during kip-82’s discussion, mentioned by James (thank you) > I have add’d some clarity over what occurs when tombstone marker is set > but topic is not a compacted topic (essentially it is kept, but the broker > takes no action) > Updated the migration stategy based on the 2 phase approach suggested by > Mayuresh. > @Mayuresh, please feel free to amend the KIP in the migration strategy > part, if I have completely miss-understood your 2 phased approach. > > Cheers > Mike > > > On 10/11/2016, 11:22, "Michael Pearce" <michael.pea...@ig.com> wrote: > > Agree on this point by Raidai, im happy having a two stage roll out a > suggested by yourself Mayuresh, so for a period we have both as obviously a > transition period will be required. > But I think the end goal is that we should just be reliant on a > tombstone marker eventually. (as in remove method denoted by Raidai) > > Also we should default new installs to have them set by default on > stage 2. > > On 10/11/2016, 05:03, "radai" <radai.rosenbl...@gmail.com> wrote: > > my personal opinion - a log compacted topic is basically a > kv-store, so a > map API. > map.put(key, null) is not the same as map.remove(key), which to me > means a > null value should not represent a delete. a delete should be > explicit > (meaning flag). > > > On Wed, Nov 9, 2016 at 11:01 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > I see the reasoning and might be inclined to agree a bit : > > If we go to stage 2, the only difference is that we can > theoretically > > support a null value non-tombstone message in a log compacted > topic, but I > > am not sure if that has any use case. > > > > But as an end goal I see that kafka should clearly specify what > it means by > > a tombstone : is it the attribute flag OR is it the null value. > If we just > > do stage 1, I don't think we are defining the end-goal > completely. > > Again this is more about semantics of correctness of end state. > > > > Thanks, > > > > Mayuresh > > > > On Wed, Nov 9, 2016 at 10:49 AM, Becket Qin < > becket....@gmail.com> wrote: > > > > > I am not sure if we need the second stage. Wouldn't it be > enough to say > > > that a message is a tombstone if one of the following is true? > > > 1. tombstone flag is set. > > > 2. value is null. > > > > > > If we go to stage 2, the only difference is that we can > theoretically > > > support a null value non-tombstone message in a log compacted > topic, but > > I > > > am not sure if that has any use case. > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > > > > On Wed, Nov 9, 2016 at 9:23 AM, Mayuresh Gharat < > > > gharatmayures...@gmail.com> > > > wrote: > > > > > > > I think it will be a good idea. +1 > > > > > > > > Thanks, > > > > > > > > Mayuresh > > > > > > > > On Wed, Nov 9, 2016 at 9:13 AM, Michael Pearce < > michael.pea...@ig.com> > > > > wrote: > > > > > > > > > +1 Mayuresh, I think this is a good solution/strategy. > > > > > > > > > > Shall we update the KIP with this? Becket/Jun/Joel any > comments to > > add > > > > > before we do? > > > > > > > > > > On 08/11/2016, 17:29, "Mayuresh Gharat" < > gharatmayures...@gmail.com> > > > > > wrote: > > > > > > > > > > I think the migration can be done in 2 stages : > > > > > > > > > > 1) In first stage the broker should understand the > attribute flag > > > as > > > > > well > > > > > as Null for the value for log compaction. > > > > > 2) In second stage we move on to supporting only the > attribute > > flag > > > > > for log > > > > > compaction. > > > > > > > > > > I agree with Becket that for older clients (consumers) > the broker > > > > might > > > > > have to down convert a message that has the attribute > flag set > > for > > > > log > > > > > compacting but has a non null value. But this should > be in first > > > > stage. > > > > > Once all the clients have upgraded (clients start > recognizing the > > > > > attribute > > > > > flag), we can move the broker to stage 2. > > > > > > > > > > Thanks, > > > > > > > > > > Mayuresh > > > > > > > > > > On Tue, Nov 8, 2016 at 12:06 AM, Michael Pearce < > > > > michael.pea...@ig.com > > > > > > > > > > > wrote: > > > > > > > > > > > Also we can add further guidance: > > > > > > > > > > > > To avoid the below caveat to organisations by > promoting of > > > > > upgrading all > > > > > > consumers first before relying on producing > tombstone messages > > > with > > > > > data > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > ________________________________________ > > > > > > From: Michael Pearce > > > > > > Sent: Tuesday, November 8, 2016 8:03:32 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction > Tombstone Flag > > > > > > > > > > > > Thanks Jun on the feedback, I think I understand the > > issue/point > > > > now. > > > > > > > > > > > > We def can add that on older client version if > tombstone marker > > > > make > > > > > the > > > > > > value null to preserve behaviour. > > > > > > > > > > > > There is one caveats to this: > > > > > > > > > > > > * we have to be clear that data is lost if reading > via old > > > > > client/message > > > > > > format - I don't think this is a big issue as mostly > the > > idea/use > > > > > case is > > > > > > around meta data transport as such would only be as > bad as > > > current > > > > > situation > > > > > > > > > > > > Re having configurable broker this was to handle > cases like you > > > > > described > > > > > > but in another way by allowing organisation choose > the > > behaviour > > > of > > > > > the > > > > > > compaction per broker or per topic so they could > manage their > > > > > transition to > > > > > > using tombstone markers. > > > > > > > > > > > > On hind sight it maybe easier to just upgrade and > downgrade the > > > > > messages > > > > > > on version as you propose. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Sent using OWA for iPhone > > > > > > ________________________________________ > > > > > > From: Jun Rao <j...@confluent.io> > > > > > > Sent: Tuesday, November 8, 2016 12:34:41 AM > > > > > > To: dev@kafka.apache.org > > > > > > Subject: Re: [DISCUSS] KIP-87 - Add Compaction > Tombstone Flag > > > > > > > > > > > > For the use case, one potential use case is for > schema > > > > registration. > > > > > For > > > > > > example, in Avro, a null value corresponds to a Null > schema. > > So, > > > if > > > > > you > > > > > > want to be able to keep the schema id in a delete > message, the > > > > value > > > > > can't > > > > > > be null. We could get around this issue by > specializing null > > > value > > > > > during > > > > > > schema registration though. > > > > > > > > > > > > Now for the proposed changes. We probably should > preserve > > client > > > > > > compatibility. If a client application is sending a > null value > > > to a > > > > > > compacted topic, ideally, it should work the same > after the > > > client > > > > > > upgrades. > > > > > > > > > > > > I am not sure about making the tombstone marker > configurable, > > > > > especially at > > > > > > the topic level. Should we allow users to change the > config > > > values > > > > > back and > > > > > > forth, and what would be the implication? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Mon, Nov 7, 2016 at 10:48 AM, Becket Qin < > > > becket....@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > Yes, changing the logic in the log cleaner makes > sense. There > > > > > could be > > > > > > some > > > > > > > other thing worth thinking (e.g. the message size > change > > after > > > > > > conversion), > > > > > > > though. > > > > > > > > > > > > > > The scenario I was thinking is the following: > > > > > > > Imagine a distributed caching system built on top > of Kafka. A > > > > user > > > > > is > > > > > > > consuming from a topic and it is guaranteed that > if the user > > > > > consume to > > > > > > the > > > > > > > log end it will get the latest value for all the > keys. > > > Currently > > > > > if the > > > > > > > consumer sees a null value it knows the key has > been removed. > > > Now > > > > > let's > > > > > > say > > > > > > > we rolled out this change. And the producer > applies a message > > > > with > > > > > the > > > > > > > tombstone flag set, but the value was not null. > When we > > append > > > > that > > > > > > message > > > > > > > to the log I suppose we will not do the down > conversion if > > the > > > > > broker has > > > > > > > set the message.format.version to the latest. > Because the log > > > > > cleaner > > > > > > won't > > > > > > > touch the active log segment, so that message will > be sitting > > > in > > > > > the > > > > > > active > > > > > > > segment as is. Now when a consumer that hasn't > upgraded yet > > > > > consumes that > > > > > > > tombstone message in the active segment, it seems > that the > > > broker > > > > > will > > > > > > need > > > > > > > to down convert that message to remove the value, > right? In > > > this > > > > > case, we > > > > > > > cannot wait for the log cleaner to do the down > conversion > > > because > > > > > that > > > > > > > message may have already been consumed before the > log > > > compaction > > > > > happens. > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Nov 7, 2016 at 9:59 AM, Michael Pearce < > > > > > michael.pea...@ig.com> > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Becket, > > > > > > > > > > > > > > > > We were thinking more about having the logic > that’s in the > > > > method > > > > > > > > shouldRetainMessage configurable via > > > http://kafka.apache.org/ > > > > > > > > documentation.html#brokerconfigs at a > broker/topic level. > > > And > > > > > then > > > > > > > scrap > > > > > > > > auto converting the message, and allow > organisations to > > > manage > > > > > the > > > > > > > rollout > > > > > > > > of enabling of the feature. > > > > > > > > (this isn’t in documentation but in response to > the > > > discussion > > > > > thread > > > > > > as > > > > > > > > an alternative approach to roll out the feature) > > > > > > > > > > > > > > > > Does this make any more sense? > > > > > > > > > > > > > > > > Thanks > > > > > > > > Mike > > > > > > > > > > > > > > > > On 11/3/16, 2:27 PM, "Becket Qin" < > becket....@gmail.com> > > > > wrote: > > > > > > > > > > > > > > > > Hi Michael, > > > > > > > > > > > > > > > > Do you mean using a new configuration it is > just the > > > > exiting > > > > > > > > message.format.version config? It seems the > > > > > message.format.version > > > > > > > > config > > > > > > > > is enough in this case. And the default > value would > > > always > > > > > be the > > > > > > > > latest > > > > > > > > version. > > > > > > > > > > > > > > > > > Message version migration would be handled > as like in > > > > > KIP-32 > > > > > > > > > > > > > > > > Also just want to confirm on this. Today if > an old > > > consumer > > > > > > consumes > > > > > > > a > > > > > > > > log > > > > > > > > compacted topic and sees an empty value, it > knows that > > > is a > > > > > > > tombstone. > > > > > > > > After we start to use the attribute bit, a > tombstone > > > > message > > > > > can > > > > > > > have a > > > > > > > > non-empty value. So by "like in KIP-32" you > mean we > > will > > > > > remove the > > > > > > > > value > > > > > > > > to down convert the message if the consumer > version is > > > old, > > > > > right? > > > > > > > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > On Wed, Nov 2, 2016 at 1:37 AM, Michael > Pearce < > > > > > > > michael.pea...@ig.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Joel , et al. > > > > > > > > > > > > > > > > > > Any comments on the below idea to handle > roll out / > > > > > compatibility > > > > > > > of > > > > > > > > this > > > > > > > > > feature, using a configuration? > > > > > > > > > > > > > > > > > > Does it make sense/clear? > > > > > > > > > Does it add value? > > > > > > > > > Do we want to enforce flag by default, or > value by > > > > > default, or > > > > > > > both? > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > On 10/27/16, 4:47 PM, "Michael Pearce" < > > > > > michael.pea...@ig.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Thanks, James, I think this is a > really good > > > addition > > > > > to the > > > > > > > KIP > > > > > > > > > details, please feel free to amend the > wiki/add the > > use > > > > > cases, > > > > > > also > > > > > > > > if any > > > > > > > > > others you think of. I definitely think its > > worthwhile > > > > > > documenting. > > > > > > > > If you > > > > > > > > > can’t let me know ill add them next week > (just > > leaving > > > > for > > > > > a long > > > > > > > > weekend > > > > > > > > > off) > > > > > > > > > > > > > > > > > > Re Joel and others comments about > upgrade and > > > > > compatibility. > > > > > > > > > > > > > > > > > > Rather than trying to auto manage this. > > > > > > > > > > > > > > > > > > Actually maybe we make a configuration > option, > > both > > > > at > > > > > server > > > > > > > > and per > > > > > > > > > topic level to control the behavior of how > the server > > > > logic > > > > > > should > > > > > > > > work out > > > > > > > > > if the record, is a tombstone record . > > > > > > > > > > > > > > > > > > e.g. > > > > > > > > > > > > > > > > > > key = compation.tombstone.marker > > > > > > > > > > > > > > > > > > value options: > > > > > > > > > > > > > > > > > > value (continues to use null value > as tombstone > > > > > marker) > > > > > > > > > flag (expects to use the tombstone > flag) > > > > > > > > > value_or_flag (if either is true it > treats the > > > record > > > > > as a > > > > > > > > tombstone) > > > > > > > > > > > > > > > > > > This way on upgrade users can keep > current > > > behavior, > > > > > and > > > > > > slowly > > > > > > > > > migrate to the new. Having a transition > period of > > using > > > > > > > > value_or_flag, > > > > > > > > > finally having flag only if an > organization wishes to > > > use > > > > > null > > > > > > > values > > > > > > > > > without it being treated as a tombstone > marker (use > > > case > > > > > noted > > > > > > > below) > > > > > > > > > > > > > > > > > > Having it both global broker level and > topic > > > override > > > > > also > > > > > > > > allows some > > > > > > > > > flexibility here. > > > > > > > > > > > > > > > > > > Cheers > > > > > > > > > Mike > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 10/27/16, 8:03 AM, "James Cheng" < > > > > > wushuja...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > This KIP would definitely address > a gap in > > the > > > > > current > > > > > > > > > functionality, where you currently can't > have a > > > tombstone > > > > > with > > > > > > any > > > > > > > > > associated content. > > > > > > > > > > > > > > > > > > That said, I'd like to talk about > use cases, > > to > > > > > make sure > > > > > > > > that > > > > > > > > > this is in fact useful. The KIP should be > updated > > with > > > > > whatever > > > > > > use > > > > > > > > cases > > > > > > > > > we come up with. > > > > > > > > > > > > > > > > > > First of all, an observation: When > we speak > > > about > > > > > log > > > > > > > > compaction, > > > > > > > > > we typically think of "the latest message > for a key > > is > > > > > retained". > > > > > > > In > > > > > > > > that > > > > > > > > > respect, a delete tombstone (i.e. a > message with a > > null > > > > > payload) > > > > > > is > > > > > > > > treated > > > > > > > > > the same as any other Kafka message: the > latest > > message > > > > is > > > > > > > retained. > > > > > > > > It > > > > > > > > > doesn't matter whether the latest message > is null, or > > > if > > > > > the > > > > > > latest > > > > > > > > message > > > > > > > > > has actual content. In all cases, the last > message is > > > > > retained. > > > > > > > > > > > > > > > > > > The only way a delete tombstone is > treated > > > > > differently > > > > > > from > > > > > > > > other > > > > > > > > > Kafka messages is that it automatically > disappears > > > after > > > > a > > > > > while. > > > > > > > > The time > > > > > > > > > of deletion is specified using > delete.retention.ms. > > > > > > > > > > > > > > > > > > So what we're really talking about > is, do we > > > want > > > > > to > > > > > > > support > > > > > > > > > messages in a log-compacted topic that > auto-delete > > > > > themselves > > > > > > after > > > > > > > > a while? > > > > > > > > > > > > > > > > > > In a thread from 2015, there was a > discussion > > > on > > > > > > > first-class > > > > > > > > > support of headers between Roger Hoover, > Felix GV, > > Jun > > > > > Rao, and > > > > > > I. > > > > > > > > See > > > > > > > > > thread at https://groups.google.com/d/ > > > > > msg/confluent-platform/ > > > > > > > > > 8xPbjyUE_7E/yQ1AeCufL_gJ < > > https://groups.google.com/d/ > > > > > > > > > > msg/confluent-platform/8xPbjyUE_7E/yQ1AeCufL_gJ> > . > > In > > > > that > > > > > > thread, > > > > > > > > Jun > > > > > > > > > raised a good question that I didn't have > a good > > answer > > > > > for at > > > > > > the > > > > > > > > time: If > > > > > > > > > a message is going to auto-delete itself > after a > > while, > > > > how > > > > > > > > important was > > > > > > > > > the message? That is, what information did > the > > message > > > > > contain > > > > > > that > > > > > > > > was > > > > > > > > > important *for a while* but not so > important that it > > > > > needed to be > > > > > > > > kept > > > > > > > > > around forever? > > > > > > > > > > > > > > > > > > Some use cases that I can think of: > > > > > > > > > > > > > > > > > > 1) Tracability. I would like to > know who > > issued > > > > > this > > > > > > delete > > > > > > > > > tombstone. It might include the hostname, > IP of the > > > > > producer of > > > > > > the > > > > > > > > delete. > > > > > > > > > 2) Timestamps. I would like to > know when this > > > > > delete was > > > > > > > > issued. > > > > > > > > > This use case is already addressed by the > > availability > > > of > > > > > > > per-message > > > > > > > > > timestamps that came in 0.10.0 > > > > > > > > > 3) Data provenance. I hope I'm > using this > > > phrase > > > > > > correctly, > > > > > > > > but > > > > > > > > > what I mean is, where did this delete come > from? What > > > > > processing > > > > > > > job > > > > > > > > > emitted it? What input to the processing > job caused > > > this > > > > > delete > > > > > > to > > > > > > > be > > > > > > > > > produced? For example, if a record in > topic A was > > > > > processed and > > > > > > > > caused a > > > > > > > > > delete tombstone to be emitted to topic B, > I might > > like > > > > the > > > > > > offset > > > > > > > > of the > > > > > > > > > topic A message to be attached to the > topic B > > message. > > > > > > > > > 4) Distributed tracing for stream > topologies. > > > > This > > > > > might > > > > > > > be a > > > > > > > > > slight repeat of the above use cases. In > the > > > > microservices > > > > > world, > > > > > > > we > > > > > > > > can > > > > > > > > > generate call-graphs of webservices using > tools like > > > > > Zipkin/ > > > > > > > > opentracing.io > > > > > > > > > <http://opentracing.io/>, or something > homegrown > > like > > > > > > > > > https://engineering.linkedin. > > > > com/distributed-service-call- > > > > > > > > > graph/real-time-distributed- > > > tracing-website-performance- > > > > > > > and-efficiency > > > > > > > > < > > > > > > > > > https://engineering.linkedin. > > > > com/distributed-service-call- > > > > > > > > > graph/real-time-distributed- > > > tracing-website-performance- > > > > > > > > and-efficiency>. > > > > > > > > > I can imagine that you might want to do > something > > > similar > > > > > for > > > > > > > stream > > > > > > > > > processing topologies, where stream > processing jobs > > > carry > > > > > along > > > > > > and > > > > > > > > forward > > > > > > > > > along a globally unique identifier, and a > distributed > > > > > topology > > > > > > > graph > > > > > > > > is > > > > > > > > > generated. > > > > > > > > > 5) Cases where processing a delete > requires > > > data > > > > > that is > > > > > > > not > > > > > > > > > available in the message key. I'm not sure > I have a > > > good > > > > > example > > > > > > of > > > > > > > > this, > > > > > > > > > though. One hand-wavy example might be > where I am > > > > > publishing > > > > > > > > documents into > > > > > > > > > Kafka where the documentId is the message > key, and > > the > > > > text > > > > > > > contents > > > > > > > > of the > > > > > > > > > document are in the message body. And I > have a > > > consuming > > > > > job that > > > > > > > > does some > > > > > > > > > analytics on the message body. If that > document gets > > > > > deleted, > > > > > > then > > > > > > > > the > > > > > > > > > consuming job might need the original > message body in > > > > > order to > > > > > > > > "delete" > > > > > > > > > that message's impact from the analytics. > But I'm not > > > > sure > > > > > that > > > > > > is > > > > > > > a > > > > > > > > great > > > > > > > > > example. If the consumer was worried about > that, the > > > > > consumer > > > > > > would > > > > > > > > > probably keep the original message around, > stored by > > > > > primary key. > > > > > > > > And then > > > > > > > > > all it would need from a delete message > would be the > > > > > primary key > > > > > > of > > > > > > > > the > > > > > > > > > message. > > > > > > > > > > > > > > > > > > Do people think these are valid > use cases? > > > > > > > > > > > > > > > > > > What are other use cases that > people can > > think > > > > of? > > > > > > > > > > > > > > > > > > -James > > > > > > > > > > > > > > > > > > > On Oct 26, 2016, at 3:46 PM, > Mayuresh > > Gharat > > > < > > > > > > > > > gharatmayures...@gmail.com> wrote: > > > > > > > > > > > > > > > > > > > > +1 @Joel. > > > > > > > > > > I think a clear migration plan > of upgrading > > > and > > > > > > > > downgrading of > > > > > > > > > server and > > > > > > > > > > clients along with handling of > issues that > > > Joel > > > > > > > mentioned, > > > > > > > > on > > > > > > > > > the KIP would > > > > > > > > > > be really great. > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > Mayuresh > > > > > > > > > > > > > > > > > > > > On Wed, Oct 26, 2016 at 3:31 PM, > Joel > > Koshy < > > > > > > > > jjkosh...@gmail.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > >> I'm not sure why it would be > useful, but > > it > > > > > should be > > > > > > > > > theoretically > > > > > > > > > >> possible if the attribute bit > alone is > > > enough > > > > > to mark > > > > > > a > > > > > > > > > tombstone. OTOH, we > > > > > > > > > >> could consider that as invalid > if we wish. > > > > > These are > > > > > > > > relevant > > > > > > > > > details that > > > > > > > > > >> I think should be added to the > KIP. > > > > > > > > > >> > > > > > > > > > >> Also, in the few odd scenarios > that I > > > > mentioned > > > > > we > > > > > > > should > > > > > > > > also > > > > > > > > > consider > > > > > > > > > >> that fetches could be coming > from other > > > > > > > yet-to-be-upgraded > > > > > > > > > brokers in a > > > > > > > > > >> cluster that is being upgraded. > So we > > would > > > > > probably > > > > > > > want > > > > > > > > to > > > > > > > > > continue to > > > > > > > > > >> support nulls as tombstones or > > down-convert > > > in > > > > > a way > > > > > > > that > > > > > > > > we > > > > > > > > > are sure works > > > > > > > > > >> with least surprise to fetchers. > > > > > > > > > >> > > > > > > > > > >> There is a slightly vague > statement under > > > > > > > "Compatibility, > > > > > > > > > Deprecation, and > > > > > > > > > >> Migration Plan" that could > benefit more > > > > details: > > > > > > *Logic > > > > > > > > would > > > > > > > > > base on > > > > > > > > > >> current behavior of null value > or if > > > tombstone > > > > > flag > > > > > > set > > > > > > > to > > > > > > > > > true, as such > > > > > > > > > >> wouldn't impact any existing > flows simply > > > > allow > > > > > new > > > > > > > > producers > > > > > > > > > to make use > > > > > > > > > >> of the feature*. It is unclear > to me based > > > on > > > > > that > > > > > > > > whether you > > > > > > > > > would > > > > > > > > > >> interpret null as a tombstone > if the > > > tombstone > > > > > > attribute > > > > > > > > bit is > > > > > > > > > off. > > > > > > > > > >> > > > > > > > > > >> On Wed, Oct 26, 2016 at 3:10 > PM, Xavier > > > > Léauté < > > > > > > > > > xav...@confluent.io> > > > > > > > > > >> wrote: > > > > > > > > > >> > > > > > > > > > >>> Does this mean that starting > with V4 > > > requests > > > > > we > > > > > > would > > > > > > > > allow > > > > > > > > > storing null > > > > > > > > > >>> messages in compacted topics? > The KIP > > > should > > > > > probably > > > > > > > > clarify > > > > > > > > > the > > > > > > > > > >> behavior > > > > > > > > > >>> for null messages where the > tombstone > > flag > > > is > > > > > not > > > > > > net. > > > > > > > > > >>> > > > > > > > > > >>> On Wed, Oct 26, 2016 at 1:32 > AM Magnus > > > > > Edenhill < > > > > > > > > > mag...@edenhill.se> > > > > > > > > > >>> wrote: > > > > > > > > > >>> > > > > > > > > > >>>> 2016-10-25 21:36 GMT+02:00 > Nacho Solis > > > > > > > > > <nso...@linkedin.com.invalid>: > > > > > > > > > >>>> > > > > > > > > > >>>>> I think you probably require > a > > MagicByte > > > > > bump if > > > > > > you > > > > > > > > expect > > > > > > > > > correct > > > > > > > > > >>>>> behavior of the system as a > whole. > > > > > > > > > >>>>> > > > > > > > > > >>>>> From a client perspective > you want to > > > make > > > > > sure > > > > > > that > > > > > > > > when you > > > > > > > > > >> deliver a > > > > > > > > > >>>>> message that the broker > supports the > > > > feature > > > > > you're > > > > > > > > expecting > > > > > > > > > >>>>> (compaction). So, depending > on the > > > > behavior > > > > > of the > > > > > > > > broker on > > > > > > > > > >>>> encountering > > > > > > > > > >>>>> a previously undefined bit > flag I would > > > > > suggest > > > > > > > making > > > > > > > > some > > > > > > > > > change to > > > > > > > > > >>>> make > > > > > > > > > >>>>> certain that flag-based > compaction is > > > > > supported. > > > > > > I'm > > > > > > > > going > > > > > > > > > to guess > > > > > > > > > >>> that > > > > > > > > > >>>>> the MagicByte would do this. > > > > > > > > > >>>>> > > > > > > > > > >>>> > > > > > > > > > >>>> I dont believe this is needed > since it > > is > > > > > already > > > > > > > > attributed > > > > > > > > > through > > > > > > > > > >> the > > > > > > > > > >>>> request's API version. > > > > > > > > > >>>> > > > > > > > > > >>>> Producer: > > > > > > > > > >>>> * if a client sends > ProduceRequest V4 > > then > > > > > > > > attributes.bit5 > > > > > > > > > indicates a > > > > > > > > > >>>> tombstone > > > > > > > > > >>>> * if a clients sends > ProduceRequest <V4 > > > then > > > > > > > > attributes.bit5 > > > > > > > > > is > > > > > > > > > >> ignored > > > > > > > > > >>>> and value==null indicates a > tombstone > > > > > > > > > >>>> * in both cases the on-disk > messages are > > > > > stored with > > > > > > > > > attributes.bit5 > > > > > > > > > >> (I > > > > > > > > > >>>> assume?) > > > > > > > > > >>>> > > > > > > > > > >>>> Consumer: > > > > > > > > > >>>> * if a clients sends > FetchRequest V4 > > > > messages > > > > > are > > > > > > > > > sendfile():ed > > > > > > > > > >> directly > > > > > > > > > >>>> from disk (with > attributes.bit5) > > > > > > > > > >>>> * if a client sends > FetchRequest <V4 > > > > messages > > > > > are > > > > > > > > slowpathed > > > > > > > > > and > > > > > > > > > >>>> translated from > attributes.bit5 to > > > > value=null > > > > > as > > > > > > > > required. > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> That's my understanding > anyway, please > > > > > correct me if > > > > > > > I'm > > > > > > > > > wrong. > > > > > > > > > >>>> > > > > > > > > > >>>> /Magnus > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>> > > > > > > > > > >>>>> On Tue, Oct 25, 2016 at > 10:17 AM, > > Magnus > > > > > Edenhill < > > > > > > > > > >> mag...@edenhill.se> > > > > > > > > > >>>>> wrote: > > > > > > > > > >>>>> > > > > > > > > > >>>>>> It is safe to assume that a > previously > > > > > undefined > > > > > > > > attributes > > > > > > > > > bit > > > > > > > > > >> will > > > > > > > > > >>> be > > > > > > > > > >>>>>> unset in protocol requests > from > > existing > > > > > clients, > > > > > > if > > > > > > > > not, > > > > > > > > > such a > > > > > > > > > >>> client > > > > > > > > > >>>>> is > > > > > > > > > >>>>>> already violating the > protocol and > > needs > > > > to > > > > > be > > > > > > > fixed. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> So I dont see a need for a > MagicByte > > > bump, > > > > > both > > > > > > > > broker and > > > > > > > > > client > > > > > > > > > >> has > > > > > > > > > >>>> the > > > > > > > > > >>>>>> information it needs to > construct or > > > parse > > > > > the > > > > > > > message > > > > > > > > > according to > > > > > > > > > >>>>> request > > > > > > > > > >>>>>> version. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> 2016-10-25 18:48 GMT+02:00 > Michael > > > Pearce > > > > < > > > > > > > > > michael.pea...@ig.com>: > > > > > > > > > >>>>>> > > > > > > > > > >>>>>>> Hi Magnus, > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> I was wondering if I even > needed to > > > > change > > > > > those > > > > > > > > also, as > > > > > > > > > >>> technically > > > > > > > > > >>>>>>> we’re just making use of a > non used > > > > > attribute > > > > > > bit, > > > > > > > > but im > > > > > > > > > not > > > > > > > > > >> 100% > > > > > > > > > >>>> that > > > > > > > > > >>>>>> it > > > > > > > > > >>>>>>> be always false currently. > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> If someone can say 100% it > will > > already > > > > be > > > > > set > > > > > > > false > > > > > > > > with > > > > > > > > > current > > > > > > > > > >>> and > > > > > > > > > >>>>>>> historic bit wise masking > techniques > > > used > > > > > over > > > > > > the > > > > > > > > time, > > > > > > > > > we could > > > > > > > > > >>> do > > > > > > > > > >>>>> away > > > > > > > > > >>>>>>> with both, and simply just > start to > > use > > > > it. > > > > > > > > Unfortunately > > > > > > > > > I don’t > > > > > > > > > >>>> have > > > > > > > > > >>>>>> that > > > > > > > > > >>>>>>> historic knowledge so was > hoping it > > > would > > > > > be > > > > > > > flagged > > > > > > > > up in > > > > > > > > > this > > > > > > > > > >>>>>> discussion > > > > > > > > > >>>>>>> thread ? > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> Cheers > > > > > > > > > >>>>>>> Mike > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> On 10/25/16, 5:36 PM, > "Magnus > > > Edenhill" < > > > > > > > > > mag...@edenhill.se> > > > > > > > > > >>> wrote: > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> Hi Michael, > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> With the version bumps > for Produce > > > and > > > > > Fetch > > > > > > > > requests, > > > > > > > > > do you > > > > > > > > > >>>>> really > > > > > > > > > >>>>>>> need > > > > > > > > > >>>>>>> to bump MagicByte too? > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> Regards, > > > > > > > > > >>>>>>> Magnus > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> 2016-10-25 18:09 > GMT+02:00 Michael > > > > > Pearce < > > > > > > > > > >>> michael.pea...@ig.com > > > > > > > > > >>>>> : > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>>> Hi All, > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> I would like to discuss > the > > following > > > > KIP > > > > > > > proposal: > > > > > > > > > >>>>>>>> https://cwiki.apache.org/ > > > > > > > > confluence/display/KAFKA/KIP- > > > > > > > > > >>>>>>>> > 87+-+Add+Compaction+Tombstone+Flag > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> This is off the back of > the > > discussion > > > > on > > > > > KIP-82 > > > > > > > / > > > > > > > > KIP > > > > > > > > > >>> meeting > > > > > > > > > >>>>>>> where it > > > > > > > > > >>>>>>>> was agreed to separate > this issue > > and > > > > > feature. > > > > > > > See: > > > > > > > > > >>>>>>>> > http://mail-archives.apache. > > > > > > > > org/mod_mbox/kafka-dev/201610 > > > > > > > > > . > > > > > > > > > >>>>>>>> mbox/%3cCAJS3ho8OcR== > > > > > EcxsJ8OP99pD2hz=iiGecWsv- > > > > > > > > > >>>>>>>> EZsBsNyDcKr= > g...@mail.gmail.com%3e > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> Thanks > > > > > > > > > >>>>>>>> Mike > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>>> The information contained > in this > > > email > > > > is > > > > > > > strictly > > > > > > > > > >>>> confidential > > > > > > > > > >>>>>> and > > > > > > > > > >>>>>>> for > > > > > > > > > >>>>>>>> the use of the addressee > only, > > unless > > > > > otherwise > > > > > > > > indicated. > > > > > > > > > >> If > > > > > > > > > >>>> you > > > > > > > > > >>>>>>> are not > > > > > > > > > >>>>>>>> the intended recipient, > please do > > not > > > > > read, > > > > > > copy, > > > > > > > > use or > > > > > > > > > >>>> disclose > > > > > > > > > >>>>>> to > > > > > > > > > >>>>>>> others > > > > > > > > > >>>>>>>> this message or any > attachment. > > Please > > > > > also > > > > > > notify > > > > > > > > the > > > > > > > > > >> sender > > > > > > > > > >>>> by > > > > > > > > > >>>>>>> replying > > > > > > > > > >>>>>>>> to this email or by > telephone > > (+44(020 > > > > > 7896 > > > > > > 0011) > > > > > > > > and then > > > > > > > > > >>>> delete > > > > > > > > > >>>>>>> the email > > > > > > > > > >>>>>>>> and any copies of it. > Opinions, > > > > > conclusion (etc) > > > > > > > > that do > > > > > > > > > >> not > > > > > > > > > >>>>> relate > > > > > > > > > >>>>>>> to the > > > > > > > > > >>>>>>>> official business of this > company > > > shall > > > > be > > > > > > > > understood as > > > > > > > > > >>>> neither > > > > > > > > > >>>>>>> given nor > > > > > > > > > >>>>>>>> endorsed by it. IG is a > trading name > > > of > > > > IG > > > > > > Markets > > > > > > > > Limited > > > > > > > > > >> (a > > > > > > > > > >>>>>> company > > > > > > > > > >>>>>>>> registered in England and > Wales, > > > company > > > > > number > > > > > > > > 04008957) > > > > > > > > > >> and > > > > > > > > > >>>> IG > > > > > > > > > >>>>>>> Index > > > > > > > > > >>>>>>>> Limited (a company > registered in > > > England > > > > > and > > > > > > > Wales, > > > > > > > > > company > > > > > > > > > >>>>> number > > > > > > > > > >>>>>>>> 01190902). Registered > address at > > > Cannon > > > > > Bridge > > > > > > > > House, 25 > > > > > > > > > >>>> Dowgate > > > > > > > > > >>>>>>> Hill, > > > > > > > > > >>>>>>>> London EC4R 2YA. Both IG > Markets > > > Limited > > > > > > (register > > > > > > > > number > > > > > > > > > >>>> 195355) > > > > > > > > > >>>>>>> and IG > > > > > > > > > >>>>>>>> Index Limited (register > number > > 114059) > > > > are > > > > > > > > authorised and > > > > > > > > > >>>>> regulated > > > > > > > > > >>>>>>> by the > > > > > > > > > >>>>>>>> Financial Conduct > Authority. > > > > > > > > > >>>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>>> The information contained > in this > > email > > > > is > > > > > > strictly > > > > > > > > > confidential > > > > > > > > > >>> and > > > > > > > > > >>>>> for > > > > > > > > > >>>>>>> the use of the addressee > only, unless > > > > > otherwise > > > > > > > > indicated. > > > > > > > > > If you > > > > > > > > > >>> are > > > > > > > > > >>>>> not > > > > > > > > > >>>>>>> the intended recipient, > please do not > > > > > read, copy, > > > > > > > > use or > > > > > > > > > disclose > > > > > > > > > >>> to > > > > > > > > > >>>>>> others > > > > > > > > > >>>>>>> this message or any > attachment. > > Please > > > > also > > > > > > notify > > > > > > > > the > > > > > > > > > sender by > > > > > > > > > >>>>> replying > > > > > > > > > >>>>>>> to this email or by > telephone > > (+44(020 > > > > > 7896 0011) > > > > > > > > and then > > > > > > > > > delete > > > > > > > > > >>> the > > > > > > > > > >>>>>> email > > > > > > > > > >>>>>>> and any copies of it. > Opinions, > > > > conclusion > > > > > (etc) > > > > > > > > that do > > > > > > > > > not > > > > > > > > > >> relate > > > > > > > > > >>>> to > > > > > > > > > >>>>>> the > > > > > > > > > >>>>>>> official business of this > company > > shall > > > > be > > > > > > > > understood as > > > > > > > > > neither > > > > > > > > > >>>> given > > > > > > > > > >>>>>> nor > > > > > > > > > >>>>>>> endorsed by it. IG is a > trading name > > of > > > > IG > > > > > > Markets > > > > > > > > Limited > > > > > > > > > (a > > > > > > > > > >>> company > > > > > > > > > >>>>>>> registered in England and > Wales, > > > company > > > > > number > > > > > > > > 04008957) > > > > > > > > > and IG > > > > > > > > > >>>> Index > > > > > > > > > >>>>>>> Limited (a company > registered in > > > England > > > > > and > > > > > > Wales, > > > > > > > > company > > > > > > > > > >> number > > > > > > > > > >>>>>>> 01190902). Registered > address at > > Cannon > > > > > Bridge > > > > > > > > House, 25 > > > > > > > > > Dowgate > > > > > > > > > >>>> Hill, > > > > > > > > > >>>>>>> London EC4R 2YA. Both IG > Markets > > > Limited > > > > > > (register > > > > > > > > number > > > > > > > > > 195355) > > > > > > > > > >>> and > > > > > > > > > >>>>> IG > > > > > > > > > >>>>>>> Index Limited (register > number > > 114059) > > > > are > > > > > > > > authorised and > > > > > > > > > >> regulated > > > > > > > > > >>>> by > > > > > > > > > >>>>>> the > > > > > > > > > >>>>>>> Financial Conduct > Authority. > > > > > > > > > >>>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> -- > > > > > > > > > >>>>> Nacho (Ignacio) Solis > > > > > > > > > >>>>> Kafka > > > > > > > > > >>>>> nso...@linkedin.com > > > > > > > > > >>>>> > > > > > > > > > >>>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > -Regards, > > > > > > > > > > Mayuresh R. Gharat > > > > > > > > > > (862) 250-7125 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this > email is > > strictly > > > > > > > confidential > > > > > > > > and > > > > > > > > > for the use of the addressee only, unless > otherwise > > > > > indicated. If > > > > > > > > you are > > > > > > > > > not the intended recipient, please do not > read, copy, > > > use > > > > > or > > > > > > > > disclose to > > > > > > > > > others this message or any attachment. > Please also > > > notify > > > > > the > > > > > > > sender > > > > > > > > by > > > > > > > > > replying to this email or by telephone > (+44(020 7896 > > > > 0011) > > > > > and > > > > > > then > > > > > > > > delete > > > > > > > > > the email and any copies of it. Opinions, > conclusion > > > > (etc) > > > > > that > > > > > > do > > > > > > > > not > > > > > > > > > relate to the official business of this > company shall > > > be > > > > > > understood > > > > > > > > as > > > > > > > > > neither given nor endorsed by it. IG is a > trading > > name > > > of > > > > > IG > > > > > > > Markets > > > > > > > > > Limited (a company registered in England > and Wales, > > > > company > > > > > > number > > > > > > > > > 04008957) and IG Index Limited (a company > registered > > in > > > > > England > > > > > > and > > > > > > > > Wales, > > > > > > > > > company number 01190902). Registered > address at > > Cannon > > > > > Bridge > > > > > > > House, > > > > > > > > 25 > > > > > > > > > Dowgate Hill, London EC4R 2YA. Both IG > Markets > > Limited > > > > > (register > > > > > > > > number > > > > > > > > > 195355) and IG Index Limited (register > number 114059) > > > are > > > > > > > authorised > > > > > > > > and > > > > > > > > > regulated by the Financial Conduct > Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > The information contained in this email is > strictly > > > > confidential > > > > > and > > > > > > for > > > > > > > > the use of the addressee only, unless otherwise > indicated. > > If > > > > > you are > > > > > > not > > > > > > > > the intended recipient, please do not read, > copy, use or > > > > > disclose to > > > > > > > others > > > > > > > > this message or any attachment. Please also > notify the > > sender > > > > by > > > > > > replying > > > > > > > > to this email or by telephone (+44(020 7896 > 0011) and then > > > > > delete the > > > > > > > email > > > > > > > > and any copies of it. Opinions, conclusion (etc) > that do > > not > > > > > relate to > > > > > > > the > > > > > > > > official business of this company shall be > understood as > > > > neither > > > > > given > > > > > > > nor > > > > > > > > endorsed by it. IG is a trading name of IG > Markets Limited > > (a > > > > > company > > > > > > > > registered in England and Wales, company number > 04008957) > > and > > > > IG > > > > > Index > > > > > > > > Limited (a company registered in England and > Wales, company > > > > > number > > > > > > > > 01190902). Registered address at Cannon Bridge > House, 25 > > > > Dowgate > > > > > Hill, > > > > > > > > London EC4R 2YA. Both IG Markets Limited > (register number > > > > > 195355) and > > > > > > IG > > > > > > > > Index Limited (register number 114059) are > authorised and > > > > > regulated by > > > > > > > the > > > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > The information contained in this email is strictly > > confidential > > > > and > > > > > for > > > > > > the use of the addressee only, unless otherwise > indicated. If > > you > > > > > are not > > > > > > the intended recipient, please do not read, copy, > use or > > disclose > > > > to > > > > > others > > > > > > this message or any attachment. Please also notify > the sender > > by > > > > > replying > > > > > > to this email or by telephone (+44(020 7896 0011) > and then > > delete > > > > > the email > > > > > > and any copies of it. Opinions, conclusion (etc) > that do not > > > relate > > > > > to the > > > > > > official business of this company shall be > understood as > > neither > > > > > given nor > > > > > > endorsed by it. IG is a trading name of IG Markets > Limited (a > > > > company > > > > > > registered in England and Wales, company number > 04008957) and > > IG > > > > > Index > > > > > > Limited (a company registered in England and Wales, > company > > > number > > > > > > 01190902). Registered address at Cannon Bridge > House, 25 > > Dowgate > > > > > Hill, > > > > > > London EC4R 2YA. Both IG Markets Limited (register > number > > 195355) > > > > > and IG > > > > > > Index Limited (register number 114059) are > authorised and > > > regulated > > > > > by the > > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -Regards, > > > > > Mayuresh R. Gharat > > > > > (862) 250-7125 > > > > > > > > > > > > > > > The information contained in this email is strictly > confidential and > > > for > > > > > the use of the addressee only, unless otherwise indicated. > If you are > > > not > > > > > the intended recipient, please do not read, copy, use or > disclose to > > > > others > > > > > this message or any attachment. Please also notify the > sender by > > > replying > > > > > to this email or by telephone (+44(020 7896 0011) and then > delete the > > > > email > > > > > and any copies of it. Opinions, conclusion (etc) that do > not relate > > to > > > > the > > > > > official business of this company shall be understood as > neither > > given > > > > nor > > > > > endorsed by it. IG is a trading name of IG Markets Limited > (a company > > > > > registered in England and Wales, company number 04008957) > and IG > > Index > > > > > Limited (a company registered in England and Wales, > company number > > > > > 01190902). Registered address at Cannon Bridge House, 25 > Dowgate > > Hill, > > > > > London EC4R 2YA. Both IG Markets Limited (register number > 195355) and > > > IG > > > > > Index Limited (register number 114059) are authorised and > regulated > > by > > > > the > > > > > Financial Conduct Authority. > > > > > > > > > > > > > > > > > > > > > -- > > > > -Regards, > > > > Mayuresh R. Gharat > > > > (862) 250-7125 > > > > > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > > > > The information contained in this email is strictly confidential and > for the use of the addressee only, unless otherwise indicated. If you are > not the intended recipient, please do not read, copy, use or disclose to > others this message or any attachment. Please also notify the sender by > replying to this email or by telephone (+44(020 7896 0011) and then delete > the email and any copies of it. Opinions, conclusion (etc) that do not > relate to the official business of this company shall be understood as > neither given nor endorsed by it. IG is a trading name of IG Markets > Limited (a company registered in England and Wales, company number > 04008957) and IG Index Limited (a company registered in England and Wales, > company number 01190902). Registered address at Cannon Bridge House, 25 > Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number > 195355) and IG Index Limited (register number 114059) are authorised and > regulated by the Financial Conduct Authority. > > > -- -Regards, Mayuresh R. Gharat (862) 250-7125
