i didnt mean to sound as insisting. what i actually mean is it would still
be a valid issue but of much less concern.
On Mon, Dec 19, 2016 at 7:50 PM, radai <radai.rosenbl...@gmail.com> wrote:
> this kip fixes a "bug" (quirk?) that arises when people implement headers
> "in V" (in the payload part of a message).
> if you have proper headers you obviously dont need to to stick them in V
> and so wont run into this, but its still a valid issue.
>
> On Mon, Dec 19, 2016 at 3:06 PM, Jay Kreps <j...@confluent.io> wrote:
>
>> Makes sense!
>>
>> -Jay
>>
>> On Mon, Dec 19, 2016 at 2:40 PM, Michael Pearce <michael.pea...@ig.com>
>> wrote:
>>
>> > Wow just read that def over tired. Hopefully it makes sense. Or you get
>> > the gist at least.
>> >
>> > ________________________________________
>> > From: Michael Pearce <michael.pea...@ig.com>
>> > Sent: Monday, December 19, 2016 9:19:02 PM
>> > To: dev@kafka.apache.org
>> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> >
>> > Hi Jay,
>> >
>> > Agreed this stemmed as offshoot from KIP-82.
>> >
>> > Which our main driver for was to be able to have some headers for a null
>> > value as such for our routing, audit, tracing and a few other bits which
>> > currently we are forced to do with a message wrapper, if we all agreed
>> on
>> > KIP-82 that we need native headers and look to implement that the push
>> for
>> > this would dissipate.
>> >
>> > This KIP would allow for though one use case that comes to mind we could
>> > see which is to have business data with a delete. Though as said this
>> isn't
>> > something we are pushing for think really we would have.
>> >
>> > As such in summary yes, if you want to fully support KIP-82 and we can
>> get
>> > that agreed in principle and a target release version, I think quite a
>> few
>> > guys at LinkedIn are quite pro it too ;) I'm happy to drop this one.
>> >
>> > Cheers
>> > Mike
>> >
>> > Sent using OWA for iPhone
>> > ________________________________________
>> > From: Jay Kreps <j...@confluent.io>
>> > Sent: Monday, December 19, 2016 8:51:23 PM
>> > To: dev@kafka.apache.org
>> > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> >
>> > Hey Michael,
>> >
>> > Here is the compatibility concern I have:
>> >
>> > 1. You have a consumer app that relies on value == null to indicate a
>> > delete (current semantics).
>> > 2. You upgrade Kafka and your clients.
>> > 3. Some producer starts using the tombstone field in combination with
>> > non-null.
>> >
>> > I share Ismael's dislike of setting tombstones on records with null
>> values.
>> > This makes sense as a transitional state, but as an end state its a bit
>> > weird. You'd expect to be able to mix null values and tombstones, and
>> have
>> > the null values remain and the tombstones get compacted. However what
>> will
>> > happen is both will be compacted and upon debugging this you'll learn
>> that
>> > we sometimes use null in the value to indicate tombstone. Ismael's
>> solution
>> > is a bigger compatibility break, though, so not sure if that is better.
>> >
>> > My other question is the relationship to KIP-82. My read is that this
>> KIP
>> > solves some but not all of the problems KIP-82 is intended for. KIP-82,
>> on
>> > the other hand, seems to address most of the motivating uses for this
>> KIP.
>> > The exception is maybe item (5) on the list where you want to
>> simultaneous
>> > delete and convey some information to subscribers, but I couldn't
>> construct
>> > a concrete examples for that one. Do we need to rationalize these two
>> KIPs?
>> > That is, do you still advocate this proposal if we do KIP-82 and vice
>> > versa? As you may have noticed I'm somewhat emotionally invested in the
>> > simplicity of the core data model, so my default position is let's try
>> to
>> > avoid stuffing more stuff in, but if we have to add stuff I like each of
>> > these individually more than doing both. :-)
>> >
>> > -Jay
>> >
>> >
>> >
>> >
>> > On Fri, Dec 16, 2016 at 12:16 PM, Michael Pearce <michael.pea...@ig.com
>> >
>> > wrote:
>> >
>> > > Hi Jay
>> > >
>> > > I disagree here that we are breaking any compatibility, we went
>> through
>> > > this on the discussion thread in fact with the help of that thread is
>> how
>> > > the kip came to the solution.
>> > >
>> > > Also on the supported combinations front you mention, we are not
>> taking
>> > > anything away afaik.
>> > >
>> > > Currently supported are only are:
>> > > Null value = delete
>> > > Non-null value = non delete
>> > >
>> > > With this kip we would support
>> > > Null value + tombstone = delete
>> > > Non null value + tombstone = delete
>> > > Non null value + no tombstone = non delete
>> > >
>> > > As for the alternative idea, this is simply a new policy, how can
>> there
>> > be
>> > > confusion here? For this policy it would be explicit that tombstone
>> > marker
>> > > would need to be set for a delete.
>> > >
>> > > I'm going to vent a little now as starting to get quite frustrated.
>> > >
>> > > We are going round in circles on kip-82 as per use cases there is now
>> > many
>> > > use cases, how many more are needed? just because confluent don't see
>> > these
>> > > doesn't mean they aren't real use cases other have, this is the point
>> of
>> > > the Apache foundation, it shouldn't be the view of just one
>> organisation.
>> > > It really is getting a feeling of the NIH syndrome. Rather than it
>> being
>> > > constructive on discussion of the implementation detail.
>> > >
>> > > kip-87 spawned from as on the kip call we all agreed this was needed.
>> And
>> > > would at least allow a custom wrapper be supported in a compacted
>> topic,
>> > > allowing meta data. Which again now I feel we are spinning wheels, and
>> > > simply finding reasons not support it.
>> > >
>> > > Cheers
>> > > Mike
>> > >
>> > >
>> > >
>> > > Sent using OWA for iPad
>> > > ________________________________________
>> > > From: Jay Kreps <j...@confluent.io>
>> > > Sent: Friday, December 16, 2016 7:09:23 PM
>> > > To: dev@kafka.apache.org
>> > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > >
>> > > Hey Michael,
>> > >
>> > > I do think it might have been better had we started with a separate
>> > concept
>> > > of null vs delete. Given that we are where we are, I'm not sure that
>> the
>> > > possible cures we explored so far are better than the disease.
>> > >
>> > > I apologize for coming to this late, but I didn't really understand
>> the
>> > > implications of the KIP and that we'd be breaking compatibility with
>> > > existing apps until the vote had begun, and in my defense I'm not sure
>> > the
>> > > other folks voting did either.
>> > >
>> > > I think we all agree there are many existing apps that are built with
>> the
>> > > assumption of "null value non-tombstone" and it isn't possible to
>> > > disambiguate these from tombstones on the producer. It isn't that
>> anyone
>> > is
>> > > saying we have to support all four possibilities at once, it's that we
>> > > simply can't orphan one of the existing combinations or our users will
>> > eat
>> > > us!
>> > >
>> > > If I've understood your alternate solution of adding another setting
>> for
>> > > compaction, I think this does fix the compatibility problem, but it
>> adds
>> > an
>> > > odd mode the user has to add on all their topics. While the current
>> state
>> > > is easily explainable, the resulting state where the meaning of
>> tombstone
>> > > and null are overlapping and ambiguous and dependent on a compaction
>> > > setting that could change out of band or not be in sync with your
>> code in
>> > > some environment seems worse to me then where we currently are. I
>> think
>> > the
>> > > question is how would this combination be explained to users and does
>> it
>> > > make sense?
>> > >
>> > > -Jay
>> > >
>> > >
>> > >
>> > > On Fri, Dec 16, 2016 at 9:25 AM, Michael Pearce <
>> michael.pea...@ig.com>
>> > > wrote:
>> > >
>> > > > Hi Chaps,
>> > > >
>> > > > Can we either get one more +1 binding (we have 2 already) on the
>> > > existing?
>> > > >
>> > > > Or have some response on the below possible alternative. We are
>> keen to
>> > > > get working on this, so we make next feature release.
>> > > >
>> > > > Cheers
>> > > > Mike
>> > > >
>> > > >
>> > > > On 13/12/2016, 16:32, "Michael Pearce" <michael.pea...@ig.com>
>> wrote:
>> > > >
>> > > > Hi Ismael
>> > > >
>> > > > Did you see our email this morning, what's your thoughts on this
>> > > > approach to instead we simply have a brand new policy?
>> > > >
>> > > > Cheers
>> > > > Mike
>> > > >
>> > > >
>> > > > Sent using OWA for iPhone
>> > > > ________________________________________
>> > > > From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael
>> > > Juma <
>> > > > ism...@juma.me.uk>
>> > > > Sent: Tuesday, December 13, 2016 11:30:05 AM
>> > > > To: dev@kafka.apache.org
>> > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > >
>> > > > Yes, this is actually tricky to do in a way where we both have
>> the
>> > > > desired
>> > > > semantics and maintain compatibility. When someone creates a
>> > > > `ProducerRecord` with a `null` value today, the producer doesn't
>> > know
>> > > > if
>> > > > it's meant to be a tombstone or not. For V3 messages, it's easy
>> > when
>> > > > the
>> > > > constructor that takes a tombstone is used. However, if any
>> other
>> > > > constructor is used, it's not clear. A couple of options I can
>> > think
>> > > > of,
>> > > > none of them particularly nice:
>> > > >
>> > > > 1. Have a third state where tombstone = unknown and the broker
>> > would
>> > > > set
>> > > > the tombstone bit if the value was null and the topic was
>> > compacted.
>> > > > People
>> > > > that wanted to pass a non-null value for the tombstone would
>> have
>> > to
>> > > > use
>> > > > the constructor that takes a tombstone. The drawbacks: third
>> state
>> > > for
>> > > > tombstone in message format, message conversion at the broker
>> for a
>> > > > common
>> > > > case.
>> > > >
>> > > > 2. Extend MetadataResponse to optionally include topic configs,
>> > which
>> > > > would
>> > > > make it possible for the producer to be smarter about setting
>> the
>> > > > tombstone. It would only do it if a tombstone was not passed
>> > > > explicitly,
>> > > > the value was null and the topic was compacted. The main
>> drawback
>> > is
>> > > > that
>> > > > the producer would be getting a bit more data for each topic
>> even
>> > > > though it
>> > > > probably won't use it most of the time. Extending
>> MetadataResponse
>> > to
>> > > > return topic configs would be useful for other reasons as well,
>> so
>> > > that
>> > > > part seems OK.
>> > > >
>> > > > In addition, for both proposals, we could consider adding
>> warnings
>> > to
>> > > > the
>> > > > documentation that the behaviour of the constructors that don't
>> > take
>> > > a
>> > > > tombstone would change in the next major release so that
>> tombstone
>> > =
>> > > > false.
>> > > > Not sure if this would be worth it though.
>> > > >
>> > > > Ismael
>> > > >
>> > > > On Sun, Dec 11, 2016 at 11:15 PM, Ewen Cheslack-Postava <
>> > > > e...@confluent.io>
>> > > > wrote:
>> > > >
>> > > > > Michael,
>> > > > >
>> > > > > It kind of depends on how you want to interpret the tombstone
>> > flag.
>> > > > If it's
>> > > > > purely a producer-facing Kafka-level thing that we treat as
>> > > internal
>> > > > to the
>> > > > > broker and log cleaner once the record is sent, then your
>> > approach
>> > > > makes
>> > > > > sense. You're just moving copying the null-indicates-delete
>> > > behavior
>> > > > of the
>> > > > > old constructor into the tombstone flag.
>> > > > >
>> > > > > However, if you want this change to more generally decouple
>> the
>> > > idea
>> > > > of
>> > > > > deletion and null values, then you are sometimes converting
>> what
>> > > > might be a
>> > > > > completely valid null value that doesn't indicate deletion
>> into a
>> > > > > tombstone. Downstream applications could potentially handle
>> these
>> > > > cases
>> > > > > differently given the separation of deletion from value.
>> > > > >
>> > > > > I guess the question is if we want to try to support the
>> latter
>> > > even
>> > > > for
>> > > > > topics where we have older produce requests. An example where
>> > this
>> > > > could
>> > > > > come up is in something like a CDC Connector. If we try to
>> > support
>> > > > the
>> > > > > semantic difference, a connector might write changes to Kafka
>> > using
>> > > > the
>> > > > > tombstone flag to indicate when a row was truly deleted (vs an
>> > > > update that
>> > > > > sets it to null but still present; this probably makes more
>> sense
>> > > > for CDC
>> > > > > from document stores or extracting single columns). There are
>> > > various
>> > > > > reasons we might want to maintain the full log and not turn
>> > > > compaction on
>> > > > > (or just use a time-based retention policy), but downstream
>> > > > applications
>> > > > > might care to know the difference between a delete and a null
>> > > value.
>> > > > In
>> > > > > fact both versions of the same log (compacted and
>> time-retention)
>> > > > could be
>> > > > > useful and I don't think it'll be uncommon to maintain both or
>> > use
>> > > > KIP-71
>> > > > > to maintain a hybrid compacted/retention topic.
>> > > > >
>> > > > > -Ewen
>> > > > >
>> > > > > On Sun, Dec 11, 2016 at 1:18 PM, Michael Pearce <
>> > > > michael.pea...@ig.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Jay,
>> > > > > >
>> > > > > > Why wouldn't that work, the tombstone value is only looked
>> at
>> > by
>> > > > the
>> > > > > > broker, on a topic configured for compaction as such is
>> benign
>> > on
>> > > > non
>> > > > > > compacted topics. This is as much as sending a null value
>> > > currently
>> > > > > >
>> > > > > >
>> > > > > > Regards
>> > > > > > Mike
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Sent using OWA for iPhone
>> > > > > > ________________________________________
>> > > > > > From: Jay Kreps <j...@confluent.io>
>> > > > > > Sent: Sunday, December 11, 2016 8:58:53 PM
>> > > > > > To: dev@kafka.apache.org
>> > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > > > >
>> > > > > > Hey Michael,
>> > > > > >
>> > > > > > I'm not quite sure that works as that would translate ALL
>> null
>> > > > values to
>> > > > > > tombstones, even for non-compacted topics that use null as
>> an
>> > > > acceptable
>> > > > > > value sent by the producer and expected by the consumer.
>> > > > > >
>> > > > > > -Jay
>> > > > > >
>> > > > > > On Sun, Dec 11, 2016 at 3:26 AM, Michael Pearce <
>> > > > michael.pea...@ig.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Ewen,
>> > > > > > >
>> > > > > > > I think the easiest way to show this is with code.
>> > > > > > >
>> > > > > > > As you can see we keep the existing behaviour for
>> > code/binaries
>> > > > calling
>> > > > > > > the pre-existing constructors, whereby if the value is
>> null
>> > the
>> > > > > tombstone
>> > > > > > > is set to true.
>> > > > > > >
>> > > > > > > Regards
>> > > > > > > Mike
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > /**
>> > > > > > > * Creates a record with a specified timestamp to be
>> sent
>> > > to
>> > > > a
>> > > > > > > specified topic and partition
>> > > > > > > *
>> > > > > > > * @param topic The topic the record will be appended
>> to
>> > > > > > > * @param partition The partition to which the record
>> > > should
>> > > > be
>> > > > > sent
>> > > > > > > * @param timestamp The timestamp of the record
>> > > > > > > * @param tombstone if the record should be treated
>> as a
>> > > > tombstone
>> > > > > if
>> > > > > > > the topic is compacted
>> > > > > > > * @param key The key that will be included in the
>> record
>> > > > > > > * @param value The record contents
>> > > > > > > */
>> > > > > > > public ProducerRecord(String topic, Integer partition,
>> > > > Boolean
>> > > > > > > tombstone, Long timestamp, K key, V value) {
>> > > > > > > if (topic == null)
>> > > > > > > throw new IllegalArgumentException("Topic
>> cannot
>> > > be
>> > > > > null.");
>> > > > > > > if (timestamp != null && timestamp < 0)
>> > > > > > > throw new IllegalArgumentException(
>> > > > > > > String.format("Invalid timestamp: %d.
>> > > > Timestamp
>> > > > > > should
>> > > > > > > always be non-negative or null.", timestamp));
>> > > > > > > if (partition != null && partition < 0)
>> > > > > > > throw new IllegalArgumentException(
>> > > > > > > String.format("Invalid partition: %d.
>> > > > Partition
>> > > > > > number
>> > > > > > > should always be non-negative or null.", partition));
>> > > > > > > if (!tombstone && value == null){
>> > > > > > > throw new IllegalArgumentException(
>> > > > > > > String.format("Tombstone must be true
>> if
>> > > null
>> > > > > > value"));
>> > > > > > > }
>> > > > > > > this.topic = topic;
>> > > > > > > this.partition = partition;
>> > > > > > > this.tombstone = tombstone;
>> > > > > > > this.key = key;
>> > > > > > > this.value = value;
>> > > > > > > this.timestamp = timestamp;
>> > > > > > > }
>> > > > > > >
>> > > > > > > public ProducerRecord(String topic, Integer partition,
>> > Long
>> > > > > > timestamp,
>> > > > > > > K key, V value) {
>> > > > > > > this(topic, partition, value == null, timestamp,
>> key,
>> > > > value);
>> > > > > > > }
>> > > > > > > ________________________________________
>> > > > > > > From: Ewen Cheslack-Postava <e...@confluent.io>
>> > > > > > > Sent: Sunday, December 11, 2016 5:45 AM
>> > > > > > > To: dev@kafka.apache.org
>> > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone Flag
>> > > > > > >
>> > > > > > > It seemed like this was addressed in the migration
>> section,
>> > > > wasn't it?
>> > > > > > The
>> > > > > > > V2 vs V3 requests and the fact that the broker will
>> downgrade
>> > > the
>> > > > > message
>> > > > > > > format (losing zero copy) if you issues a V2 request to a
>> > > broker
>> > > > using
>> > > > > V3
>> > > > > > > format handles compatibility, does it not? The existing
>> > > > consumers won't
>> > > > > > see
>> > > > > > > the extra metadata in the value, but they will get a null
>> > > > instead and
>> > > > > > treat
>> > > > > > > it as a tombstone. Certainly there is a performance
>> impact,
>> > but
>> > > > it
>> > > > > seemed
>> > > > > > > compatible.
>> > > > > > >
>> > > > > > > I'm worried about this though:
>> > > > > > >
>> > > > > > >
>> > > > > > > - *NOTE *: With the new version of producer client
>> using
>> > > > > > ProduceRequest
>> > > > > > > V3 (magic byte = 2), a non tombstone (tombstone bit not
>> > set)
>> > > > and
>> > > > > null
>> > > > > > > value
>> > > > > > > should not be allowed as the older version of consumer
>> > using
>> > > > > > > FetchRequest
>> > > > > > > V2 will think of this as a tombstone when its actually
>> > not.
>> > > > > > >
>> > > > > > >
>> > > > > > > Unless I'm misunderstanding, this ends up breaking binary
>> > > > compatibility
>> > > > > > for
>> > > > > > > the Java API. It sounds like this suggests that passing a
>> > null
>> > > > value to
>> > > > > > the
>> > > > > > > existing ProducerRecord constructors would generate an
>> > > exception
>> > > > since
>> > > > > > you
>> > > > > > > didn't explicitly enable the tombstone (via whatever new
>> > > > constructor is
>> > > > > > > provided). But that means you can't swap in a newer client
>> > jar
>> > > > without
>> > > > > > > recompiling and get the same behavior if your app deletes
>> > keys
>> > > > using
>> > > > > the
>> > > > > > > current approach because your app will start throwing
>> > > > exceptions. Maybe
>> > > > > > > this is a tradeoff we're ok with, but we've tried pretty
>> hard
>> > > to
>> > > > avoid
>> > > > > > > breaking compatibility like this so far -- it makes
>> picking
>> > up
>> > > > bug
>> > > > > fixes
>> > > > > > in
>> > > > > > > the clients harder for users of frameworks that have to
>> pin
>> > to
>> > > > earlier
>> > > > > > > default versions for compatibility.
>> > > > > > >
>> > > > > > > But then later the KIP says:
>> > > > > > >
>> > > > > > >
>> > > > > > > - The old constructors for ProducerRecord without this
>> > > > parameter
>> > > > > will
>> > > > > > be
>> > > > > > > kept but updated so that their default behaviour if
>> > setting
>> > > > null
>> > > > > value
>> > > > > > > will
>> > > > > > > be the tombstone will be set to true to keep existing
>> > > > behaviour.
>> > > > > > >
>> > > > > > >
>> > > > > > > So maybe I am misinterpreting something.
>> > > > > > >
>> > > > > > > And just a nit re: motivation section, item 6 would be
>> > clearer
>> > > > for a
>> > > > > > union
>> > > > > > > schema with null (or optional schemas in other formats),
>> e.g.
>> > > > [null,
>> > > > > > > string], in which case losing the schema truly is losing
>> > > > information
>> > > > > > > (whereas null is already the only valid value for a pure
>> null
>> > > > schema).
>> > > > > > >
>> > > > > > > -Ewen
>> > > > > > >
>> > > > > > >
>> > > > > > > On Sat, Dec 10, 2016 at 9:24 PM, Michael Pearce <
>> > > > michael.pea...@ig.com
>> > > > > >
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > Hi Jay,
>> > > > > > > >
>> > > > > > > > Good point this detail is missing in the KIP write up.
>> Ive
>> > > > added this
>> > > > > > > now.
>> > > > > > > >
>> > > > > > > > Essentially simply just upgrading the clients we do not
>> > > expect
>> > > > any
>> > > > > > client
>> > > > > > > > app code change needed.
>> > > > > > > >
>> > > > > > > > Cheers
>> > > > > > > > Mike
>> > > > > > > > ________________________________________
>> > > > > > > > From: Jay Kreps <j...@confluent.io>
>> > > > > > > > Sent: Saturday, December 10, 2016 10:51 PM
>> > > > > > > > To: dev@kafka.apache.org
>> > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction Tombstone
>> Flag
>> > > > > > > >
>> > > > > > > > Michael,
>> > > > > > > >
>> > > > > > > > The compatibility section goes through the migration
>> path,
>> > > but
>> > > > isn't
>> > > > > > the
>> > > > > > > > bigger compatibility issue with existing apps? There are
>> > many
>> > > > > (probably
>> > > > > > > > thousands) of apps in production that use this feature
>> and
>> > > > send null
>> > > > > to
>> > > > > > > > mean delete. It seems like this would break
>> compatibility
>> > > with
>> > > > them,
>> > > > > > and
>> > > > > > > > they would have to be rewritten to right?
>> > > > > > > >
>> > > > > > > > -Jay
>> > > > > > > >
>> > > > > > > > On Thu, Dec 8, 2016 at 12:12 AM, Michael Pearce <
>> > > > > michael.pea...@ig.com
>> > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > Hi Jun,
>> > > > > > > > >
>> > > > > > > > > 4) On v3 we honour the tombstone. As such we expect
>> it to
>> > > be
>> > > > set
>> > > > > > > > correctly
>> > > > > > > > > as per the KIP.
>> > > > > > > > >
>> > > > > > > > > 4.1) why would we want to produce an error when its
>> v3?
>> > > This
>> > > > is the
>> > > > > > > exact
>> > > > > > > > > purpose to support non-null tombstone's
>> > > > > > > > > 4.2) again here im unclear on the question on the v3,
>> > > produce
>> > > > > request
>> > > > > > > we
>> > > > > > > > > expect the tombstone flag to be set correctly.
>> > > > > > > > >
>> > > > > > > > > 4.4) compaction only occurs on compacted topics, the
>> bit
>> > > > makes no
>> > > > > > > > > difference and not looked at on un-compacted
>> (time/size
>> > > based
>> > > > > > > eviction).
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > On 06/12/2016, 20:08, "Jun Rao" <j...@confluent.io>
>> > wrote:
>> > > > > > > > >
>> > > > > > > > > Hi, Michael,
>> > > > > > > > >
>> > > > > > > > > 4. Then, I think I misunderstood this point. Could
>> > you
>> > > > document
>> > > > > > the
>> > > > > > > > > following points in the wiki?
>> > > > > > > > > 4.1 If producer V3 sets tombstone, but provides a
>> > > > non-null
>> > > > > value,
>> > > > > > > > does
>> > > > > > > > > the
>> > > > > > > > > send() get an error or does the producer
>> > automatically
>> > > > set the
>> > > > > > > value
>> > > > > > > > to
>> > > > > > > > > null?
>> > > > > > > > > 4.2 If producer V3 doesn't set tombstone, but
>> > provides
>> > > a
>> > > > null
>> > > > > > > value,
>> > > > > > > > > does
>> > > > > > > > > the send() get an error or does the producer
>> > > > automatically sets
>> > > > > > the
>> > > > > > > > > tombstone?
>> > > > > > > > > 4.3 Does the broker only expect messages that (a)
>> > have
>> > > no
>> > > > > > tombstone
>> > > > > > > > and
>> > > > > > > > > non-null value; (b) have tombstone and null value
>> and
>> > > > reject
>> > > > > the
>> > > > > > > > > messages
>> > > > > > > > > with an error code otherwise?
>> > > > > > > > > 4.4 Do 4.1, 4.2, 4.3 depend on whether the topic
>> is
>> > > > compacted
>> > > > > on
>> > > > > > > > not?
>> > > > > > > > >
>> > > > > > > > > Thanks,
>> > > > > > > > >
>> > > > > > > > > Jun
>> > > > > > > > >
>> > > > > > > > > On Tue, Dec 6, 2016 at 10:35 AM, Michael Pearce <
>> > > > > > > > michael.pea...@ig.com
>> > > > > > > > > >
>> > > > > > > > > wrote:
>> > > > > > > > >
>> > > > > > > > > > Not at all. This only acts on compacted topics
>> > just
>> > > > as what
>> > > > > > > occurs
>> > > > > > > > > today
>> > > > > > > > > >
>> > > > > > > > > > Sent using OWA for iPhone
>> > > > > > > > > > ________________________________________
>> > > > > > > > > > From: Jun Rao <j...@confluent.io>
>> > > > > > > > > > Sent: Tuesday, December 6, 2016 6:25:28 PM
>> > > > > > > > > > To: dev@kafka.apache.org
>> > > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > Tombstone
>> > > > Flag
>> > > > > > > > > >
>> > > > > > > > > > Hi, Michael,
>> > > > > > > > > >
>> > > > > > > > > > 4. Hmm, does that mean the new client library
>> can
>> > > > never send
>> > > > > a
>> > > > > > > null
>> > > > > > > > > message
>> > > > > > > > > > even to a regular topic? This seems like a
>> change
>> > of
>> > > > the
>> > > > > > existing
>> > > > > > > > > behavior.
>> > > > > > > > > >
>> > > > > > > > > > Thanks,
>> > > > > > > > > >
>> > > > > > > > > > Jun
>> > > > > > > > > >
>> > > > > > > > > > On Tue, Dec 6, 2016 at 9:51 AM, Michael Pearce <
>> > > > > > > > > michael.pea...@ig.com>
>> > > > > > > > > > wrote:
>> > > > > > > > > >
>> > > > > > > > > > > Hi Jun,
>> > > > > > > > > > >
>> > > > > > > > > > > Re 4) That's because we expect the tombstone
>> > value
>> > > > to be
>> > > > > set
>> > > > > > > > > correctly if
>> > > > > > > > > > > message bit is 2, as such if an older client
>> > sends
>> > > > in on
>> > > > > old
>> > > > > > > > > message the
>> > > > > > > > > > > message is upcast and the bit is set
>> correctly.
>> > And
>> > > > such no
>> > > > > > > > longer
>> > > > > > > > > need
>> > > > > > > > > > to
>> > > > > > > > > > > check the value. Mayuresh can you confirm my
>> > > > thinking and
>> > > > > > > > > understanding
>> > > > > > > > > > of
>> > > > > > > > > > > what we've discussed?
>> > > > > > > > > > >
>> > > > > > > > > > > The second point I understand what you're
>> getting
>> > > at
>> > > > now my
>> > > > > > > > > apologies.
>> > > > > > > > > > Yes
>> > > > > > > > > > > this makes sense to save on touching the
>> message,
>> > > if
>> > > > we're
>> > > > > > the
>> > > > > > > > > only kip
>> > > > > > > > > > > going in, in this release.
>> > > > > > > > > > >
>> > > > > > > > > > > Cheers
>> > > > > > > > > > > Mike
>> > > > > > > > > > >
>> > > > > > > > > > > Sent using OWA for iPhone
>> > > > > > > > > > > ________________________________________
>> > > > > > > > > > > From: Jun Rao <j...@confluent.io>
>> > > > > > > > > > > Sent: Tuesday, December 6, 2016 5:22:13 PM
>> > > > > > > > > > > To: dev@kafka.apache.org
>> > > > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > > > Tombstone Flag
>> > > > > > > > > > >
>> > > > > > > > > > > Hi, Michael,
>> > > > > > > > > > >
>> > > > > > > > > > > 4. Is this updated in the wiki? The text "If
>> the
>> > > > magic byte
>> > > > > > on
>> > > > > > > > > message is
>> > > > > > > > > > > 2, the broker should use the tombstone bit for
>> > log
>> > > > > > compaction."
>> > > > > > > > > doesn't
>> > > > > > > > > > > seem to have changed.
>> > > > > > > > > > >
>> > > > > > > > > > > 2. My point is that if we change the message
>> > format
>> > > > just
>> > > > > for
>> > > > > > > this
>> > > > > > > > > KIP, we
>> > > > > > > > > > > should consider whether it's worth optimizing
>> the
>> > > > down
>> > > > > > > conversion
>> > > > > > > > > path
>> > > > > > > > > > > (i.e., decide whether a conversion is needed
>> by
>> > > just
>> > > > > looking
>> > > > > > at
>> > > > > > > > the
>> > > > > > > > > > > tombstone bit in the wrapper message) since
>> > > > tombstone will
>> > > > > be
>> > > > > > > > used
>> > > > > > > > > > rarely.
>> > > > > > > > > > > However, if the message format change here is
>> > > > combined with
>> > > > > > > other
>> > > > > > > > > KIPs,
>> > > > > > > > > > > then this optimization likely won't be needed.
>> > The
>> > > > latter
>> > > > > > > > probably
>> > > > > > > > > makes
>> > > > > > > > > > > the code simpler. Jiangjie, Mayuresh, what do
>> you
>> > > > think?
>> > > > > > > > > > >
>> > > > > > > > > > > Other than those, +1 from me,
>> > > > > > > > > > >
>> > > > > > > > > > > Thanks,
>> > > > > > > > > > >
>> > > > > > > > > > > Jun
>> > > > > > > > > > >
>> > > > > > > > > > > On Tue, Dec 6, 2016 at 8:54 AM, Michael
>> Pearce <
>> > > > > > > > > michael.pea...@ig.com>
>> > > > > > > > > > > wrote:
>> > > > > > > > > > >
>> > > > > > > > > > > > Hi Jun
>> > > > > > > > > > > >
>> > > > > > > > > > > > do we have your vote on this now?
>> > > > > > > > > > > >
>> > > > > > > > > > > > Any other concerns?
>> > > > > > > > > > > >
>> > > > > > > > > > > > Cheers
>> > > > > > > > > > > > Mike
>> > > > > > > > > > > >
>> > > > > > > > > > > > Sent using OWA for iPhone
>> > > > > > > > > > > > ________________________________________
>> > > > > > > > > > > > From: Michael Pearce <michael.pea...@ig.com
>> >
>> > > > > > > > > > > > Sent: Saturday, December 3, 2016 1:37:45 AM
>> > > > > > > > > > > > To: dev@kafka.apache.org
>> > > > > > > > > > > > Subject: Re: [VOTE] KIP-87 - Add Compaction
>> > > > Tombstone
>> > > > > Flag
>> > > > > > > > > > > >
>> > > > > > > > > > > > Hi Jun,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Have updated. Thanks again for the feedback.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Agree yes we should align up when it gets to
>> > > that,
>> > > > I
>> > > > > assume
>> > > > > > > > > you've
>> > > > > > > > > > > flagged
>> > > > > > > > > > > > the same in the other KIP?
>> > > > > > > > > > > >
>> > > > > > > > > > > > I think for now let's get this KIP approved,
>> > then
>> > > > we can
>> > > > > > > start
>> > > > > > > > > the work
>> > > > > > > > > > > to
>> > > > > > > > > > > > get an initial PR, then we can discuss how
>> to
>> > > > align the
>> > > > > two
>> > > > > > > if
>> > > > > > > > > needed.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Cheers,
>> > > > > > > > > > > > Mike
>> > > > > > > > > > > >
>> > > > > > > > > > > > On 02/12/2016, 18:26, "Jun Rao" <
>> > > j...@confluent.io>
>> > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > Hi, Michael,
>> > > > > > > > > > > >
>> > > > > > > > > > > > For 2), this is fine. Could you update
>> the
>> > > KIP
>> > > > wiki
>> > > > > to
>> > > > > > > make
>> > > > > > > > > this
>> > > > > > > > > > and
>> > > > > > > > > > > > other
>> > > > > > > > > > > > points clearer? Other than that, the KIP
>> > > looks
>> > > > good
>> > > > > to
>> > > > > > > me.
>> > > > > > > > > > > >
>> > > > > > > > > > > > An orthogonal thing is that there are
>> other
>> > > > KIPs such
>> > > > > > as
>> > > > > > > > > KIP-98
>> > > > > > > > > > that
>> > > > > > > > > > > > also
>> > > > > > > > > > > > intend to change the message format. If
>> > they
>> > > > all get
>> > > > > > > > > approved, we
>> > > > > > > > > > > > should
>> > > > > > > > > > > > think about whether it's better to just
>> > bump
>> > > > up the
>> > > > > > magic
>> > > > > > > > > byte once
>> > > > > > > > > > > to
>> > > > > > > > > > > > incorporate multiple format changes
>> like we
>> > > > did in
>> > > > > > > > > KIP-31/KIP-32.
>> > > > > > > > > > > >
>> > > > > > > > > > > > Thanks,
>> > > > > > > > > > > >
>> > > > > > > > > > > > Jun
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > On Fri, Dec 2, 2016 at 3:19 AM, Michael
>> > > Pearce
>> > > > <
>> > > > > > > > > > > michael.pea...@ig.com>
>> > > > > > > > > > > > wrote:
>> > > > > > > > > > > >
>> > > > > > > > > > > > > Hi Jao
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for the response. Sorry for
>> slow
>> > > > reply, both
>> > > > > > > with
>> > > > > > > > > personal
>> > > > > > > > > > > > sickness
>> > > > > > > > > > > > > and also battling some critical issues
>> > > > encountered
>> > > > > > > since
>> > > > > > > > > > upgrading
>> > > > > > > > > > > to
>> > > > > > > > > > > > > 0.10.1.0
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 1) Thans for spotting, Document error
>> > where
>> > > > we
>> > > > > > branched
>> > > > > > > > > this KIP
>> > > > > > > > > > > from
>> > > > > > > > > > > > > KIP-82, will get that removed.
>> > > > > > > > > > > > > 2) Intent is to do this just at the
>> > record
>> > > > message
>> > > > > > > level.
>> > > > > > > > > > > > > 3) Thanks for spotting, Will ensure
>> this
>> > is
>> > > > > > corrected.
>> > > > > > > > > > > > > 4) As per discussion thread we will
>> > support
>> > > > > > tombstone +
>> > > > > > > > > null
>> > > > > > > > > > value,
>> > > > > > > > > > > > > tombstone + non null value, no
>> tombstone
>> > +
>> > > > null
>> > > > > > value.
>> > > > > > > > > > > > > 5) I believe this was in the
>> discussion
>> > > > thread,
>> > > > > > > @Mayuresh
>> > > > > > > > > is this
>> > > > > > > > > > > > > something we've overlooked? I thought
>> we
>> > > > would down
>> > > > > > > > > convert and
>> > > > > > > > > > > > remove the
>> > > > > > > > > > > > > value so the old consumer had existing
>> > > > behavior, or
>> > > > > > is
>> > > > > > > > > there
>> > > > > > > > > > > > something we
>> > > > > > > > > > > > > haven't thought about?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Cheers
>> > > > > > > > > > > > > Mike
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On 30/11/2016, 18:12, "Jun Rao" <
>> > > > j...@confluent.io>
>> > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Hi, Michael,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks for the KIP. A few comments
>> > > below.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 1. The message format change
>> contains
>> > > > > > > "HeadersLength
>> > > > > > > > > > Headers".
>> > > > > > > > > > > > Is that
>> > > > > > > > > > > > > intended?
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 2. For compressed messageset, is
>> the
>> > > > tombstone
>> > > > > > bit
>> > > > > > > > > only set
>> > > > > > > > > > at
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > shallow
>> > > > > > > > > > > > > level? Do we always leave that
>> bit in
>> > > the
>> > > > > wrapper
>> > > > > > > > > message
>> > > > > > > > > > > unset?
>> > > > > > > > > > > > An
>> > > > > > > > > > > > > alternative is to set the
>> tombstone
>> > bit
>> > > > in the
>> > > > > > > > wrapper
>> > > > > > > > > if at
>> > > > > > > > > > > > least one
>> > > > > > > > > > > > > inner message has the tombstone
>> bit
>> > > set.
>> > > > This
>> > > > > > makes
>> > > > > > > > > things a
>> > > > > > > > > > > bit
>> > > > > > > > > > > > more
>> > > > > > > > > > > > > complicated, but we could
>> potentially
>> > > > exploit
>> > > > > > that
>> > > > > > > > for
>> > > > > > > > > > > > optimizing down
>> > > > > > > > > > > > > conversion. For example, we only
>> need
>> > > to
>> > > > > convert
>> > > > > > > > > messages
>> > > > > > > > > > with
>> > > > > > > > > > > > magic 2
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > magic 1 if the wrapper's tombstone
>> > bit
>> > > > is set
>> > > > > > > > > (conversion is
>> > > > > > > > > > > > always
>> > > > > > > > > > > > > needed
>> > > > > > > > > > > > > from magic 2 to magic 0). Not
>> sure if
>> > > the
>> > > > > > > > optimization
>> > > > > > > > > is
>> > > > > > > > > > worth
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > complexity though.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 3. The referencing of the new
>> version
>> > > of
>> > > > > > > > > > > > ProducerRequest/FetchRequest
>> > > > > > > > > > > > > is
>> > > > > > > > > > > > > inconsistent (v4 vs v3). Since our
>> > > > convention
>> > > > > > > starts
>> > > > > > > > at
>> > > > > > > > > > version
>> > > > > > > > > > > > at 0, I
>> > > > > > > > > > > > > think the new version would be 3.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 4. "If the magic byte on message
>> is
>> > 2,
>> > > > the
>> > > > > broker
>> > > > > > > > > should use
>> > > > > > > > > > > the
>> > > > > > > > > > > > > tombstone
>> > > > > > > > > > > > > bit for log compaction." What
>> about
>> > > null
>> > > > value?
>> > > > > > My
>> > > > > > > > > > > understanding
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > that
>> > > > > > > > > > > > > null value will be treated the
>> same
>> > as
>> > > > setting
>> > > > > > the
>> > > > > > > > > tombstone
>> > > > > > > > > > > bit.
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > 5. For the migration path, it
>> would
>> > be
>> > > > useful
>> > > > > to
>> > > > > > > > > describe the
>> > > > > > > > > > > > down
>> > > > > > > > > > > > > conversion path to consumers
>> (i.e.,
>> > > > brokers on
>> > > > > > > > message
>> > > > > > > > > format
>> > > > > > > > > > > > 0.10.2
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > consumers on older version).
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > Jun
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > On Tue, Nov 29, 2016 at 3:18 AM,
>> > > Michael
>> > > > > Pearce <
>> > > > > > > > > > > > michael.pea...@ig.com
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi All,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > We have been discussing in the
>> > below
>> > > > thread
>> > > > > and
>> > > > > > > > final
>> > > > > > > > > > changes
>> > > > > > > > > > > > have
>> > > > > > > > > > > > > been
>> > > > > > > > > > > > > > made to the KIP wiki based on
>> these
>> > > > > > discussions.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > We would now like to put to the
>> > vote
>> > > > the
>> > > > > > > following
>> > > > > > > > > KIP:
>> > > > > > > > > > > > > > https://cwiki.apache.org/
>> > > > > > > > > confluence/display/KAFKA/KIP-
>> > > > > > > > > > 87+-+
>> > > > > > > > > > > > > > Add+Compaction+Tombstone+Flag
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > This kip is for having a
>> distinct
>> > > > compaction
>> > > > > > > > > attribute
>> > > > > > > > > > > > "tombstone"
>> > > > > > > > > > > > > flag
>> > > > > > > > > > > > > > instead of relying on null
>> value,
>> > > > allowing
>> > > > > > > non-null
>> > > > > > > > > value
>> > > > > > > > > > > > delete
>> > > > > > > > > > > > > messages.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Many thanks,
>> > > > > > > > > > > > > > Michael
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On 22/11/2016, 15:52, "Michael
>> > > Pearce"
>> > > > <
>> > > > > > > > > > > michael.pea...@ig.com>
>> > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi Mayuresh,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > LGTM. Ive just made one
>> small
>> > > > adjustment
>> > > > > > > > > updating the
>> > > > > > > > > > > wire
>> > > > > > > > > > > > > protocol to
>> > > > > > > > > > > > > > show the magic byte bump.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Do we think we're good to
>> put
>> > to
>> > > a
>> > > > vote?
>> > > > > Is
>> > > > > > > > > there any
>> > > > > > > > > > > > other bits
>> > > > > > > > > > > > > > needing discussion?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Cheers
>> > > > > > > > > > > > > > Mike
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On 21/11/2016, 18:26,
>> "Mayuresh
>> > > > Gharat" <
>> > > > > > > > > > > > > gharatmayures...@gmail.com>
>> > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Hi Michael,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > I have updated the
>> > migration
>> > > > section
>> > > > > of
>> > > > > > > the
>> > > > > > > > > KIP.
>> > > > > > > > > > Can
>> > > > > > > > > > > > you
>> > > > > > > > > > > > > please
>> > > > > > > > > > > > > > take a look?
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > Mayuresh
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > On Fri, Nov 18, 2016 at
>> > 9:07
>> > > > AM,
>> > > > > > Mayuresh
>> > > > > > > > > Gharat <
>> > > > > > > > > > > > > > gharatmayures...@gmail.com
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Hi Michael,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > That whilst sending
>> > > > tombstone and
>> > > > > non
>> > > > > > > > null
>> > > > > > > > > value,
>> > > > > > > > > > > the
>> > > > > > > > > > > > > consumer
>> > > > > > > > > > > > > > can expect
>> > > > > > > > > > > > > > > only to receive the
>> > > non-null
>> > > > > message
>> > > > > > > only
>> > > > > > > > > in step
>> > > > > > > > > > > > (3) is
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > correct?
>> > > > > > > > > > > > > > > ---> I do agree with
>> you
>> > > > here.
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Becket, Ismael : can
>> you
>> > > guys
>> > > > > review
>> > > > > > > the
>> > > > > > > > > > migration
>> > > > > > > > > > > > plan
>> > > > > > > > > > > > > listed
>> > > > > > > > > > > > > > above using
>> > > > > > > > > > > > > > > magic byte?
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Thanks,
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > Mayuresh
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > On Fri, Nov 18, 2016
>> at
>> > > 8:58
>> > > > AM,
>> > > > > > > Michael
>> > > > > > > > > Pearce <
>> > > > > > > > > > > > > > michael.pea...@ig.com>
>> > > > > > > > > > > > > > > wrote:
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >> Many thanks for this
>> > > > Mayuresh. I
>> > > > > > don't
>> > > > > > > > > have any
>> > > > > > > > > > > > > objections.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> I assume we should
>> > state:
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> That whilst sending
>> > > > tombstone and
>> > > > > > non
>> > > > > > > > null
>> > > > > > > > > > value,
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > consumer
>> > > > > > > > > > > > > > can expect
>> > > > > > > > > > > > > > >> only to receive the
>> > > non-null
>> > > > > message
>> > > > > > > > only
>> > > > > > > > > in
>> > > > > > > > > > step
>> > > > > > > > > > > > (3) is
>> > > > > > > > > > > > > this
>> > > > > > > > > > > > > > correct?
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Cheers
>> > > > > > > > > > > > > > >> Mike
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Sent using OWA for
>> > iPhone
>> > > > > > > > > > > > > > >>
>> > > > ______________________________
>> > > > > > > > __________
>> > > > > > > > > > > > > > >> From: Mayuresh
>> Gharat <
>> > > > > > > > > > gharatmayures...@gmail.com
>> > > > > > > > > > > >
>> > > > > > > > > > > > > > >> Sent: Thursday,
>> November
>> > > > 17, 2016
>> > > > > > > > 5:18:41
>> > > > > > > > > PM
>> > > > > > > > > > > > > > >> To:
>> > dev@kafka.apache.org
>> > > > > > > > > > > > > > >> Subject: Re:
>> [DISCUSS]
>> > > > KIP-87 -
>> > > > > Add
>> > > > > > > > > Compaction
>> > > > > > > > > > > > Tombstone
>> > > > > > > > > > > > > Flag
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Hi Ismael,
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Thanks for the
>> > > explanation.
>> > > > > > > > > > > > > > >> Specially I like this
>> > part
>> > > > where
>> > > > > in
>> > > > > > > you
>> > > > > > > > > > mentioned
>> > > > > > > > > > > > we can
>> > > > > > > > > > > > > get
>> > > > > > > > > > > > > > rid of the
>> > > > > > > > > > > > > > >> older null value
>> support
>> > > > for log
>> > > > > > > > > compaction
>> > > > > > > > > > later
>> > > > > > > > > > > > on,
>> > > > > > > > > > > > > here :
>> > > > > > > > > > > > > > >> We can't change
>> > semantics
>> > > > of the
>> > > > > > > message
>> > > > > > > > > format
>> > > > > > > > > > > > without
>> > > > > > > > > > > > > having
>> > > > > > > > > > > > > > a long
>> > > > > > > > > > > > > > >> transition period.
>> And
>> > we
>> > > > can't
>> > > > > rely
>> > > > > > > > > > > > > > >> on people reading
>> > > > documentation or
>> > > > > > > > acting
>> > > > > > > > > on a
>> > > > > > > > > > > > warning for
>> > > > > > > > > > > > > > something so
>> > > > > > > > > > > > > > >> fundamental. As
>> such, my
>> > > > take is
>> > > > > > that
>> > > > > > > we
>> > > > > > > > > need to
>> > > > > > > > > > > > bump the
>> > > > > > > > > > > > > magic
>> > > > > > > > > > > > > > byte. The
>> > > > > > > > > > > > > > >> good news is
>> > > > > > > > > > > > > > >> that we don't have to
>> > > > support all
>> > > > > > > > versions
>> > > > > > > > > > > forever.
>> > > > > > > > > > > > We
>> > > > > > > > > > > > > have
>> > > > > > > > > > > > > > said that we
>> > > > > > > > > > > > > > >> will support direct
>> > > > upgrades for 2
>> > > > > > > > years.
>> > > > > > > > > That
>> > > > > > > > > > > > means that
>> > > > > > > > > > > > > > message format
>> > > > > > > > > > > > > > >> version n could, in
>> > > theory,
>> > > > be
>> > > > > > > removed 2
>> > > > > > > > > years
>> > > > > > > > > > > > after the
>> > > > > > > > > > > > > it's
>> > > > > > > > > > > > > > introduced.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Just a heads up, I
>> would
>> > > > like to
>> > > > > > > mention
>> > > > > > > > > that
>> > > > > > > > > > even
>> > > > > > > > > > > > without
>> > > > > > > > > > > > > > bumping magic
>> > > > > > > > > > > > > > >> byte, we will *NOT*
>> > loose
>> > > > zero
>> > > > > copy
>> > > > > > as
>> > > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > > > client(x+1)
>> > > > > > > > > > > > > in my
>> > > > > > > > > > > > > > >> explanation
>> > > > > > > > > > > > > > >> above will convert
>> > > > internally a
>> > > > > null
>> > > > > > > > > value to
>> > > > > > > > > > > have a
>> > > > > > > > > > > > > tombstone
>> > > > > > > > > > > > > > bit set and
>> > > > > > > > > > > > > > >> a tombstone bit set
>> to
>> > > have
>> > > > a null
>> > > > > > > value
>> > > > > > > > > > > > automatically
>> > > > > > > > > > > > > > internally and by
>> > > > > > > > > > > > > > >> the time we move to
>> > > version
>> > > > (x+2),
>> > > > > > the
>> > > > > > > > > clients
>> > > > > > > > > > > > would have
>> > > > > > > > > > > > > > upgraded.
>> > > > > > > > > > > > > > >> Obviously if we
>> support
>> > a
>> > > > request
>> > > > > > from
>> > > > > > > > > > > consumer(x),
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > will
>> > > > > > > > > > > > > > loose zero
>> > > > > > > > > > > > > > >> copy
>> > > > > > > > > > > > > > >> but that is the same
>> > case
>> > > > with
>> > > > > magic
>> > > > > > > > byte.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> But if magic byte
>> bump
>> > > > makes life
>> > > > > > > easier
>> > > > > > > > > for
>> > > > > > > > > > > > transition
>> > > > > > > > > > > > > for the
>> > > > > > > > > > > > > > above
>> > > > > > > > > > > > > > >> reasons that you
>> > > explained,
>> > > > I am
>> > > > > OK
>> > > > > > > with
>> > > > > > > > > it
>> > > > > > > > > > since
>> > > > > > > > > > > > we are
>> > > > > > > > > > > > > going
>> > > > > > > > > > > > > > to meet the
>> > > > > > > > > > > > > > >> end goal down the
>> road
>> > :)
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> On a side note can we
>> > > > update the
>> > > > > doc
>> > > > > > > > here
>> > > > > > > > > on
>> > > > > > > > > > magic
>> > > > > > > > > > > > byte
>> > > > > > > > > > > > > to say
>> > > > > > > > > > > > > > that "*it
>> > > > > > > > > > > > > > >> should be bumped
>> > whenever
>> > > > the
>> > > > > > message
>> > > > > > > > > format is
>> > > > > > > > > > > > changed
>> > > > > > > > > > > > > or the
>> > > > > > > > > > > > > > >> interpretation of
>> > message
>> > > > format
>> > > > > > > (usage
>> > > > > > > > > of the
>> > > > > > > > > > > > reserved
>> > > > > > > > > > > > > bits as
>> > > > > > > > > > > > > > well) is
>> > > > > > > > > > > > > > >> changed*".
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Hi Michael,
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Here is the update
>> plan
>> > > > that we
>> > > > > > > > discussed
>> > > > > > > > > > offline
>> > > > > > > > > > > > > yesterday :
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Currently the
>> magic-byte
>> > > > which
>> > > > > > > > > corresponds to
>> > > > > > > > > > the
>> > > > > > > > > > > > > > "message.format.version"
>> > > > > > > > > > > > > > >> is set to 1.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> 1) On broker it will
>> be
>> > > set
>> > > > to 1
>> > > > > > > > > initially.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> 2) When a producer
>> > client
>> > > > sends a
>> > > > > > > > message
>> > > > > > > > > with
>> > > > > > > > > > > > magic-byte
>> > > > > > > > > > > > > = 2,
>> > > > > > > > > > > > > > since the
>> > > > > > > > > > > > > > >> broker is on
>> magic-byte
>> > =
>> > > > 1, we
>> > > > > will
>> > > > > > > > down
>> > > > > > > > > > convert
>> > > > > > > > > > > > it,
>> > > > > > > > > > > > > which
>> > > > > > > > > > > > > > means if the
>> > > > > > > > > > > > > > >> tombstone bit is set,
>> > the
>> > > > value
>> > > > > will
>> > > > > > > be
>> > > > > > > > > set to
>> > > > > > > > > > > > null. A
>> > > > > > > > > > > > > consumer
>> > > > > > > > > > > > > > >> understanding
>> > magic-byte =
>> > > > 1, will
>> > > > > > > still
>> > > > > > > > > work
>> > > > > > > > > > with
>> > > > > > > > > > > > this. A
>> > > > > > > > > > > > > > consumer
>> > > > > > > > > > > > > > >> working
>> > > > > > > > > > > > > > >> with magic-byte =2
>> will
>> > > > also be
>> > > > > able
>> > > > > > > to
>> > > > > > > > > > understand
>> > > > > > > > > > > > this,
>> > > > > > > > > > > > > since
>> > > > > > > > > > > > > > it
>> > > > > > > > > > > > > > >> understands the
>> > tombstone.
>> > > > > > > > > > > > > > >> Now there is still
>> the
>> > > > question of
>> > > > > > > > > supporting a
>> > > > > > > > > > > > > non-tombstone
>> > > > > > > > > > > > > > and null
>> > > > > > > > > > > > > > >> value from producer
>> > client
>> > > > with
>> > > > > > > > > magic-byte = 2.*
>> > > > > > > > > > > (I
>> > > > > > > > > > > > am
>> > > > > > > > > > > > > not sure
>> > > > > > > > > > > > > > if we
>> > > > > > > > > > > > > > >> should support this.
>> > > > Ismael/Becket
>> > > > > > can
>> > > > > > > > > comment
>> > > > > > > > > > > > here)*
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> 3) When almost all
>> the
>> > > > clients
>> > > > > have
>> > > > > > > > > upgraded,
>> > > > > > > > > > the
>> > > > > > > > > > > > > > message.format.version
>> > > > > > > > > > > > > > >> on
>> > > > > > > > > > > > > > >> the broker can be
>> > changed
>> > > > to 2,
>> > > > > > where
>> > > > > > > in
>> > > > > > > > > the
>> > > > > > > > > > down
>> > > > > > > > > > > > > conversion in
>> > > > > > > > > > > > > > the above
>> > > > > > > > > > > > > > >> step will not
>> happen. If
>> > > at
>> > > > this
>> > > > > > point
>> > > > > > > > we
>> > > > > > > > > get a
>> > > > > > > > > > > > consumer
>> > > > > > > > > > > > > > request from a
>> > > > > > > > > > > > > > >> older consumer, we
>> might
>> > > > have to
>> > > > > > down
>> > > > > > > > > convert
>> > > > > > > > > > > where
>> > > > > > > > > > > > in we
>> > > > > > > > > > > > > loose
>> > > > > > > > > > > > > > zero copy,
>> > > > > > > > > > > > > > >> but these cases
>> should
>> > be
>> > > > rare.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Becket can you review
>> > this
>> > > > plan
>> > > > > and
>> > > > > > > add
>> > > > > > > > > more
>> > > > > > > > > > > > details if I
>> > > > > > > > > > > > > have
>> > > > > > > > > > > > > > >> missed/wronged
>> > something,
>> > > > before
>> > > > > we
>> > > > > > > put
>> > > > > > > > > it on
>> > > > > > > > > > KIP.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Thanks,
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> Mayuresh
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> On Wed, Nov 16, 2016
>> at
>> > > > 11:07 PM,
>> > > > > > > > Michael
>> > > > > > > > > > Pearce <
>> > > > > > > > > > > > > > michael.pea...@ig.com>
>> > > > > > > > > > > > > > >> wrote:
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> > Thanks guys, for
>> > > > discussing this
>> > > > > > > > > offline and
>> > > > > > > > > > > > getting
>> > > > > > > > > > > > > some
>> > > > > > > > > > > > > > consensus.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > So its clear for
>> > myself
>> > > > and
>> > > > > others
>> > > > > > > > what
>> > > > > > > > > is
>> > > > > > > > > > > > proposed now
>> > > > > > > > > > > > > (i
>> > > > > > > > > > > > > > think i
>> > > > > > > > > > > > > > >> > understand, but
>> want
>> > to
>> > > > make
>> > > > > sure)
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > Could i ask either
>> > > > directly
>> > > > > update
>> > > > > > > the
>> > > > > > > > > kip to
>> > > > > > > > > > > > detail the
>> > > > > > > > > > > > > > migration
>> > > > > > > > > > > > > > >> > strategy, or
>> > (re-)state
>> > > > your
>> > > > > > offline
>> > > > > > > > > discussed
>> > > > > > > > > > > and
>> > > > > > > > > > > > > agreed
>> > > > > > > > > > > > > > migration
>> > > > > > > > > > > > > > >> > strategy based on a
>> > > magic
>> > > > byte
>> > > > > is
>> > > > > > in
>> > > > > > > > > this
>> > > > > > > > > > > thread.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > The main original
>> > driver
>> > > > for the
>> > > > > > KIP
>> > > > > > > > > was to
>> > > > > > > > > > > > support
>> > > > > > > > > > > > > > compaction where
>> > > > > > > > > > > > > > >> value
>> > > > > > > > > > > > > > >> > isn't null, based
>> off
>> > > the
>> > > > > > > discussions
>> > > > > > > > on
>> > > > > > > > > > KIP-82
>> > > > > > > > > > > > thread.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > We should be able
>> to
>> > > > support
>> > > > > > > > > non-tombstone +
>> > > > > > > > > > > null
>> > > > > > > > > > > > value
>> > > > > > > > > > > > > by the
>> > > > > > > > > > > > > > >> completion
>> > > > > > > > > > > > > > >> > of the KIP, as we
>> > noted
>> > > > when
>> > > > > > > > discussing
>> > > > > > > > > this
>> > > > > > > > > > > kip,
>> > > > > > > > > > > > having
>> > > > > > > > > > > > > > logic based on
>> > > > > > > > > > > > > > >> a
>> > > > > > > > > > > > > > >> > null value isn't
>> very
>> > > > clean and
>> > > > > > also
>> > > > > > > > > separates
>> > > > > > > > > > > the
>> > > > > > > > > > > > > concerns.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > As discussed
>> already
>> > > > though we
>> > > > > can
>> > > > > > > > > split this
>> > > > > > > > > > > into
>> > > > > > > > > > > > > KIP-87a
>> > > > > > > > > > > > > > and KIP-87b
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > Where we look to
>> > deliver
>> > > > KIP-87a
>> > > > > > on
>> > > > > > > a
>> > > > > > > > > > compacted
>> > > > > > > > > > > > topic
>> > > > > > > > > > > > > (to
>> > > > > > > > > > > > > > address the
>> > > > > > > > > > > > > > >> > immediate issues)
>> > > > > > > > > > > > > > >> > * tombstone + null
>> > value
>> > > > > > > > > > > > > > >> > * tombstone +
>> non-null
>> > > > value
>> > > > > > > > > > > > > > >> > * non-tombstone +
>> > > > non-null value
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > Then we can discuss
>> > once
>> > > > KIP-87a
>> > > > > > is
>> > > > > > > > > completed
>> > > > > > > > > > > > options
>> > > > > > > > > > > > > later
>> > > > > > > > > > > > > > and how we
>> > > > > > > > > > > > > > >> > support the second
>> > part
>> > > > KIP-87b
>> > > > > to
>> > > > > > > > > deliver:
>> > > > > > > > > > > > > > >> > * non-tombstone +
>> null
>> > > > value
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > Cheers
>> > > > > > > > > > > > > > >> > Mike
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> >
>> > > > ______________________________
>> > > > > > > > > __________
>> > > > > > > > > > > > > > >> > From: Becket Qin <
>> > > > > > > > becket....@gmail.com>
>> > > > > > > > > > > > > > >> > Sent: Thursday,
>> > November
>> > > > 17,
>> > > > > 2016
>> > > > > > > 1:43
>> > > > > > > > > AM
>> > > > > > > > > > > > > > >> > To:
>> > > dev@kafka.apache.org
>> > > > > > > > > > > > > > >> > Subject: Re:
>> [DISCUSS]
>> > > > KIP-87 -
>> > > > > > Add
>> > > > > > > > > Compaction
>> > > > > > > > > > > > > Tombstone Flag
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > Renu, Mayuresh and
>> I
>> > had
>> > > > an
>> > > > > > offline
>> > > > > > > > > > discussion,
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > following
>> > > > > > > > > > > > > > is a brief
>> > > > > > > > > > > > > > >> > summary.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > 1. We agreed that
>> not
>> > > > bumping up
>> > > > > > > magic
>> > > > > > > > > value
>> > > > > > > > > > may
>> > > > > > > > > > > > result
>> > > > > > > > > > > > > in
>> > > > > > > > > > > > > > losing zero
>> > > > > > > > > > > > > > >> copy
>> > > > > > > > > > > > > > >> > during migration.
>> > > > > > > > > > > > > > >> > 2. Given that
>> bumping
>> > up
>> > > > magic
>> > > > > > value
>> > > > > > > > is
>> > > > > > > > > almost
>> > > > > > > > > > > > free and
>> > > > > > > > > > > > > has
>> > > > > > > > > > > > > > benefit of
>> > > > > > > > > > > > > > >> > avoiding potential
>> > > > performance
>> > > > > > > issue.
>> > > > > > > > > It is
>> > > > > > > > > > > > probably
>> > > > > > > > > > > > > worth
>> > > > > > > > > > > > > > doing.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > One issue we still
>> > need
>> > > > to think
>> > > > > > > about
>> > > > > > > > > is
>> > > > > > > > > > > whether
>> > > > > > > > > > > > we
>> > > > > > > > > > > > > want to
>> > > > > > > > > > > > > > support a
>> > > > > > > > > > > > > > >> > non-tombstone
>> message
>> > > > with null
>> > > > > > > value.
>> > > > > > > > > > > > > > >> > Currently it is not
>> > > > supported by
>> > > > > > > > Kafka.
>> > > > > > > > > If we
>> > > > > > > > > > > > allow a
>> > > > > > > > > > > > > > non-tombstone null
>> > > > > > > > > > > > > > >> > value message to
>> exist
>> > > > after
>> > > > > > KIP-87.
>> > > > > > > > The
>> > > > > > > > > > problem
>> > > > > > > > > > > > is
>> > > > > > > > > > > > > that such
>> > > > > > > > > > > > > > message
>> > > > > > > > > > > > > > >> will
>> > > > > > > > > > > > > > >> > not be supported by
>> > the
>> > > > > consumers
>> > > > > > > > prior
>> > > > > > > > > to
>> > > > > > > > > > > KIP-87.
>> > > > > > > > > > > > > Because a
>> > > > > > > > > > > > > > null value
>> > > > > > > > > > > > > > >> > will always be
>> > > > interpreted to a
>> > > > > > > > > tombstone.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > One option is that
>> we
>> > > > keep the
>> > > > > > > current
>> > > > > > > > > way,
>> > > > > > > > > > i.e.
>> > > > > > > > > > > > do not
>> > > > > > > > > > > > > > support such
>> > > > > > > > > > > > > > >> > message. It would
>> be
>> > > good
>> > > > to
>> > > > > know
>> > > > > > if
>> > > > > > > > > there is
>> > > > > > > > > > a
>> > > > > > > > > > > > > concrete use
>> > > > > > > > > > > > > > case for
>> > > > > > > > > > > > > > >> such
>> > > > > > > > > > > > > > >> > message. If there
>> is
>> > > not,
>> > > > we can
>> > > > > > > > > probably just
>> > > > > > > > > > > not
>> > > > > > > > > > > > > support it.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > Thanks,
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > JIangjie (Becket)
>> Qin
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > On Wed, Nov 16,
>> 2016
>> > at
>> > > > 1:28 PM,
>> > > > > > > > > Mayuresh
>> > > > > > > > > > > Gharat <
>> > > > > > > > > > > > > > >> >
>> > > > gharatmayures...@gmail.com
>> > > > > > > > > > > > > > >> > > wrote:
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >> > > Hi Ismael,
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > This is
>> something I
>> > > can
>> > > > think
>> > > > > of
>> > > > > > > for
>> > > > > > > > > > migration
>> > > > > > > > > > > > plan:
>> > > > > > > > > > > > > > >> > > So the migration
>> > plan
>> > > > can look
>> > > > > > > > > something
>> > > > > > > > > > like
>> > > > > > > > > > > > this,
>> > > > > > > > > > > > > with up
>> > > > > > > > > > > > > > >> conversion :
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > 1) Currently lets
>> > say
>> > > > we have
>> > > > > > > Broker
>> > > > > > > > > at
>> > > > > > > > > > > version
>> > > > > > > > > > > > x.
>> > > > > > > > > > > > > > >> > > 2) Currently we
>> have
>> > > > clients
>> > > > > at
>> > > > > > > > > version x.
>> > > > > > > > > > > > > > >> > > 3) a) We move the
>> > > > version to
>> > > > > > > > > Broker(x+1) :
>> > > > > > > > > > > > supports
>> > > > > > > > > > > > > both
>> > > > > > > > > > > > > > tombstone and
>> > > > > > > > > > > > > > >> > null
>> > > > > > > > > > > > > > >> > > for log
>> compaction.
>> > > > > > > > > > > > > > >> > > b) We upgrade
>> > the
>> > > > client
>> > > > > to
>> > > > > > > > > version
>> > > > > > > > > > > > client(x+1) :
>> > > > > > > > > > > > > if in
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > >> producer
>> > > > > > > > > > > > > > >> > > client(x+1) the
>> > value
>> > > > is set
>> > > > > to
>> > > > > > > > null,
>> > > > > > > > > we
>> > > > > > > > > > will
>> > > > > > > > > > > > > automatically
>> > > > > > > > > > > > > > set the
>> > > > > > > > > > > > > > >> > > Tombstone bit
>> > > > internally. If
>> > > > > the
>> > > > > > > > > producer
>> > > > > > > > > > > > client(x+1)
>> > > > > > > > > > > > > sets
>> > > > > > > > > > > > > > the
>> > > > > > > > > > > > > > >> tombstone
>> > > > > > > > > > > > > > >> > > itself, well and
>> > good.
>> > > > For
>> > > > > > > producer
>> > > > > > > > > > client(x),
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > broker
>> > > > > > > > > > > > > > will up
>> > > > > > > > > > > > > > >> convert
>> > > > > > > > > > > > > > >> > > to have the
>> > tombstone
>> > > > bit.
>> > > > > > > > > Broker(x+1) is
>> > > > > > > > > > > > supporting
>> > > > > > > > > > > > > both.
>> > > > > > > > > > > > > > Consumer
>> > > > > > > > > > > > > > >> > > client(x+1) will
>> be
>> > > > aware of
>> > > > > > this
>> > > > > > > > and
>> > > > > > > > > should
>> > > > > > > > > > > be
>> > > > > > > > > > > > able
>> > > > > > > > > > > > > to
>> > > > > > > > > > > > > > handle this.
>> > > > > > > > > > > > > > >> For
>> > > > > > > > > > > > > > >> > > consumer
>> client(x)
>> > we
>> > > > will
>> > > > > down
>> > > > > > > > > convert the
>> > > > > > > > > > > > message
>> > > > > > > > > > > > > on the
>> > > > > > > > > > > > > > broker
>> > > > > > > > > > > > > > >> side.
>> > > > > > > > > > > > > > >> > > c) At this
>> point
>> > > we
>> > > > will
>> > > > > > have
>> > > > > > > to
>> > > > > > > > > > specify a
>> > > > > > > > > > > > > warning or
>> > > > > > > > > > > > > > clearly
>> > > > > > > > > > > > > > >> specify
>> > > > > > > > > > > > > > >> > > in docs that this
>> > > > behavior is
>> > > > > > > about
>> > > > > > > > > to be
>> > > > > > > > > > > > changed for
>> > > > > > > > > > > > > log
>> > > > > > > > > > > > > > compaction.
>> > > > > > > > > > > > > > >> > > 4) a) In next
>> > release
>> > > > of the
>> > > > > > > > > Broker(x+2), we
>> > > > > > > > > > > > say that
>> > > > > > > > > > > > > only
>> > > > > > > > > > > > > > Tombstone
>> > > > > > > > > > > > > > >> is
>> > > > > > > > > > > > > > >> > > used for log
>> > > compaction
>> > > > on the
>> > > > > > > > Broker
>> > > > > > > > > side.
>> > > > > > > > > > > > > Clients(x+1)
>> > > > > > > > > > > > > > still is
>> > > > > > > > > > > > > > >> > > supported.
>> > > > > > > > > > > > > > >> > > b) We upgrade
>> > the
>> > > > client
>> > > > > to
>> > > > > > > > > version
>> > > > > > > > > > > > client(x+2) :
>> > > > > > > > > > > > > if
>> > > > > > > > > > > > > > value is set
>> > > > > > > > > > > > > > >> to
>> > > > > > > > > > > > > > >> > > null, tombstone
>> will
>> > > > not be
>> > > > > set
>> > > > > > > > > > automatically.
>> > > > > > > > > > > > The
>> > > > > > > > > > > > > client
>> > > > > > > > > > > > > > will have to
>> > > > > > > > > > > > > > >> > call
>> > > > > > > > > > > > > > >> > > setTombstone() to
>> > > > actually set
>> > > > > > the
>> > > > > > > > > > tombstone.
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > We should compare
>> > this
>> > > > > migration
>> > > > > > > > plan
>> > > > > > > > > with
>> > > > > > > > > > the
>> > > > > > > > > > > > > migration
>> > > > > > > > > > > > > > plan for
>> > > > > > > > > > > > > > >> magic
>> > > > > > > > > > > > > > >> > > byte bump and do
>> > > > whatever
>> > > > > looks
>> > > > > > > > good.
>> > > > > > > > > > > > > > >> > > I am just worried
>> > that
>> > > > if we
>> > > > > go
>> > > > > > > down
>> > > > > > > > > magic
>> > > > > > > > > > > byte
>> > > > > > > > > > > > route,
>> > > > > > > > > > > > > > unless I am
>> > > > > > > > > > > > > > >> > missing
>> > > > > > > > > > > > > > >> > > something, it
>> sounds
>> > > > like
>> > > > > kafka
>> > > > > > > will
>> > > > > > > > > be
>> > > > > > > > > > stuck
>> > > > > > > > > > > > with
>> > > > > > > > > > > > > > supporting both
>> > > > > > > > > > > > > > >> null
>> > > > > > > > > > > > > > >> > > value and
>> tombstone
>> > > bit
>> > > > for
>> > > > > log
>> > > > > > > > > compaction
>> > > > > > > > > > for
>> > > > > > > > > > > > life
>> > > > > > > > > > > > > long,
>> > > > > > > > > > > > > > which does
>> > > > > > > > > > > > > > >> not
>> > > > > > > > > > > > > > >> > > look like a good
>> end
>> > > > state.
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > Thanks,
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > Mayuresh
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > On Wed, Nov 16,
>> 2016
>> > > at
>> > > > 9:32
>> > > > > AM,
>> > > > > > > > > Mayuresh
>> > > > > > > > > > > > Gharat <
>> > > > > > > > > > > > > > >> > >
>> > > > gharatmayures...@gmail.com
>> > > > > > > > > > > > > > >> > > > wrote:
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > > Hi Ismael,
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > That's a very
>> good
>> > > > point
>> > > > > > which I
>> > > > > > > > > might
>> > > > > > > > > > have
>> > > > > > > > > > > > not
>> > > > > > > > > > > > > > considered earlier.
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > Here is a plan
>> > that
>> > > I
>> > > > can
>> > > > > > think
>> > > > > > > > of:
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > Stage 1) The
>> > broker
>> > > > from now
>> > > > > > on,
>> > > > > > > > up
>> > > > > > > > > > converts
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > message
>> > > > > > > > > > > > > > to have the
>> > > > > > > > > > > > > > >> > > > tombstone
>> marker.
>> > > The
>> > > > log
>> > > > > > > > compaction
>> > > > > > > > > > thread
>> > > > > > > > > > > > does log
>> > > > > > > > > > > > > > compaction
>> > > > > > > > > > > > > > >> based
>> > > > > > > > > > > > > > >> > on
>> > > > > > > > > > > > > > >> > > > both null and
>> > > > tombstone
>> > > > > > marker.
>> > > > > > > > > This is
>> > > > > > > > > > our
>> > > > > > > > > > > > > transition
>> > > > > > > > > > > > > > period.
>> > > > > > > > > > > > > > >> > > > Stage 2) The
>> next
>> > > > release we
>> > > > > > > only
>> > > > > > > > > say that
>> > > > > > > > > > > log
>> > > > > > > > > > > > > compaction
>> > > > > > > > > > > > > > is based
>> > > > > > > > > > > > > > >> on
>> > > > > > > > > > > > > > >> > > > tombstone
>> marker.
>> > > > (Open
>> > > > > source
>> > > > > > > > > kafka makes
>> > > > > > > > > > > > this as a
>> > > > > > > > > > > > > > policy). By
>> > > > > > > > > > > > > > >> this
>> > > > > > > > > > > > > > >> > > time,
>> > > > > > > > > > > > > > >> > > > the
>> organization
>> > > > which is
>> > > > > > moving
>> > > > > > > > to
>> > > > > > > > > this
>> > > > > > > > > > > > release
>> > > > > > > > > > > > > will be
>> > > > > > > > > > > > > > sure that
>> > > > > > > > > > > > > > >> they
>> > > > > > > > > > > > > > >> > > > have gone
>> through
>> > > the
>> > > > entire
>> > > > > > > > > transition
>> > > > > > > > > > > > period.
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > My only goal of
>> > > doing
>> > > > this
>> > > > > is
>> > > > > > > that
>> > > > > > > > > Kafka
>> > > > > > > > > > > > clearly
>> > > > > > > > > > > > > > specifies the end
>> > > > > > > > > > > > > > >> > state
>> > > > > > > > > > > > > > >> > > > about what log
>> > > > compaction
>> > > > > > means
>> > > > > > > > (is
>> > > > > > > > > it
>> > > > > > > > > > null
>> > > > > > > > > > > > value
>> > > > > > > > > > > > > or a
>> > > > > > > > > > > > > > tombstone
>> > > > > > > > > > > > > > >> > marker,
>> > > > > > > > > > > > > > >> > > > but not both).
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > What do you
>> think?
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > Thanks,
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > Mayuresh
>> > > > > > > > > > > > > > >> > > > .
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > On Wed, Nov 16,
>> > 2016
>> > > > at 9:17
>> > > > > > AM,
>> > > > > > > > > Ismael
>> > > > > > > > > > > Juma <
>> > > > > > > > > > > > > > ism...@juma.me.uk>
>> > > > > > > > > > > > > > >> > wrote:
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > >> One comment
>> > below.
>> > > > > > > > > > > > > > >> > > >>
>> > > > > > > > > > > > > > >> > > >> On Wed, Nov
>> 16,
>> > > 2016
>> > > > at
>> > > > > 5:08
>> > > > > > > PM,
>> > > > > > > > > Mayuresh
>> > > > > > > > > > > > Gharat <
>> > > > > > > > > > > > > > >> > > >>
>> > > > gharatmayures...@gmail.com
>> > > > > > > > > > > > > > >> > > >> > wrote:
>> > > > > > > > > > > > > > >> > > >>
>> > > > > > > > > > > > > > >> > > >> > - If we
>> > don't
>> > > > bump up
>> > > > > > the
>> > > > > > > > > magic
>> > > > > > > > > > byte,
>> > > > > > > > > > > > on the
>> > > > > > > > > > > > > broker
>> > > > > > > > > > > > > > side, the
>> > > > > > > > > > > > > > >> > > broker
>> > > > > > > > > > > > > > >> > > >> > will
>> always
>> > > > have to
>> > > > > look
>> > > > > > > at
>> > > > > > > > > both
>> > > > > > > > > > > > tombstone
>> > > > > > > > > > > > > bit and
>> > > > > > > > > > > > > > the value
>> > > > > > > > > > > > > > >> when
>> > > > > > > > > > > > > > >> > > do
>> > > > > > > > > > > > > > >> > > >> the
>> > > > > > > > > > > > > > >> > > >> >
>> compaction.
>> > > > Assuming
>> > > > > we
>> > > > > > do
>> > > > > > > > > not bump
>> > > > > > > > > > up
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > magic
>> > > > > > > > > > > > > > byte,
>> > > > > > > > > > > > > > >> > > >> > imagine
>> the
>> > > > broker
>> > > > > sees
>> > > > > > a
>> > > > > > > > > message
>> > > > > > > > > > > which
>> > > > > > > > > > > > does
>> > > > > > > > > > > > > not
>> > > > > > > > > > > > > > have a
>> > > > > > > > > > > > > > >> tombstone
>> > > > > > > > > > > > > > >> > > bit
>> > > > > > > > > > > > > > >> > > >> > set. The
>> > > broker
>> > > > does
>> > > > > not
>> > > > > > > > know
>> > > > > > > > > when
>> > > > > > > > > > the
>> > > > > > > > > > > > > message was
>> > > > > > > > > > > > > > produced
>> > > > > > > > > > > > > > >> (i.e.
>> > > > > > > > > > > > > > >> > > >> > whether
>> > > > > > > > > > > > > > >> > > >> > the
>> message
>> > > has
>> > > > been
>> > > > > up
>> > > > > > > > > converted or
>> > > > > > > > > > > > not), it
>> > > > > > > > > > > > > has
>> > > > > > > > > > > > > > to take a
>> > > > > > > > > > > > > > >> > further
>> > > > > > > > > > > > > > >> > > >> > look at
>> > > > > > > > > > > > > > >> > > >> > the
>> value to
>> > > > see if it
>> > > > > > is
>> > > > > > > > > null or
>> > > > > > > > > > not
>> > > > > > > > > > > in
>> > > > > > > > > > > > > order to
>> > > > > > > > > > > > > > determine
>> > > > > > > > > > > > > > >> if it
>> > > > > > > > > > > > > > >> > > is
>> > > > > > > > > > > > > > >> > > >> a
>> > > > > > > > > > > > > > >> > > >> >
>> tombstone.
>> > The
>> > > > same
>> > > > > > logic
>> > > > > > > > has
>> > > > > > > > > to be
>> > > > > > > > > > > put
>> > > > > > > > > > > > on the
>> > > > > > > > > > > > > > consumer as
>> > > > > > > > > > > > > > >> well
>> > > > > > > > > > > > > > >> > > >> because
>> > > > > > > > > > > > > > >> > > >> > the
>> > > > > > > > > > > > > > >> > > >> > consumer
>> > does
>> > > > not know
>> > > > > > if
>> > > > > > > > the
>> > > > > > > > > > message
>> > > > > > > > > > > > has
>> > > > > > > > > > > > > been up
>> > > > > > > > > > > > > > converted or
>> > > > > > > > > > > > > > >> > not.
>> > > > > > > > > > > > > > >> > > >> > - If
>> we
>> > > > upconvert
>> > > > > > while
>> > > > > > > > > > appending,
>> > > > > > > > > > > > this is
>> > > > > > > > > > > > > not
>> > > > > > > > > > > > > > the case,
>> > > > > > > > > > > > > > >> > right?
>> > > > > > > > > > > > > > >> > > >>
>> > > > > > > > > > > > > > >> > > >>
>> > > > > > > > > > > > > > >> > > >> If I
>> understand
>> > you
>> > > > > > correctly,
>> > > > > > > > > this is
>> > > > > > > > > > not
>> > > > > > > > > > > > > sufficient
>> > > > > > > > > > > > > > because the
>> > > > > > > > > > > > > > >> log
>> > > > > > > > > > > > > > >> > > may
>> > > > > > > > > > > > > > >> > > >> have messages
>> > > > appended
>> > > > > before
>> > > > > > > it
>> > > > > > > > > was
>> > > > > > > > > > > > upgraded to
>> > > > > > > > > > > > > include
>> > > > > > > > > > > > > > KIP-87.
>> > > > > > > > > > > > > > >> > > >>
>> > > > > > > > > > > > > > >> > > >> Ismael
>> > > > > > > > > > > > > > >> > > >>
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > > > --
>> > > > > > > > > > > > > > >> > > > -Regards,
>> > > > > > > > > > > > > > >> > > > Mayuresh R.
>> Gharat
>> > > > > > > > > > > > > > >> > > > (862) 250-7125
>> > > > > > > > > > > > > > >> > > >
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > > --
>> > > > > > > > > > > > > > >> > > -Regards,
>> > > > > > > > > > > > > > >> > > Mayuresh R.
>> Gharat
>> > > > > > > > > > > > > > >> > > (862) 250-7125
>> > > > > > > > > > > > > > >> > >
>> > > > > > > > > > > > > > >> > The information
>> > > contained
>> > > > in
>> > > > > this
>> > > > > > > > email
>> > > > > > > > > is
>> > > > > > > > > > > > strictly
>> > > > > > > > > > > > > > confidential and for
>> > > > > > > > > > > > > > >> > the use of the
>> > addressee
>> > > > only,
>> > > > > > > unless
>> > > > > > > > > > otherwise
>> > > > > > > > > > > > > indicated. If
>> > > > > > > > > > > > > > you are
>> > > > > > > > > > > > > > >> not
>> > > > > > > > > > > > > > >> > the intended
>> > recipient,
>> > > > please
>> > > > > do
>> > > > > > > not
>> > > > > > > > > read,
>> > > > > > > > > > > copy,
>> > > > > > > > > > > > use or
>> > > > > > > > > > > > > > disclose to
>> > > > > > > > > > > > > > >> others
>> > > > > > > > > > > > > > >> > this message or any
>> > > > attachment.
>> > > > > > > Please
>> > > > > > > > > also
>> > > > > > > > > > > > notify the
>> > > > > > > > > > > > > sender
>> > > > > > > > > > > > > > by
>> > > > > > > > > > > > > > >> replying
>> > > > > > > > > > > > > > >> > to this email or by
>> > > > telephone
>> > > > > > > (+44(020
>> > > > > > > > > 7896
>> > > > > > > > > > > 0011)
>> > > > > > > > > > > > and
>> > > > > > > > > > > > > then
>> > > > > > > > > > > > > > delete the
>> > > > > > > > > > > > > > >> email
>> > > > > > > > > > > > > > >> > and any copies of
>> it.
>> > > > Opinions,
>> > > > > > > > > conclusion
>> > > > > > > > > > (etc)
>> > > > > > > > > > > > that
>> > > > > > > > > > > > > do not
>> > > > > > > > > > > > > > relate to
>> > > > > > > > > > > > > > >> the
>> > > > > > > > > > > > > > >> > official business
>> of
>> > > this
>> > > > > company
>> > > > > > > > shall
>> > > > > > > > > be
>> > > > > > > > > > > > understood as
>> > > > > > > > > > > > > > neither given
>> > > > > > > > > > > > > > >> nor
>> > > > > > > > > > > > > > >> > endorsed by it. IG
>> is
>> > a
>> > > > trading
>> > > > > > name
>> > > > > > > > of
>> > > > > > > > > IG
>> > > > > > > > > > > Markets
>> > > > > > > > > > > > > Limited (a
>> > > > > > > > > > > > > > company
>> > > > > > > > > > > > > > >> > registered in
>> England
>> > > and
>> > > > Wales,
>> > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > > > 04008957) and
>> > > > > > > > > > > > > > IG Index
>> > > > > > > > > > > > > > >> > Limited (a company
>> > > > registered in
>> > > > > > > > > England and
>> > > > > > > > > > > > Wales,
>> > > > > > > > > > > > > company
>> > > > > > > > > > > > > > number
>> > > > > > > > > > > > > > >> > 01190902).
>> Registered
>> > > > address at
>> > > > > > > > Cannon
>> > > > > > > > > Bridge
>> > > > > > > > > > > > House, 25
>> > > > > > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > > > > >> > London EC4R 2YA.
>> Both
>> > IG
>> > > > Markets
>> > > > > > > > Limited
>> > > > > > > > > > > (register
>> > > > > > > > > > > > > number
>> > > > > > > > > > > > > > 195355) and IG
>> > > > > > > > > > > > > > >> > Index Limited
>> > (register
>> > > > number
>> > > > > > > 114059)
>> > > > > > > > > are
>> > > > > > > > > > > > authorised
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > regulated by
>> > > > > > > > > > > > > > >> the
>> > > > > > > > > > > > > > >> > Financial Conduct
>> > > > Authority.
>> > > > > > > > > > > > > > >> >
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >> --
>> > > > > > > > > > > > > > >> -Regards,
>> > > > > > > > > > > > > > >> Mayuresh R. Gharat
>> > > > > > > > > > > > > > >> (862) 250-7125
>> > > > > > > > > > > > > > >> The information
>> > contained
>> > > > in this
>> > > > > > > email
>> > > > > > > > is
>> > > > > > > > > > > strictly
>> > > > > > > > > > > > > > confidential and for
>> > > > > > > > > > > > > > >> the use of the
>> addressee
>> > > > only,
>> > > > > > unless
>> > > > > > > > > otherwise
>> > > > > > > > > > > > > indicated. If
>> > > > > > > > > > > > > > you are not
>> > > > > > > > > > > > > > >> the intended
>> recipient,
>> > > > please do
>> > > > > > not
>> > > > > > > > > read,
>> > > > > > > > > > copy,
>> > > > > > > > > > > > use or
>> > > > > > > > > > > > > > disclose to others
>> > > > > > > > > > > > > > >> this message or any
>> > > > attachment.
>> > > > > > Please
>> > > > > > > > > also
>> > > > > > > > > > notify
>> > > > > > > > > > > > the
>> > > > > > > > > > > > > sender
>> > > > > > > > > > > > > > by replying
>> > > > > > > > > > > > > > >> to this email or by
>> > > > telephone
>> > > > > > (+44(020
>> > > > > > > > > 7896
>> > > > > > > > > > 0011)
>> > > > > > > > > > > > and then
>> > > > > > > > > > > > > > delete the email
>> > > > > > > > > > > > > > >> and any copies of it.
>> > > > Opinions,
>> > > > > > > > > conclusion (etc)
>> > > > > > > > > > > > that do
>> > > > > > > > > > > > > not
>> > > > > > > > > > > > > > relate to the
>> > > > > > > > > > > > > > >> official business of
>> > this
>> > > > company
>> > > > > > > shall
>> > > > > > > > be
>> > > > > > > > > > > > understood as
>> > > > > > > > > > > > > > neither given nor
>> > > > > > > > > > > > > > >> endorsed by it. IG
>> is a
>> > > > trading
>> > > > > name
>> > > > > > > of
>> > > > > > > > IG
>> > > > > > > > > > Markets
>> > > > > > > > > > > > > Limited (a
>> > > > > > > > > > > > > > company
>> > > > > > > > > > > > > > >> registered in England
>> > and
>> > > > Wales,
>> > > > > > > company
>> > > > > > > > > number
>> > > > > > > > > > > > 04008957)
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > IG Index
>> > > > > > > > > > > > > > >> Limited (a company
>> > > > registered in
>> > > > > > > England
>> > > > > > > > > and
>> > > > > > > > > > > Wales,
>> > > > > > > > > > > > > company
>> > > > > > > > > > > > > > number
>> > > > > > > > > > > > > > >> 01190902). Registered
>> > > > address at
>> > > > > > > Cannon
>> > > > > > > > > Bridge
>> > > > > > > > > > > > House, 25
>> > > > > > > > > > > > > > Dowgate Hill,
>> > > > > > > > > > > > > > >> London EC4R 2YA.
>> Both IG
>> > > > Markets
>> > > > > > > Limited
>> > > > > > > > > > (register
>> > > > > > > > > > > > number
>> > > > > > > > > > > > > > 195355) and IG
>> > > > > > > > > > > > > > >> Index Limited
>> (register
>> > > > number
>> > > > > > 114059)
>> > > > > > > > are
>> > > > > > > > > > > > authorised and
>> > > > > > > > > > > > > > regulated by the
>> > > > > > > > > > > > > > >> Financial Conduct
>> > > Authority.
>> > > > > > > > > > > > > > >>
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > > -Regards,
>> > > > > > > > > > > > > > > Mayuresh R. Gharat
>> > > > > > > > > > > > > > > (862) 250-7125
>> > > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > --
>> > > > > > > > > > > > > > -Regards,
>> > > > > > > > > > > > > > Mayuresh R. Gharat
>> > > > > > > > > > > > > > (862) 250-7125
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > > The information contained in
>> > this
>> > > > email
>> > > > > is
>> > > > > > > > > strictly
>> > > > > > > > > > > > confidential
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > for the use of the addressee
>> only,
>> > > > unless
>> > > > > > > otherwise
>> > > > > > > > > > > indicated.
>> > > > > > > > > > > > If
>> > > > > > > > > > > > > you are
>> > > > > > > > > > > > > > not the intended recipient,
>> please
>> > do
>> > > > not
>> > > > > read,
>> > > > > > > > > copy, use
>> > > > > > > > > > or
>> > > > > > > > > > > > > disclose to
>> > > > > > > > > > > > > > others this message or any
>> > > attachment.
>> > > > Please
>> > > > > > > also
>> > > > > > > > > notify
>> > > > > > > > > > the
>> > > > > > > > > > > > sender
>> > > > > > > > > > > > > by
>> > > > > > > > > > > > > > replying to this email or by
>> > > telephone
>> > > > > (+44(020
>> > > > > > > > 7896
>> > > > > > > > > 0011)
>> > > > > > > > > > > and
>> > > > > > > > > > > > then
>> > > > > > > > > > > > > delete
>> > > > > > > > > > > > > > the email and any copies of it.
>> > > > Opinions,
>> > > > > > > > conclusion
>> > > > > > > > > (etc)
>> > > > > > > > > > > > that do
>> > > > > > > > > > > > > not
>> > > > > > > > > > > > > > relate to the official business
>> of
>> > > this
>> > > > > company
>> > > > > > > > > shall be
>> > > > > > > > > > > > understood
>> > > > > > > > > > > > > as
>> > > > > > > > > > > > > > neither given nor endorsed by
>> it.
>> > IG
>> > > > is a
>> > > > > > trading
>> > > > > > > > > name of
>> > > > > > > > > > IG
>> > > > > > > > > > > > Markets
>> > > > > > > > > > > > > > Limited (a company registered in
>> > > > England and
>> > > > > > > Wales,
>> > > > > > > > > company
>> > > > > > > > > > > > number
>> > > > > > > > > > > > > > 04008957) and IG Index Limited
>> (a
>> > > > company
>> > > > > > > > registered
>> > > > > > > > > in
>> > > > > > > > > > > > England and
>> > > > > > > > > > > > > Wales,
>> > > > > > > > > > > > > > company number 01190902).
>> > Registered
>> > > > address
>> > > > > at
>> > > > > > > > > Cannon
>> > > > > > > > > > Bridge
>> > > > > > > > > > > > House,
>> > > > > > > > > > > > > 25
>> > > > > > > > > > > > > > Dowgate Hill, London EC4R 2YA.
>> Both
>> > > IG
>> > > > > Markets
>> > > > > > > > > Limited
>> > > > > > > > > > > > (register
>> > > > > > > > > > > > > number
>> > > > > > > > > > > > > > 195355) and IG Index Limited
>> > > (register
>> > > > number
>> > > > > > > > > 114059) are
>> > > > > > > > > > > > authorised
>> > > > > > > > > > > > > and
>> > > > > > > > > > > > > > regulated by the Financial
>> Conduct
>> > > > Authority.
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > >
>> > > > > > > > > > > > > The information contained in this
>> email
>> > is
>> > > > strictly
>> > > > > > > > > confidential
>> > > > > > > > > > > and
>> > > > > > > > > > > > for
>> > > > > > > > > > > > > the use of the addressee only, unless
>> > > > otherwise
>> > > > > > > > indicated.
>> > > > > > > > > If you
>> > > > > > > > > > > > are not
>> > > > > > > > > > > > > the intended recipient, please do not
>> > read,
>> > > > copy,
>> > > > > use
>> > > > > > > or
>> > > > > > > > > disclose
>> > > > > > > > > > > to
>> > > > > > > > > > > > others
>> > > > > > > > > > > > > this message or any attachment. Please
>> > also
>> > > > notify
>> > > > > > the
>> > > > > > > > > sender by
>> > > > > > > > > > > > replying
>> > > > > > > > > > > > > to this email or by telephone (+44(020
>> > 7896
>> > > > 0011
>> > > > > <020%207896%200011>) and
>> > > > > > > > then
>> > > > > > > > > delete
>> > > > > > > > > > > > the email
>> > > > > > > > > > > > > and any copies of it. Opinions,
>> > conclusion
>> > > > (etc)
>> > > > > that
>> > > > > > > do
>> > > > > > > > > not
>> > > > > > > > > > relate
>> > > > > > > > > > > > to the
>> > > > > > > > > > > > > official business of this company
>> shall
>> > be
>> > > > > understood
>> > > > > > > as
>> > > > > > > > > neither
>> > > > > > > > > > > > given nor
>> > > > > > > > > > > > > endorsed by it. IG is a trading name
>> of
>> > IG
>> > > > Markets
>> > > > > > > > Limited
>> > > > > > > > > (a
>> > > > > > > > > > > company
>> > > > > > > > > > > > > registered in England and Wales,
>> company
>> > > > number
>> > > > > > > 04008957)
>> > > > > > > > > and IG
>> > > > > > > > > > > > Index
>> > > > > > > > > > > > > Limited (a company registered in
>> England
>> > > and
>> > > > Wales,
>> > > > > > > > company
>> > > > > > > > > > number
>> > > > > > > > > > > > > 01190902). Registered address at
>> Cannon
>> > > > Bridge
>> > > > > House,
>> > > > > > > 25
>> > > > > > > > > Dowgate
>> > > > > > > > > > > > Hill,
>> > > > > > > > > > > > > London EC4R 2YA. Both IG Markets
>> Limited
>> > > > (register
>> > > > > > > number
>> > > > > > > > > 195355)
>> > > > > > > > > > > > and IG
>> > > > > > > > > > > > > Index Limited (register number 114059)
>> > are
>> > > > > authorised
>> > > > > > > and
>> > > > > > > > > > regulated
>> > > > > > > > > > > > by the
>> > > > > > > > > > > > > Financial Conduct Authority.
>> > > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > >
>> > > > > > > > > > > > The information contained in this email is
>> > > strictly
>> > > > > > > > confidential
>> > > > > > > > > and
>> > > > > > > > > > for
>> > > > > > > > > > > > the use of the addressee only, unless
>> otherwise
>> > > > > indicated.
>> > > > > > If
>> > > > > > > > > you are
>> > > > > > > > > > not
>> > > > > > > > > > > > the intended recipient, please do not read,
>> > copy,
>> > > > use or
>> > > > > > > > > disclose to
>> > > > > > > > > > > others
>> > > > > > > > > > > > this message or any attachment. Please also
>> > > notify
>> > > > the
>> > > > > > sender
>> > > > > > > > by
>> > > > > > > > > > replying
>> > > > > > > > > > > > to this email or by telephone (+44(020 7896
>> > 0011
>> > > > > <020%207896%200011>) and then
>> > > > > > > > > delete the
>> > > > > > > > > > > email
>> > > > > > > > > > > > and any copies of it. Opinions, conclusion
>> > (etc)
>> > > > that do
>> > > > > > not
>> > > > > > > > > relate to
>> > > > > > > > > > > the
>> > > > > > > > > > > > official business of this company shall be
>> > > > understood as
>> > > > > > > > neither
>> > > > > > > > > given
>> > > > > > > > > > > nor
>> > > > > > > > > > > > endorsed by it. IG is a trading name of IG
>> > > Markets
>> > > > > Limited
>> > > > > > (a
>> > > > > > > > > company
>> > > > > > > > > > > > registered in England and Wales, company
>> number
>> > > > 04008957)
>> > > > > > and
>> > > > > > > > IG
>> > > > > > > > > Index
>> > > > > > > > > > > > Limited (a company registered in England and
>> > > Wales,
>> > > > > company
>> > > > > > > > > number
>> > > > > > > > > > > > 01190902). Registered address at Cannon
>> Bridge
>> > > > House, 25
>> > > > > > > > Dowgate
>> > > > > > > > > Hill,
>> > > > > > > > > > > > London EC4R 2YA. Both IG Markets Limited
>> > > (register
>> > > > number
>> > > > > > > > > 195355) and
>> > > > > > > > > > IG
>> > > > > > > > > > > > Index Limited (register number 114059) are
>> > > > authorised and
>> > > > > > > > > regulated by
>> > > > > > > > > > > the
>> > > > > > > > > > > > Financial Conduct Authority.
>> > > > > > > > > > > >
>> > > > > > > > > > > The information contained in this email is
>> > strictly
>> > > > > > > confidential
>> > > > > > > > > and for
>> > > > > > > > > > > the use of the addressee only, unless
>> otherwise
>> > > > indicated.
>> > > > > If
>> > > > > > > you
>> > > > > > > > > are not
>> > > > > > > > > > > the intended recipient, please do not read,
>> copy,
>> > > > use or
>> > > > > > > disclose
>> > > > > > > > > to
>> > > > > > > > > > others
>> > > > > > > > > > > this message or any attachment. Please also
>> > notify
>> > > > the
>> > > > > sender
>> > > > > > > by
>> > > > > > > > > replying
>> > > > > > > > > > > to this email or by telephone (+44(020 7896
>> 0011
>> > > > > <020%207896%200011>) and then
>> > > > > > > delete
>> > > > > > > > > the
>> > > > > > > > > > email
>> > > > > > > > > > > and any copies of it. Opinions, conclusion
>> (etc)
>> > > > that do
>> > > > > not
>> > > > > > > > > relate to
>> > > > > > > > > > the
>> > > > > > > > > > > official business of this company shall be
>> > > > understood as
>> > > > > > > neither
>> > > > > > > > > given
>> > > > > > > > > > nor
>> > > > > > > > > > > endorsed by it. IG is a trading name of IG
>> > Markets
>> > > > Limited
>> > > > > (a
>> > > > > > > > > company
>> > > > > > > > > > > registered in England and Wales, company
>> number
>> > > > 04008957)
>> > > > > and
>> > > > > > > IG
>> > > > > > > > > Index
>> > > > > > > > > > > Limited (a company registered in England and
>> > Wales,
>> > > > company
>> > > > > > > > number
>> > > > > > > > > > > 01190902). Registered address at Cannon Bridge
>> > > > House, 25
>> > > > > > > Dowgate
>> > > > > > > > > Hill,
>> > > > > > > > > > > London EC4R 2YA. Both IG Markets Limited
>> > (register
>> > > > number
>> > > > > > > 195355)
>> > > > > > > > > and IG
>> > > > > > > > > > > Index Limited (register number 114059) are
>> > > > authorised and
>> > > > > > > > > regulated by
>> > > > > > > > > > the
>> > > > > > > > > > > Financial Conduct Authority.
>> > > > > > > > > > >
>> > > > > > > > > > >
>> > > > > > > > > > The information contained in this email is
>> strictly
>> > > > > > confidential
>> > > > > > > > and
>> > > > > > > > > for
>> > > > > > > > > > the use of the addressee only, unless otherwise
>> > > > indicated. If
>> > > > > > you
>> > > > > > > > > are not
>> > > > > > > > > > the intended recipient, please do not read,
>> copy,
>> > use
>> > > > or
>> > > > > > disclose
>> > > > > > > > to
>> > > > > > > > > others
>> > > > > > > > > > this message or any attachment. Please also
>> notify
>> > > the
>> > > > sender
>> > > > > > by
>> > > > > > > > > replying
>> > > > > > > > > > to this email or by telephone (+44(020 7896 0011
>> > > > > <020%207896%200011>) and then
>> > > > > > delete
>> > > > > > > > > the email
>> > > > > > > > > > and any copies of it. Opinions, conclusion (etc)
>> > that
>> > > > do not
>> > > > > > > relate
>> > > > > > > > > to the
>> > > > > > > > > > official business of this company shall be
>> > understood
>> > > > as
>> > > > > > neither
>> > > > > > > > > given nor
>> > > > > > > > > > endorsed by it. IG is a trading name of IG
>> Markets
>> > > > Limited (a
>> > > > > > > > company
>> > > > > > > > > > registered in England and Wales, company number
>> > > > 04008957) and
>> > > > > > IG
>> > > > > > > > > Index
>> > > > > > > > > > Limited (a company registered in England and
>> Wales,
>> > > > company
>> > > > > > > number
>> > > > > > > > > > 01190902). Registered address at Cannon Bridge
>> > House,
>> > > > 25
>> > > > > > Dowgate
>> > > > > > > > > Hill,
>> > > > > > > > > > London EC4R 2YA. Both IG Markets Limited
>> (register
>> > > > number
>> > > > > > 195355)
>> > > > > > > > > and IG
>> > > > > > > > > > Index Limited (register number 114059) are
>> > authorised
>> > > > and
>> > > > > > > regulated
>> > > > > > > > > by the
>> > > > > > > > > > Financial Conduct Authority.
>> > > > > > > > > >
>> > > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > The information contained in this email is strictly
>> > > > confidential
>> > > > > and
>> > > > > > > for
>> > > > > > > > > the use of the addressee only, unless otherwise
>> > indicated.
>> > > > If you
>> > > > > are
>> > > > > > > not
>> > > > > > > > > the intended recipient, please do not read, copy, use
>> or
>> > > > disclose
>> > > > > to
>> > > > > > > > others
>> > > > > > > > > this message or any attachment. Please also notify the
>> > > > sender by
>> > > > > > > replying
>> > > > > > > > > to this email or by telephone (+44(020 7896 0011
>> > > > > <020%207896%200011>) and then delete the
>> > > > > > > > email
>> > > > > > > > > and any copies of it. Opinions, conclusion (etc) that
>> do
>> > > not
>> > > > relate
>> > > > > > to
>> > > > > > > > the
>> > > > > > > > > official business of this company shall be understood
>> as
>> > > > neither
>> > > > > > given
>> > > > > > > > nor
>> > > > > > > > > endorsed by it. IG is a trading name of IG Markets
>> > Limited
>> > > (a
>> > > > > company
>> > > > > > > > > registered in England and Wales, company number
>> 04008957)
>> > > > and IG
>> > > > > > Index
>> > > > > > > > > Limited (a company registered in England and Wales,
>> > company
>> > > > number
>> > > > > > > > > 01190902). Registered address at Cannon Bridge House,
>> 25
>> > > > Dowgate
>> > > > > > Hill,
>> > > > > > > > > London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > > 195355)
>> > > > > and
>> > > > > > > IG
>> > > > > > > > > Index Limited (register number 114059) are authorised
>> and
>> > > > regulated
>> > > > > > by
>> > > > > > > > the
>> > > > > > > > > Financial Conduct Authority.
>> > > > > > > > >
>> > > > > > > > The information contained in this email is strictly
>> > > > confidential and
>> > > > > > for
>> > > > > > > > the use of the addressee only, unless otherwise
>> indicated.
>> > If
>> > > > you are
>> > > > > > not
>> > > > > > > > the intended recipient, please do not read, copy, use or
>> > > > disclose to
>> > > > > > > others
>> > > > > > > > this message or any attachment. Please also notify the
>> > sender
>> > > > by
>> > > > > > replying
>> > > > > > > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > > > and then delete the
>> > > > > > > email
>> > > > > > > > and any copies of it. Opinions, conclusion (etc) that do
>> > not
>> > > > relate
>> > > > > to
>> > > > > > > the
>> > > > > > > > official business of this company shall be understood as
>> > > > neither
>> > > > > given
>> > > > > > > nor
>> > > > > > > > endorsed by it. IG is a trading name of IG Markets
>> Limited
>> > (a
>> > > > company
>> > > > > > > > registered in England and Wales, company number
>> 04008957)
>> > and
>> > > > IG
>> > > > > Index
>> > > > > > > > Limited (a company registered in England and Wales,
>> company
>> > > > number
>> > > > > > > > 01190902). Registered address at Cannon Bridge House, 25
>> > > > Dowgate
>> > > > > Hill,
>> > > > > > > > London EC4R 2YA. Both IG Markets Limited (register
>> number
>> > > > 195355) and
>> > > > > > IG
>> > > > > > > > Index Limited (register number 114059) are authorised
>> and
>> > > > regulated
>> > > > > by
>> > > > > > > the
>> > > > > > > > Financial Conduct Authority.
>> > > > > > > >
>> > > > > > > The information contained in this email is strictly
>> > > confidential
>> > > > and
>> > > > > for
>> > > > > > > the use of the addressee only, unless otherwise
>> indicated. If
>> > > > you are
>> > > > > not
>> > > > > > > the intended recipient, please do not read, copy, use or
>> > > > disclose to
>> > > > > > others
>> > > > > > > this message or any attachment. Please also notify the
>> sender
>> > > by
>> > > > > replying
>> > > > > > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > > > and then delete the
>> > > > > > email
>> > > > > > > and any copies of it. Opinions, conclusion (etc) that do
>> not
>> > > > relate to
>> > > > > > the
>> > > > > > > official business of this company shall be understood as
>> > > neither
>> > > > given
>> > > > > > nor
>> > > > > > > endorsed by it. IG is a trading name of IG Markets
>> Limited (a
>> > > > company
>> > > > > > > registered in England and Wales, company number 04008957)
>> and
>> > > IG
>> > > > Index
>> > > > > > > Limited (a company registered in England and Wales,
>> company
>> > > > number
>> > > > > > > 01190902). Registered address at Cannon Bridge House, 25
>> > > Dowgate
>> > > > Hill,
>> > > > > > > London EC4R 2YA. Both IG Markets Limited (register number
>> > > > 195355) and
>> > > > > IG
>> > > > > > > Index Limited (register number 114059) are authorised and
>> > > > regulated by
>> > > > > > the
>> > > > > > > Financial Conduct Authority.
>> > > > > > >
>> > > > > > The information contained in this email is strictly
>> > confidential
>> > > > and for
>> > > > > > the use of the addressee only, unless otherwise indicated.
>> If
>> > you
>> > > > are not
>> > > > > > the intended recipient, please do not read, copy, use or
>> > disclose
>> > > > to
>> > > > > others
>> > > > > > this message or any attachment. Please also notify the
>> sender
>> > by
>> > > > replying
>> > > > > > to this email or by telephone (+44(020 7896 0011
>> > > > <020%207896%200011>)
>> > > > > and then delete the email
>> > > > > > and any copies of it. Opinions, conclusion (etc) that do not
>> > > > relate to
>> > > > > the
>> > > > > > official business of this company shall be understood as
>> > neither
>> > > > given
>> > > > > nor
>> > > > > > endorsed by it. IG is a trading name of IG Markets Limited
>> (a
>> > > > company
>> > > > > > registered in England and Wales, company number 04008957)
>> and
>> > IG
>> > > > Index
>> > > > > > Limited (a company registered in England and Wales, company
>> > > number
>> > > > > > 01190902). Registered address at Cannon Bridge House, 25
>> > Dowgate
>> > > > Hill,
>> > > > > > London EC4R 2YA. Both IG Markets Limited (register number
>> > 195355)
>> > > > and IG
>> > > > > > Index Limited (register number 114059) are authorised and
>> > > > regulated by
>> > > > > the
>> > > > > > Financial Conduct Authority.
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > The information contained in this email is strictly confidential and
>> > for
>> > > > the use of the addressee only, unless otherwise indicated. If you
>> are
>> > not
>> > > > the intended recipient, please do not read, copy, use or disclose to
>> > > others
>> > > > this message or any attachment. Please also notify the sender by
>> > replying
>> > > > to this email or by telephone (+44(020 7896 0011) and then delete
>> the
>> > > email
>> > > > and any copies of it. Opinions, conclusion (etc) that do not relate
>> to
>> > > the
>> > > > official business of this company shall be understood as neither
>> given
>> > > nor
>> > > > endorsed by it. IG is a trading name of IG Markets Limited (a
>> company
>> > > > registered in England and Wales, company number 04008957) and IG
>> Index
>> > > > Limited (a company registered in England and Wales, company number
>> > > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
>> Hill,
>> > > > London EC4R 2YA. Both IG Markets Limited (register number 195355)
>> and
>> > IG
>> > > > Index Limited (register number 114059) are authorised and regulated
>> by
>> > > the
>> > > > Financial Conduct Authority.
>> > > >
>> > > The information contained in this email is strictly confidential and
>> for
>> > > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > > the intended recipient, please do not read, copy, use or disclose to
>> > others
>> > > this message or any attachment. Please also notify the sender by
>> replying
>> > > to this email or by telephone (+44(020 7896 0011) and then delete the
>> > email
>> > > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> > the
>> > > official business of this company shall be understood as neither given
>> > nor
>> > > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > > registered in England and Wales, company number 04008957) and IG Index
>> > > Limited (a company registered in England and Wales, company number
>> > > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > > London EC4R 2YA. Both IG Markets Limited (register number 195355) and
>> IG
>> > > Index Limited (register number 114059) are authorised and regulated by
>> > the
>> > > Financial Conduct Authority.
>> > >
>> > The information contained in this email is strictly confidential and for
>> > the use of the addressee only, unless otherwise indicated. If you are
>> not
>> > the intended recipient, please do not read, copy, use or disclose to
>> others
>> > this message or any attachment. Please also notify the sender by
>> replying
>> > to this email or by telephone (+44(020 7896 0011) and then delete the
>> email
>> > and any copies of it. Opinions, conclusion (etc) that do not relate to
>> the
>> > official business of this company shall be understood as neither given
>> nor
>> > endorsed by it. IG is a trading name of IG Markets Limited (a company
>> > registered in England and Wales, company number 04008957) and IG Index
>> > Limited (a company registered in England and Wales, company number
>> > 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
>> > London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
>> > Index Limited (register number 114059) are authorised and regulated by
>> the
>> > Financial Conduct Authority.
>> >
>>
>
>
