Hi Ismael, That's a very good point which I might have not considered earlier.
Here is a plan that I can think of: Stage 1) The broker from now on, up converts the message to have the tombstone marker. The log compaction thread does log compaction based on both null and tombstone marker. This is our transition period. Stage 2) The next release we only say that log compaction is based on tombstone marker. (Open source kafka makes this as a policy). By this time, the organization which is moving to this release will be sure that they have gone through the entire transition period. My only goal of doing this is that Kafka clearly specifies the end state about what log compaction means (is it null value or a tombstone marker, but not both). What do you think? Thanks, Mayuresh . On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <ism...@juma.me.uk> wrote: > One comment below. > > On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > - If we don't bump up the magic byte, on the broker side, the broker > > will always have to look at both tombstone bit and the value when do > the > > compaction. Assuming we do not bump up the magic byte, > > imagine the broker sees a message which does not have a tombstone bit > > set. The broker does not know when the message was produced (i.e. > > whether > > the message has been up converted or not), it has to take a further > > look at > > the value to see if it is null or not in order to determine if it is a > > tombstone. The same logic has to be put on the consumer as well > because > > the > > consumer does not know if the message has been up converted or not. > > - If we upconvert while appending, this is not the case, right? > > > If I understand you correctly, this is not sufficient because the log may > have messages appended before it was upgraded to include KIP-87. > > Ismael > -- -Regards, Mayuresh R. Gharat (862) 250-7125
