Hi Mayuresh, Thanks for describing your plan in detail. I understand the concern about having to support both old and new way for a long time. In my opinion, we don't have a choice in that matter. We can't change semantics of the message format without having a long transition period. And we can't rely on people reading documentation or acting on a warning for something so fundamental.
As such, my take is that we need to bump the magic byte. The good news is that we don't have to support all versions forever. We have said that we will support direct upgrades for 2 years. That means that message format version n could, in theory, be removed 2 years after the it's introduced. So far, we only have 2 message format versions and we have never removed any. So, we'll see what will actually happen, in practice. Ismael On Wed, Nov 16, 2016 at 9:28 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Hi Ismael, > > This is something I can think of for migration plan: > So the migration plan can look something like this, with up conversion : > > 1) Currently lets say we have Broker at version x. > 2) Currently we have clients at version x. > 3) a) We move the version to Broker(x+1) : supports both tombstone and null > for log compaction. > b) We upgrade the client to version client(x+1) : if in the producer > client(x+1) the value is set to null, we will automatically set the > Tombstone bit internally. If the producer client(x+1) sets the tombstone > itself, well and good. For producer client(x), the broker will up convert > to have the tombstone bit. Broker(x+1) is supporting both. Consumer > client(x+1) will be aware of this and should be able to handle this. For > consumer client(x) we will down convert the message on the broker side. > c) At this point we will have to specify a warning or clearly specify > in docs that this behavior is about to be changed for log compaction. > 4) a) In next release of the Broker(x+2), we say that only Tombstone is > used for log compaction on the Broker side. Clients(x+1) still is > supported. > b) We upgrade the client to version client(x+2) : if value is set to > null, tombstone will not be set automatically. The client will have to call > setTombstone() to actually set the tombstone. > > We should compare this migration plan with the migration plan for magic > byte bump and do whatever looks good. > I am just worried that if we go down magic byte route, unless I am missing > something, it sounds like kafka will be stuck with supporting both null > value and tombstone bit for log compaction for life long, which does not > look like a good end state. > > Thanks, > > Mayuresh > > > > > On Wed, Nov 16, 2016 at 9:32 AM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Hi Ismael, > > > > That's a very good point which I might have not considered earlier. > > > > Here is a plan that I can think of: > > > > Stage 1) The broker from now on, up converts the message to have the > > tombstone marker. The log compaction thread does log compaction based on > > both null and tombstone marker. This is our transition period. > > Stage 2) The next release we only say that log compaction is based on > > tombstone marker. (Open source kafka makes this as a policy). By this > time, > > the organization which is moving to this release will be sure that they > > have gone through the entire transition period. > > > > My only goal of doing this is that Kafka clearly specifies the end state > > about what log compaction means (is it null value or a tombstone marker, > > but not both). > > > > What do you think? > > > > Thanks, > > > > Mayuresh > > . > > > > On Wed, Nov 16, 2016 at 9:17 AM, Ismael Juma <ism...@juma.me.uk> wrote: > > > >> One comment below. > >> > >> On Wed, Nov 16, 2016 at 5:08 PM, Mayuresh Gharat < > >> gharatmayures...@gmail.com > >> > wrote: > >> > >> > - If we don't bump up the magic byte, on the broker side, the > broker > >> > will always have to look at both tombstone bit and the value when > do > >> the > >> > compaction. Assuming we do not bump up the magic byte, > >> > imagine the broker sees a message which does not have a tombstone > bit > >> > set. The broker does not know when the message was produced (i.e. > >> > whether > >> > the message has been up converted or not), it has to take a further > >> > look at > >> > the value to see if it is null or not in order to determine if it > is > >> a > >> > tombstone. The same logic has to be put on the consumer as well > >> because > >> > the > >> > consumer does not know if the message has been up converted or not. > >> > - If we upconvert while appending, this is not the case, right? > >> > >> > >> If I understand you correctly, this is not sufficient because the log > may > >> have messages appended before it was upgraded to include KIP-87. > >> > >> Ismael > >> > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
