Xavier, Yeah I think that post KIP-58 it is possible to depend on the delivery of messages in compacted topics, if you override the default compaction time. Prior to that it was true that you could control the delete retention, but any message including the tombstone could be compacted away prior to delivery. That was what I meant by non-deterministic. Now that we have that KIP I agree that at least it can be given an SLA like any other message (which was what I was meaning by deterministic).
-Jay On Fri, Oct 28, 2016 at 10:15 AM, Xavier Léauté <xav...@confluent.io> wrote: > > > > I kind of agree with James that it is a bit questionable how valuable any > > data in a delete marker can be since it will be deleted somewhat > > nondeterministically. > > > > One could argue that even in normal topics, assuming a time-based log > retention policy is in place, any message will be deleted somewhat > nondeterministally, so why treat the compacted ones any differently? To me > at least, the retention setting for delete messages seems to be the > counterpart to the time-based retention setting for normal topics. > > Currently the semantics of the messages are in the eye of the beholder--you > > can choose to interpret a stream as either being appends or revisions as > > you choose. This proposal is changing that so that the semantics are > > determined by the sender. > > > Let's imagine someone wanted to augment this stream to include audit logs > for each record update, e.g. which user made the change. One would want to > include that information as part of the message, and have the ability to > mark a deletion. > > I don't think it changes the semantics in this case, you can still choose > to interpret the data as a stream of audit log entries (inserts), ignoring > the tombstone flag, or you can interpret it as a table modeling only the > latest version of each record. Whether a compacted or normal topic is used > shouldn't matter to the sender. >
