Bill, Thanks for the proposal. A couple of comments.
1. It seems that this new policy should work for CreateTime as well. If a topic is configured with CreateTime, messages may not be added in strict order in the log. However, to build a time-based index, we will be maintaining the largest timestamp for all messages in a log segment. We can delete a segment if its largest timestamp is less than log.retention.min.timestamp. This guarantees that no messages newer than log.retention.min.timestamp will be deleted, which is probably what the user wants. 2. Right now, the user can specify "delete" as the retention policy and a log segment will be deleted either when the size of a partition exceeds a threshold or the timestamp of a segment is older than a relative period of time (say 7 days) from now. What you are proposing is not a new retention policy, but an additional check that will cause a segment to be deleted when the timestamp of a segment is older than an absolute timestamp? If so, could you update the wiki accordingly? Jun On Sat, Feb 13, 2016 at 3:23 PM, Bill Warshaw <wdwars...@gmail.com> wrote: > Hello, > > That is a good catch, thanks for pointing it out. If this KIP is accepted, > we'd need to document this and make the log cleaner not run timestamp-based > deletion unless message.timestamp.type=LogAppendTime. > > On Sat, Feb 13, 2016 at 5:38 AM, Andrew Schofield < > andrew_schofield_j...@outlook.com> wrote: > > > This KIP is related to KIP-32, but I strikes me that it only makes sense > > with one of the two proposed message timestamp types. If I understand > > correctly, message timestamps are only certain to be monotonically > > increasing in the log if message.timestamp.type=LogAppendTime. > > > > > > > > Does timestamp-based auto-expiration require use of > > message.timestamp.type=LogAppendTime? > > > > > > > > > > I think this KIP is a good idea, but I think it relies on strict ordering > > of timestamps to be workable. > > > > > > > > Andrew Schofield > > > > > > > > > > > Date: Fri, 12 Feb 2016 10:38:46 -0800 > > > Subject: Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy > > > From: n...@confluent.io > > > To: dev@kafka.apache.org > > > > > > Adding a timestamp based auto-expiration is useful and this proposal > > makes > > > sense. Thx! > > > > > > On Wed, Feb 10, 2016 at 3:35 PM, Jay Kreps wrote: > > > > > >> I think this makes a lot of sense and won't be hard to implement and > > >> doesn't create too much in the way of new interfaces. > > >> > > >> -Jay > > >> > > >> On Tue, Feb 9, 2016 at 8:13 AM, Bill Warshaw wrote: > > >> > > >>> Hello, > > >>> > > >>> I just submitted KIP-47 for adding a new log deletion policy based > on a > > >>> minimum timestamp of messages to retain. > > >>> > > >>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy > > >>> > > >>> I'm open to any comments or suggestions. > > >>> > > >>> Thanks, > > >>> Bill Warshaw > > >>> > > >> > > > > > > > > > > > > -- > > > Thanks, > > > Neha > > >
