We initially looked at exposing an API through AdminUtils that would manually delete everything in a given partition before a specified offset. We ran into several difficulties in figuring out how to efficiently communicate this "truncate partition" command to all brokers in Zookeeper, without adding a large amount of Zookeeper watches or storing large amounts of data in Zookeeper.
To clarify, the minimum timestamp will be a per-topic configuration. On Sat, Feb 13, 2016 at 5:02 PM, Enrico Olivelli <eolive...@gmail.com> wrote: > Hi, > I'm currently using kafka 0.9 as a commit log. I would find more useful to > set an offset for every partition (actually I am using only one partition > per topic) instead of a global timestamp, has this option been already > considered? Thanks > -- Enrico > > Il giorno Sab 13 Feb 2016 11:38 Andrew Schofield < > andrew_schofield_j...@outlook.com> ha scritto: > > > This KIP is related to KIP-32, but I strikes me that it only makes sense > > with one of the two proposed message timestamp types. If I understand > > correctly, message timestamps are only certain to be monotonically > > increasing in the log if message.timestamp.type=LogAppendTime. > > > > > > > > Does timestamp-based auto-expiration require use of > > message.timestamp.type=LogAppendTime? > > > > > > > > > > I think this KIP is a good idea, but I think it relies on strict ordering > > of timestamps to be workable. > > > > > > > > Andrew Schofield > > > > > > > > > > > Date: Fri, 12 Feb 2016 10:38:46 -0800 > > > Subject: Re: [DISCUSS] KIP-47 - Add timestamp-based log deletion policy > > > From: n...@confluent.io > > > To: dev@kafka.apache.org > > > > > > Adding a timestamp based auto-expiration is useful and this proposal > > makes > > > sense. Thx! > > > > > > On Wed, Feb 10, 2016 at 3:35 PM, Jay Kreps wrote: > > > > > >> I think this makes a lot of sense and won't be hard to implement and > > >> doesn't create too much in the way of new interfaces. > > >> > > >> -Jay > > >> > > >> On Tue, Feb 9, 2016 at 8:13 AM, Bill Warshaw wrote: > > >> > > >>> Hello, > > >>> > > >>> I just submitted KIP-47 for adding a new log deletion policy based > on a > > >>> minimum timestamp of messages to retain. > > >>> > > >>> > > >>> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy > > >>> > > >>> I'm open to any comments or suggestions. > > >>> > > >>> Thanks, > > >>> Bill Warshaw > > >>> > > >> > > > > > > > > > > > > -- > > > Thanks, > > > Neha > > -- > > > -- Enrico Olivelli >
