1. Initially I looked at using the actual offset, by adding a call to AdminUtils to just delete anything in a given topic/partition to a given offset. I ran into a lot of trouble here trying to work out how the system would recognize that every broker had successfully deleted that range from the partition before returning to the client. If we were ok treating this as a completely asynchronous operation I would be open to revisiting this approach.
2. For our use case, we would be updating the config every few hours for a given topic, and there would not a be a sizable amount of consumers. I imagine that this would not scale well if someone was adjusting this config very frequently on a large system, but I don't know if there are any use cases where that would occur. I imagine most use cases would involve truncating the log after taking a snapshot or doing some other expensive operation that didn't occur very frequently. On Mon, May 2, 2016 at 2:23 PM, Jay Kreps <j...@confluent.io> wrote: > Two comments: > > 1. Is there a reason to use physical time rather than offset? The idea > is for the consumer to say when it has consumed something so it can be > deleted, right? It seems like offset would be a much more precise way > to do > this--i.e. the consumer says "I have checkpointed state up to offset X > you > can get rid of anything prior to that". Doing this by timestamp seems > like > it is just more error prone... > 2. Is this mechanism practical to use at scale? It requires several ZK > writes per config change, so I guess that depends on how frequently the > consumers would update the value and how many consumers there are...any > thoughts on this? > > -Jay > > On Thu, Apr 28, 2016 at 8:28 AM, Bill Warshaw <wdwars...@gmail.com> wrote: > > > I'd like to re-initiate the vote for KIP-47 now that KIP-33 has been > > accepted and is in-progress. I've updated the KIP ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy > > ). > > I have a commit with the functionality for KIP-47 ready to go once KIP-33 > > is complete; it's a fairly minor change. > > > > On Wed, Mar 9, 2016 at 8:42 PM, Gwen Shapira <g...@confluent.io> wrote: > > > > > For convenience, the KIP is here: > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-47+-+Add+timestamp-based+log+deletion+policy > > > > > > Do you mind updating the KIP with time formats we plan on supporting > > > in the configuration? > > > > > > On Wed, Mar 9, 2016 at 11:44 AM, Bill Warshaw <wdwars...@gmail.com> > > wrote: > > > > Hello, > > > > > > > > I'd like to initiate the vote for KIP-47. > > > > > > > > Thanks, > > > > Bill Warshaw > > > > > >
