bq. an older timestamp that allowed I guess you meant 'than allowed'
Cheers On Tue, Nov 21, 2017 at 2:57 PM, Matthias J. Sax <matth...@confluent.io> wrote: > This is possible, but I think you don't need the time-based index for it :) > > You will just buffer up all messages for a 5 minute sliding-window and > maintain all message sorted by timestamp in this window. Each time the > window "moves" you write the oldest records that "drop out" of the > window to the topic. If you get a record with an older timestamp that > allowed, you don't insert in into the window but drop it. > > The timestamp index is useful if you want to seek to a specific offset > base on timestamp. But I don't think you need this for your use case. > > > > -Matthias > > On 11/21/17 1:39 PM, Ray Ruvinskiy wrote: > > I’ve been reading https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 33+-+Add+a+time+based+log+index and trying to determine whether I can use > the time-based index as an efficient way to sort a stream of messages into > timestamp (CreateTime) order. > > > > I am dealing with a number of sources emitting messages that are then > processed in a distributed fashion and written to a Kafka topic. During > this processing, the original order of the messages is not strictly > maintained. Each message has an embedded timestamp. I’d like to be able to > sort these messages back into timestamp order, allowing for a certain > lateness interval, before processing them further. For example, supposing > the lateness interval is 5 minutes, at time T I’d like to consume from the > topic all messages with timestamp up to (T - 5 minutes), in timestamp > order. The assumption is that a message should be no more than 5 minutes > late; if it is more than 5 minutes late, it can be discarded. Is this > something that can be done with the time-based index? > > > > Thanks, > > > > Ray > > > >
