Hey Boerge, Thanks for the blog link. I will read this blog later.
Here is another alternative solution which may be worth thinking. We know that the Unix time 0 corresponds to January 1, 1970. Let's say the earliest time you may want to use as the timestamp of the Kafka message is within X milliseconds before the January 1, 1970. Then you can add X to the timestamp before you produce Kafka message. And you can also make similar conversion when you use `offsetsForTimes()` or after you consume messages. This seems to address your use-case without introducing negative timestamp. IMO, this solution requires a bit more logic in your application code. But it keeps the Kafka timestamp logic simple and we reserve the capability to use timestamp -1 for messages without timestamp for most Kafka users who do not need negative timestamp. Do you think this would be a good alternative solution? Thanks, Dong On Tue, Dec 5, 2017 at 5:39 PM, Boerge Svingen <bsvin...@borkdal.com> wrote: > > Yes. To provide a little more detail, we are using Kafka to store > everything ever published by The New York Times, and to make this content > available to a range of systems and applications. Assets are published to > Kafka chronologically, so that consumers can seek to any point in time and > start consuming from there, like Konstantin is describing, all the way back > to our beginning in 1851. > > https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ < > https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/> > has more information on the use case. > > > Thanks, > Boerge. > > > -- > > Boerge Svingen > Director of Engineering > The New York Times > > > > > > On 2017-12-05, at 19:35, Dong Lin <lindon...@gmail.com> wrote: > > > > Hey Konstantin, > > > > According to KIP-32 the timestamp is also used for log rolling and log > > retention. Therefore, unless broker is configured to never delete any > > message based on time, messages produced with negative timestamp in your > > use-case will be deleted by the broker anyway. Do you actually plan to > use > > Kafka as a persistent storage system that never delete messages? > > > > Thanks, > > Dong > > > > > > > > > > On Tue, Dec 5, 2017 at 1:24 PM, Konstantin Chukhlomin < > chuhlo...@gmail.com> > > wrote: > > > >> Hi Dong, > >> > >> Currently we are storing historical timestamp in the message. > >> > >> What we are trying to achieve is to make it possible to do Kafka lookup > >> by timestamp. Ideally I would do `offsetsForTimes` to find articles > >> published > >> in 1910s (if we are storing articles on the log). > >> > >> So first two suggestions aren't really covering our use-case. > >> > >> We could create a new timestamp type like "HistoricalTimestamp" or > >> "MaybeNegativeTimestamp". > >> And the only difference between this one and CreateTime is that it could > >> be negative. > >> I tend to use CreateTime for this purpose because it's easier to > >> understand from > >> user perspective as a timestamp which publisher can set. > >> > >> Thanks, > >> Konstantin > >> > >>> On Dec 5, 2017, at 3:47 PM, Dong Lin <lindon...@gmail.com> wrote: > >>> > >>> Hey Konstantin, > >>> > >>> Thanks for the KIP. I have a few questions below. > >>> > >>> Strictly speaking Kafka actually allows you to store historical data. > And > >>> user are free to encode arbitrary timestamp field in their Kafka > message. > >>> For example, your Kafka message can currently have Json or Avro format > >> and > >>> you can put a timestamp field there. Do you think that could address > your > >>> use-case? > >>> > >>> Alternatively, KIP-82 introduced Record Header in Kafka and you can > also > >>> define your customized key/value pair in the header. Do you think this > >> can > >>> address your use-case? > >>> > >>> Also, currently there are two types of timestamp according to KIP-32. > If > >>> the type is LogAppendTime then the timestamp value is the time when > >> broker > >>> receives the message. If the type is CreateTime then the timestamp > value > >> is > >>> determined when producer produces message. With these two definitions, > >> the > >>> timestamp should always be positive. We probably need a new type here > if > >> we > >>> can not put timestamp in the Record Header or the message payload. Does > >>> this sound reasonable? > >>> > >>> Thanks, > >>> Dong > >>> > >>> > >>> > >>> On Tue, Dec 5, 2017 at 8:40 AM, Konstantin Chukhlomin < > >> chuhlo...@gmail.com> > >>> wrote: > >>> > >>>> Hi all, > >>>> > >>>> I have created a KIP to support negative timestamp: > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>> 228+Negative+record+timestamp+support <https://cwiki.apache.org/ > >>>> confluence/display/KAFKA/KIP-228+Negative+record+timestamp+support> > >>>> > >>>> Here are proposed changes: https://github.com/apache/ > >>>> kafka/compare/trunk...chuhlomin:trunk <https://github.com/apache/ > >>>> kafka/compare/trunk...chuhlomin:trunk> > >>>> > >>>> I'm pretty sure that not cases are covered, so comments and > suggestions > >>>> are welcome. > >>>> > >>>> Thank you, > >>>> Konstantin > >> > >> > >
