Re: Tumbling windows with long retention

2016-12-14 Thread Matthias J. Sax
Mikael, by "growing out of bounds" we refer to the fact, that the changelog encodes the keys as pair of . Thus, over time as we create more and more window, storage requirement grows and grows and will eventually hit a wall. How fast this happens, depends mainly on your window advance time (and nu

Re: Tumbling windows with long retention

2016-12-14 Thread Mikael Högqvist
Hi Matthias, kind of :) I'm interested in the retention mechanisms and my use case is to keep old windows around for a long time (up to a year or longer) and access them via interactive queries. As I understand from the documentation, the retention mechanism is used to avoid changelogs from "grow

Re: Tumbling windows with long retention

2016-12-14 Thread Matthias J. Sax
I am not sure if I can follow. However, in Kafka Streams using window aggregation, the windowed KTable uses a key-value store internally -- it's only called windowed store because it encodes the key for the store as pair of and also applies a couple of other mechanism with regard to retention tim

Tumbling windows with long retention

2016-12-14 Thread Mikael Högqvist
Hi, I'm wondering about the tradeoffs when implementing a tumbling window with a long retention, e.g. 1 year. Is it better to use a normal key value store and aggregate the time bucket using a group by instead of a window store? Best, Mikael