Re: Kafka Streams window retention period question

2017-01-05 Thread Alexander Demidko
Great, thanks for the info guys! On Thu, Jan 5, 2017 at 10:09 PM, Matthias J. Sax wrote: > Hi Alex, > > if a window was purged because its retention time passed it will not > accept any records anymore -- thus, if a very late record arrives, it > will get dropped without any further notice. > >

Re: Kafka Streams window retention period question

2017-01-05 Thread Matthias J. Sax
Hi Alex, if a window was purged because its retention time passed it will not accept any records anymore -- thus, if a very late record arrives, it will get dropped without any further notice. About stream time and partition: yes. And how time is advanced/tracked in independent for the window typ

Re: Kafka Streams window retention period question

2017-01-05 Thread Sachin Mittal
What should happen when the window 00:00..01:00 will finally get purged (and the internal stream time will get bumped to say time 10:00) but then I receive an event ? Will it create the 00:00..01:00 window again or the event will be dropped because it's way older than the internal stream time? Onc

Re: Kafka Streams window retention period question

2017-01-05 Thread Alexander Demidko
Hi Matthias, Thanks for such a thorough response! I guess there are cases when a determinism might be preferred over computing "more correct results" (e.g. in unit tests, where one manually lays out an order of incoming events and wants to get an exact output), but from now on I can simply assume

Re: Kafka Streams window retention period question

2017-01-04 Thread Matthias J. Sax
Hi Alexander, first, both mailing list should be fine :) About internal time tracking: Kafka Streams tracks an internal "stream time" that is determined as the minimum "partition time" over all its input partitions. The "partition time" is tracked for each input partition individually and is the

Kafka Streams window retention period question

2017-01-04 Thread Alexander Demidko
Hi folks, I'm experimenting with Kafka Streams windowed aggregation and came across window retention period behavior I don't fully understand. I'm using custom timestamp extractor which gets the timestamp from the payload. Values are aggregated using tumbling time windows and summed by the key. I