Re: Kafka streams for out of order density aggregation

2016-06-24 Thread Matthias J. Sax
I just want to add something: If I understand the question correctly, you are asking for a strong ordering guarantee. I personally doubt that out-of-order on count-based windows can be supported with strong consistency guarantee in an efficient manner. If a late record arrives for window X, the n

Re: Kafka streams for out of order density aggregation

2016-06-23 Thread Guozhang Wang
Hello Ryan, On the DSL layer, currently there is not support for record window yet; and we are discussing about adding such support in the future, maybe first session windows then others. On the Processor API layer, you can definitely implement this "record window" feature yourself by keeping tra

Kafka streams for out of order density aggregation

2016-06-23 Thread Ryan Thompson
Hello, Say I have a stream, and want to determine whether or not a given "density" of of records match a given condition. For example, let's say I want to how many of the last 10 records have a numerical value greater than 100. Does the kafka streams DSL (or processor API) provide a way to do th