Kafka Streams reducebykey and tumbling window - need final windowed KTable values

2016-06-13 Thread Clive Cox
Hi,  I would like to process a stream with a tumbling window of 5secs, create aggregated stats for keys and push the final aggregates at the end of each window period to a analytics backend. I have tried doing something like:   stream     .map         .reduceByKey(...           , TimeWindows.

Re: Kafka Streams reducebykey and tumbling window - need final windowed KTable values

2016-06-13 Thread Clive Cox
escribed in KIP-63 <https://cwiki.apache.org/confluence/display/KAFKA/KIP-63:+Unify+store+and+downstream+caching+in+streams>), however they are not implemented yet. So your code needs to handle each update. Thanks Eno > On 13 Jun 2016, at 11:13, Clive Cox wrote: > > Hi, >  I would lik

Re: Kafka Streams reducebykey and tumbling window - need final windowed KTable values

2016-06-26 Thread Clive Cox
re to deduplicate the final output before sending to the backend. It could have a simple policy like "output all values at 1 second intervals" or "output after 10 records have been received". Eno > On 13 Jun 2016, at 13:36, Clive Cox wrote: > > > Thanks

Re: Kafka Streams reducebykey and tumbling window - need final windowed KTable values

2016-06-30 Thread Clive Cox
gt; structures. > > Thanks, > Eno > >> On 26 Jun 2016, at 18:42, Clive Cox wrote: >> >> Following on from this thread, if I want to iterate over a KTable at the end >> of its hopping/tumbling Time Window how can I do this at present myself? Is >> th