Re: flatMapValues does not calculate timestamp for each record generated

2018-07-10 Thread Matthias J. Sax
By default, Kafka uses the record metadata timestamp. Thus, I assume that you use a custom timestamp extractor? If yes, what timestamp do you extract, as it seems you have multiple? To make it work, you will need to write the data back to Kafka into a new topic after the flatpMap() and read each r

Re: flatMapValues does not calculate timestamp for each record generated

2018-07-10 Thread Sicheng Liu
This is a specific example: We are sending metrics to Kafka Stream with the following layout: Record Key: -- |metric name|tags| -- Record Value: ---

Re: flatMapValues does not calculate timestamp for each record generated

2018-07-10 Thread Matthias J. Sax
Not sure what you mean by "does not reset recordContext". Note, that the "contract" for `flatMapValues` is that the output records inherit the timestamp of the input record. Not sure what behavior you expect? Maybe you can elaborate? -Matthias On 7/9/18 7:27 PM, Sicheng Liu wrote: > Hi, > > I

flatMapValues does not calculate timestamp for each record generated

2018-07-09 Thread Sicheng Liu
Hi, I found out that doing windowed aggregation on records generated by flatMapValues gets incorrect result. Take this topology as an example: myStream.flatMapValues(...) .groupByKey(...) .aggregate(...) Since inside KStreamWindowAggregateProcessor, the timestamp