Hi there, We're using Flink to read from a Kinesis stream. The stream contains messages that themselves contain lists of events and we want our Flink jobs (using the event time characteristic) to process those events individually. We have this working using flatMap in the DataStream but we're having trouble correctly assigning timestamps to the events.
We have been using FlinkKinesisConsumer.setPeriodicWatermarkAssigner() as that should mean the watermarks are generated correctly, but it results in all events in one message sharing a timestamp, resulting in some events being assigned to the wrong window. Using DataStream.assignTimestampsAndWatermarks() after the flatMap means we can assign the correct timestamps, but the watermarks may not necessarily be correct with respect to the Kinesis shards. Is there are strategy we can use that gets us both watermarks from the Kinesis consumer and correct timestamps for individual events? Best regards, Randal. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
