Hi Randal, Can you share the code for the 1st approach (FlinkKinesisConsumer.setPeriodicWatermarkAssigner))? I think the 2nd approach (flatMap) can be improved by partitioning the stream the same way kinesis does (i.e. same partition key).
Regards, Roman On Mon, Dec 7, 2020 at 2:44 PM Randal Pitt <randal.p...@foresite.com> wrote: > Hi there, > > We're using Flink to read from a Kinesis stream. The stream contains > messages that themselves contain lists of events and we want our Flink jobs > (using the event time characteristic) to process those events individually. > We have this working using flatMap in the DataStream but we're having > trouble correctly assigning timestamps to the events. > > We have been using FlinkKinesisConsumer.setPeriodicWatermarkAssigner() as > that should mean the watermarks are generated correctly, but it results in > all events in one message sharing a timestamp, resulting in some events > being assigned to the wrong window. > > Using DataStream.assignTimestampsAndWatermarks() after the flatMap means we > can assign the correct timestamps, but the watermarks may not necessarily > be > correct with respect to the Kinesis shards. > > Is there are strategy we can use that gets us both watermarks from the > Kinesis consumer and correct timestamps for individual events? > > Best regards, > > Randal. > > > > -- > Sent from: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ >
