Re: Timestamp and key preservation over operators

Thank you Fabian. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Timestamp and key preservation over operators

The window operator cannot configured to use the max timestamp of the events in the window as the timestamp of the output record. The reason is that such a behavior can produce late records. If you want to do that, you have to track the max timestamp and assign it yourself with a timestamp assigne

Re: Timestamp and key preservation over operators

Thank you Fabian. One more question from me on this topic: as I send out early messages in my window function, the timestamp assigned by window function (to the end-time of the window) is not like my expectation. I want it to be the time of the (last) message that triggered the output. Is there a

Re: Timestamp and key preservation over operators

Hi Averell, Yes, timestamps and watermarks do not (completely) move together. The watermark should always be lower than the timestamps of the currently processed records. Otherwise, the records might be processed as late records (depending on the logic). The easiest way to check the timestamp of

Re: Timestamp and key preservation over operators

Thank you Fabian. I have one more question about timestamp: In the previous email, you asked how did I check the timestamp - I don't have an answer. Then I only checked the watermark, not the timestamp. I had the (wrong) assumption that watermarks advance along with timestamps. Today I played with

Re: Timestamp and key preservation over operators

Hi Averell, The watermark of a stream is always the low watermark of all its input streams. If one of the input streams does not have watermarks, Flink does not compute a watermark for the merged stream. If you do not need time-based operations on streams 3 and 4, setting the watermark to MAX_WATE

Re: Timestamp and key preservation over operators

Hi Fabian, Guowei, I have some updates: 1. I added timestamp&watermark extractor on all of my remaining sources (3 & 4), and the watermark does propagate to my final operator. 2. As I could not find a way to set my file sources as IDLE, I tried to tweak the class ContinuousFileReaderOperator to be

Re: Timestamp and key preservation over operators

Hi Fabian, Guowei Thanks for the help. My flow is as the attached photo. Where (1) and (2) are the main data streams from file sources, while (3) and (4) are the enrichment data, also from file sources.

Re: Timestamp and key preservation over operators

Hi, Actually all operators should preserve record timestamps if set the correct TimeCharacteritics to event time. A window operator will set the timestamp of all emitted records to the end-timestamp of the window. Not sure what happens if you use a processing time window in an event time applicati

Re: Timestamp and key preservation over operators

Hi, Most operators will preserve the input elements timestamp if it has. Window is a special case. The timestamp of elements emitted by window is the maxTimestamp of the Window which is triggered. Different Window will have different implementation.(GlobalWindow/TimeWindow/CustomizedWindow). Keyby

Timestamp and key preservation over operators

Hello, I extracted timestamps using BoundedOutOfOrdernessTimestampExtractor from my sources, have a WindowFunction, and found that my timestamps has been lost. To do another Window operation, I need to extract timestamp again. I tried to find a document for that but haven't found one. Could you pl