Re: Non-temporal watermarks

2023-02-03 Thread David Anderson
system always > represents a forward moving moment in time - it is just that it means an > historic date for data during recovery from the databases and then a > current timestamp when the system is processing live data. > > Hope that gives you some ideas and help. > > James. > ---

Re: Non-temporal watermarks

2023-02-02 Thread James Sandys-Lumsdaine
ata. Hope that gives you some ideas and help. James. From: Gen Luo Sent: 02 February 2023 09:52 To: Jan Lukavský Cc: user@flink.apache.org Subject: Re: Non-temporal watermarks Hi, This is an interesting topic. I suppose the watermark is defined based on the eve

Re: Non-temporal watermarks

2023-02-02 Thread Gen Luo
Hi, This is an interesting topic. I suppose the watermark is defined based on the event time since it's mainly used, or designed, for the event time processing. Flink provides the event time processing mechanism because it's widely needed. Every event has its event time and we usually need to grou

Re: Non-temporal watermarks

2023-02-02 Thread Jan Lukavský
Hi, I will not speak about details related to Flink specifically, the concept of watermarks is more abstract, so I'll leave implementation details aside. Speaking generally, yes, there is a set of requirements that must be met in order to be able to generate a system that uses watermarks.