Re: Watermarks event time vs processing time

2022-03-29 Thread HG
atermark strategy > - and replaces it with something that is arbitrary (at this point > it is hard to guess the correct max lateness that is a mixture of the > events from multiple Kafka partitions) > > > > Concusion: > > The only way to make the event tim

RE: Watermarks event time vs processing time

2022-03-29 Thread Schwalbe Matthias
manner. Hope this helps Thias From: HG Sent: Tuesday, March 29, 2022 1:07 PM To: Schwalbe Matthias Cc: user Subject: Re: Watermarks event time vs processing time ⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠ Hello Matthias, When I remove all the watermark strategies it does not

Re: Watermarks event time vs processing time

2022-03-29 Thread HG
Hello Matthias, When I remove all the watermark strategies it does not process anything . For example when I use WatermarkStrategy.noWatermarks() instead of the one I build nothing seems to happen at all. Also when I skip the part where I add wmStrategy to create tuple4dswm: DataStream> tuple4

RE: Watermarks event time vs processing time

2022-03-16 Thread Schwalbe Matthias
ing of the watermarks on single operators / per subtask useful: Look for subtasks that don’t have watermarks, or too low watermarks for a specific session window to trigger. Thias From: HG Sent: Mittwoch, 16. März 2022 16:41 To: Schwalbe Matthias Cc: user Subject: Re: Watermarks event time

Re: Watermarks event time vs processing time

2022-03-16 Thread HG
Hi Matthias and others Thanks for the answer. I will remove the Idleness. However I am not doing max/min etc. Unfortunately most examples are about aggregations. The inputs are like this {"handling_time":1646992800260,"transaction_id":"017f6af1548e-119dfb",} {"handling_time":164699280

RE: Watermarks event time vs processing time

2022-03-16 Thread Schwalbe Matthias
Hi Hanspeter, Let me relate some hints that might help you getting concepts clearer. From your description I make following assumptions where your are not specific enough (please confirm or correct in your answer): 1. You store incoming events in state per transaction_id to be sorted/aggreg