Hi,
Yes it was a watermarking issue. There were few out of order records in my
stream and as per watermarking strategy the watermark was advanced to the
future and hence current events were getting discarded.
I have fixed this by not processing future timestamped records.
Thanks
Sachin
On Mon, Apr 8, 2024 at 11:55 AM <dominik.buen...@swisscom.com> wrote:
> Hi Sachin
>
>
>
> What exactly does the MyReducer do? Can you provide us with some code?
>
>
>
> Just a wild guess from my side, did you check the watermarking? If the
> Watermarks aren't progressing there's no way for Flink to know when to emit
> a window and therefore you won't see any outgoing events.
>
> Kind Regards
>
> Dominik
>
>
>
> *From: *Sachin Mittal <sjmit...@gmail.com>
> *Date: *Monday, 8 April 2024 at 08:17
> *To: *user@flink.apache.org <user@flink.apache.org>
> *Subject: *How to debug window step in flink
>
> *Be aware:* This is an external email.
>
>
>
> Hi,
>
>
>
> I have a following windowing step in my pipeline:
>
>
>
> inputData
> .keyBy(new MyKeySelector())
> .window(
> TumblingEventTimeWindows.*of*(Time.*seconds*(60)))
> .reduce(new MyReducer())
> .name("MyReducer");
>
>
>
> Same step when I see in Flink UI shows as:
>
>
>
> Window(TumblingEventTimeWindows(60000), EventTimeTrigger, MyReducer,
> PassThroughWindowFunction) :- Sink: MyData sink +- Filter
>
>
>
>
>
> So far I don't see any errors in the pipeline.
>
>
>
> However when I check the following metrics:
>
> 0.MyReducer.numRecordsInPerSecond = 600 / s
> 0.MyReducer.numRecordsOutPerSecond = 0 / s
>
>
>
> It shows that the step is receiving data but it is not outputting anything.
>
> Looks like some problem in the step.
>
>
>
> Most obvious thing that I can deduce is that the window step may not be
> outputting any record.
>
> Also MyReducer is a pretty simple function and it does not catch any
> exception, so if there was a problem in this function it would have been
> thrown to the pipeline
>
> and I would have known.
>
>
>
> Please let me know how I can debug this better.
>
>
>
> Thanks
>
> Sachin
>
>
>
>
>
>
>
