Hi Yarden,
can you please provide all flink-related PipelineOptions you use for the
job?
Jan
On 5/16/24 13:44, Yarden BenMoshe wrote:
Hi all,
I have a project running with Beam 2.51, using Flink runner. In one of
my pipelines i have a FixedWindow and had a problem upgrading until
now, with a timers issue now resolved, and hopefully allowing me to
upgrade to version 2.56
However, I encounter another problem now which I believe is related to
watermarking(?).
My pipeline's source is a kafka topic.
My basic window definition is:
PCollection<KV<String, Iterable<CustomObject>>> windowCustomObjectInfo
= customObject.apply("windowCustomObjectInfo",
Window.into(FixedWindows.of(Duration.standardSeconds(60)))).apply(GroupByKey.create());
and ever since upgrading to version 2.56 I am not getting any output
from that window. when enabling TRACE logs, i have this message:
2024-05-12 13:50:55,257 TRACE org.apache.beam.sdk.util.WindowTracing
[] - WatermarkHold.addHolds: element hold at 2024-05-12T13:50:59.999Z
is on time for key:test-12345;
window:[2024-05-12T13:50:00.000Z..2024-05-12T13:51:00.000Z);
inputWatermark:-290308-12-21T19:59:05.225Z;
outputWatermark:-290308-12-21T19:59:05.225Z
Any hints on where should I look or maybe how I can adjust my window
definition? Are you familiar with any change that might be the cause
for my issue?
Thanks
