Hi Jan, my PipelineOptions is as follows: options.setStreaming(true); options.setAttachedMode(false); options.setRunner(FlinkRunner.class);
I've also tried adding: options.setAutoWatermarkInterval(100L); as seen in some github issue, without any success so far. other than that, i am working with parallelism:3 and number of task slots: 3 Thanks! Yarden בתאריך יום ה׳, 16 במאי 2024 ב-15:05 מאת Jan Lukavský <[email protected] >: > Hi Yarden, > > can you please provide all flink-related PipelineOptions you use for the > job? > > Jan > > On 5/16/24 13:44, Yarden BenMoshe wrote: > > Hi all, > > I have a project running with Beam 2.51, using Flink runner. In one of > > my pipelines i have a FixedWindow and had a problem upgrading until > > now, with a timers issue now resolved, and hopefully allowing me to > > upgrade to version 2.56 > > However, I encounter another problem now which I believe is related to > > watermarking(?). > > My pipeline's source is a kafka topic. > > My basic window definition is: > > > > PCollection<KV<String, Iterable<CustomObject>>> windowCustomObjectInfo > > = customObject.apply("windowCustomObjectInfo", > > > Window.into(FixedWindows.of(Duration.standardSeconds(60)))).apply(GroupByKey.create()); > > > > and ever since upgrading to version 2.56 I am not getting any output > > from that window. when enabling TRACE logs, i have this message: > > > > 2024-05-12 13:50:55,257 TRACE org.apache.beam.sdk.util.WindowTracing > > [] - WatermarkHold.addHolds: element hold at 2024-05-12T13:50:59.999Z > > is on time for key:test-12345; > > window:[2024-05-12T13:50:00.000Z..2024-05-12T13:51:00.000Z); > > inputWatermark:-290308-12-21T19:59:05.225Z; > > outputWatermark:-290308-12-21T19:59:05.225Z > > > > > > Any hints on where should I look or maybe how I can adjust my window > > definition? Are you familiar with any change that might be the cause > > for my issue? > > Thanks >
