I am not running my pipeline from command-line, so used instead:
options.setExperiments(Collections.singletonList("use_deprecated_read"));with ExperimentalOptions added to my options interface, however I dont think there's any effect to using it. in terms of the watermark, i received again: WatermarkHold.addHolds: element hold at 2024-05-19T07:52:59.999Z is on time for key:aaa-bbb-ccc; window:[2024-05-19T07:52:00.000Z..2024-05-19T07:53:00.000Z); inputWatermark:-290308-12-21T19:59:05.225Z; outputWatermark:-290308-12-21T19:59:05.225Z בתאריך יום ה׳, 16 במאי 2024 ב-17:06 מאת Jan Lukavský <[email protected] >: > Does using --experiments=use_deprecated_read have any effect? > On 5/16/24 14:30, Yarden BenMoshe wrote: > > Hi Jan, my PipelineOptions is as follows: > options.setStreaming(true); > options.setAttachedMode(false); > options.setRunner(FlinkRunner.class); > > I've also tried adding: > options.setAutoWatermarkInterval(100L); > as seen in some github issue, without any success so far. > > other than that, i am working with parallelism:3 and number of task > slots: 3 > > Thanks! > Yarden > > בתאריך יום ה׳, 16 במאי 2024 ב-15:05 מאת Jan Lukavský <[email protected] > >: > >> Hi Yarden, >> >> can you please provide all flink-related PipelineOptions you use for the >> job? >> >> Jan >> >> On 5/16/24 13:44, Yarden BenMoshe wrote: >> > Hi all, >> > I have a project running with Beam 2.51, using Flink runner. In one of >> > my pipelines i have a FixedWindow and had a problem upgrading until >> > now, with a timers issue now resolved, and hopefully allowing me to >> > upgrade to version 2.56 >> > However, I encounter another problem now which I believe is related to >> > watermarking(?). >> > My pipeline's source is a kafka topic. >> > My basic window definition is: >> > >> > PCollection<KV<String, Iterable<CustomObject>>> windowCustomObjectInfo >> > = customObject.apply("windowCustomObjectInfo", >> > >> Window.into(FixedWindows.of(Duration.standardSeconds(60)))).apply(GroupByKey.create()); >> > >> > and ever since upgrading to version 2.56 I am not getting any output >> > from that window. when enabling TRACE logs, i have this message: >> > >> > 2024-05-12 13:50:55,257 TRACE org.apache.beam.sdk.util.WindowTracing >> > [] - WatermarkHold.addHolds: element hold at 2024-05-12T13:50:59.999Z >> > is on time for key:test-12345; >> > window:[2024-05-12T13:50:00.000Z..2024-05-12T13:51:00.000Z); >> > inputWatermark:-290308-12-21T19:59:05.225Z; >> > outputWatermark:-290308-12-21T19:59:05.225Z >> > >> > >> > Any hints on where should I look or maybe how I can adjust my window >> > definition? Are you familiar with any change that might be the cause >> > for my issue? >> > Thanks >> >
