I agree it looks very similar, will continue to look at it with the help of this issue. Thanks, Jan!
Does the description in [1] match your case? > > [1] https://github.com/apache/beam/issues/31085#issuecomment-2115304242 > On 5/19/24 10:07, Yarden BenMoshe wrote: > > I am not running my pipeline from command-line, so used instead: > options.setExperiments(Collections.singletonList("use_deprecated_read")); > > with ExperimentalOptions added to my options interface, however I dont > think there's any effect to using it. in terms of the watermark, i received > again: > WatermarkHold.addHolds: element hold at 2024-05-19T07:52:59.999Z is on > time for key:aaa-bbb-ccc; > window:[2024-05-19T07:52:00.000Z..2024-05-19T07:53:00.000Z); > inputWatermark:-290308-12-21T19:59:05.225Z; > outputWatermark:-290308-12-21T19:59:05.225Z > > > > בתאריך יום ה׳, 16 במאי 2024 ב-17:06 מאת Jan Lukavský <[email protected] > >: > >> Does using --experiments=use_deprecated_read have any effect? >> On 5/16/24 14:30, Yarden BenMoshe wrote: >> >> Hi Jan, my PipelineOptions is as follows: >> options.setStreaming(true); >> options.setAttachedMode(false); >> options.setRunner(FlinkRunner.class); >> >> I've also tried adding: >> options.setAutoWatermarkInterval(100L); >> as seen in some github issue, without any success so far. >> >> other than that, i am working with parallelism:3 and number of task >> slots: 3 >> >> Thanks! >> Yarden >> >> בתאריך יום ה׳, 16 במאי 2024 ב-15:05 מאת Jan Lukavský < >> [email protected]>: >> >>> Hi Yarden, >>> >>> can you please provide all flink-related PipelineOptions you use for the >>> job? >>> >>> Jan >>> >>> On 5/16/24 13:44, Yarden BenMoshe wrote: >>> > Hi all, >>> > I have a project running with Beam 2.51, using Flink runner. In one of >>> > my pipelines i have a FixedWindow and had a problem upgrading until >>> > now, with a timers issue now resolved, and hopefully allowing me to >>> > upgrade to version 2.56 >>> > However, I encounter another problem now which I believe is related to >>> > watermarking(?). >>> > My pipeline's source is a kafka topic. >>> > My basic window definition is: >>> > >>> > PCollection<KV<String, Iterable<CustomObject>>> windowCustomObjectInfo >>> > = customObject.apply("windowCustomObjectInfo", >>> > >>> Window.into(FixedWindows.of(Duration.standardSeconds(60)))).apply(GroupByKey.create()); >>> > >>> > and ever since upgrading to version 2.56 I am not getting any output >>> > from that window. when enabling TRACE logs, i have this message: >>> > >>> > 2024-05-12 13:50:55,257 TRACE org.apache.beam.sdk.util.WindowTracing >>> > [] - WatermarkHold.addHolds: element hold at 2024-05-12T13:50:59.999Z >>> > is on time for key:test-12345; >>> > window:[2024-05-12T13:50:00.000Z..2024-05-12T13:51:00.000Z); >>> > inputWatermark:-290308-12-21T19:59:05.225Z; >>> > outputWatermark:-290308-12-21T19:59:05.225Z >>> > >>> > >>> > Any hints on where should I look or maybe how I can adjust my window >>> > definition? Are you familiar with any change that might be the cause >>> > for my issue? >>> > Thanks >>> >>
