Does using --experiments=use_deprecated_read have any effect?
On 5/16/24 14:30, Yarden BenMoshe wrote:
Hi Jan, my PipelineOptions is as follows:
options.setStreaming(true);
options.setAttachedMode(false);
options.setRunner(FlinkRunner.class);
I've also tried adding:
options.setAutoWatermarkInterval(100L);
as seen in some github issue, without any success so far.
other than that, i am working with parallelism:3 and number of task
slots: 3
Thanks!
Yarden
בתאריך יום ה׳, 16 במאי 2024 ב-15:05 מאת Jan Lukavský
<[email protected]>:
Hi Yarden,
can you please provide all flink-related PipelineOptions you use
for the
job?
Jan
On 5/16/24 13:44, Yarden BenMoshe wrote:
> Hi all,
> I have a project running with Beam 2.51, using Flink runner. In
one of
> my pipelines i have a FixedWindow and had a problem upgrading until
> now, with a timers issue now resolved, and hopefully allowing me to
> upgrade to version 2.56
> However, I encounter another problem now which I believe is
related to
> watermarking(?).
> My pipeline's source is a kafka topic.
> My basic window definition is:
>
> PCollection<KV<String, Iterable<CustomObject>>>
windowCustomObjectInfo
> = customObject.apply("windowCustomObjectInfo",
>
Window.into(FixedWindows.of(Duration.standardSeconds(60)))).apply(GroupByKey.create());
>
> and ever since upgrading to version 2.56 I am not getting any
output
> from that window. when enabling TRACE logs, i have this message:
>
> 2024-05-12 13:50:55,257 TRACE
org.apache.beam.sdk.util.WindowTracing
> [] - WatermarkHold.addHolds: element hold at
2024-05-12T13:50:59.999Z
> is on time for key:test-12345;
> window:[2024-05-12T13:50:00.000Z..2024-05-12T13:51:00.000Z);
> inputWatermark:-290308-12-21T19:59:05.225Z;
> outputWatermark:-290308-12-21T19:59:05.225Z
>
>
> Any hints on where should I look or maybe how I can adjust my
window
> definition? Are you familiar with any change that might be the
cause
> for my issue?
> Thanks
