Hello,
I am currently updating from Beam 2.25.0 to Beam 2.27.0 and from Flink
1.10.3 to Flink 1.11.3.
My pipeline does read from 2 Kafka topics and windowing them via:
Window.<KV<String, AssetProto.AssetEnvelope>>into(
FixedWindows.of(Duration.standardMinutes(1)))
.withAllowedLateness(Duration.standardMinutes(30)
.discardingFiredPanes())
followed by a CoGroupByKey - when testing it I see the following behavior:
1. When starting with an offset that goes back 48 hours the pipeline
consumes everything up to the latest message before its starts the
operators after the CoGroupByKey. This seem to have caused OOM-errors.
2. This is visible in the Flink frontend like this
[image: Screenshot 2021-02-12 at 09.56.39.png]
The behavior I would expect is that as soon as the watermark passes the end
of the window and the allowed lateness it should start trickling down into
the other operators. Or is this kind of a memory and speed optimization?
Best,
Tobi
