processFunction will just emit watermarks from upstream as they come. No
function/operator in Flink is a black hole w.r.t. watermarks. It's just
important to remember that watermark after a network shuffle is always the
min of all inputs (ignoring idle inputs). So if any connected upstream part
is
Thanks again Arvid,
I am getting closer to the culprit as I've found some interesting
scenarios. Still no exact answer yet. We are indeed also using
.withIdleness to mitigate slow/issues with partitions.
I did have a few clarifying questions though w.r.t watermarks if you don't
mind.
*Watermark pr
Hi Ahmad,
The ProcessFunction is simply forwarding the Watermark [1]. So I don't have
any explanation as to why it would not advance anymore as soon as you emit
data. My assumption was that by emitting in the process function causes
backpressure and thus halts the advancement of the watermark upst
Thanks Arvid.
Getting the easy stuff out of the way, I certainly wait for longer than 10s
(typically observe what happens over a few minutes) so the bounded
watermark issue isn't in play here.
The Async IO as it stands today has timeouts so it doesn't run
indefinitely. WIth that said, I replaced t
Hi Ahmad,
>From your description, I'd look in a different direction: Could it be that
your Sink/Async IO is not processing data (fast enough)?
Since you have a bounded watermark strategy, you'd need to see 10s of data
being processed before the first watermark is emitted.
To test that, can you ple
Flink 1.11
I have a simple Flink application that reads from Kafka, uses event
timestamps, assigns timestamps and watermarks and then key's by a field and
uses a KeyedProcessFunciton.
The keyed process function outputs events from with the `processElement`
method using `out.collect`. No timers are
