If data flows normally, watermarks will progress to downstream operators.
If you have no records flowing in a partition of a stream and you don't
have an idleness configured, that partition will hold back the progress of
watermarks. Watermarks are essential for the checkpointing mechanism, which
ul
Can you elaborate a bit more? While idleness is not what we’re seeing now, it
could perhaps be an issue later on. What about a certain partition going idle
will result in state buildup?
Thanks,
Vishal
On 25 Jan 2023 at 9:14 PM +0530, Martijn Visser ,
wrote:
> Hi Vishal,
>
> Could idleness be an
Hi Vishal,
Could idleness be an issue? I could see that if idleness occurs and the
Kafka Source not going in an idle state, that more internal state (to
commit Kafka transactions) can build up over time that ultimately causes an
out of memory problem. See
https://nightlies.apache.org/flink/flink-d
Could it be that link is unable to commit offsets to Kafka? I know that blinks
checkpoint mechanism isn’t tied to its ability to commit offset but at the same
time, we’ve seen that the job can take hours to commit offsets while
checkpoints go through successfully during that period. But with Kaf
