Yes you are correct Dominik. The committed Kafka offsets tell you what the
program has read as input from the Kafka topic. But depending on the actual
program logic this does not mean that you have output the results of
processing these input events up to this point. As you have said, there are
Fli
Hi Dom,
There are sync phase and async phase in checkpointing. When a operator receives
a barrier, it performs snapshot aka the sync phase. And when the barriers pass
through all the operators including sinks, the operators will get a
notification, after which they do the async part, like commi
Okay, thanks for clarifying. I have some followup question here. If we
consider Kafka offsets commits, this basically means that
the offsets committed during the checkpoint are not necessarily the
offsets that were really processed by the pipeline and written to sink ? I
mean If there is a window i
Hi Dominik,
A record is already processed once it enters the window. Thus the
checkpoint barrier does not get blocked before the window containing the
leading records is triggered.
A window is actually part of the states of the WindowOperator and the data
records processing is to build up this sta
When a WindowOperator receives all the barrier from the upstream, it will
forward the barrier to downstream operator and perform the checkpoint
asynchronously.
It doesn't have to wait the window to trigger before sending out the barrier.
Regards,
Dian
> 在 2019年9月6日，下午8:02，Dominik Wosiński 写道：
