Hi all,
Thanks for the detailed responses!
I’ve taken a closer look, and as far as I can tell, this is an existing problem
with the checkpointing mechanism. This is also tracked in this JIRA [1], but
not fixed. The PR attached to the JIRA temporarily blocks the events being sent
from Coordinato
Hi all,
Thanks for the detailed responses!
I’ve taken a closer look, and as far as I can tell, this is an existing problem
with the checkpointing mechanism. This is also tracked in this JIRA [1], but
not fixed. The PR attached to the JIRA temporarily blocks the events being sent
from Coordinato
Hi all,
Thanks for the detailed responses!
I’ve taken a closer look, and as far as I can tell, this is an existing problem
with the checkpointing mechanism. This is also tracked in this JIRA [1], but
not fixed. The PR attached to the JIRA temporarily blocks the events being sent
from Coordinato
Hi Hong,
Great question! Afaik, it depends on the implementation. Speaking of the
"loopback" event sending to the SplitEnumerator, I guess you meant here [1]
(It might be good, if you could point out the right position in the source
code to help us understand the question better:)), which will end
Hi Hong,
The checkpoint is triggered by the timer executor of CheckpointCoordinator.
It triggers the checkpoint in SourceCoordinator (which is passed to
SplitEnumerator) and then in SourceOperator. The checkpoint event is put in
SplitEnumerator's event loop to be executed. You can see the details
Hi Hong,
In addition, I guess the inconsistency could also be handled by the
reader.start() method such that the operator event is re-sent during
restore.
Best,
Mason
On Tue, May 23, 2023 at 7:58 AM Piotr Nowojski wrote:
> Hi,
>
> I vaguely remember someone implementing a mechanism to deal wit
Hi,
I vaguely remember someone implementing a mechanism to deal with it. I
think at least at some point (it might have changed since I looked at it),
it was solving the problem via canceling the checkpoint in the scenario
that you described. However I can not remember from the top of my head
neith
Hi all,
I’m writing a new source based on the FLIP-27 Source API, and I had some
questions on the checkpointing mechanisms and associated guarantees. Would
appreciate if someone more familiar with the API would be able to provide
insights here!
In FLIP-27 Source, we now have a SplitEnumerator
