Hi Paul,
I guess it depends on your use case whether the transactional sink "needs
to cooperate with a trigger that fires right before each checkpoint".
You could use a regular Trigger that fires every minute (in processing
time) and configure checkpointing to an interval of 1 minute. In
worst-cas
Hi Fabian,
Thanks for your reply!
It takes me a while to reconsider the proposal, and I think you’re right. The
checkpoint hook would affect a lot of mechanisms we already have, and it’s
unworthy.
> Wouldn't a transactional sink provide exactly the same guarantees?
Yes, it is. But it needs to
Hi Paul,
I think this would be very tricky to implement and interfere with many
parts of the system like state backends, checkpointing logic, etc.
We would need to maintain a copy (or version) of the state at the time of a
checkpoint. There might be multiple checkpoints in flight. Checkpoints
migh
Hi Fabian,
Perhaps I didn’t explain that clearly. Actually I want a trigger to fire when a
checkpoint is completed, and emit the intermediate results in consistency
with the completed checkpoint.
It works like this:
1) Once the window operator receives a barrier, it performs the snapshot as
us
Hi Paul,
If I got your proposal right, you'd like to fire a Trigger right before a
checkpoint is taken, correct?
So, before taking a checkpoint, a Trigger would fire and the operator would
process and emit some intermediate results.
This approach would not completely solve the consistency issue b
