Qingsheng Ren created FLINK-18578:
-------------------------------------
Summary: Add rejecting checkpoint logic in source
Key: FLINK-18578
URL: https://issues.apache.org/jira/browse/FLINK-18578
Project: Flink
Issue Type: New Feature
Components: Runtime / Checkpointing
Reporter: Qingsheng Ren
Under some database's change data capture (CDC) case, the process is usually
divided into two phases: snapshotting phase (lock captured tables and scan all
records in them) and log streaming phase (read all changes starting from the
moment of locking tables) in order to build a complete view of captured tables.
The first snapshotting phase should be atomic so we have to give up all records
created in snapshotting phase if any failure happen, because contents in
captured tables might have changed during recovery. And checkpointing within
snapshotting phase is meaningless too.
As a result, we need to add a new feature in the source to reject checkpoint if
the source is currently within an atomic operation or some other processes that
cannot do a checkpoint currently. This rejection should not be treated as a
failure that could lead to failure of the entire job.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
