Then how to implement exactly-once async io? That is, neither missing data or duplicating data.
Is there some way to index data by checkpoint id and records which checkpoints already commit to db? But that means we need MapState, right? However, the async-io operator normally follows other operators, e.g. fold, so it normally faces the DataStream but not KeyedStream, and DataStream only supports ListState, right? 2018-01-03 18:43 GMT+08:00 Stefan Richter <s.rich...@data-artisans.com>: > > >> Am 01.01.2018 um 15:22 schrieb Jinhua Luo <luajit...@gmail.com>: >> >> 2017-12-08 18:25 GMT+08:00 Stefan Richter <s.rich...@data-artisans.com>: >>> You need to be a bit careful if your sink needs exactly-once semantics. In >>> this case things should either be idempotent or the db must support rolling >>> back changes between checkpoints, e.g. via transactions. Commits should be >>> triggered for confirmed checkpoints („notifyCheckpointComplete“). >> >> I doubt if we have a risk here: in notifyCheckpointComplete, the >> checkpoint was completed, and if the process crashes (or machine >> failure) before it commits the db, the flink would restart the app, >> restoring the state from the last checkpoint, but it would not invoke >> notifyCheckpointComplete again? correct? if so, we would miss the >> database ingress for the data between the last two checkpoints, am I >> correct? > > Yes, that is correct. What I was talking about was more the opposite > problem,i.e. committing too early. In that case, you could have committed for > a checkpoint that failed afterwards, and recovery will start from an earlier > checkpoint but with your commit already applied. You should only commit after > you received the notification or else your semantics can be down to > „at-least-once". >
