Martijn Visser created FLINK-41060:
--------------------------------------
Summary: Exactly-once JdbcSink commits its XA branch at the
checkpoint barrier
Key: FLINK-41060
URL: https://issues.apache.org/jira/browse/FLINK-41060
Project: Flink
Issue Type: Bug
Components: Connectors / JDBC
Affects Versions: jdbc-3.4.0, jdbc-4.1.0, jdbc-4.0.0, jdbc-3.3.0, jdbc-3.2.0
Reporter: Martijn Visser
Assignee: Martijn Visser
Fix For: jdbc-5.0.0, jdbc-4.1.1
With the exactly-once {{JdbcSink}}, {{JdbcWriter.prepareCommit()}} prepares the
XA branch, resets {{pendingRecords}} and returns the committable, after which
{{JdbcWriter.snapshotState()}} sees {{pendingRecords == false}} and calls
{{commitTxUntil(lastCheckpointId)}}, which commits the branch that was just
prepared. The data is committed at the barrier, before the checkpoint
completes. A checkpoint that fails after the barrier passed the sink leaves its
records committed, and the restart replays and writes them again.
The committer does not compensate: {{JdbcCommitableSerializer}} only writes the
{{Xid}}, so a deserialized committable carries no transaction and
{{JdbcCommitter.commit()}} falls back to {{commitTx()}} on its own empty
{{XaTransaction}}. With writer and committer in one chain the committable still
carries the writer's transaction object, which is why
{{ExactlyOnceJdbcSinkTest}} passes; with {{pipeline.operator-chaining=false}}
the same job leaves its only branch in-doubt and commits no row (Derby: 0 of 10
rows, 1 in-doubt branch).
We should let {{snapshotState}} commit only when {{prepareCommit}} found
nothing to prepare, and let {{JdbcCommitter.commit}} commit the handed xid
through {{xaConnectionProvider.commit(xid, true)}}.
{{ExactlyOnceJdbcWriterTest.testCheckpoint}} asserts the barrier-time
visibility today and needs to change with it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
