wuhainan commented on issue #12212:
URL: https://github.com/apache/seatunnel/issues/12212#issuecomment-5586664309
@DanielLeens
# Additional Investigation Results
We completed the requested native binlog decoding and checkpoint
investigation.
## Environment Correction
The actual runtime environment is:
- SeaTunnel: `2.3.13`
- Flink: `1.20`
- Scala: `2.12`
- Debezium: `1.9.8.Final`
- mysql-binlog-connector-java: `0.27.2`
- Source: MySQL Community Server `5.7.29-log`
- Target: TiDB
## Native Binlog Decoding Result
The failing event reported by Debezium was:
```text
file: mysql-bin.002163
last recorded offset: 350499092
event start position: 350555013
next position: 350563202
event type: EXT_WRITE_ROWS
data length: 8170
event timestamp: 2026-09-08 18:04:41
```
We decoded the complete transaction using the native MySQL tool:
```bash
mysqlbinlog \
--read-from-remote-server \
--base64-output=DECODE-ROWS \
--verbose --verbose \
--start-position=350499092 \
--stop-position=350594360 \
mysql-bin.002163
```
The transaction structure is:
```text
350499092 BEGIN
350499155 Table_map
...
350546742 Table_map
350546817 Write_rows
350555013 Write_rows <-- event reported by Debezium
350563202 Write_rows
350571375 Write_rows
350579550 Write_rows
350587754 Write_rows, STMT_END_F
350594329 Xid
350594360 COMMIT completed
```
`mysqlbinlog -vv` successfully decoded all row events through `COMMIT`. No
CRC or row-decoding error was reported.
Therefore, the stored MySQL binlog does not appear to be deterministically
malformed. The EOF was only observed in `mysql-binlog-connector-java`.
## Confirmed Missing Rows
Two consecutive source rows are present in the native MySQL binlog. They
belong to an earlier transaction that committed at:
```text
mysql-bin.002163:349812844
```
The target contains the surrounding records, including the immediately
preceding and following IDs, but these two rows are absent by both:
- primary key;
- business unique key.
The pattern is therefore:
```text
... previous records present
record A present
missing record 1
missing record 2
record B present
... following records present
```
This is a two-row gap rather than the loss of the entire source transaction.
## Checkpoint and Failure Timeline
Flink checkpoint configuration:
```text
execution.checkpointing.mode = EXACTLY_ONCE
execution.checkpointing.interval = 30 s
```
The relevant JobManager and TaskManager timeline is:
```text
18:15:37.606
Checkpoint 15278 was triggered.
18:15:46.971
The Debezium binlog reader failed.
Last offset stored:
mysql-bin.002163:350499092
Binlog reader near:
mysql-bin.002163:350555013
18:15:46.998
Checkpoint 15278 completed successfully.
18:16:07.606
Checkpoint 15279 was triggered after the Debezium reader thread had
already reported the deserialization failure.
18:16:21.077
The MySQL CDC Source task changed from RUNNING to FAILED.
18:16:21.101
Checkpoint 15279 still completed successfully.
18:16:21.103
SourceCoordinator marked checkpoint 15279 as completed.
18:16:31.102
The job was restored from checkpoint 15279.
```
The relevant source error is:
```text
Error during binlog processing.
Last offset stored = {
file=mysql-bin.002163,
pos=350499092,
...
}
binlog reader near position =
mysql-bin.002163/350555013
Caused by:
java.io.EOFException:
Failed to read next byte from position -649582912
at AbstractRowsEventDataDeserializer.deserializeLongLong(...)
```
The negative `ByteArrayInputStream` position appears to be an overflowing
cumulative integer counter, not a negative MySQL binlog position.
No JDBC `BatchUpdateException`, `SQLException`, or target write error was
found around the September 8 failure. An unrelated JDBC communication error
exists in logs from September 3.
## Current Hypothesis
The missing-row transaction committed at position `349812844`, while
Debezium's last recorded offset at failure was already `350499092`.
The suspicious behavior is that checkpoints 15278 and 15279 completed after
the binlog reader had already encountered the fatal exception, and the job then
restored from checkpoint 15279.
We have not found a post-restore log that explicitly prints the restored
binlog offset. However, the missing rows were not replayed after recovery.
This suggests a possible race between:
1. the SplitFetcher/binlog reader encountering an exception;
2. propagation of that exception to the Source task;
3. SourceReader checkpoint state being acknowledged;
4. records already read by the fetcher but not yet durably delivered
downstream.
A checkpoint may therefore contain a source offset later than records that
were not successfully delivered or persisted.
## Questions
1. Can `SourceReader.snapshotState()` complete while a `SplitFetcher` has
already failed but the failure has not yet reached
`SourceReaderBase.getNextFetch()`?
2. Can the MySQL CDC split offset advance ahead of records still pending in
the fetcher/source-reader queue?
3. Should `SplitFetcherManager.checkErrors()` also be called before
acknowledging a checkpoint?
4. Is this a known issue in SeaTunnel 2.3.13, Debezium 1.9.8.Final, or
mysql-binlog-connector-java 0.27.2?
5. Is there a recommended way to prevent restoring from a checkpoint that
completed after the fetcher had already encountered a fatal error?
We can provide additional redacted logs or run a targeted build/test if
needed.
> Note: `execution.checkpointing.mode=EXACTLY_ONCE` only describes the Flink
checkpoint mode. We have not yet confirmed that the JDBC Sink
connector-specific `exactly_once` option is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
