Yanquan Lv created FLINK-41057:
----------------------------------
Summary: Iceberg append-only tables incorrectly append
before-images from UPDATE and DELETE events
Key: FLINK-41057
URL: https://issues.apache.org/jira/browse/FLINK-41057
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.7.0
Reporter: Yanquan Lv
Assignee: Yanquan Lv
h2. BACKGROUND
The Iceberg sink must handle CDC events according to the target table's write
capabilities.
For tables with identifier fields, UPDATE events require deleting the
before-image and inserting the after-image. For append-only tables without
identifier fields, UPDATE events should only append the after-image, and DELETE
events should be ignored.
The fix for FLINK-40640 introduced unconditional conversion of UPDATE events
into DELETE(before) followed by INSERT(after), without distinguishing these two
types of target tables.
h2.
PROBLEM
Append-only writers do not apply deletion semantics to rows marked as DELETE.
Instead, they append those rows as ordinary records.
As a result:
- UPDATE events append both the before-image and the after-image, introducing
an extra copy of the old record.
- DELETE events append the deleted record instead of leaving the target table
unchanged.
The UPDATE behavior is a regression introduced by the unconditional UPDATE
splitting. The DELETE behavior is a pre-existing issue.
h2.
REPRODUCTION SCENARIO
Use an Iceberg target table without identifier fields.
1. Insert a record: (1, "old").
2. Update the record from (1, "old") to (1, "new").
3. Commit the changes.
Actual table contents:
- (1, "old")
- (1, "old")
- (1, "new")
Expected table contents:
- (1, "old")
- (1, "new")
If a subsequent DELETE event for (1, "new") is received, the table contents
should remain unchanged. No additional record should be appended.
h2.
EXPECTED BEHAVIOR
For append-only target tables:
- INSERT and REPLACE: append the after-image.
- UPDATE: append only the after-image.
- DELETE: ignore the event.
For target tables with identifier fields:
- Preserve existing equality-delete behavior.
- Keep partition-changing UPDATE events working correctly.
Determine the write capability from the actual target Iceberg table's
identifier fields, rather than solely from primary keys declared in the
incoming CDC schema.
Retaining historical records in an append-only table is intentional. This issue
does not introduce upsert or delete support for such tables.
h2.
ACCEPTANCE CRITERIA
- Cover partitioned and unpartitioned append-only tables.
- Cover mixed target-table capabilities within the same writer.
- Cover mismatches between incoming CDC primary keys and target-table
identifier fields.
- Preserve correct UPDATE and DELETE behavior for tables with identifier fields.
- Correctly invalidate cached writer factories on schema changes, including
when an ignored DELETE has not created a task writer.
h2.
RELATED ISSUE
FLINK-40640
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
