Yanquan Lv created FLINK-40651:
----------------------------------
Summary: Support configurable Iceberg file rolling within a
checkpoint
Key: FLINK-40651
URL: https://issues.apache.org/jira/browse/FLINK-40651
Project: Flink
Issue Type: Improvement
Components: Flink CDC
Reporter: Yanquan Lv
h2. Description
The Flink CDC Iceberg sink currently hard-codes the target file size to 256 MB
and does not pass Iceberg write properties to {{{}RowDataTaskWriterFactory{}}}.
As a result, configurations such as:
{{table.properties.write.target-file-size-bytes: "4194304"
table.properties.write.parquet.row-group-size-bytes: "4194304"}}
do not affect runtime file rolling as expected.
h2. Expected Behavior
* Honor {{write.target-file-size-bytes}} when creating the Iceberg writer.
* Pass Iceberg/Parquet write properties to the underlying writer.
* Allow multiple files to be rolled within one checkpoint while committing
them only after the checkpoint succeeds.
* Optionally support a connector-specific record-count rolling option.
h2. Acceptance Criteria
* Multiple files can be generated within one checkpoint when the configured
target size is reached.
* Exactly-once checkpoint semantics are preserved.
* Existing behavior remains unchanged when no related configuration is
provided.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
