JNSimba commented on PR #3767: URL: https://github.com/apache/flink-cdc/pull/3767#issuecomment-2783640913
> @JNSimba Could we provide this feature by `SupportsProjectionPushDown`? `debezium.column.include.list` and `debezium.column.exclude.list` is hard for users to understand and use. We could use `SupportsProjectionPushDown` interface to automatically generate the debezium setting. In this way we don't need to validate whether there is a conflict between the provided schema and the debezium setting. @ruanhang1993 `SupportsProjectionPushDown` is indeed an elegant approach, but some of the cdc logic seems to have problems in the push-down phase, for example, here If you run the `MySqlConnectorITCase.testConsumingAllEvents` case, the chunkkey will be adjusted here. For example, if my project column does not have a chunkkey, an error will be reported https://github.com/apache/flink-cdc/blob/b437a49e67a2ef84c59cde09f2bfc372f45b31fa/flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/main/java/org/apache/flink/cdc/connectors/mysql/debezium/reader/BinlogSplitReader.java#L267C1-L274C37 ```java Object[] chunkKey = RecordUtils.getSplitKey( splitKeyType, statefulTaskContext.getSchemaNameAdjuster(), target); for (FinishedSnapshotSplitInfo splitInfo : finishedSplitsInfo.get(tableId)) { if (RecordUtils.splitKeyRangeContains( chunkKey, splitInfo.getSplitStart(), splitInfo.getSplitEnd()) && position.isAfter(splitInfo.getHighWatermark())) { return true; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
