Sergei Morozov created FLINK-37371:
--------------------------------------
Summary: Postgres CDC incremental source fails to handle
upper-case table and column names
Key: FLINK-37371
URL: https://issues.apache.org/jira/browse/FLINK-37371
Project: Flink
Issue Type: Bug
Components: Flink CDC
Affects Versions: cdc-3.2.0
Reporter: Sergei Morozov
Example table schema:
{code:java}
create table "Customers" (
"Id" int not null primary key,
"Name" varchar(32) not null
);
{code}
Snapshotting fails with the following exception:
{noformat}
java.sql.SQLException: No result returned after running query [SELECT
reltuples::bigint FROM pg_class WHERE oid = to_regclass('public.Customers')]
at
org.apache.flink.cdc.connectors.postgres.source.utils.PostgresQueryUtils.lambda$queryApproximateRowCnt$1(PostgresQueryUtils.java:81)
at
org.apache.flink.cdc.connectors.shaded.io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:642)
at
org.apache.flink.cdc.connectors.shaded.io.debezium.jdbc.JdbcConnection.queryAndMap(JdbcConnection.java:510)
at
org.apache.flink.cdc.connectors.postgres.source.utils.PostgresQueryUtils.queryApproximateRowCnt(PostgresQueryUtils.java:76)
at
org.apache.flink.cdc.connectors.postgres.source.PostgresChunkSplitter.queryApproximateRowCnt(PostgresChunkSplitter.java:77)
at
org.apache.flink.cdc.connectors.base.source.assigner.splitter.JdbcSourceChunkSplitter.splitTableIntoChunks(JdbcSourceChunkSplitter.java:295)
at
org.apache.flink.cdc.connectors.base.source.assigner.splitter.JdbcSourceChunkSplitter.generateSplits(JdbcSourceChunkSplitter.java:81)
... 14 common frames omitted
{noformat}
Also:
{noformat}
org.apache.flink.util.FlinkRuntimeException: Read split
SnapshotSplit{tableId=public.Customers, splitId='public.Customers:0',
splitKeyType=[`Id` INT NOT NULL], splitStart=null, splitEnd=[8097],
highWatermark=null} error due to org.apache.flink.util.FlinkRuntimeException:
Snapshotting of table public.Customers failed.
at
org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceScanFetcher.checkReadException(IncrementalSourceScanFetcher.java:181)
at
org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceScanFetcher.pollSplitRecords(IncrementalSourceScanFetcher.java:128)
at
org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.pollSplitRecords(IncrementalSourceSplitReader.java:161)
at
org.apache.flink.cdc.connectors.base.source.reader.IncrementalSourceSplitReader.fetch(IncrementalSourceSplitReader.java:98)
at
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
at java.base/java.lang.Thread.run(Thread.java:831)
Caused by:
org.apache.flink.cdc.connectors.shaded.io.debezium.DebeziumException:
org.apache.flink.util.FlinkRuntimeException: Snapshotting of table
public.Customers failed
at
org.apache.flink.cdc.connectors.shaded.io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:85)
at
org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask.executeDataSnapshot(PostgresScanFetchTask.java:112)
at
org.apache.flink.cdc.connectors.base.source.reader.external.AbstractScanFetchTask.execute(AbstractScanFetchTask.java:71)
at
org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask.execute(PostgresScanFetchTask.java:86)
at
org.apache.flink.cdc.connectors.base.source.reader.external.IncrementalSourceScanFetcher.lambda$submitTask$1(IncrementalSourceScanFetcher.java:99)
... 3 common frames omitted
Caused by: org.apache.flink.util.FlinkRuntimeException: Snapshotting of table
public.Customers failed
at
org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.createDataEventsForTable(PostgresScanFetchTask.java:350)
at
org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.createDataEvents(PostgresScanFetchTask.java:265)
at
org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.doExecute(PostgresScanFetchTask.java:255)
at
org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.doExecute(PostgresScanFetchTask.java:211)
at
org.apache.flink.cdc.connectors.shaded.io.debezium.pipeline.source.AbstractSnapshotChangeEventSource.execute(AbstractSnapshotChangeEventSource.java:76)
... 7 common frames omitted
Caused by: org.postgresql.util.PSQLException: ERROR: column "id" does not exist
Hint: Perhaps you meant to reference the column "Customers.Id".
Position: 45
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2553)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2285)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:323)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:481)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:401)
at
org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:164)
at
org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:114)
at
org.apache.flink.cdc.connectors.postgres.source.fetch.PostgresScanFetchTask$PostgresSnapshotSplitReadTask.createDataEventsForTable(PostgresScanFetchTask.java:313)
... 11 common frames omitted
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
