yux created FLINK-35983:
---------------------------
Summary: Job crashes when using wildcard (*) match with metadata
column
Key: FLINK-35983
URL: https://issues.apache.org/jira/browse/FLINK-35983
Project: Flink
Issue Type: Bug
Components: Flink CDC
Reporter: yux
One may write such a projection rule:
transform:
- projection: '*, __namespace_name__, __schema_name__, __table_name__'
to append some metadata columns at the end of existing columns. However this
will crash the job since once a metadata column was declared, since wildcard
matches metadata column, too:
Caused by: java.lang.IllegalArgumentException: Field names must be unique.
Found duplicates: [__namespace_name__, __schema_name__, __table_name__]
at
org.apache.flink.cdc.common.types.RowType.validateFields(RowType.java:158)
at org.apache.flink.cdc.common.types.RowType.<init>(RowType.java:54)
at org.apache.flink.cdc.common.types.RowType.of(RowType.java:183)
at org.apache.flink.cdc.common.types.RowType.of(RowType.java:175)
at
org.apache.flink.cdc.runtime.typeutils.DataTypeConverter.toRowType(DataTypeConverter.java:55)
at
org.apache.flink.cdc.runtime.operators.transform.TableChangeInfo.of(TableChangeInfo.java:100)
at
org.apache.flink.cdc.runtime.operators.transform.TransformSchemaOperator.cacheCreateTable(TransformSchemaOperator.java:183)
at
org.apache.flink.cdc.runtime.operators.transform.TransformSchemaOperator.processElement(TransformSchemaOperator.java:168)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:75)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:50)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at
org.apache.flink.streaming.runtime.tasks.SourceOperatorStreamTask$AsyncDataOutputToOutput.emitRecord(SourceOperatorStreamTask.java:309)
at
org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:110)
at
org.apache.flink.streaming.api.operators.source.SourceOutputWithWatermarks.collect(SourceOutputWithWatermarks.java:101)
at
org.apache.flink.api.connector.source.lib.util.IteratorSourceReaderBase.pollNext(IteratorSourceReaderBase.java:111)
at
org.apache.flink.cdc.connectors.values.source.ValuesDataSource$EventIteratorReader.pollNext(ValuesDataSource.java:294)
at
org.apache.flink.streaming.api.operators.SourceOperator.emitNext(SourceOperator.java:419)
at
org.apache.flink.streaming.runtime.io.StreamTaskSourceInput.emitNext(StreamTaskSourceInput.java:68)
at
org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:65)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:562)
at
org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:231)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:858)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:807)
at
org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:953)
at
org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:932)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:746)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
