Hi Ran,
what would be the data type of this dynamic metadata column? The planner
and many parts of the stack will require a data type.
Personally, I feel connector developers can already have the same
functionality by declaring a metadata column as `MAP<STRING, STRING>`.
This is what we expose already as `debezium.source.properties`. Whatever
Debezium adds will be available through this property and can be
accessed via `SELECT col['my-new-property'] FROM x` including being NULL
be default if not present.
Regards,
Timo
On 25.08.22 14:04, Ran Tao wrote:
```
create table test_source(
__test_metadata__ varchar METADATA,
f0 varchar,
f1 varchar,
f2 bigint,
ts as CURRENT_TIMESTAMP
) with(
'connector'='test',
...
)
```
If we not pre define `__test_metadata__` as meta keys by implementing
listReadableMetadata, run the above sql, it will cause exception like this:
org.apache.flink.table.api.ValidationException: Invalid metadata key
'__test_metadata__' in column '__test_metadata__' of table
'default_catalog.default_database.test_source'. The DynamicTableSource
class 'com.alipay.flink.connectors.test.source.TestDynamicTableSource'
supports the following metadata keys for reading:
xxx, yyy
at
org.apache.flink.table.planner.connectors.DynamicSourceUtils.lambda$validateAndApplyMetadata$5(DynamicSourceUtils.java:409)
Because the current flink metadata column must exist in results returned by
`listReadableMetadata`. But when a certain connector adds some metadatas,
we can not use it directly unless we modify this connector code and support
it. In some situations, It can be intolerable. Can we support 'DYNAMIC
MetadataColumn'? Its basic mechanism is not to check a column with
existing metadatas and users can define it dynamically. If a certain
connector without this metadata, the column value will return null
otherwise return it's concrete value. It has great benefits in some
scenarios.
Looking forward to your opinions.
