[
https://issues.apache.org/jira/browse/FLINK-27846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630272#comment-17630272
]
Shammon edited comment on FLINK-27846 at 11/8/22 10:49 AM:
-----------------------------------------------------------
Hi [~lzljs3620320] [~TsReaper]
Currently flink table store writes and reads data according to the column name
in avro/orc/parquet. When performing column modification ddl, such as renaming
column, deleting column, and adding new column with old name, it will cause
data reading errors.
One current idea is to store the column id as the name to avro/orc/parquet. The
problem with this modification is that it will cause incompatibility between
the new version and the previous version of the data file. What do you think of
this solution? Hope to hear from you, THX
was (Author: zjureel):
Hi [~lzljs3620320] [~TsReaper]
Currently flink table store writes and reads according to the column name in
avro/orc/parquet. When performing column modification ddl, such as renaming
column, deleting column, and adding new column with old name, it will cause
data reading errors.
One current idea is to store the column id as the name to avro/orc/parquet. The
problem with this modification is that it will cause incompatibility between
the new version and the previous version of the data file. What do you think of
this solution? Hope to hear from you, THX
> Schema evolution for data file reading
> --------------------------------------
>
> Key: FLINK-27846
> URL: https://issues.apache.org/jira/browse/FLINK-27846
> Project: Flink
> Issue Type: Sub-task
> Components: Table Store
> Reporter: Jingsong Lee
> Priority: Minor
> Fix For: table-store-0.3.0
>
>
> For file reads, we need to.
> - Adjust the correspondence of specific fields
> - If there is a type evolution, we need to upgrade the corresponding data
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
