Hi devs, I want to start a discussion about Schema Evolution on the Flink Table Store. [1]
In FLINK-21634, We plan to support many schema changes in Flink SQL. But for the current Table Store, it may result in wrong data, unclear evolutions. In general, the user has these operations for schema: - Add column: Adding a column to a table. - Modify column type. - Drop column: Drop a column. - Rename column: For example, rename the "name_1" column to "name_2". Another schema change is partition keys, the data is changing over time, for example, a table with day partition, as the business continues to grow, the new partition of the table by day will become larger and the business wants to change to hourly partitions. A simple approach is to rewrite all the existing data when modifying the schema. But this expensive way is not acceptable to the user, so we need to support and define it clearly. Modifying the schema does not rewrite the existing data, when reading the original data needs to evolve to the current schema. Look forward to your feedback! [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-226%3A+Introduce+Schema+Evolution+on+Table+Store Best, Jingsong
