Re: [DISCUSS] State Schema Evolution for RowData

2025-05-06 Thread Hangxiang Yu
Hi, Weiqing. Thanks for driving this FLIP. I'm +1 for supporting schema evolution for SQL RowData type. I just have some questions: 1. Could we consider defining a method returning *SchemaEvolutionSerializer* in *TypeSerializerSchemaCompatibility* (like compatibleAfterMigration(TypeSerializer sche

Re: [DISCUSS] State Schema Evolution for RowData

2025-04-28 Thread Weiqing Yang
Thanks for the suggestions, Zakelly! Regarding *migrateElement* - it is specifically needed for ListState, which stores elements individually with delimiters. Its implementation deserializes and processes each element one by one during migration, so I introduced the *migrateElement* API to handle

Re: [DISCUSS] State Schema Evolution for RowData

2025-04-28 Thread Zakelly Lan
Hi Weiqiang, Thanks for your answers! It seems a simple deserialization-serialization lacks flexibility, thus I'd agree to introduce new methods. I'd suggest changing the signature to: ``` public void migrateState( TypeSerializerSnapshot oldSerializerSnapshot, DataI

Re: [DISCUSS] State Schema Evolution for RowData

2025-04-27 Thread Weiqing Yang
Hi Zakelly, Thanks for your feedback. You're right - *resolveSchemaCompatibility* is critical for identifying schema compatibility. However, our challenge extends beyond detection to handling the actual migration process, particularly given RowData’s complex requirements. The standard migration

Re: [DISCUSS] State Schema Evolution for RowData

2025-04-26 Thread Zakelly Lan
Hi, Weiqing Thanks for the FLIP! In general I'd +1 for schema evolution for RowData types, which will enhance the user experience of SQL jobs. I have one questions for now: You suggested introducing new methods in `TypeSerializerSnapshot`, but is it possible to leverage existing state migration