Hi David and Galen, To the best of my knowledge, the state or checkpoint itself has not changed much (from 1.20 to 2.0), and will be compatible if you are using the same statebackend. But there still some problems:
1. Due to numerous API breaking changes in 2.0, users may need to rewrite their job, resulting in auto-assigned operator ids changing. This means that the previous state cannot be mapped to the operators and restored. However, manually specifying the operator ids can avoid this problem. 2. The SQL jobs are using auto-assigned operator ids, I'm not sure if it still generates the ids in the same way as previously. 3. The serializers might not be compatible. Fortunately, most of the built-in serializers are IIUC compatible. 4. If you are using newly introduced functions in 2.0, such as the disaggregated state, the state cannot be migrated. For 1,2 and 3, I believe the State Processor API[1] can help users migrate their state. While we cannot guarantee that Flink V1 state can be migrated to V2, in many cases the state can be successfully migrated. Also, the newer the Flink version is, the easier migration would be. It would be helpful for someone to conduct migration tests from 1.20 to 2.0-preview and see if there are more issues on this. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/ Best, Zakelly On Sat, Nov 9, 2024 at 12:48 AM Galen Warren <ga...@cvillewarrens.com.invalid> wrote: > Yes, I'd like to second that it would be very helpful to have a way to > migrate state from V1 to V2. > > On Thu, Nov 7, 2024 at 12:11 PM David Radley <david_rad...@uk.ibm.com> > wrote: > > > Hello, > > At Flink Forward I learnt that Fink V1 state could not be migrated to V2. > > I think this would be a big migration inhibitor for current Flink users, > as > > they would need to throw away their existing state. As such I think this > is > > a critical possible blocking issue. Prior to Flink 2 going out, I think > > this needs to be looked at in case we need to amend the v2 state in some > > way to facilitate easier migration. > > > > Is this a task that is already in hand in the community? If not, I am > > happy to contribute this or be involved with the contribution – but would > > need guidance from the Flink PMCs on the approach, > > Kind regards, David. > > > > Unless otherwise stated above: > > > > IBM United Kingdom Limited > > Registered in England and Wales with number 741598 > > Registered office: Building C, IBM Hursley Office, Hursley Park Road, > > Winchester, Hampshire SO21 2JN > > >
