[
https://issues.apache.org/jira/browse/FLINK-37551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17938144#comment-17938144
]
Zakelly Lan commented on FLINK-37551:
-------------------------------------
[~mayuehappy] Basically, these should be no other than what you listed. Note
that the metainfo also differs in StateMetaInfoSnapshot.BackendStateType for
state v1/v2, which should also be addressed. We can make this jira an umbrella
one and add sub-tasks if there are further ones found.
> Improve the state compatibility of ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend
> --------------------------------------------------------------------------------------
>
> Key: FLINK-37551
> URL: https://issues.apache.org/jira/browse/FLINK-37551
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 2.0.0
> Reporter: Yue Ma
> Assignee: Yue Ma
> Priority: Major
> Fix For: 2.0.1
>
>
> Currently, when switching between ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend, the states may not be compatible. This may cause a
> lot of upgrade costs for users. The main purpose of this ticket is to improve
> the compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend.
> In order to support the sync/async mode compatibility between
> ForStKeyedStateBackend, there may be many details involved. For the
> convenience of future tracking, the required changes will be summarized here.
> (The current list may still be incomplete, but it will be continuously
> updated until full compatibility)
> * {*}Made StateMetaInfo consistent for ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend{*}. Currently, if MapState or ListState is used,
> the StateMeta information saved by ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend when making cp may be inconsistent, which may cause
> incompatible states. For example, when using MapState, ForStKeyeStateBackend
> will save the ValueSerializer information in StateMeta, but
> ForStSyncKeyeStateBackend will save the MapSerializer information. So we need
> to unify this place and perform a transformation when ForStKeyeStateBackend
> creates stateMeta
> * {*}Let ForStKeyedStateBackend consider namespaceSerializer compatibility
> when restoring state{*}. ForStKeyedStateBackend currently does not consider
> namespaceSerializer compatibility when restoring state. This may cause the
> namespaceSerializer to change when it is not known whether it should be
> compatible.
> * {*}Make StateMeta in ForState restore from oldStateInfo from
> Checkpoint{*}.(https://issues.apache.org/jira/browse/FLINK-37524) Currently,
> when creating using ForStKeyedStateBackend create a new State ,
> AbstractKeyedState will directly use the serializer in StateDescriptor
> regardless of whether it is restored from a Checkpoint.However, this may
> cause data to be unable to be restored when the last serializer changes (for
> example, the class registration order in KryoSerializer has changed, for
> specific cases, please refer
> toStateBackendTestBase#testKryoRestoreResilienceWithDifferentRegistrationOrder).
> Therefore, when creating a State from a Checkpoint, oldStateInfo should be
> used as the new State's Serializer.
> * *Add unit tests for state compatibility between ForStKeyeStateBackend and
> ForStSyncKeyeStateBackend*
> * *Make serialization consistent between ForStMapState and
> ForStSyncMapState* (https://issues.apache.org/jira/browse/FLINK-37419)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
