Yue Ma created FLINK-37551:
------------------------------
Summary: Improve the state compatibility of ForStKeyeStateBackend
and ForStSyncKeyeStateBackend
Key: FLINK-37551
URL: https://issues.apache.org/jira/browse/FLINK-37551
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 2.0.0
Reporter: Yue Ma
Fix For: 2.0.1
Currently, when switching between ForStKeyeStateBackend and
ForStSyncKeyeStateBackend, the states may not be compatible. This may cause a
lot of upgrade costs for users. The main purpose of this ticket is to improve
the compatibility of ForStKeyeStateBackend and ForStSyncKeyeStateBackend.
In order to support the sync/async mode compatibility between
ForStKeyedStateBackend, there may be many details involved. For the convenience
of future tracking, the required changes will be summarized here. (The current
list may still be incomplete, but it will be continuously updated until full
compatibility)
* {*}Made StateMetaInfo consistent for ForStKeyeStateBackend and
ForStSyncKeyeStateBackend{*}. Currently, if MapState or ListState is used, the
StateMeta information saved by ForStKeyeStateBackend and
ForStSyncKeyeStateBackend when making cp may be inconsistent, which may cause
incompatible states. For example, when using MapState, ForStKeyeStateBackend
will save the ValueSerializer information in StateMeta, but
ForStSyncKeyeStateBackend will save the MapSerializer information. So we need
to unify this place.
* {*}Let ForStKeyedStateBackend consider namespaceSerializer compatibility
when restoring state{*}. ForStKeyedStateBackend currently does not consider
namespaceSerializer compatibility when restoring state. This may cause the
namespaceSerializer to change when it is not known whether it should be
compatible.
* {*}Make StateMeta in ForState restore from oldStateInfo from
Checkpoint{*}.(https://issues.apache.org/jira/browse/FLINK-37524) Currently,
when creating using ForStKeyedStateBackend create a new State ,
AbstractKeyedState will directly use the serializer in StateDescriptor
regardless of whether it is restored from a Checkpoint.However, this may cause
data to be unable to be restored when the last serializer changes (for example,
the class registration order in KryoSerializer has changed, for specific cases,
please refer
toStateBackendTestBase#testKryoRestoreResilienceWithDifferentRegistrationOrder).
Therefore, when creating a State from a Checkpoint, oldStateInfo should be
used as the new State's Serializer.
* *Add unit tests for state compatibility between ForStKeyeStateBackend and
ForStSyncKeyeStateBackend*
* *Make serialization consistent between ForStMapState and ForStSyncMapState*
(https://issues.apache.org/jira/browse/FLINK-37419)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
