[
https://issues.apache.org/jira/browse/FLINK-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17529581#comment-17529581
]
Roman Khachatryan commented on FLINK-23143:
-------------------------------------------
Do you mind sharing your POC?
IIRC, getOrCreateKeyedState of a nested backend is not called by the changelog
backend.
My concern was that calling createInternalState() twice creates two state
objects (e.g. HeapValueState).
Another concern is that besides of serializers, the state itself also needs to
be migrated (in case of RocksDB). So maybe it makes sense to pull
HeapKeyedStateBackend.tryRegisterStateTable and
RocksDBKeyedStateBackend.tryRegisterKvStateInformation into
AbstractKeyedStateBackend (or some interface) and call it directly from
ChangelogKeyedStateBackend (delegation approach wouldn't be needed in this
case). Or extract only the migration part, e.g. upgrade(StateDescriptor).
Besides that, updating the TTL policy should also be taken into account.
Do you mind creating a short design doc describing what needs to be upgraded,
and one or more options of how to do it?
> Support state migration
> -----------------------
>
> Key: FLINK-23143
> URL: https://issues.apache.org/jira/browse/FLINK-23143
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Roman Khachatryan
> Priority: Minor
> Fix For: 1.16.0
>
>
> ChangelogKeyedStateBackend.getOrCreateKeyedState is currently used during
> recovery; on 1st user access, it doesn't update metadata nor migrate state
> (as opposed to other backends).
>
> The proposed solution is to
> # wrap serializers (and maybe other objects) in getOrCreateKeyedState
> # store wrapping objects in a new map keyed by state name
> # pass wrapped objects to delegatedBackend.createInternalState
> # on 1st user access, lookup wrapper and upgrade its wrapped serializer
> This should be done for both KV/PQ states.
>
> See also [https://github.com/apache/flink/pull/15420#discussion_r656934791]
>
> cc: [~yunta]
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
